You've already forked UnrealEngineUWP
mirror of
https://github.com/izzy2lost/UnrealEngineUWP.git
synced 2026-03-26 18:15:20 -07:00
ddff3a39f5
#rb PJ.Kack #rnx #preflight 61a4b235405273b2c3daa7c7 ### Virtualization - The idea is to move from our current set up, where the virtualization happens when a package is saved and the payloads are saved to a local virtualization storage system with the final push to persistent storage occurring when the package is submitted to a system where we save payloads into the package file, but move them to persistent virtualization storage when the package file is submitted. - The main advantage is that if someone submits a package file to source control without virtualizing it, we don't have to worry that others might not be able to load the package, the worst case scenario is that data sizes get bigger until the package is virtualized again. - This is only the first pass, in the future we can do further optimizations, like not storing payloads locally if we know that they are already in the persistent storage system etc. - In order to keep the virtualization process simple we want to be able to do it without needing to re-save the package, so to this end we will storage the payloads at the end of the package file as before, but instead of storing them inside of the package file format we will use a new container, the FPackageTrailer that is appended to the package file instead. - In theory this will make future sidecar work easier as the trailer can just be moved to the sidecar file as as long as we know where to find the trailer things will just work (tm) - For now VBD will continue to serialize it's offset/size to disk and use those values to read the payload directly when not virtualizing. Ideally we'd use the trailer but this will help reduce the risk. ### Current Issues - The system does not work with the editor domain and has minor issues with text based assets - The trailer system is disabled by default via the config system [Core]UsePackageTrailer=False and can then be enabled on select test projects until fully functional. ### PackageTrailer - The trailer is split into two parts FPackageTrailerBuilder and FPackageTrailer - FPackageTrailerBuilder is used when building the trailer during package save and FPackageTrailer is the structure we load and actually use. - When saving the trailer we try to avoid using containers and such so that we can try to keep the specification for the file format very clear so that licensees can build their own tools for the virtualization process if they so desire. - the header file contains a breakdown of the new format. ### VirtualizedBulkData - FPayloadToc is now only used by the sidecar experimental feature. In a future update the sidecar will be changed to basically be the package trailer but stored in it's own file. - FindPayloadsInPackageFile has been moved to PackageTrailer.h as it has a closer association with that code. - Added a new method, ::LoadFromPackageTrailer, to load payloads from the package trailer without needing to know where the payload is on disk. Using this is opt in with the cvar "Serialization.LoadFromTrailer" and is only provided for debugging purposes. Although once this system has matured it will likely become the preferred way to access payloads. - We no longer allow virtualization of bulkdata payloads on save, but I am not sure if we might want to allow this as an option in the future. So for now the code branch is disabled by the global constant bAllowVirtualizationOnSave. - Added utility functions ::GetLinkerLoadFromOwner and ::GetTrailerFromOwner for easy access to the LinkerLoad/Trailer from the owning object. - Added a utility ::UpdateArchiveData for seeking back to a known position in an archive, writing over it, then seeking back, which is fairly common in the package saving code paths. This might be worth moving to Archive.h for general use. - ::LoadFromPackageFile has been refactored to prefer returning error values early over nested ifs. - We no longer serialize the EFlags::IsVirtualized flag to disk. Instead it is applied when the package is loaded if we detect that a payload is virtualized and then removed before saved to disk. This allows the virtualization process to occur without resaving the package. - We currently support the old serialize to disk path which is enabled when the FPackageTrailer system is disabled and the newer system when it is enabled. - We do however continue to serialize the offset in the package file to disk when saved. Strictly speaking this is not needed as we can look this up from the package trailer when the package is loaded but continuing to serialize it out to disk helps keep the old and newer data format paths working together without adding too much special case code. -- Once the FPackageTrailer feature is no longer optional we can consider changing this. ### SavePackage/SavePackage2 - Removed the functions that would create the FPayloadToc - Added new functions to help create the trailer. - Moved the end of package tag out of the if to it's own scope (still won't run if a package writer exists), the reasoning is to make it clearer where the package format ends and where we can start writing the payload trailer. - We have checks to make sure that the trailer does not try to write to disk for both text based assets and if the editor domain is enabled. Support for these features will be added later, but since the package trailer system is disabled by default this shouldn't cause problems. ### LinkerLoad - After we parse the FPackageFileSummary we now also parse the header of the package trailer. In the workspace domain this will incur additional seek costs (which may or not actually seek and invalidate the internal file cache depending on how large the package's exports are) as the trailer is found after the package -- In the future, the package trailer header can be stored right after the FPackageFileSummary in the editor domain which will remove this overhead. - By parsing the header of the trailer at this point, virtualized bulkdata objects in the package can use the look up info to determine if their payloads are virtualized or not, and if not then where they reside inside the package file. ### LinkerSave - We no longer collect a list of all virtualized bulkdata in a package while saving as we no longer generate the FPayloadToc so BulkDataInPackage has been removed. - We now gather data to be appended to the package trailer via TrailerBuilder. ### DumpPayloadToc - This command now dumps info based on the package trailer if one can be found. #ROBOMERGE-AUTHOR: paul.chipchase #ROBOMERGE-SOURCE: CL 18307893 in //UE5/Release-5.0/... via CL 18307905 #ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469) [CL 18307913 by paul chipchase in ue5-release-engine-test branch]
87 lines
2.8 KiB
C++
87 lines
2.8 KiB
C++
// Copyright Epic Games, Inc. All Rights Reserved.
|
|
|
|
#include "HAL/IConsoleManager.h"
|
|
#include "Logging/LogMacros.h"
|
|
#include "Misc/PackagePath.h"
|
|
#include "UObject/PackageTrailer.h"
|
|
|
|
namespace UE
|
|
{
|
|
|
|
#if WITH_EDITORONLY_DATA
|
|
|
|
/**
|
|
* This function is used to write information about package's payloads to the log file. This has no
|
|
* practical development use and should only be used for debugging purposes.
|
|
*
|
|
* @param Args The function expects each arg to be a valid package path. Failure to provide a valid
|
|
* package path will result in errors being written to the log.
|
|
*/
|
|
void DumpPayloadToc(const TArray<FString>& Args)
|
|
{
|
|
if (Args.Num() == 0)
|
|
{
|
|
UE_LOG(LogVirtualization, Error, TEXT("Command 'DumpPackagePayloadInfo' called without any arguments"));
|
|
return;
|
|
}
|
|
|
|
for (const FString& Arg : Args)
|
|
{
|
|
FPackagePath Path;
|
|
|
|
if (FPackagePath::TryFromMountedName(Arg, Path))
|
|
{
|
|
TArray<Virtualization::FPayloadId> LocalPayloadIds;
|
|
TArray<Virtualization::FPayloadId> VirtualizedPayloadIds;
|
|
|
|
if (!UE::FindPayloadsInPackageFile(Path, UE::EPayloadFilter::Local, LocalPayloadIds))
|
|
{
|
|
UE_LOG(LogVirtualization, Error, TEXT("Failed to find local payload information from package: '%s'"), *Path.GetDebugName());
|
|
continue;
|
|
}
|
|
|
|
if (!UE::FindPayloadsInPackageFile(Path, UE::EPayloadFilter::Virtualized, VirtualizedPayloadIds))
|
|
{
|
|
UE_LOG(LogVirtualization, Error, TEXT("Failed to find virtualized payload information from package: '%s'"), *Path.GetDebugName());
|
|
continue;
|
|
}
|
|
|
|
UE_LOG(LogVirtualization, Display, TEXT("Package: '%s' has %d local and %d virtualized payloads"), *Path.GetDebugName(), LocalPayloadIds.Num(), VirtualizedPayloadIds.Num());
|
|
|
|
if (LocalPayloadIds.Num() > 0)
|
|
{
|
|
UE_LOG(LogVirtualization, Display, TEXT("LocalPayloads:"));
|
|
for (int32 Index = 0; Index < LocalPayloadIds.Num(); ++Index)
|
|
{
|
|
UE_LOG(LogVirtualization, Display, TEXT("%02d: '%s'"), Index, *LocalPayloadIds[Index].ToString());
|
|
}
|
|
}
|
|
|
|
if (VirtualizedPayloadIds.Num() > 0)
|
|
{
|
|
UE_LOG(LogVirtualization, Display, TEXT("VirtualizedPayloads:"));
|
|
for (int32 Index = 0; Index < VirtualizedPayloadIds.Num(); ++Index)
|
|
{
|
|
UE_LOG(LogVirtualization, Display, TEXT("%02d: '%s'"), Index, *VirtualizedPayloadIds[Index].ToString());
|
|
}
|
|
}
|
|
}
|
|
else
|
|
{
|
|
UE_LOG(LogVirtualization, Error, TEXT("Arg '%s' could not be converted to a valid package path"), *Arg);
|
|
}
|
|
}
|
|
}
|
|
|
|
/**
|
|
* Note that this command is only valid when 'WITH_EDITORONLY_DATA 1' as virtualized payloads are not
|
|
* expected to exist at runtime.
|
|
*/
|
|
static FAutoConsoleCommand CCmdDumpPayloadToc = FAutoConsoleCommand(
|
|
TEXT("DumpPackagePayloadInfo"),
|
|
TEXT("Writes out information about a package's payloads to the log."),
|
|
FConsoleCommandWithArgsDelegate::CreateStatic(DumpPayloadToc));
|
|
|
|
#endif //WITH_EDITORONLY_DATA
|
|
|
|
} // namespace UE
|