izzy/UnrealEngineUWP
0
0
Fork 0
mirror of https://github.com/izzy2lost/UnrealEngineUWP.git synced 2026-03-26 18:15:20 -07:00
Code Issues Packages Projects Releases Wiki Activity
Files
3c2010d281269994deefcc86297a79247a9aeb34
UnrealEngineUWP/Engine/Source/Developer/Virtualization/Private/DumpPackagePayloadInfoCommand.cpp

109 lines
3.6 KiB
C++
Raw Normal View History

Add a console command 'DumpPackagePayloadInfo' that will take a list of package name or paths and then write info about each package's payloads to the log file. #rb CarlMagnus.Nordin #jira none #rnx #preflight 6156fd08475173000182663d - This command is intended for debug use as an easy way to dump out the payload info in a package file that can then be inspected. #ROBOMERGE-AUTHOR: paul.chipchase #ROBOMERGE-SOURCE: CL 17691116 in //UE5/Main/... #ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v875-17642767) #ROBOMERGE[STARSHIP]: UE5-Release-Engine-Staging Release-5.0 [CL 17691124 by paul chipchase in ue5-release-engine-test branch]
2021-10-01 09:30:43 -04:00
// Copyright Epic Games, Inc. All Rights Reserved.
#include "HAL/IConsoleManager.h"
#include "Logging/LogMacros.h"
#include "Misc/PackagePath.h"
Reworked the virtualization system so that we only virtualize a package when it is submitted to source control, which makes the overall system much less dangerous. #rb PJ.Kack #rnx #preflight 61a4b235405273b2c3daa7c7 ### Virtualization - The idea is to move from our current set up, where the virtualization happens when a package is saved and the payloads are saved to a local virtualization storage system with the final push to persistent storage occurring when the package is submitted to a system where we save payloads into the package file, but move them to persistent virtualization storage when the package file is submitted. - The main advantage is that if someone submits a package file to source control without virtualizing it, we don't have to worry that others might not be able to load the package, the worst case scenario is that data sizes get bigger until the package is virtualized again. - This is only the first pass, in the future we can do further optimizations, like not storing payloads locally if we know that they are already in the persistent storage system etc. - In order to keep the virtualization process simple we want to be able to do it without needing to re-save the package, so to this end we will storage the payloads at the end of the package file as before, but instead of storing them inside of the package file format we will use a new container, the FPackageTrailer that is appended to the package file instead. - In theory this will make future sidecar work easier as the trailer can just be moved to the sidecar file as as long as we know where to find the trailer things will just work (tm) - For now VBD will continue to serialize it's offset/size to disk and use those values to read the payload directly when not virtualizing. Ideally we'd use the trailer but this will help reduce the risk. ### Current Issues - The system does not work with the editor domain and has minor issues with text based assets - The trailer system is disabled by default via the config system [Core]UsePackageTrailer=False and can then be enabled on select test projects until fully functional. ### PackageTrailer - The trailer is split into two parts FPackageTrailerBuilder and FPackageTrailer - FPackageTrailerBuilder is used when building the trailer during package save and FPackageTrailer is the structure we load and actually use. - When saving the trailer we try to avoid using containers and such so that we can try to keep the specification for the file format very clear so that licensees can build their own tools for the virtualization process if they so desire. - the header file contains a breakdown of the new format. ### VirtualizedBulkData - FPayloadToc is now only used by the sidecar experimental feature. In a future update the sidecar will be changed to basically be the package trailer but stored in it's own file. - FindPayloadsInPackageFile has been moved to PackageTrailer.h as it has a closer association with that code. - Added a new method, ::LoadFromPackageTrailer, to load payloads from the package trailer without needing to know where the payload is on disk. Using this is opt in with the cvar "Serialization.LoadFromTrailer" and is only provided for debugging purposes. Although once this system has matured it will likely become the preferred way to access payloads. - We no longer allow virtualization of bulkdata payloads on save, but I am not sure if we might want to allow this as an option in the future. So for now the code branch is disabled by the global constant bAllowVirtualizationOnSave. - Added utility functions ::GetLinkerLoadFromOwner and ::GetTrailerFromOwner for easy access to the LinkerLoad/Trailer from the owning object. - Added a utility ::UpdateArchiveData for seeking back to a known position in an archive, writing over it, then seeking back, which is fairly common in the package saving code paths. This might be worth moving to Archive.h for general use. - ::LoadFromPackageFile has been refactored to prefer returning error values early over nested ifs. - We no longer serialize the EFlags::IsVirtualized flag to disk. Instead it is applied when the package is loaded if we detect that a payload is virtualized and then removed before saved to disk. This allows the virtualization process to occur without resaving the package. - We currently support the old serialize to disk path which is enabled when the FPackageTrailer system is disabled and the newer system when it is enabled. - We do however continue to serialize the offset in the package file to disk when saved. Strictly speaking this is not needed as we can look this up from the package trailer when the package is loaded but continuing to serialize it out to disk helps keep the old and newer data format paths working together without adding too much special case code. -- Once the FPackageTrailer feature is no longer optional we can consider changing this. ### SavePackage/SavePackage2 - Removed the functions that would create the FPayloadToc - Added new functions to help create the trailer. - Moved the end of package tag out of the if to it's own scope (still won't run if a package writer exists), the reasoning is to make it clearer where the package format ends and where we can start writing the payload trailer. - We have checks to make sure that the trailer does not try to write to disk for both text based assets and if the editor domain is enabled. Support for these features will be added later, but since the package trailer system is disabled by default this shouldn't cause problems. ### LinkerLoad - After we parse the FPackageFileSummary we now also parse the header of the package trailer. In the workspace domain this will incur additional seek costs (which may or not actually seek and invalidate the internal file cache depending on how large the package's exports are) as the trailer is found after the package -- In the future, the package trailer header can be stored right after the FPackageFileSummary in the editor domain which will remove this overhead. - By parsing the header of the trailer at this point, virtualized bulkdata objects in the package can use the look up info to determine if their payloads are virtualized or not, and if not then where they reside inside the package file. ### LinkerSave - We no longer collect a list of all virtualized bulkdata in a package while saving as we no longer generate the FPayloadToc so BulkDataInPackage has been removed. - We now gather data to be appended to the package trailer via TrailerBuilder. ### DumpPayloadToc - This command now dumps info based on the package trailer if one can be found. #ROBOMERGE-AUTHOR: paul.chipchase #ROBOMERGE-SOURCE: CL 18307893 in //UE5/Release-5.0/... via CL 18307905 #ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469) [CL 18307913 by paul chipchase in ue5-release-engine-test branch]
2021-11-29 07:02:24 -05:00
#include "UObject/PackageTrailer.h"
Add a console command 'DumpPackagePayloadInfo' that will take a list of package name or paths and then write info about each package's payloads to the log file. #rb CarlMagnus.Nordin #jira none #rnx #preflight 6156fd08475173000182663d - This command is intended for debug use as an easy way to dump out the payload info in a package file that can then be inspected. #ROBOMERGE-AUTHOR: paul.chipchase #ROBOMERGE-SOURCE: CL 17691116 in //UE5/Main/... #ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v875-17642767) #ROBOMERGE[STARSHIP]: UE5-Release-Engine-Staging Release-5.0 [CL 17691124 by paul chipchase in ue5-release-engine-test branch]
2021-10-01 09:30:43 -04:00
namespace UE
{
#if WITH_EDITORONLY_DATA
Add 'size of disk' to the output for local payloads when running the 'DumpPackagePayloadInfo command. #rb trivial #rnx #preflight 6294aa916601713a4f75e17d [CL 20429874 by paul chipchase in ue5-main branch]
2022-05-30 07:35:08 -04:00
FString BytesToString(int64 SizeInBytes)
{
if (SizeInBytes < (8 *1024))
{
return FString::Printf(TEXT("%4d bytes"), SizeInBytes);
}
else if (SizeInBytes < (1024 * 1024))
{
double SizeInKb = SizeInBytes / (1024.0);
return FString::Printf(TEXT("%.2f KB"), SizeInKb);
}
else
{
double SizeInMB = SizeInBytes / (1024.0 * 1024.0);
return FString::Printf(TEXT("%.2f MB"), SizeInMB);
}
}
Add a console command 'DumpPackagePayloadInfo' that will take a list of package name or paths and then write info about each package's payloads to the log file. #rb CarlMagnus.Nordin #jira none #rnx #preflight 6156fd08475173000182663d - This command is intended for debug use as an easy way to dump out the payload info in a package file that can then be inspected. #ROBOMERGE-AUTHOR: paul.chipchase #ROBOMERGE-SOURCE: CL 17691116 in //UE5/Main/... #ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v875-17642767) #ROBOMERGE[STARSHIP]: UE5-Release-Engine-Staging Release-5.0 [CL 17691124 by paul chipchase in ue5-release-engine-test branch]
2021-10-01 09:30:43 -04:00
/**
* This function is used to write information about package's payloads to the log file. This has no
* practical development use and should only be used for debugging purposes.
*
* @param Args The function expects each arg to be a valid package path. Failure to provide a valid
* package path will result in errors being written to the log.
*/
Rename DumpPayloadTocCommand.cpp to DumpPackagePayloadInfoCommand.cpp to match the new name we use for the actual command. #rb trivial #rnx #preflight 61e7e9fcb875abaf33196c62 #ROBOMERGE-AUTHOR: paul.chipchase #ROBOMERGE-SOURCE: CL 18656571 in //UE5/Release-5.0/... via CL 18656590 via CL 18656598 #ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v900-18638592) [CL 18656605 by paul chipchase in ue5-main branch]
2022-01-19 06:21:21 -05:00
void DumpPackagePayloadInfo(const TArray<FString>& Args)
Add a console command 'DumpPackagePayloadInfo' that will take a list of package name or paths and then write info about each package's payloads to the log file. #rb CarlMagnus.Nordin #jira none #rnx #preflight 6156fd08475173000182663d - This command is intended for debug use as an easy way to dump out the payload info in a package file that can then be inspected. #ROBOMERGE-AUTHOR: paul.chipchase #ROBOMERGE-SOURCE: CL 17691116 in //UE5/Main/... #ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v875-17642767) #ROBOMERGE[STARSHIP]: UE5-Release-Engine-Staging Release-5.0 [CL 17691124 by paul chipchase in ue5-release-engine-test branch]
2021-10-01 09:30:43 -04:00
{
if (Args.Num() == 0)
{
UE_LOG(LogVirtualization, Error, TEXT("Command 'DumpPackagePayloadInfo' called without any arguments"));
return;
}
for (const FString& Arg : Args)
{
FPackagePath Path;
if (FPackagePath::TryFromMountedName(Arg, Path))
{
It is now possible to disable virtualization for all payloads owned by a specific asset type by adding the name of the asset to [Core.ContentVirtualization]DisabledAsset string array in the engine ini file. #rb Per.Larsson #rnx #jira UE-151377 #preflight 628364050039ea57a52d6989 ### Virtualization - [Core.ContentVirtualization] in the engine ini file now supports an array called 'DisabledAsset' which can be used to name asset types that should not virtualize their payloads. -- By default (in BaseEngine.ini) we have disabled the StaticMesh asset as we know it will crash if a payload is missing and the SoundWave asset as it still is pending testing. - This new way to disable virtualization is data driven. The older hard coded method has not been removed but will likely be reworked in a future submit. - Now when an editor bulkdata is adding it's payload to the package trailer builder during package save it will poll the virtualization system with a call to the new method ::IsDisabledForObject by passing in it's owner. -- If the owner is valid and was present in the 'DisabledAsset' array then the method will return true and the EPayloadFlags::DisableVirtualization flag will be applied. ### Package Trailer - The pre-existing functionality of enum EPayloadFilter has been moved to a new enum EPayloadStorageType as will only filter payloads based on their storage type. - EPayloadFilter has been modified to filter payloads based on functionality although at the moment the only thing it can filter for is to return payloads that can be virtualized, it is left for future expansion. - EPayloadFlags has been reduced to a uint16 with the remaining 2bytes being turned into a new member EPayloadFilterReason. - This new member allows us to record the exact reason why a payload is excluded from virtualization. If it is zero then the payload can virtualize, otherwise it will contain one or more reasons as to why it is being excluded. For this reason the enum is a bitfield. - Added overloads of ::GetPayloads and ::GetNumPayloads that take EPayloadFilter rather than a EPayloadStorageType - Added wrappers around all AccessMode types for FLookupTableEntry. - FPackageTrailerBuilder has been extended to take a EPayloadFilterReason so that the caller can already provide a reason why the payload cannot be virtualized. -- As a future peace of work this will probably be changed and we will ask the caller to pass in the owner UObject pointer instead and then we will process the filtering when building the package trailer to a) keep all of the filtering code in one place b) keep the filtering consistent ### PackageSubmissionChecks - The virtualization process in will now request the payloads that can be virtualized from the package trailer, which respects the new payload flag, rather than requesting all locally stored payloads. ### UObjects - There is no need for the SoundWave or MeshDescription classes to opt out of virtualization on construction. This will be done when the package is saved and is now data driven rather than being hardcoded. ### DumpPackagePayloadInfo - The command has been updated to also display the filter reasons applied to each payload [CL 20240971 by paul chipchase in ue5-main branch]
2022-05-17 07:54:28 -04:00
FPackageTrailer Trailer;
if (!FPackageTrailer::TryLoadFromPackage(Path, Trailer))
Add a console command 'DumpPackagePayloadInfo' that will take a list of package name or paths and then write info about each package's payloads to the log file. #rb CarlMagnus.Nordin #jira none #rnx #preflight 6156fd08475173000182663d - This command is intended for debug use as an easy way to dump out the payload info in a package file that can then be inspected. #ROBOMERGE-AUTHOR: paul.chipchase #ROBOMERGE-SOURCE: CL 17691116 in //UE5/Main/... #ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v875-17642767) #ROBOMERGE[STARSHIP]: UE5-Release-Engine-Staging Release-5.0 [CL 17691124 by paul chipchase in ue5-release-engine-test branch]
2021-10-01 09:30:43 -04:00
{
It is now possible to disable virtualization for all payloads owned by a specific asset type by adding the name of the asset to [Core.ContentVirtualization]DisabledAsset string array in the engine ini file. #rb Per.Larsson #rnx #jira UE-151377 #preflight 628364050039ea57a52d6989 ### Virtualization - [Core.ContentVirtualization] in the engine ini file now supports an array called 'DisabledAsset' which can be used to name asset types that should not virtualize their payloads. -- By default (in BaseEngine.ini) we have disabled the StaticMesh asset as we know it will crash if a payload is missing and the SoundWave asset as it still is pending testing. - This new way to disable virtualization is data driven. The older hard coded method has not been removed but will likely be reworked in a future submit. - Now when an editor bulkdata is adding it's payload to the package trailer builder during package save it will poll the virtualization system with a call to the new method ::IsDisabledForObject by passing in it's owner. -- If the owner is valid and was present in the 'DisabledAsset' array then the method will return true and the EPayloadFlags::DisableVirtualization flag will be applied. ### Package Trailer - The pre-existing functionality of enum EPayloadFilter has been moved to a new enum EPayloadStorageType as will only filter payloads based on their storage type. - EPayloadFilter has been modified to filter payloads based on functionality although at the moment the only thing it can filter for is to return payloads that can be virtualized, it is left for future expansion. - EPayloadFlags has been reduced to a uint16 with the remaining 2bytes being turned into a new member EPayloadFilterReason. - This new member allows us to record the exact reason why a payload is excluded from virtualization. If it is zero then the payload can virtualize, otherwise it will contain one or more reasons as to why it is being excluded. For this reason the enum is a bitfield. - Added overloads of ::GetPayloads and ::GetNumPayloads that take EPayloadFilter rather than a EPayloadStorageType - Added wrappers around all AccessMode types for FLookupTableEntry. - FPackageTrailerBuilder has been extended to take a EPayloadFilterReason so that the caller can already provide a reason why the payload cannot be virtualized. -- As a future peace of work this will probably be changed and we will ask the caller to pass in the owner UObject pointer instead and then we will process the filtering when building the package trailer to a) keep all of the filtering code in one place b) keep the filtering consistent ### PackageSubmissionChecks - The virtualization process in will now request the payloads that can be virtualized from the package trailer, which respects the new payload flag, rather than requesting all locally stored payloads. ### UObjects - There is no need for the SoundWave or MeshDescription classes to opt out of virtualization on construction. This will be done when the package is saved and is now data driven rather than being hardcoded. ### DumpPackagePayloadInfo - The command has been updated to also display the filter reasons applied to each payload [CL 20240971 by paul chipchase in ue5-main branch]
2022-05-17 07:54:28 -04:00
UE_LOG(LogVirtualization, Error, TEXT("Failed to find load the package trailer from: '%s'"), *Path.GetDebugName());
Add a console command 'DumpPackagePayloadInfo' that will take a list of package name or paths and then write info about each package's payloads to the log file. #rb CarlMagnus.Nordin #jira none #rnx #preflight 6156fd08475173000182663d - This command is intended for debug use as an easy way to dump out the payload info in a package file that can then be inspected. #ROBOMERGE-AUTHOR: paul.chipchase #ROBOMERGE-SOURCE: CL 17691116 in //UE5/Main/... #ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v875-17642767) #ROBOMERGE[STARSHIP]: UE5-Release-Engine-Staging Release-5.0 [CL 17691124 by paul chipchase in ue5-release-engine-test branch]
2021-10-01 09:30:43 -04:00
continue;
}
It is now possible to disable virtualization for all payloads owned by a specific asset type by adding the name of the asset to [Core.ContentVirtualization]DisabledAsset string array in the engine ini file. #rb Per.Larsson #rnx #jira UE-151377 #preflight 628364050039ea57a52d6989 ### Virtualization - [Core.ContentVirtualization] in the engine ini file now supports an array called 'DisabledAsset' which can be used to name asset types that should not virtualize their payloads. -- By default (in BaseEngine.ini) we have disabled the StaticMesh asset as we know it will crash if a payload is missing and the SoundWave asset as it still is pending testing. - This new way to disable virtualization is data driven. The older hard coded method has not been removed but will likely be reworked in a future submit. - Now when an editor bulkdata is adding it's payload to the package trailer builder during package save it will poll the virtualization system with a call to the new method ::IsDisabledForObject by passing in it's owner. -- If the owner is valid and was present in the 'DisabledAsset' array then the method will return true and the EPayloadFlags::DisableVirtualization flag will be applied. ### Package Trailer - The pre-existing functionality of enum EPayloadFilter has been moved to a new enum EPayloadStorageType as will only filter payloads based on their storage type. - EPayloadFilter has been modified to filter payloads based on functionality although at the moment the only thing it can filter for is to return payloads that can be virtualized, it is left for future expansion. - EPayloadFlags has been reduced to a uint16 with the remaining 2bytes being turned into a new member EPayloadFilterReason. - This new member allows us to record the exact reason why a payload is excluded from virtualization. If it is zero then the payload can virtualize, otherwise it will contain one or more reasons as to why it is being excluded. For this reason the enum is a bitfield. - Added overloads of ::GetPayloads and ::GetNumPayloads that take EPayloadFilter rather than a EPayloadStorageType - Added wrappers around all AccessMode types for FLookupTableEntry. - FPackageTrailerBuilder has been extended to take a EPayloadFilterReason so that the caller can already provide a reason why the payload cannot be virtualized. -- As a future peace of work this will probably be changed and we will ask the caller to pass in the owner UObject pointer instead and then we will process the filtering when building the package trailer to a) keep all of the filtering code in one place b) keep the filtering consistent ### PackageSubmissionChecks - The virtualization process in will now request the payloads that can be virtualized from the package trailer, which respects the new payload flag, rather than requesting all locally stored payloads. ### UObjects - There is no need for the SoundWave or MeshDescription classes to opt out of virtualization on construction. This will be done when the package is saved and is now data driven rather than being hardcoded. ### DumpPackagePayloadInfo - The command has been updated to also display the filter reasons applied to each payload [CL 20240971 by paul chipchase in ue5-main branch]
2022-05-17 07:54:28 -04:00
TArray<FIoHash> LocalPayloadIds = Trailer.GetPayloads(UE::EPayloadStorageType::Local);
TArray<FIoHash> VirtualizedPayloadIds = Trailer.GetPayloads(UE::EPayloadStorageType::Virtualized);
UE_LOG(LogVirtualization, Display, TEXT("")); // Blank line to make the output easier to read
Add a console command 'DumpPackagePayloadInfo' that will take a list of package name or paths and then write info about each package's payloads to the log file. #rb CarlMagnus.Nordin #jira none #rnx #preflight 6156fd08475173000182663d - This command is intended for debug use as an easy way to dump out the payload info in a package file that can then be inspected. #ROBOMERGE-AUTHOR: paul.chipchase #ROBOMERGE-SOURCE: CL 17691116 in //UE5/Main/... #ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v875-17642767) #ROBOMERGE[STARSHIP]: UE5-Release-Engine-Staging Release-5.0 [CL 17691124 by paul chipchase in ue5-release-engine-test branch]
2021-10-01 09:30:43 -04:00
UE_LOG(LogVirtualization, Display, TEXT("Package: '%s' has %d local and %d virtualized payloads"), *Path.GetDebugName(), LocalPayloadIds.Num(), VirtualizedPayloadIds.Num());
if (LocalPayloadIds.Num() > 0)
{
UE_LOG(LogVirtualization, Display, TEXT("LocalPayloads:"));
Add 'size of disk' to the output for local payloads when running the 'DumpPackagePayloadInfo command. #rb trivial #rnx #preflight 6294aa916601713a4f75e17d [CL 20429874 by paul chipchase in ue5-main branch]
2022-05-30 07:35:08 -04:00
UE_LOG(LogVirtualization, Display, TEXT("Index | %-40s | SizeOnDisk | FilterReason"), TEXT("PayloadIdentifier"));
Add a console command 'DumpPackagePayloadInfo' that will take a list of package name or paths and then write info about each package's payloads to the log file. #rb CarlMagnus.Nordin #jira none #rnx #preflight 6156fd08475173000182663d - This command is intended for debug use as an easy way to dump out the payload info in a package file that can then be inspected. #ROBOMERGE-AUTHOR: paul.chipchase #ROBOMERGE-SOURCE: CL 17691116 in //UE5/Main/... #ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v875-17642767) #ROBOMERGE[STARSHIP]: UE5-Release-Engine-Staging Release-5.0 [CL 17691124 by paul chipchase in ue5-release-engine-test branch]
2021-10-01 09:30:43 -04:00
for (int32 Index = 0; Index < LocalPayloadIds.Num(); ++Index)
{
It is now possible to disable virtualization for all payloads owned by a specific asset type by adding the name of the asset to [Core.ContentVirtualization]DisabledAsset string array in the engine ini file. #rb Per.Larsson #rnx #jira UE-151377 #preflight 628364050039ea57a52d6989 ### Virtualization - [Core.ContentVirtualization] in the engine ini file now supports an array called 'DisabledAsset' which can be used to name asset types that should not virtualize their payloads. -- By default (in BaseEngine.ini) we have disabled the StaticMesh asset as we know it will crash if a payload is missing and the SoundWave asset as it still is pending testing. - This new way to disable virtualization is data driven. The older hard coded method has not been removed but will likely be reworked in a future submit. - Now when an editor bulkdata is adding it's payload to the package trailer builder during package save it will poll the virtualization system with a call to the new method ::IsDisabledForObject by passing in it's owner. -- If the owner is valid and was present in the 'DisabledAsset' array then the method will return true and the EPayloadFlags::DisableVirtualization flag will be applied. ### Package Trailer - The pre-existing functionality of enum EPayloadFilter has been moved to a new enum EPayloadStorageType as will only filter payloads based on their storage type. - EPayloadFilter has been modified to filter payloads based on functionality although at the moment the only thing it can filter for is to return payloads that can be virtualized, it is left for future expansion. - EPayloadFlags has been reduced to a uint16 with the remaining 2bytes being turned into a new member EPayloadFilterReason. - This new member allows us to record the exact reason why a payload is excluded from virtualization. If it is zero then the payload can virtualize, otherwise it will contain one or more reasons as to why it is being excluded. For this reason the enum is a bitfield. - Added overloads of ::GetPayloads and ::GetNumPayloads that take EPayloadFilter rather than a EPayloadStorageType - Added wrappers around all AccessMode types for FLookupTableEntry. - FPackageTrailerBuilder has been extended to take a EPayloadFilterReason so that the caller can already provide a reason why the payload cannot be virtualized. -- As a future peace of work this will probably be changed and we will ask the caller to pass in the owner UObject pointer instead and then we will process the filtering when building the package trailer to a) keep all of the filtering code in one place b) keep the filtering consistent ### PackageSubmissionChecks - The virtualization process in will now request the payloads that can be virtualized from the package trailer, which respects the new payload flag, rather than requesting all locally stored payloads. ### UObjects - There is no need for the SoundWave or MeshDescription classes to opt out of virtualization on construction. This will be done when the package is saved and is now data driven rather than being hardcoded. ### DumpPackagePayloadInfo - The command has been updated to also display the filter reasons applied to each payload [CL 20240971 by paul chipchase in ue5-main branch]
2022-05-17 07:54:28 -04:00
FPayloadInfo Info = Trailer.GetPayloadInfo(LocalPayloadIds[Index]);
Add 'size of disk' to the output for local payloads when running the 'DumpPackagePayloadInfo command. #rb trivial #rnx #preflight 6294aa916601713a4f75e17d [CL 20429874 by paul chipchase in ue5-main branch]
2022-05-30 07:35:08 -04:00
UE_LOG(LogVirtualization, Display, TEXT("%02d | %s | %-10s | %s"),
Index,
*LexToString(LocalPayloadIds[Index]),
*BytesToString(Info.CompressedSize),
*LexToString(Info.FilterFlags));
Add a console command 'DumpPackagePayloadInfo' that will take a list of package name or paths and then write info about each package's payloads to the log file. #rb CarlMagnus.Nordin #jira none #rnx #preflight 6156fd08475173000182663d - This command is intended for debug use as an easy way to dump out the payload info in a package file that can then be inspected. #ROBOMERGE-AUTHOR: paul.chipchase #ROBOMERGE-SOURCE: CL 17691116 in //UE5/Main/... #ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v875-17642767) #ROBOMERGE[STARSHIP]: UE5-Release-Engine-Staging Release-5.0 [CL 17691124 by paul chipchase in ue5-release-engine-test branch]
2021-10-01 09:30:43 -04:00
}
}
if (VirtualizedPayloadIds.Num() > 0)
{
UE_LOG(LogVirtualization, Display, TEXT("VirtualizedPayloads:"));
It is now possible to disable virtualization for all payloads owned by a specific asset type by adding the name of the asset to [Core.ContentVirtualization]DisabledAsset string array in the engine ini file. #rb Per.Larsson #rnx #jira UE-151377 #preflight 628364050039ea57a52d6989 ### Virtualization - [Core.ContentVirtualization] in the engine ini file now supports an array called 'DisabledAsset' which can be used to name asset types that should not virtualize their payloads. -- By default (in BaseEngine.ini) we have disabled the StaticMesh asset as we know it will crash if a payload is missing and the SoundWave asset as it still is pending testing. - This new way to disable virtualization is data driven. The older hard coded method has not been removed but will likely be reworked in a future submit. - Now when an editor bulkdata is adding it's payload to the package trailer builder during package save it will poll the virtualization system with a call to the new method ::IsDisabledForObject by passing in it's owner. -- If the owner is valid and was present in the 'DisabledAsset' array then the method will return true and the EPayloadFlags::DisableVirtualization flag will be applied. ### Package Trailer - The pre-existing functionality of enum EPayloadFilter has been moved to a new enum EPayloadStorageType as will only filter payloads based on their storage type. - EPayloadFilter has been modified to filter payloads based on functionality although at the moment the only thing it can filter for is to return payloads that can be virtualized, it is left for future expansion. - EPayloadFlags has been reduced to a uint16 with the remaining 2bytes being turned into a new member EPayloadFilterReason. - This new member allows us to record the exact reason why a payload is excluded from virtualization. If it is zero then the payload can virtualize, otherwise it will contain one or more reasons as to why it is being excluded. For this reason the enum is a bitfield. - Added overloads of ::GetPayloads and ::GetNumPayloads that take EPayloadFilter rather than a EPayloadStorageType - Added wrappers around all AccessMode types for FLookupTableEntry. - FPackageTrailerBuilder has been extended to take a EPayloadFilterReason so that the caller can already provide a reason why the payload cannot be virtualized. -- As a future peace of work this will probably be changed and we will ask the caller to pass in the owner UObject pointer instead and then we will process the filtering when building the package trailer to a) keep all of the filtering code in one place b) keep the filtering consistent ### PackageSubmissionChecks - The virtualization process in will now request the payloads that can be virtualized from the package trailer, which respects the new payload flag, rather than requesting all locally stored payloads. ### UObjects - There is no need for the SoundWave or MeshDescription classes to opt out of virtualization on construction. This will be done when the package is saved and is now data driven rather than being hardcoded. ### DumpPackagePayloadInfo - The command has been updated to also display the filter reasons applied to each payload [CL 20240971 by paul chipchase in ue5-main branch]
2022-05-17 07:54:28 -04:00
UE_LOG(LogVirtualization, Display, TEXT("Index|\t%-40s|\tFilterReason"), TEXT("PayloadIdentifier"));
Add a console command 'DumpPackagePayloadInfo' that will take a list of package name or paths and then write info about each package's payloads to the log file. #rb CarlMagnus.Nordin #jira none #rnx #preflight 6156fd08475173000182663d - This command is intended for debug use as an easy way to dump out the payload info in a package file that can then be inspected. #ROBOMERGE-AUTHOR: paul.chipchase #ROBOMERGE-SOURCE: CL 17691116 in //UE5/Main/... #ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v875-17642767) #ROBOMERGE[STARSHIP]: UE5-Release-Engine-Staging Release-5.0 [CL 17691124 by paul chipchase in ue5-release-engine-test branch]
2021-10-01 09:30:43 -04:00
for (int32 Index = 0; Index < VirtualizedPayloadIds.Num(); ++Index)
{
It is now possible to disable virtualization for all payloads owned by a specific asset type by adding the name of the asset to [Core.ContentVirtualization]DisabledAsset string array in the engine ini file. #rb Per.Larsson #rnx #jira UE-151377 #preflight 628364050039ea57a52d6989 ### Virtualization - [Core.ContentVirtualization] in the engine ini file now supports an array called 'DisabledAsset' which can be used to name asset types that should not virtualize their payloads. -- By default (in BaseEngine.ini) we have disabled the StaticMesh asset as we know it will crash if a payload is missing and the SoundWave asset as it still is pending testing. - This new way to disable virtualization is data driven. The older hard coded method has not been removed but will likely be reworked in a future submit. - Now when an editor bulkdata is adding it's payload to the package trailer builder during package save it will poll the virtualization system with a call to the new method ::IsDisabledForObject by passing in it's owner. -- If the owner is valid and was present in the 'DisabledAsset' array then the method will return true and the EPayloadFlags::DisableVirtualization flag will be applied. ### Package Trailer - The pre-existing functionality of enum EPayloadFilter has been moved to a new enum EPayloadStorageType as will only filter payloads based on their storage type. - EPayloadFilter has been modified to filter payloads based on functionality although at the moment the only thing it can filter for is to return payloads that can be virtualized, it is left for future expansion. - EPayloadFlags has been reduced to a uint16 with the remaining 2bytes being turned into a new member EPayloadFilterReason. - This new member allows us to record the exact reason why a payload is excluded from virtualization. If it is zero then the payload can virtualize, otherwise it will contain one or more reasons as to why it is being excluded. For this reason the enum is a bitfield. - Added overloads of ::GetPayloads and ::GetNumPayloads that take EPayloadFilter rather than a EPayloadStorageType - Added wrappers around all AccessMode types for FLookupTableEntry. - FPackageTrailerBuilder has been extended to take a EPayloadFilterReason so that the caller can already provide a reason why the payload cannot be virtualized. -- As a future peace of work this will probably be changed and we will ask the caller to pass in the owner UObject pointer instead and then we will process the filtering when building the package trailer to a) keep all of the filtering code in one place b) keep the filtering consistent ### PackageSubmissionChecks - The virtualization process in will now request the payloads that can be virtualized from the package trailer, which respects the new payload flag, rather than requesting all locally stored payloads. ### UObjects - There is no need for the SoundWave or MeshDescription classes to opt out of virtualization on construction. This will be done when the package is saved and is now data driven rather than being hardcoded. ### DumpPackagePayloadInfo - The command has been updated to also display the filter reasons applied to each payload [CL 20240971 by paul chipchase in ue5-main branch]
2022-05-17 07:54:28 -04:00
FPayloadInfo Info = Trailer.GetPayloadInfo(VirtualizedPayloadIds[Index]);
UE_LOG(LogVirtualization, Display, TEXT("%02d: |\t%s|\t%s"), Index, *LexToString(VirtualizedPayloadIds[Index]), *LexToString(Info.FilterFlags));
Add a console command 'DumpPackagePayloadInfo' that will take a list of package name or paths and then write info about each package's payloads to the log file. #rb CarlMagnus.Nordin #jira none #rnx #preflight 6156fd08475173000182663d - This command is intended for debug use as an easy way to dump out the payload info in a package file that can then be inspected. #ROBOMERGE-AUTHOR: paul.chipchase #ROBOMERGE-SOURCE: CL 17691116 in //UE5/Main/... #ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v875-17642767) #ROBOMERGE[STARSHIP]: UE5-Release-Engine-Staging Release-5.0 [CL 17691124 by paul chipchase in ue5-release-engine-test branch]
2021-10-01 09:30:43 -04:00
}
}
}
else
{
UE_LOG(LogVirtualization, Error, TEXT("Arg '%s' could not be converted to a valid package path"), *Arg);
}
}
}
/**
* Note that this command is only valid when 'WITH_EDITORONLY_DATA 1' as virtualized payloads are not
* expected to exist at runtime.
*/
static FAutoConsoleCommand CCmdDumpPayloadToc = FAutoConsoleCommand(
TEXT("DumpPackagePayloadInfo"),
TEXT("Writes out information about a package's payloads to the log."),
Rename DumpPayloadTocCommand.cpp to DumpPackagePayloadInfoCommand.cpp to match the new name we use for the actual command. #rb trivial #rnx #preflight 61e7e9fcb875abaf33196c62 #ROBOMERGE-AUTHOR: paul.chipchase #ROBOMERGE-SOURCE: CL 18656571 in //UE5/Release-5.0/... via CL 18656590 via CL 18656598 #ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v900-18638592) [CL 18656605 by paul chipchase in ue5-main branch]
2022-01-19 06:21:21 -05:00
FConsoleCommandWithArgsDelegate::CreateStatic(DumpPackagePayloadInfo));
Add a console command 'DumpPackagePayloadInfo' that will take a list of package name or paths and then write info about each package's payloads to the log file. #rb CarlMagnus.Nordin #jira none #rnx #preflight 6156fd08475173000182663d - This command is intended for debug use as an easy way to dump out the payload info in a package file that can then be inspected. #ROBOMERGE-AUTHOR: paul.chipchase #ROBOMERGE-SOURCE: CL 17691116 in //UE5/Main/... #ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v875-17642767) #ROBOMERGE[STARSHIP]: UE5-Release-Engine-Staging Release-5.0 [CL 17691124 by paul chipchase in ue5-release-engine-test branch]
2021-10-01 09:30:43 -04:00
#endif //WITH_EDITORONLY_DATA
} // namespace UE
Reference in New Issue Copy Permalink