#rb Per.Larsson
#jira UE-156189
#rnx
#preflight 62a33c245a0ab464fb020897
- The main goal is to split the ini file option that selects the virtualization system to use (SystemName) from the options for our virtualization module.
- The old values should continue to be read and used, but if we detect that ini file values are still under [Core.ContentVirtualization] we will warn about it.
[CL 20594207 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
#jira UE-151377
#preflight 628364050039ea57a52d6989
### Virtualization
- [Core.ContentVirtualization] in the engine ini file now supports an array called 'DisabledAsset' which can be used to name asset types that should not virtualize their payloads.
-- By default (in BaseEngine.ini) we have disabled the StaticMesh asset as we know it will crash if a payload is missing and the SoundWave asset as it still is pending testing.
- This new way to disable virtualization is data driven. The older hard coded method has not been removed but will likely be reworked in a future submit.
- Now when an editor bulkdata is adding it's payload to the package trailer builder during package save it will poll the virtualization system with a call to the new method ::IsDisabledForObject by passing in it's owner.
-- If the owner is valid and was present in the 'DisabledAsset' array then the method will return true and the EPayloadFlags::DisableVirtualization flag will be applied.
### Package Trailer
- The pre-existing functionality of enum EPayloadFilter has been moved to a new enum EPayloadStorageType as will only filter payloads based on their storage type.
- EPayloadFilter has been modified to filter payloads based on functionality although at the moment the only thing it can filter for is to return payloads that can be virtualized, it is left for future expansion.
- EPayloadFlags has been reduced to a uint16 with the remaining 2bytes being turned into a new member EPayloadFilterReason.
- This new member allows us to record the exact reason why a payload is excluded from virtualization. If it is zero then the payload can virtualize, otherwise it will contain one or more reasons as to why it is being excluded. For this reason the enum is a bitfield.
- Added overloads of ::GetPayloads and ::GetNumPayloads that take EPayloadFilter rather than a EPayloadStorageType
- Added wrappers around all AccessMode types for FLookupTableEntry.
- FPackageTrailerBuilder has been extended to take a EPayloadFilterReason so that the caller can already provide a reason why the payload cannot be virtualized.
-- As a future peace of work this will probably be changed and we will ask the caller to pass in the owner UObject pointer instead and then we will process the filtering when building the package trailer to a) keep all of the filtering code in one place b) keep the filtering consistent
### PackageSubmissionChecks
- The virtualization process in will now request the payloads that can be virtualized from the package trailer, which respects the new payload flag, rather than requesting all locally stored payloads.
### UObjects
- There is no need for the SoundWave or MeshDescription classes to opt out of virtualization on construction. This will be done when the package is saved and is now data driven rather than being hardcoded.
### DumpPackagePayloadInfo
- The command has been updated to also display the filter reasons applied to each payload
[CL 20240971 by paul chipchase in ue5-main branch]
#rb trivial
#rnx
#preflight 6274be518d32cd80d8cfeaa6
- Adding this so that we can more easily track in a build machine log when a payload gets pulled when debugging unexpected behavior.
[CL 20072222 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
#jira UE-151000
#preflight 627271c83b2ed6f85363f53a
- In the old code when submitting packages for virtualization we would load all local payloads into memory then pass them to the virtualization system to be pushed to persistent storage. This works fine for an average users submits but when resaving large projects we can have submits with thousands of packages containing gigabytes of payloads.
- FPushRequest has been changed to accept either a payload directly (in the form of FCompressedBuffer) or accept a reference to a IPayloadProvider which will return the payload when asked.
- Most of the existing backends make no use of the batched pushing feature and only push one payload at a time. The default implementation of this will be to ask each request one at a time for the payload and then pass it to the backend for a single push. If the IPayloadProvider is being used we will therefor only ever have one payload loaded at a time avoiding memory spikes.
- The source control backend currently does implement the batched pushing feature. This backend writes each payload to disk before submitting them to perforce, so all we have to do is make sure that request one payload, write it to disk, then discard the payload to avoid memory spikes.
- This change required that FPushRequest make it's members private and provide accessors to them, as the caller shouldn't need to care if the payload is loaded or comes via the provider.
- NOTE that this implementation is less efficient if we have packages containing many different payloads as we will end up opening the package, parsing the trailer then load a payload many times over, where as the original code would load all payloads from the file in one go. In practice, given the overhead of the source control backend this didn't make a huge difference and is probably not worth the effort to optimize at this point.
[CL 20040609 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
#jira UE-148223
#preflight 6268efeb2f53f9169aa9b5c5
- Moving this to use the format that the other command lines do will make it easier to find.
- Moved the parsing code from ::ApplySettingsFromCmdline to ::ApplyDebugSettingsFromFromCmdline to indicate that it is a debug command and not expected for production use.
- Removed ::ApplySettingsFromCmdline as we no longer have non-debug commandlines.
- There is no console command version because we cannot change the graph mid process.
[CL 19936091 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
#jira UE-148223
#preflight 6267c4e2853fdb6fddb20be3
- Remove the Core.ContentVirtualizationDebugOptions section from the ini files entirely.
- Remove the ::ApplyDebugSettingsFromConfigFiles method as we no longer load any debug values from the config files.
[CL 19923780 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-148223
#rnx
#preflight 62666fc90634d0904cca8756
- It was probably a bad idea to have debug settings like this applied from the config file as it would be fairly easy for people to accidently forget about them or submit them.
- The single threaded command can now be set via the commandline on start up using the '-VA-XXX' synctax that the system has started to adopt.
- Additionally it has been exposed as a console command so that it can be toggles on and off at runtime.
- Removed 'ForceSingleThreaded' from BaseEngine.ini and removed the parsing code for this from FVirtualizationManager::ApplyDebugSettingsFromConfigFiles.
- Removed 'FailPayloadPullOperations' from BaseEngine.ini, it hasn't been a valid setting for a while.
- Changed ConsoleCommands to ConsoleObjects and use it to store the base class IConsoleObject instead so it can contain console variables
- When pushing payloads we now only take the singlethreaded lock after checking if we have any backends/are enabled.
[CL 19900622 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-148223
#rnx
#preflight 62629b8fbc24759dc73452dc
- Moved the debug values to their own structure to make it clearer that they are for debug purposes.
- Moved the code registering the console commands to a specific method.
[CL 19898824 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#jira UE-148223
#rnx
#preflight 626252fed2644b9ff7d3167b
- Added verbose logging to show when a payload is skipped due to the miss chance, or when a backend skips a payload due to the 'VA.MissBackends' command
[CL 19862821 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#jira UE-148223
#rnx
#preflight 6260f6e4360b45c32a84fae8
- The commandline -VA-MissBackends has been extended to a console command 'VA.MissBackends' that can be used at runtime to disable specific backends or reset the list and enable all of them again.
-- Entering VA.MissBackends without any args will print help documentation to the console/output log and show the user the names of the backends that they can disable.
- Technically we don't need the command line parsing anymore and users could use -dpcvars=VA.MissBackends=X instead but since we already have the code we might as well leave it untouched.
- Disabling backends from pulling payloads for debug purposes has been split off to it's own system stored as a
- IVirtualizationBackend::EOperations has been turned into bit flags, so we no longer need the 'both' entry
- IVirtualizationBackendnow has two EOperations members, one to record the operations that the backend supports and another that allows the operations to be disabled for debugging purposes.
- IVirtualizationBackend now has one method to poll if an operation is supported ::IsOperationSupported, which should not change over the lift time of the backend. In addition there is also a method for disabling/enabling a backend for debugging purposes as well as polling the debug state.
[CL 19845175 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#rnx
#preflight 623ae40e7b69b01ec15da75c
#robomerge FNNC
- We need to expose the code that checks a set of packages for non virtualized payloads and virtualizes them so that the stand alone submission tool can access it.
- Currently I am adding this to the IVirtualizationSystem interface to avoid adding additional interface, but we might want to consider moving this some day as it is not a great fit with the rest of the interface.
[CL 19479757 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#rnx
#preflight 62276138671c913c0515c3b4
- To make it easier to extend the number of parameters when initializing the virtualization system, the function has been changed to accept a single struct, FInitParams, which will contains all potential parameters.
- Calling the version of UE::Virtualization::Initialize without any parameters will fallback to using the default values.
- The virtualization manager now passes the provided project name onto each backend that it creates.
- The source control backend now stored the provided project name and uses that when creating the submit description for new payloads rather than polling FApp for the current project name.
-- This is required for the stand alone virtualization application which will not have a specific project set.
[CL 19303638 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#rnx
#preflight 6225cf4cb57e715222ca15aa
- IVirtualizationSystem now has a new ::Initialize method which is called after it is first created. This allows the system to return a proper error when setting up which was not possible when the set up code ran in the constructor.
- The method also takes a FConfigFile reference which should be used when parsing settings from the config file system.
-- The config file being passed to this method, is either the optional config file provided when calling UE::Virtualization::Initialize, or (if that was not set) we find the GEngineIni config file from the global cache to use.
- This should ensure that the entire virtualization system is now loading the settings from the same set of config files where as before the code was a bit inconsistent (loading it in place, using the GConfig global, or trying to find the file from the global cache)
- If the system fails to create we now log if the problem was that we could not find a valid factory for the given system (meaning the module the system lives in was not yet loaded) or if the initialization phase failed.
- FVirtualizationManager now passes a single FConfigFile into all of it's methods that are parsing settings from it.
[CL 19283763 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#jira UE-142748
#rnx
#preflight 620e06d53e74a28b341e72b6
#lockdown aurel.cordonnier
- Turns out that the option did work fine, it just needed to be tested properly, so removed the assert that was guarding against mixed package trailers when submitting virtualized assets.
- Improved verbose logging when a payload is filtered based on payload size.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 19073541 in //UE5/Release-5.0/... via CL 19090494
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v921-19075845)
[CL 19134901 by paul chipchase in ue5-main branch]
#rb trivial
#jira UE-134434
#rnx
#preflight 620d1a10742ffef420223b15
- Changing a number of log items from 'Log' to 'Display' so that the info shows up more easily.
- Removed a number of log items that added no useful infomation
- Edited some of the existing log items to be clearer.
[CL 19031890 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#jira UE-133473
#preflight 620bf6ae483ff0ae5ec22257
### VirtualizationManager
- Added EPackageFilterMode that can be set via [Core.ContentVirtualization] in the config file to control the default virtualization behavior.
-- When set to EPackageFilterMode::OptOut(the default) all package paths will virtualize unless excluded by a pattern in UVirtualizationFilterSettings::ExcludePackagePaths.
-- When set to EPackageFilterMode::OptIn then no package path will virtualize unless included by a pattern in UVirtualizationFilterSettings::IncludePackagePaths.
- Added a TRACE_CPUPROFILER_EVENT_SCOPE to the constructor and ::MountBackends to better track the set up time costs.
- Change use of FConfigCacheIni::LoadLocalIniFile to use GConfig, there is no need to load our own.
- Improved verbose logging to show when a payload is rejected via filtering.
- We now early out if all payloads being requested in a push are rejected during validation.
- Renamed the FString overload for ::ShouldVirtualizePackage to ::ShouldVirtualize to make the difference clearer.
- Added support for UVirtualizationFilterSettings::ExcludePackagePaths when filtering
### UVirtualizationFilterSettings
- Now has a new FString array ExcludePackagePaths, which contains the paths/patterns used to force packages to be virtualized when filtering.
[CL 19010992 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
#preflight 61fabeb8ad2ae6c3b763a2b9
- Renamed to ::QueryPayloadStatuses since the output data is in the form of an array of FPayloadStatus
- The method now returns EQueryResult rather than bool, which has 'Success' as value 0, all other values indicate an error.
-- This can be expanded in the future if we start to provide more info from backends as to what actually failed.
- FPayloadStatus has been cleaned up so that 'Partial' is not 'FoundPartial' to give a better indication that the payload was found in at least one storage backend, but not all of them.
[CL 18861036 by paul chipchase in ue5-main branch]
#rb Per.Larsson, Devin.Doucette
#jira UE-133497
#rnx
#preflight 61f835491c5ac5523462810a
- A lot of files have been touched but most of this is changing FPayload::IsValid to !FIoHash::IsZero, the main changes are in EditorBulkData/EditorBulkDataTests
- The only difference between FPayloadId and FIoHash is that the former considered the hash of an invalid or empty buffer to be 'invalid' too where as FIoHash would return a valid hash
-- To keep behaviour the same, we only hash payloads in EditorBulkData using a utility method ::HashBuffer which will not hash empty or invalid payloads and return a default FIoHash instead.
-- Unit tests have been extended to prove that the behaviour has not changed.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18806362 in //UE5/Release-5.0/... via CL 18808527 via CL 18821790
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v908-18788545)
[CL 18822154 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#rnx
#jira UE-136758
#preflight 61f3e3626b5aea38e5b4cab7
- Package filtering was disabled when the api was changed to take batches of payloads to push to the backend as the context per payload had to be changed to a raw FString rather than FPackagePath.
- We now convert each context to a FPackagePath and if it is valid test against filtering, if it is not valid we assume that the payload did not come from a package.
- Still tempted to move filtering out of virtualization manager entirely and move it to PackageSubmissionChecks, but that can always be done at a later date.
- Retested the filtering system, plus the ability to add filters to the config files for GameFeature plugins.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18776038 in //UE5/Release-5.0/... via CL 18777356 via CL 18777639
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v903-18687472)
[CL 18777650 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#rnx
#jira UE-140366
#preflight 61f2a6f752396dbfeeede332
- Add a way to poll the virtualization system to see if pushing to a backend storage type is possible or not.
- If we cannot push to persistent backend storage when submitting packages to source control we just log in verbose mode and return rather than giving an error. Submitting a non-virtualized package is no longer the problem it once was.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18769119 in //UE5/Release-5.0/... via CL 18769125 via CL 18769151
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v903-18687472)
[CL 18769154 by paul chipchase in ue5-main branch]
#rb none
#jira UE-140126
#rnx
#preflight 61efbb8e8e5c8d4eaabb987c
- FScopedStatsCounter can only add a single hit/size in it's destructor, so a for loop calling ::AddHit on it was just overwriting the values each time.
- Use TrackCyclesOnly so that 'Timer' only adds the time taken to the stats.
- Now the for loop adds hits to the stats manually for each successful request.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18720805 in //UE5/Release-5.0/... via CL 18720806 via CL 18720836
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v903-18687472)
[CL 18720839 by paul chipchase in ue5-main branch]
