#rb PJ.Kack
#jira UE-148223
#rnx
#preflight 626252fed2644b9ff7d3167b
- Added verbose logging to show when a payload is skipped due to the miss chance, or when a backend skips a payload due to the 'VA.MissBackends' command
[CL 19862821 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#jira UE-148223
#rnx
#preflight 6260f6e4360b45c32a84fae8
- The commandline -VA-MissBackends has been extended to a console command 'VA.MissBackends' that can be used at runtime to disable specific backends or reset the list and enable all of them again.
-- Entering VA.MissBackends without any args will print help documentation to the console/output log and show the user the names of the backends that they can disable.
- Technically we don't need the command line parsing anymore and users could use -dpcvars=VA.MissBackends=X instead but since we already have the code we might as well leave it untouched.
- Disabling backends from pulling payloads for debug purposes has been split off to it's own system stored as a
- IVirtualizationBackend::EOperations has been turned into bit flags, so we no longer need the 'both' entry
- IVirtualizationBackendnow has two EOperations members, one to record the operations that the backend supports and another that allows the operations to be disabled for debugging purposes.
- IVirtualizationBackend now has one method to poll if an operation is supported ::IsOperationSupported, which should not change over the lift time of the backend. In addition there is also a method for disabling/enabling a backend for debugging purposes as well as polling the debug state.
[CL 19845175 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#rnx
#preflight 623ae40e7b69b01ec15da75c
#robomerge FNNC
- We need to expose the code that checks a set of packages for non virtualized payloads and virtualizes them so that the stand alone submission tool can access it.
- Currently I am adding this to the IVirtualizationSystem interface to avoid adding additional interface, but we might want to consider moving this some day as it is not a great fit with the rest of the interface.
[CL 19479757 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#rnx
#preflight 62276138671c913c0515c3b4
- To make it easier to extend the number of parameters when initializing the virtualization system, the function has been changed to accept a single struct, FInitParams, which will contains all potential parameters.
- Calling the version of UE::Virtualization::Initialize without any parameters will fallback to using the default values.
- The virtualization manager now passes the provided project name onto each backend that it creates.
- The source control backend now stored the provided project name and uses that when creating the submit description for new payloads rather than polling FApp for the current project name.
-- This is required for the stand alone virtualization application which will not have a specific project set.
[CL 19303638 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#rnx
#preflight 6225cf4cb57e715222ca15aa
- IVirtualizationSystem now has a new ::Initialize method which is called after it is first created. This allows the system to return a proper error when setting up which was not possible when the set up code ran in the constructor.
- The method also takes a FConfigFile reference which should be used when parsing settings from the config file system.
-- The config file being passed to this method, is either the optional config file provided when calling UE::Virtualization::Initialize, or (if that was not set) we find the GEngineIni config file from the global cache to use.
- This should ensure that the entire virtualization system is now loading the settings from the same set of config files where as before the code was a bit inconsistent (loading it in place, using the GConfig global, or trying to find the file from the global cache)
- If the system fails to create we now log if the problem was that we could not find a valid factory for the given system (meaning the module the system lives in was not yet loaded) or if the initialization phase failed.
- FVirtualizationManager now passes a single FConfigFile into all of it's methods that are parsing settings from it.
[CL 19283763 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#jira UE-142748
#rnx
#preflight 620e06d53e74a28b341e72b6
#lockdown aurel.cordonnier
- Turns out that the option did work fine, it just needed to be tested properly, so removed the assert that was guarding against mixed package trailers when submitting virtualized assets.
- Improved verbose logging when a payload is filtered based on payload size.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 19073541 in //UE5/Release-5.0/... via CL 19090494
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v921-19075845)
[CL 19134901 by paul chipchase in ue5-main branch]
#rb trivial
#jira UE-134434
#rnx
#preflight 620d1a10742ffef420223b15
- Changing a number of log items from 'Log' to 'Display' so that the info shows up more easily.
- Removed a number of log items that added no useful infomation
- Edited some of the existing log items to be clearer.
[CL 19031890 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#jira UE-133473
#preflight 620bf6ae483ff0ae5ec22257
### VirtualizationManager
- Added EPackageFilterMode that can be set via [Core.ContentVirtualization] in the config file to control the default virtualization behavior.
-- When set to EPackageFilterMode::OptOut(the default) all package paths will virtualize unless excluded by a pattern in UVirtualizationFilterSettings::ExcludePackagePaths.
-- When set to EPackageFilterMode::OptIn then no package path will virtualize unless included by a pattern in UVirtualizationFilterSettings::IncludePackagePaths.
- Added a TRACE_CPUPROFILER_EVENT_SCOPE to the constructor and ::MountBackends to better track the set up time costs.
- Change use of FConfigCacheIni::LoadLocalIniFile to use GConfig, there is no need to load our own.
- Improved verbose logging to show when a payload is rejected via filtering.
- We now early out if all payloads being requested in a push are rejected during validation.
- Renamed the FString overload for ::ShouldVirtualizePackage to ::ShouldVirtualize to make the difference clearer.
- Added support for UVirtualizationFilterSettings::ExcludePackagePaths when filtering
### UVirtualizationFilterSettings
- Now has a new FString array ExcludePackagePaths, which contains the paths/patterns used to force packages to be virtualized when filtering.
[CL 19010992 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
#preflight 61fabeb8ad2ae6c3b763a2b9
- Renamed to ::QueryPayloadStatuses since the output data is in the form of an array of FPayloadStatus
- The method now returns EQueryResult rather than bool, which has 'Success' as value 0, all other values indicate an error.
-- This can be expanded in the future if we start to provide more info from backends as to what actually failed.
- FPayloadStatus has been cleaned up so that 'Partial' is not 'FoundPartial' to give a better indication that the payload was found in at least one storage backend, but not all of them.
[CL 18861036 by paul chipchase in ue5-main branch]
#rb Per.Larsson, Devin.Doucette
#jira UE-133497
#rnx
#preflight 61f835491c5ac5523462810a
- A lot of files have been touched but most of this is changing FPayload::IsValid to !FIoHash::IsZero, the main changes are in EditorBulkData/EditorBulkDataTests
- The only difference between FPayloadId and FIoHash is that the former considered the hash of an invalid or empty buffer to be 'invalid' too where as FIoHash would return a valid hash
-- To keep behaviour the same, we only hash payloads in EditorBulkData using a utility method ::HashBuffer which will not hash empty or invalid payloads and return a default FIoHash instead.
-- Unit tests have been extended to prove that the behaviour has not changed.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18806362 in //UE5/Release-5.0/... via CL 18808527 via CL 18821790
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v908-18788545)
[CL 18822154 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#rnx
#jira UE-136758
#preflight 61f3e3626b5aea38e5b4cab7
- Package filtering was disabled when the api was changed to take batches of payloads to push to the backend as the context per payload had to be changed to a raw FString rather than FPackagePath.
- We now convert each context to a FPackagePath and if it is valid test against filtering, if it is not valid we assume that the payload did not come from a package.
- Still tempted to move filtering out of virtualization manager entirely and move it to PackageSubmissionChecks, but that can always be done at a later date.
- Retested the filtering system, plus the ability to add filters to the config files for GameFeature plugins.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18776038 in //UE5/Release-5.0/... via CL 18777356 via CL 18777639
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v903-18687472)
[CL 18777650 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#rnx
#jira UE-140366
#preflight 61f2a6f752396dbfeeede332
- Add a way to poll the virtualization system to see if pushing to a backend storage type is possible or not.
- If we cannot push to persistent backend storage when submitting packages to source control we just log in verbose mode and return rather than giving an error. Submitting a non-virtualized package is no longer the problem it once was.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18769119 in //UE5/Release-5.0/... via CL 18769125 via CL 18769151
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v903-18687472)
[CL 18769154 by paul chipchase in ue5-main branch]
#rb none
#jira UE-140126
#rnx
#preflight 61efbb8e8e5c8d4eaabb987c
- FScopedStatsCounter can only add a single hit/size in it's destructor, so a for loop calling ::AddHit on it was just overwriting the values each time.
- Use TrackCyclesOnly so that 'Timer' only adds the time taken to the stats.
- Now the for loop adds hits to the stats manually for each successful request.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18720805 in //UE5/Release-5.0/... via CL 18720806 via CL 18720836
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v903-18687472)
[CL 18720839 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#jira UE-136126
#rnx
#preflight
### VirtualizationSystem
- Added a new overload for Push to VirtualizationSystem that takes an array of FPushRequest, which is a new structure representing a single payload request.
- Filtering by package name is currently disabled, this is because the API has been forced into changing and passing the package name in via a FString rather than FPackagePath which means we would need to be more careful. This will be done in a future submit.
- The backend interface has been extended to also have a batch version of PushData, by default this will attempt to submit each request one at a time so payloads don't have to try and implement a batched version if there is no need.
- The context being passed with a payload when being pushed has been changed from FPackagePath to FString due to include order issues, as the FPackagePath lives in CoreUObject and the API for virtualization lives in Core. Additionally in the future the payloads might not be owned by a package (there is nothing specifically enforcing this) so the context being a string makes more sense.
- NOTE: Due to the context change we currently no longer support the filtering feature, which allows for payloads belonging to packages under specific directories to be excluded from virtualization. This is something that will be solved in a future submit.
### SourceControlBackend
- Now that we can submit multiple payloads in the same submit, the CL description has been changed slightly. We will now print a list of payload identifiers -> the package trying to submit that payload. This will only tell the users which package originally caused the payload to submit. If a user submits a new package at a later date that contains the same payload we will not be updating the description.
### PackageSubmissionChecks
- Converted the submission process to use the new batch push operation in VirtualizationSystem.
-- This means that we do a single push and then have to update the package trailers to convert the now pushed payloads from local to virtualized.
- Added new define UE_PRECHECK_PAYLOAD_STATUS that makes it easy to toggle off the checks to see which payloads need to be submitted to the persistent backend. This is useful to test if it actually helps speed up the overall operations or if it is faster to just perform the batch push operations on all payloads and check the return values.
-- The hope is that over time the submission processes will become fast enough that we can remove the precheck.
- Fixed up logging to not always assume more than one package or payload.
### General Notes
- Errors and logging is now a bit more vague as we often not just report that X payloads failed etc rather than specific payload identifiers. This probably doesn't affect the user too much since those identifiers as fairly meaningless to them anyway.
- The source control submission could be further optimized by first checking the status of the files in thge depot and only then creating/switching workspace etc.
- As currently written, we need to load all of the payloads into memory, then the backends will do what they need (in the case of source control this results in the payloads being written to disk then submitted) which could create quite a large memory spike when submitting a large number of packages.
-- One solution would be to change the batch push API to take a "payload provider" interface and have the payloads requested as needed rather than passing in the FCompressedBuffer directly. This would let us immediately write the payload to disk for submission then discard it from memory, preventing larger spikes. Although it could cause overhead if there are multiple backends being submitted to. Internally we are unlikely to have more than one backend per storage solution so maybe we should just make it a config option?
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18403735 in //UE5/Release-5.0/... via CL 18403737
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v896-18170469)
[CL 18403738 by paul chipchase in ue5-release-engine-test branch]
#rb PJ.Kack
#rnx
#preflight 61a795773c29b3cf13cd8250
### PackageSubmissionChecks
- Under the old model submitting a large number of packages could be very slow as each package would check each payload that it owns and is currently stored locally one at a time. For the source control backend this created quite a large overhead even when all of the payloads were already virtualized.
- Now we do a pass over the submitted files to find all valid packages that have package trailers with locally stored payloads, gather the payloads into a single list and then query that in one large batch.
- Once we find which payloads are not in permanent storage (note that in the case where a project is using multiple permanent storage solutions, if a payload is missing in one backend it counts as not being in permanent storage) we then attempt to virtualized them.
- Only after all of this is done will we create the truncated copy of each package and then append the updated trailer to each one. In theory doing it in this order this might slightly increase the change of submit failures that occur after virtualization that result in a package never being submitted and orphaned payloads being added to permanent storage, but this will always be a risk.
- Added an assert to fire if we detect a trailer with some virtualized and some local payloads. This should be a supported feature but needs proper testing first before we can allow it. With out current project settings no project should actually encounter this scenario.
- To make the code easier to follow we now early out of the entire check when errors are encountered.
- Added logging at various stages in the process to help show the user that something is happening and make problems easier to identify in the future.
- Notes
-- There is a lot of handling of invalid FPayloads. This is because it is currently possible to add empty payloads to the trailer which is inefficient and wastes space. The trailer will be modified to reject empty payloads in a future update at which point a lot of this handling can be removed.
-- This could've also been solved by not fully rehydrating a package on save by the end user, which will be added as a project setting in a future piece of work, but this approach will solve the edge case when the user does have a large amount of hydrated packages which contain payloads that are already virtualized so it was better to fix that now while we have good test cases for it.
-- We still have scaling problems with large number of package being submitted that do have payloads that need to be virtualized, this will be fixed by extending IVirtualizationSystem::Push to also accept batches of payloads in future work.
-- OnPrePackageSubmission could be broken up into smaller chunks to make the code easier to follow. This will be done after the batch payload submission work is done.
### VirtualizationSystem
- EStorageType has been promoted to enum class.
- Added a new enum FPayloadStatus to be used when querying if a payload exists in a backend storage system or not.
- Add a new method ::DoPayloadsExist which allows the caller to query if one or more payloads exists in the given backend storage system.
### VirtualizationManager
- Implemented ::DoPayloadsExist. First we get the results from each backend in the storage system (which return as true or false from each backend) then total how many backends found the payload in order to set the correct status.
### IVirtualizationBackend
- ::DoesPayloadExist which queries the existence of a single payload has been added to the interface. Most backends already implemented this for private use and if so have had their implementation renamed to match this.
- Also added ::DoPayloadsExist which takes a batch of FpayloadIdsto query. Some backends can deal with a batch of payload ids much more efficiently than one at a time, although the default implementation does call ::DoesPayloadExist for each requested payload.
-- The default implementation prevents every backend from needing to implement the same for loop but does allow backends that can gain from batching to override it.
### VirtualizationSourceControlBackend
- This backend does override ::DoPayloadsExist and implements it's own version as it tends to perform very poorly when not operating on larger batches.
- In this case ::DoesPayloadExist calls back to ::DoPayloadsExist to check each payload rather than implement as specific version.
### PackageTrailer
- The trailer can now be queries to request how many payloads of a given type it contains
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18339847 in //UE5/Release-5.0/... via CL 18339852
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)
[CL 18339859 by paul chipchase in ue5-release-engine-test branch]
This represents UE4/Main @18073326, Release-5.0 @18081140 and Dev-PerfTest @18045971
[CL 18081471 by aurel cordonnier in ue5-release-engine-test branch]
This represents UE4/Main @17911760, Release-5.0 @17915875 and Dev-PerfTest @17914035
[CL 17918595 by aurel cordonnier in ue5-release-engine-test branch]
