#rb Per.Larsson
#rnx
#jira UE-151377
#preflight 628364050039ea57a52d6989
### Virtualization
- [Core.ContentVirtualization] in the engine ini file now supports an array called 'DisabledAsset' which can be used to name asset types that should not virtualize their payloads.
-- By default (in BaseEngine.ini) we have disabled the StaticMesh asset as we know it will crash if a payload is missing and the SoundWave asset as it still is pending testing.
- This new way to disable virtualization is data driven. The older hard coded method has not been removed but will likely be reworked in a future submit.
- Now when an editor bulkdata is adding it's payload to the package trailer builder during package save it will poll the virtualization system with a call to the new method ::IsDisabledForObject by passing in it's owner.
-- If the owner is valid and was present in the 'DisabledAsset' array then the method will return true and the EPayloadFlags::DisableVirtualization flag will be applied.
### Package Trailer
- The pre-existing functionality of enum EPayloadFilter has been moved to a new enum EPayloadStorageType as will only filter payloads based on their storage type.
- EPayloadFilter has been modified to filter payloads based on functionality although at the moment the only thing it can filter for is to return payloads that can be virtualized, it is left for future expansion.
- EPayloadFlags has been reduced to a uint16 with the remaining 2bytes being turned into a new member EPayloadFilterReason.
- This new member allows us to record the exact reason why a payload is excluded from virtualization. If it is zero then the payload can virtualize, otherwise it will contain one or more reasons as to why it is being excluded. For this reason the enum is a bitfield.
- Added overloads of ::GetPayloads and ::GetNumPayloads that take EPayloadFilter rather than a EPayloadStorageType
- Added wrappers around all AccessMode types for FLookupTableEntry.
- FPackageTrailerBuilder has been extended to take a EPayloadFilterReason so that the caller can already provide a reason why the payload cannot be virtualized.
-- As a future peace of work this will probably be changed and we will ask the caller to pass in the owner UObject pointer instead and then we will process the filtering when building the package trailer to a) keep all of the filtering code in one place b) keep the filtering consistent
### PackageSubmissionChecks
- The virtualization process in will now request the payloads that can be virtualized from the package trailer, which respects the new payload flag, rather than requesting all locally stored payloads.
### UObjects
- There is no need for the SoundWave or MeshDescription classes to opt out of virtualization on construction. This will be done when the package is saved and is now data driven rather than being hardcoded.
### DumpPackagePayloadInfo
- The command has been updated to also display the filter reasons applied to each payload
[CL 20240971 by paul chipchase in ue5-main branch]
#rb trivial
#rnx
#preflight 62820fb0046b81bf93921c6b
- When adding the check to see if a package file could be written to, I tried to do this as early as possible to avoid virtualizing payloads for packages that can't be modified.
- This didn't take into account submits via the editor, when the package is loaded which means the editor will have the package file locked for edit.
- For now I have gone with the simple fix, we check if we can write to the package once we have called ResetLoader on the package. This means that we will virtualize the payloads then potentially skip removing them from the package file if another process has a file lock.
- This approach will minimize impact to the users for now but we should revisit this in the future to try and reduce the error scope further.
[CL 20221653 by paul chipchase in ue5-main branch]
#rb trivial
#jira UE-143675
#rnx
#preflight 6274d9b7732d07cf93f55621
### Problem
- When using the stand alone virtualization tool, to virtualize packages directly from p4v it is possible that the editor will have locked the package files. This means we cannot modify it and remove the payload making the virtualization of the package file impossible.
- At some point in the future we will add the ability for the tool to communicate with the editor to ask that the lock be relinquished OR there is the possibility that we will be moving away from the editor permantly locking package files.
- As this problem will be eventually be solved we'd rather this problem not present to the end users as an error as this will cause them to view the tool as annoying and fiddly to use.
- So for now we will treat this problem as a warning and allow the end user to continue and submit the package non-virtualized.
### Fix
- At the moment once we detect a package has local payloads that might need virtualization we check to see if the file could be written to.
- This might raise some false positives if the local payloads are filtered out or otherwise not suppose to be virtualized.
- At the moment this sort of filtering is done when pushing the payloads meaning we could end up pushing payloads then being unable to modify the package.
- A future work item will be added to clean this up at some point in the future (depending on when we will start work to remove the problem entirely)
[CL 20073356 by paul chipchase in ue5-main branch]
#rb trivial
#rnx
#preflight 6274be518d32cd80d8cfeaa6
- Adding this so that we can more easily track in a build machine log when a payload gets pulled when debugging unexpected behavior.
[CL 20072222 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-151000
#rnx
#preflight 627291732f6d177be3c292e3
- When submitting large batches it was found that once a cl description reached a certain length, that p4v would lock up attempting to display it in the history tab, so we want to be able to make smaller submits to keep the description to a managable size.
- In addition it is better for our server set up to make multiple smaller batch submits than insanely large ones.
- The max size of the batch can be set when creating the virtualization graph in the config file.
[CL 20055325 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
#jira UE-151000
#preflight 627271c83b2ed6f85363f53a
- In the old code when submitting packages for virtualization we would load all local payloads into memory then pass them to the virtualization system to be pushed to persistent storage. This works fine for an average users submits but when resaving large projects we can have submits with thousands of packages containing gigabytes of payloads.
- FPushRequest has been changed to accept either a payload directly (in the form of FCompressedBuffer) or accept a reference to a IPayloadProvider which will return the payload when asked.
- Most of the existing backends make no use of the batched pushing feature and only push one payload at a time. The default implementation of this will be to ask each request one at a time for the payload and then pass it to the backend for a single push. If the IPayloadProvider is being used we will therefor only ever have one payload loaded at a time avoiding memory spikes.
- The source control backend currently does implement the batched pushing feature. This backend writes each payload to disk before submitting them to perforce, so all we have to do is make sure that request one payload, write it to disk, then discard the payload to avoid memory spikes.
- This change required that FPushRequest make it's members private and provide accessors to them, as the caller shouldn't need to care if the payload is loaded or comes via the provider.
- NOTE that this implementation is less efficient if we have packages containing many different payloads as we will end up opening the package, parsing the trailer then load a payload many times over, where as the original code would load all payloads from the file in one go. In practice, given the overhead of the source control backend this didn't make a huge difference and is probably not worth the effort to optimize at this point.
[CL 20040609 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
#jira UE-147948
#preflight 6271209be16e280be60b5a62
- In the worst case scenario (where a machine can only access the source control backend) we will want to limit the potential number of concurrent connections that they can make.
- In testing, not limiting the connections could cause the occasional connection failure and I imagine having a full team running in the worse case scenario would quickly overwhelm the server.
- Currently we default to 8 connections (the limit allowed in UGS)
- The semaphore is a quick and dirty implementation similar to std::counting_semaphore. I am not spending much time on it as the worst case scenario is not expected to be common and if it does occur, the system is going to be painfully slow no matter how good the thread throttling.
### Future work
- This work could be taken a lot further by either
-- a) reusing connections rather than creating one for each pull, which will require changes to our source control api
-- b) gathering requests from many requesting threads once the limit has been reached and then sending the requests in batches
--c) reserve one connection for the game thread to prevent the user waiting?
[CL 20024594 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
#jira UE-148223
#preflight 6268efeb2f53f9169aa9b5c5
- Moving this to use the format that the other command lines do will make it easier to find.
- Moved the parsing code from ::ApplySettingsFromCmdline to ::ApplyDebugSettingsFromFromCmdline to indicate that it is a debug command and not expected for production use.
- Removed ::ApplySettingsFromCmdline as we no longer have non-debug commandlines.
- There is no console command version because we cannot change the graph mid process.
[CL 19936091 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
#jira UE-148223
#preflight 6267c4e2853fdb6fddb20be3
- Remove the Core.ContentVirtualizationDebugOptions section from the ini files entirely.
- Remove the ::ApplyDebugSettingsFromConfigFiles method as we no longer load any debug values from the config files.
[CL 19923780 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-148223
#rnx
#preflight 62666fc90634d0904cca8756
- It was probably a bad idea to have debug settings like this applied from the config file as it would be fairly easy for people to accidently forget about them or submit them.
- The single threaded command can now be set via the commandline on start up using the '-VA-XXX' synctax that the system has started to adopt.
- Additionally it has been exposed as a console command so that it can be toggles on and off at runtime.
- Removed 'ForceSingleThreaded' from BaseEngine.ini and removed the parsing code for this from FVirtualizationManager::ApplyDebugSettingsFromConfigFiles.
- Removed 'FailPayloadPullOperations' from BaseEngine.ini, it hasn't been a valid setting for a while.
- Changed ConsoleCommands to ConsoleObjects and use it to store the base class IConsoleObject instead so it can contain console variables
- When pushing payloads we now only take the singlethreaded lock after checking if we have any backends/are enabled.
[CL 19900622 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-148223
#rnx
#preflight 62629b8fbc24759dc73452dc
- Moved the debug values to their own structure to make it clearer that they are for debug purposes.
- Moved the code registering the console commands to a specific method.
[CL 19898824 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#jira UE-148223
#rnx
#preflight 626252fed2644b9ff7d3167b
- Added verbose logging to show when a payload is skipped due to the miss chance, or when a backend skips a payload due to the 'VA.MissBackends' command
[CL 19862821 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#jira UE-148223
#rnx
#preflight 6260f6e4360b45c32a84fae8
- The commandline -VA-MissBackends has been extended to a console command 'VA.MissBackends' that can be used at runtime to disable specific backends or reset the list and enable all of them again.
-- Entering VA.MissBackends without any args will print help documentation to the console/output log and show the user the names of the backends that they can disable.
- Technically we don't need the command line parsing anymore and users could use -dpcvars=VA.MissBackends=X instead but since we already have the code we might as well leave it untouched.
- Disabling backends from pulling payloads for debug purposes has been split off to it's own system stored as a
- IVirtualizationBackend::EOperations has been turned into bit flags, so we no longer need the 'both' entry
- IVirtualizationBackendnow has two EOperations members, one to record the operations that the backend supports and another that allows the operations to be disabled for debugging purposes.
- IVirtualizationBackend now has one method to poll if an operation is supported ::IsOperationSupported, which should not change over the lift time of the backend. In addition there is also a method for disabling/enabling a backend for debugging purposes as well as polling the debug state.
[CL 19845175 by paul chipchase in ue5-main branch]
#rb Per.Larsson, Sebastian.Nordgren
#jira UE-147948
#preflight 6256ba2f647ad886b380d007
- In some cases people are getting minor connection failures to the perforce server which causes the p4 print command to fail. In these cases it would be better to retry the command rather than failing to get the payload entirely
- Currently the default will be 2 attempts to pull a payload with a 100ms wait between them, this can be changed when setting up the backend graph as optional config values "RetryCount=" and "RetryWaitTime="
-- Note that RetryWaitTime is in ms
- Moved the parsing of the command line to it's own method in the backend.
[CL 19752733 by paul chipchase in ue5-main branch]
#rb trivial
#rnx
#preflight 623c5912c3399da9533ffd30
- The package trailer will be on by default soon and the option to remove it disabled, so this check is a bit pointless anyway.
[CL 19493442 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#rnx
#preflight 623ae40e7b69b01ec15da75c
#robomerge FNNC
- We need to expose the code that checks a set of packages for non virtualized payloads and virtualizes them so that the stand alone submission tool can access it.
- Currently I am adding this to the IVirtualizationSystem interface to avoid adding additional interface, but we might want to consider moving this some day as it is not a great fit with the rest of the interface.
[CL 19479757 by paul chipchase in ue5-main branch]
#rb Sebastian.Nordgren
#rnx
#preflight 622f645a19287beb2c359176
- Use the SetRequireDirPathEndWithSeperator option with the FUpdateStatus operation as we are checking depot file paths. If we do not use this option they can be seen as network paths and will be checked against the file system which can take some time to time out.
- Wrapped this up in a utility function so that it can be more easily updated in the future.
[CL 19373558 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
#jira UE-136480
#preflight 62289b59f4469cadac10eb91
- The backend can now connect to source control without forcing the rest of the editor to connect to source control.
- If the user disconnects the editor from source control, the backend will still be able to function using it's own connection.
-When the backend needs to change the current workspace in order to submit payloads we no longer invalidate the file cache used by the editor and so the user will no longer have to manually refresh files in the content browsers to restore the source control status icon.
[CL 19318970 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#rnx
#preflight 62276138671c913c0515c3b4
- To make it easier to extend the number of parameters when initializing the virtualization system, the function has been changed to accept a single struct, FInitParams, which will contains all potential parameters.
- Calling the version of UE::Virtualization::Initialize without any parameters will fallback to using the default values.
- The virtualization manager now passes the provided project name onto each backend that it creates.
- The source control backend now stored the provided project name and uses that when creating the submit description for new payloads rather than polling FApp for the current project name.
-- This is required for the stand alone virtualization application which will not have a specific project set.
[CL 19303638 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#rnx
#preflight 6225cf4cb57e715222ca15aa
- IVirtualizationSystem now has a new ::Initialize method which is called after it is first created. This allows the system to return a proper error when setting up which was not possible when the set up code ran in the constructor.
- The method also takes a FConfigFile reference which should be used when parsing settings from the config file system.
-- The config file being passed to this method, is either the optional config file provided when calling UE::Virtualization::Initialize, or (if that was not set) we find the GEngineIni config file from the global cache to use.
- This should ensure that the entire virtualization system is now loading the settings from the same set of config files where as before the code was a bit inconsistent (loading it in place, using the GConfig global, or trying to find the file from the global cache)
- If the system fails to create we now log if the problem was that we could not find a valid factory for the given system (meaning the module the system lives in was not yet loaded) or if the initialization phase failed.
- FVirtualizationManager now passes a single FConfigFile into all of it's methods that are parsing settings from it.
[CL 19283763 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#jira UE-142926
#preflight 621e1f177f2803279b575764
- Now when payloads are submitted, we will submit them from within the project's saved directory (by default) this now works because we write a .p4ignore file to the submission directory which will override any additional ignore file higher up in the directory structure.
-- A new utility function ::CreateSubmissionSessionDirectory was added to make sure that the directory we are submitting from has a p4ignore file. It is added in it's own scope as it is assumed that additional functionality will be added here later.
- If a user wishes to choose a different location to submit from, they can override the environment variable "UE-VirtualizationWorkingDir" to provide a different location. This might be useful if the user does not have a huge amount of spare space on the disk that the project is running from.
- There is also an optional entry to the ini file set up 'SubmitFromTempDir' (false by default) which when set to true will cause the system to attempt to submit the payload files from the machines temp directory instead.
-- There isn't an easy way for users to override this locally, which reduces the usefulness of the option. This might be worth improving in the future although that time would be better spent removing the need for the directory in the first place.
- We now try to find/create the root directory to submit payloads from when the backend is created. Failure at this point is not expected and will prevent the backend from working and so is considered an error.
- Added more logging to track the session guid when submitting and to be clear about where the files are being submitted from.
[CL 19197367 by paul chipchase in ue5-main branch]
