#rb Per.Larsson
#jira UE-182205
#preflight 643932b2211b661dc4f594c6
- The enums don't contain any entries at the moment, the goal of this change is to reduce the friction that future API changes might have.
- The older versions of PullData/PushData have been deprecated but this won't flag compiler warnings if people have derived their own implementations of IVirtualizationBackend, so in addition to deprecating the methods have been set to 'final' and have checkNoEntry implementations which is the best we can do at the moment.
-- It is unlikely that people have actually done this as the documentation on adding new backend types will only go out in 5.3.
[CL 25039959 by paul chipchase in ue5-main branch]
#rb none
#jira UE-180383
#preflight 64183a83691c5ebc155d21a2
- The payload pull error dialog now suports an additional custom line in the dialog that can be set via Engine:[Core.VirtualizationModule]:AdditionalPullErrorMsg. The intent is to allow teams to provide additional help and info to their users.
- The default error text is now localized.
[CL 24714178 by paul chipchase in ue5-main branch]
#rb none
#jira UE-180383
#preflight 64147be21c44ff98b969b1aa
- The new flow will display an error dialog to the user informing them of the problem and ask if they want to retry or quit the process. This is quite a shift if how we have been handling failed payload pulls up until and this is the first rough implementation so for now it is off by default.
- We only show one dialog at a time and if a new error occurs while we are waiting for user input, that thread is set to wait on the existing dialogs outcome.
[CL 24689856 by paul chipchase in ue5-main branch]
#rb trivial
#jira none
#rnx
#preflight 6413305ef81cce1dd8daae8b
- We no longer error if the values cannot be parsed from the config file, our defaults in BaseEngine.ini are the same defaults that we have in code anyway.
- We now trust that FConfigFile will not write to the variable passed into it on parse failure so there is no need to parse to a local variable first, then apply that to the member if the parse was a success.
[CL 24671648 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#jira UE-177013
#preflight 63ece283956709374a03eb8a
- The existing config file option 'EnableCacheAfterPull' is now deprecated. If we detect that it is still in the config file we will warn the user about it but continue to apply the value to maintain compatibility.
- Add a new config file option 'EnableCacheOnPull' which controls if a payload is pushed to cached storage after it has been pulled from persistent storage. (Default true)
- Add a new config file option 'EnableCacheOnPush' which controls if a payload is pushed to cached storage before it is pushed to persistent storage. (Default true)
- We now store the info about caching in a single enum bitfield ECachingPolicy.
- Added an up to date comment section about ini file set up for FVirtualizationManager which was previously missing.
[CL 24230636 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-177015
#preflight 63eb71ea956709374ab61d5e
- So far nobody has proposed any worthwhile use case for virtualizing engine content given the following downsides:
-- We cannot ever do so internally as we would be unable to distribute the virtualized packages.
-- Projects and the engine packages cannot always be accessed via the same client spec, leading to problems if we try.
-- Engine content is typically small and not suited to virtualization in the first place.
- If we remove the option then we can exclude the engine packages from our VA commandlets which will give less confusing output (at the moment if you try to virtualize a project containing a single package we will report that several thousand packages were found as we need to parse engine content too)
- Deprecated the ini file options, if we detect them we will warn the user that the option is still present and that engine content now will never virtualize.
[CL 24208899 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
#preflight 63d7b0135428dc67b160b3aa
- CL 23859876 introduced optimizations that consider a filepath to be invalid if the path contains consecutive path separators (DirA//DirB for example instead of DirA/DirB) and although no issues were seen when testing we fairly quickly ran into a user facing errors when trying to submit work from the editor.
- The users client spec root had a '\' at the end of it where as most internal devs do not. This in turn caused problems when parsing file info from source control as the code was assuming that the client spec root had no trailing separator and was adding one.
- Although the system providing the bad filepaths will be fixed it makes sense to sanitize the filepath input for these two processes as any system, including 3rd party ones could be providing the paths.
[CL 23919107 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-169626
#rnx
#preflight 63bd670a71079a8d1c0e837b
- Since the API was forcing the caller to pass in a results structure to be filled in, we might as well make it the return value.
- Added a ::WasSuccessful method to the results structures that can be used instead of checking if the result had errors or not.
- Remove the reset method from FVirtualizationResult/FRehydrationResult as they no longer need it.
- The older deprecated methods still use the results enum, so we cannot easily deprecate those enums yet.
[CL 23626072 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-169626
#rnx
#preflight 63bd0ebfd862fdd347bce1fe
### VA System
- This allows us to provide the user with more ways to customize the rehydration and return more detailed info about it if the calling code wishes to log additional info. In both cases we can extend the options and the data returned without changing the API.
- At the moment the only flag we support is 'Checkout', which requests that the rehydration process checkout any package that it needs to modify rather than giving an error. This means that the user does not need to check packages out from revision control before running the rehydration process.
-- We still check if packages can be modified and warn the user if they can't, as package files could be locked in other ways.
- The rehydration process will now long the time taken if verbose logging is set for the category 'LogVirtualization'
### UnrealVirtualizationTool
- The virtualize command now reports how many packages were checked out if the flag was set.
- The rehydration command now supports a '-Checkout' commandline flag, which when enabled will use the new api to checkout the packages that need to be checked out when rehydrated.
[CL 23625132 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-169626
#rnx
#preflight 639c4112012902cb8db43e13
- This allows us to provide the user with more ways to customize the virtualization and return more detailed info about it if the calling code wishes to log additional info. In both cases we can extend the options and the data returned without changing the API.
- Previously if we virtualized a package that was not checked out in revision control we would warn the user and then skip updating the package file on disk. This means the payloads would be uploaded but the user would be left with no local changes. Since sometimes we know we don't need to check out any package (virtualizing the packages in a change list for example) we don't want to always incur the cost of polling reivision control to see which packages do need checking out. This is why we now allow the caller to request package files be checked out via the new options enum EVirtualizationOptions.
-- If the EVirtualizationOptions::Checkout flag is provided we will poll the revision control status of all package files and then check out those which need it.
-- We still check if packages can be modified and warn the user if they can't, as package files could be locked in other ways.
- Added a new utility function to SourceControlUtilties to make it easier to check out packages. There is similar functionality elsewhere in the code base but the virtualization module is too low level to make use of it.
- Updated existing code that calls ::TryVirtualizePackages and add cases of ''using namespace UE::Virtualization' where required to improve readability.
- The UnrealVirtualizationTool now supports a new cmdline option "-checkout" that can be used when virtualizing packages. This will checkout any package that was actually virtualized so the result can be saved back out to the workspace domain. This means we no longer require the caller to have checked out the packages before running the tool.
[CL 23536832 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-171492
#rnx
#preflight 6391f02d67018b14b581c0b0
- It is possible that a project might want to have persistent storage enabled but no caching storage, it is probably a bad idea but it should work. However before this fix doing so would cause an error to be logged during the virtualization of a package and fail the process.
- During the virtualization process we now check if pushing is enabled for cache storage before trying to push there, if not then we log that the phase is being skipped.
- Even though the above fixes the problem we would still get odd results when pushing payloads to cached storage without any backends enabled.
-- We now check if the pushing process is disabled before processing requests, so we can fail faster. There is now also a proper error value for this that can be stored in each request (FPushRequest) where as previously they'd all still display "pending"
-- We also check to see if the requested push type has any associated backend before processing requests and have a specific error value for this. Note that we consider
- We also now force the visibility of the slow task progress bar, as otherwise we might not end up displaying the virtualization progress to the progress dialog (this is a work around to a long standing problem with the slow task system due to the time limit associated with updating the dialog via EnterProgressFrame, this needs to be fixed elsewhere)
#ushell-cherrypick of 23445439 by paul.chipchase
[CL 23461405 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-168452
#rnx
#preflight 635a4773f1493f97864fb660
- IVirtualizationBackend can now have its operations disabled after it is created.
-- This means that a backend that only supported pushing and not pulling would end up with no valid operations if marked as read only. This will not assert because we do this after the backend is created and we only check on creation.
[CL 22810918 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-168057
#rnx
#preflight 635656e6e6096564af1bc117
- Work on UE-165308 broke the following commandlines as they were accidently converted to cmdline switches rather than values
-- VABackendGraph
-- VAMissBackends
-- VAMissChance
- ::IsCmdLineSet has now been split into two utility wrappers, one for params (bools) and one for values.
[CL 22792790 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-163093
#rnx
#preflight 63526abcae33b04ec1ee4a65
- The caller can now request more than one payload to be pulled froim the system at a time. In theory this will allow backends to pull data quicker.
-- The file system backend works with the new system but makes no attempt to take advantage as the backend is not intended for serious production use.
-- The DDC backend should be taking full advantage of the new batching API
-- Note that although the source control API does attempt to take advantage of batching, the internal source control API implementation is not doing so. This will be addressed in a future submit.
- This does make the pulling logic a bit more complicated in FVirtualizationManager as we need to deal with some payloads being found in the first backend, some in the second and some in the third etc.
-- To help deal with this a new class FPullRequestCollection has been added which abstracts a lot of complexity away.
[CL 22707955 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-165308
#rnx
#preflight 634ff17e8449f85a47390fbe
- It was pointed out that a command line option containing '-' characters could confuse parsers and it is not worth the risk to keep them.
- Now we accept versions of each command line without the '-' character which will be the version that we document for users in 5.1
[CL 22650940 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
#preflight 634805f7f93be0f634b8ed5b
- Added a new FArchive 'FFixedBufferWriterArchive' that will let me serialize to an already allocated, fixed sized memory buffer.
- Moved the code for opening and reading packages to shared utility functions so that it is easier to keep the error messages consistent.
- Moved common code to a new function ::TryRehydrateBuilder so that both rehydration methods can share it.
- Note that technically we are changing the API since now we accept a TConstArrayView for the package file paths for both the virtualization and rehydration paths, but existing code will be compatible with this change.
[CL 22511406 by paul chipchase in ue5-main branch]
#rb trivial
#jira UE-160942
#rnx
#preflight 634557f5c272487ed4ab2014
- In addition to adding the hit counter (which can be very useful when debugging) the stats logging code has been factored out to a single lambda tht is called for each set of stats to make it easier to update in future.
[CL 22457126 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
#preflight 633ad936c250f60502b2d8cb
- Everything was working but when a cvar is set via a device profile we expect the cvar to be registered sooner, hence the warning.
- Added a new define UE_USE_GLOBAL_CVAR to FVirtualizationManager.cpp, when set we use a static global cvar for 'VA.AllowPkgVirtualization' rather then creating and registering one when the system is mounted.
- Since the cvar can now outlive the manager we also needed to add a way to remove the bound lambda from the cvar when the manager is destroyed.
[CL 22322164 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-160942
#rnx
#preflight 6336f2ab5c2225fe5f6c08a5
- Removed the status enum from FPushRequest and replaced it with a class that represents the push result instead.
-- This lets us return more context about problems. At the moment it is only being used to return the filter reason if a payload is filtered out. In the future we could return per payload error messages etc.
- Now that we return more detailed info about the payload push we no longer report payloads that were already in the backend as being pushed which could cause some very misleading stats to be shown.
- Changed the backends to use the new results system.
-- Not all paths were correctly setting the state in the file system backend.
-- The source control backend does not set the result in all paths, which will leave the results in the "pending" state. Which will still mark the payload as not uploaded and so is safe to do.
- Removed the pushing of a single method from IVirtualizationBackend as it was only used in one place internally, so we might as well just change that code to call the batch push overload.
[CL 22279127 by paul chipchase in ue5-main branch]
