#rb none
#jira UE-174785
#rnx
#preflight 63ff0ceeae54ee4ce912c17c
- Recently how we discover packages for the VA commandlets was changed to be much faster but it was also changed to automatically filter out all engine content as we removed the optional (opt in) feature of virtualizing engine content.
- This meant that 'CheckForVirtualizedContent' would not be able to check engine packages if '-CheckEngine' was set as we were not providing the commandlet with them.
- ::FindAllPackages has been renamed ::FindPackages and both it and ::DiscoverPackages now accept a flags parameter that allows the caller to request that engine content be excluded.
- Although most commandlets do not want engine content, having the engine content filter as opt in closer matches existing apis and will probably lead to less mistakes.
[CL 24459148 by paul chipchase in ue5-main branch]
#rb trivial
#jira UE-176611
#preflight 63fe18fcf43e53f68189d869
- Renamed UValidateVirtualizedContentCommandlet to UCheckForVirtualizedContentCommandlet. P4 rename was not done because I wanted to repurpose UValidateVirtualizedContentCommandlet and adding new versions of the code files after renaming them is likely to cause weird p4 merge issues in the future.
[CL 24442899 by paul chipchase in ue5-main branch]
#rb trivial
#jira UE-178518
#preflight 63fcb6f560163ed3c22e3d35
- The option will only checkin packages that were checked out by the commandlet. Local files or packages that were already open for edit will not be submitted.
[CL 24422913 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-167454
#rnx
#preflight 63f5e5b9ac06ce789f047431
- Replace use of ::NormalizePackageNames with the asset registry for finding packages.
- We can use the async version of SearchAllAssets so that we don't have to wait on the asset registry cache being saved to disk.
- ::FindAllPackages now returns all paths in our standard form (relative to Engine\Binaries\<Platform>) where as NormalizePackageNames would return a mix of formats depending on the mount point location.
[CL 24359197 by paul chipchase in ue5-main branch]
#rb trivial
#jira UE-177015
#preflight 63eb87720a06073fef3110dd
- In CL 24208899 we removed the option to virtualize engine content which means that the virtualization commandlets can use NORMALIZE_ExcludeEnginePackages when searching for packages.
- This helps make our logging a bit more readable, for example if someone were to run the virtualize project commandlet on a new project containing only a single package we will no longer return the number of engine packages + 1 when reporting how many packages are being considered for virtualization. This should help the user to have more confidence in the numbers that we are reporting.
[CL 24209315 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-174785
#rnx
#preflight 63d3d027f6267152017a68dd
- There are some circumstances where we want to be sure that locations do not contain virtualized data. As an example we were to accidently virtualize an engine package and then distribute that with an engine release it will not work for people. This commandlet is designed to be run as part of a CIS to detect this sort of thing and raise errors so that they can be corrected.
- We support three forms of validation
-- '-CheckEngine' which errors if the engine or its plugins contain virtualized data
-- '-CheckProject' which errors if the current project or its plugins contain virtualized data
-- '-CheckDir' which errors if the given directory or its subdirectories contain virtualized data. More than one path can be supplied by using '+' as the delimiter.
[CL 23905198 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-172167
#rnx
#preflight 63c649b50225f00e1466bf92
- Cmdline to use '-run=VirtualizationEditor.RehydrateProject'.
- The commandlet will first scan the project to find all packages (.uasset/.umap) and then attempt to rehydrate them.
- If revision control is enabled then virtualized packages will be checked out using the current settings for the project (so things like -P4Changelist= will work)
[CL 23735823 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-172166
#rnx
#preflight 63c64982d040694ab83efdd1
- Cmdline to use '-run=VirtualizationEditor.VirtualizeProject'.
- The commandlet will first scan the project to find all packages (.uasset/.umap) and then attempt to virtualize them.
- If revision control is enabled then virtualized packages will be checked out using the current settings for the project (so things like -P4Changelist= will work)
[CL 23735820 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-169626
#rnx
#preflight 63bd670a71079a8d1c0e837b
- Since the API was forcing the caller to pass in a results structure to be filled in, we might as well make it the return value.
- Added a ::WasSuccessful method to the results structures that can be used instead of checking if the result had errors or not.
- Remove the reset method from FVirtualizationResult/FRehydrationResult as they no longer need it.
- The older deprecated methods still use the results enum, so we cannot easily deprecate those enums yet.
[CL 23626072 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-169626
#rnx
#preflight 63bd0ebfd862fdd347bce1fe
### VA System
- This allows us to provide the user with more ways to customize the rehydration and return more detailed info about it if the calling code wishes to log additional info. In both cases we can extend the options and the data returned without changing the API.
- At the moment the only flag we support is 'Checkout', which requests that the rehydration process checkout any package that it needs to modify rather than giving an error. This means that the user does not need to check packages out from revision control before running the rehydration process.
-- We still check if packages can be modified and warn the user if they can't, as package files could be locked in other ways.
- The rehydration process will now long the time taken if verbose logging is set for the category 'LogVirtualization'
### UnrealVirtualizationTool
- The virtualize command now reports how many packages were checked out if the flag was set.
- The rehydration command now supports a '-Checkout' commandline flag, which when enabled will use the new api to checkout the packages that need to be checked out when rehydrated.
[CL 23625132 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-169895
#rnx
#preflight 639204815624e6da5ef1ba93
- This doesn't have great potential for production use but can be useful for teams when trying to work out which filters and limits to use when turning on asset virtualization.
[CL 23461404 by paul chipchase in ue5-main branch]
Tested compiling fortnite, unrealeditor, lyra, qagame with non-unity/pch
#preflight 63635997876630122adeab9f
#rb none
[CL 22958990 by henrik karlsson in ue5-main branch]
#rb Per.Larsson
#jira UE-167457
#rnx
#preflight 635a75130d3a231123c51720
- Use ParallelForWithTaskContext to iterate ove rthe list of packages and extra the payload information.
- This has been shown to givea speed up of 5x-10x depending on the machine being tested.
[CL 22810930 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-163093
#rnx
#preflight 63528929ae33b04ec1f4d892
- Iteration method 4 requests all found payloads in one single large batch. Useful for testing but would easily OOM systems when used on real projects.
- Iteration method 5 is similar to method 3, it splits the work into smaller jobs of 64 payloads at a time and spreads them out over our cores. This is the new method and in tests has managed to reach about 800mbit/s from HordeStorage.
[CL 22707967 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-164010
#rnx
#preflight 631f33368c5f7af11bc2a6b5
The commandlet is run via the cmdline '-run=VirtualizationEditor.PrecachePayloads' (the VirtualizationEditor part will load the module before executing the PrecachePayloads commandlet)
At the moment this is very slow to run as the VA API only allows one payload to be pulled in a single call and although we can make requests on many threads, there is a lot of overhead when requesting data from some backends and a batch pull API would be substantially quicker. This will be looked at in a future work item.
There are currently 4 different ways we can iterate over the payloads and precache them, since the commandlet is still too slow to use in general production, the different ways of iterating are provided so we can experiment later.
There is no way to customize what is precached, we just iterate over all the packaged in a project. We might want to improve this in the future but it would perhaps make more sense to move the package iteration options from the resave commandlet and make it something general purpose that different commandlets can also use. This would help build up a consistent way of choosing which packages are affected by a commandlet.
[CL 22050364 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-162600
#rnx
#preflight 6317334c7b21ad1e0a1448be
- When the statistics panel lived in the DDC editor module it was easier to keep all virtualization functionality in one place. Now that we have our own module we can move it to its own code file for better readability.
- Added a utility function ::SetupMenuEntries called when the module is started, we should add all future menu entries here.
[CL 21816542 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-162600
#rnx
#preflight 631597667562a90dfa991913
- Up until now we have been piggybacking off the DDC editor module as the VA statistics panel was originally based on the DDC statistic panel but now we are starting to do more VA editor/UX work so we should probably have our own module.
- This does mean we need to remove the the VA statistics panel from the DDC status bar, but it didn't really belong there and the panel can still be opened via the menu 'Tool->Audit->Virtual Assets'
[CL 21790583 by paul chipchase in ue5-main branch]
