Tested compiling fortnite, unrealeditor, lyra, qagame with non-unity/pch
#preflight 63635997876630122adeab9f
#rb none
[CL 22958990 by henrik karlsson in ue5-main branch]
#rb Per.Larsson
#jira UE-167457
#rnx
#preflight 635a75130d3a231123c51720
- Use ParallelForWithTaskContext to iterate ove rthe list of packages and extra the payload information.
- This has been shown to givea speed up of 5x-10x depending on the machine being tested.
[CL 22810930 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-163093
#rnx
#preflight 63528929ae33b04ec1f4d892
- Iteration method 4 requests all found payloads in one single large batch. Useful for testing but would easily OOM systems when used on real projects.
- Iteration method 5 is similar to method 3, it splits the work into smaller jobs of 64 payloads at a time and spreads them out over our cores. This is the new method and in tests has managed to reach about 800mbit/s from HordeStorage.
[CL 22707967 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-164010
#rnx
#preflight 631f33368c5f7af11bc2a6b5
The commandlet is run via the cmdline '-run=VirtualizationEditor.PrecachePayloads' (the VirtualizationEditor part will load the module before executing the PrecachePayloads commandlet)
At the moment this is very slow to run as the VA API only allows one payload to be pulled in a single call and although we can make requests on many threads, there is a lot of overhead when requesting data from some backends and a batch pull API would be substantially quicker. This will be looked at in a future work item.
There are currently 4 different ways we can iterate over the payloads and precache them, since the commandlet is still too slow to use in general production, the different ways of iterating are provided so we can experiment later.
There is no way to customize what is precached, we just iterate over all the packaged in a project. We might want to improve this in the future but it would perhaps make more sense to move the package iteration options from the resave commandlet and make it something general purpose that different commandlets can also use. This would help build up a consistent way of choosing which packages are affected by a commandlet.
[CL 22050364 by paul chipchase in ue5-main branch]
