izzy/UnrealEngineUWP
0
0
Fork 0
mirror of https://github.com/izzy2lost/UnrealEngineUWP.git synced 2026-03-26 18:15:20 -07:00
Code Issues Packages Projects Releases Wiki Activity
Files
5.3
UnrealEngineUWP/Engine/Source/Editor/VirtualizationEditor/Private/PrecachePayloadsCommandlet.cpp

48 lines
1.5 KiB
C++
Raw Permalink Normal View History

Add a very basic way to precache all virtualized payloads in a project and store them in the cache storage backends (usually the local DDC in our internal set up) #rb Per.Larsson #jira UE-164010 #rnx #preflight 631f33368c5f7af11bc2a6b5 The commandlet is run via the cmdline '-run=VirtualizationEditor.PrecachePayloads' (the VirtualizationEditor part will load the module before executing the PrecachePayloads commandlet) At the moment this is very slow to run as the VA API only allows one payload to be pulled in a single call and although we can make requests on many threads, there is a lot of overhead when requesting data from some backends and a batch pull API would be substantially quicker. This will be looked at in a future work item. There are currently 4 different ways we can iterate over the payloads and precache them, since the commandlet is still too slow to use in general production, the different ways of iterating are provided so we can experiment later. There is no way to customize what is precached, we just iterate over all the packaged in a project. We might want to improve this in the future but it would perhaps make more sense to move the package iteration options from the resave commandlet and make it something general purpose that different commandlets can also use. This would help build up a consistent way of choosing which packages are affected by a commandlet. [CL 22050364 by paul chipchase in ue5-main branch]
2022-09-16 11:35:57 -04:00
// Copyright Epic Games, Inc. All Rights Reserved.
Rename virtualization commandlet files to a more standard naming convention. #rb Sebastian.Nordgren #rnx #preflight 63c120f370575f890018a069 [CL 23675497 by paul chipchase in ue5-main branch]
2023-01-13 04:29:19 -05:00
#include "PrecachePayloadsCommandlet.h"
Add a very basic way to precache all virtualized payloads in a project and store them in the cache storage backends (usually the local DDC in our internal set up) #rb Per.Larsson #jira UE-164010 #rnx #preflight 631f33368c5f7af11bc2a6b5 The commandlet is run via the cmdline '-run=VirtualizationEditor.PrecachePayloads' (the VirtualizationEditor part will load the module before executing the PrecachePayloads commandlet) At the moment this is very slow to run as the VA API only allows one payload to be pulled in a single call and although we can make requests on many threads, there is a lot of overhead when requesting data from some backends and a batch pull API would be substantially quicker. This will be looked at in a future work item. There are currently 4 different ways we can iterate over the payloads and precache them, since the commandlet is still too slow to use in general production, the different ways of iterating are provided so we can experiment later. There is no way to customize what is precached, we just iterate over all the packaged in a project. We might want to improve this in the future but it would perhaps make more sense to move the package iteration options from the resave commandlet and make it something general purpose that different commandlets can also use. This would help build up a consistent way of choosing which packages are affected by a commandlet. [CL 22050364 by paul chipchase in ue5-main branch]
2022-09-16 11:35:57 -04:00
Move some common commandlet code to a utility file so that the virtualization commandlets can share. #rb Sebastian.Nordgren #rnx #preflight 63c11303305002c641cda826 [CL 23675149 by paul chipchase in ue5-main branch]
2023-01-13 03:31:00 -05:00
#include "CommandletUtils.h"
Add more profiling scopes to the PrecachePayloads commandlet #rb trivial #rnx #preflight 635b9fef7515720a5fb1bfe7 [CL 22827291 by paul chipchase in ue5-main branch]
2022-10-28 08:34:16 -04:00
#include "UObject/PackageTrailer.h"
#include "Virtualization/VirtualizationSystem.h"
Add a very basic way to precache all virtualized payloads in a project and store them in the cache storage backends (usually the local DDC in our internal set up) #rb Per.Larsson #jira UE-164010 #rnx #preflight 631f33368c5f7af11bc2a6b5 The commandlet is run via the cmdline '-run=VirtualizationEditor.PrecachePayloads' (the VirtualizationEditor part will load the module before executing the PrecachePayloads commandlet) At the moment this is very slow to run as the VA API only allows one payload to be pulled in a single call and although we can make requests on many threads, there is a lot of overhead when requesting data from some backends and a batch pull API would be substantially quicker. This will be looked at in a future work item. There are currently 4 different ways we can iterate over the payloads and precache them, since the commandlet is still too slow to use in general production, the different ways of iterating are provided so we can experiment later. There is no way to customize what is precached, we just iterate over all the packaged in a project. We might want to improve this in the future but it would perhaps make more sense to move the package iteration options from the resave commandlet and make it something general purpose that different commandlets can also use. This would help build up a consistent way of choosing which packages are affected by a commandlet. [CL 22050364 by paul chipchase in ue5-main branch]
2022-09-16 11:35:57 -04:00
UPrecachePayloadsCommandlet::UPrecachePayloadsCommandlet(const FObjectInitializer& ObjectInitializer)
: Super(ObjectInitializer)
{
}
int32 UPrecachePayloadsCommandlet::Main(const FString& Params)
{
TRACE_CPUPROFILER_EVENT_SCOPE(UPrecachePayloadsCommandlet);
Fix the 'CheckForVirtualizedContent ' commandlet to be able to check engine content once again. #rb none #jira UE-174785 #rnx #preflight 63ff0ceeae54ee4ce912c17c - Recently how we discover packages for the VA commandlets was changed to be much faster but it was also changed to automatically filter out all engine content as we removed the optional (opt in) feature of virtualizing engine content. - This meant that 'CheckForVirtualizedContent' would not be able to check engine packages if '-CheckEngine' was set as we were not providing the commandlet with them. - ::FindAllPackages has been renamed ::FindPackages and both it and ::DiscoverPackages now accept a flags parameter that allows the caller to request that engine content be excluded. - Although most commandlets do not want engine content, having the engine content filter as opt in closer matches existing apis and will probably lead to less mistakes. [CL 24459148 by paul chipchase in ue5-main branch]
2023-03-01 03:47:24 -05:00
TArray<FString> PackageNames = UE::Virtualization::FindPackages(UE::Virtualization::EFindPackageFlags::ExcludeEngineContent);
Add a very basic way to precache all virtualized payloads in a project and store them in the cache storage backends (usually the local DDC in our internal set up) #rb Per.Larsson #jira UE-164010 #rnx #preflight 631f33368c5f7af11bc2a6b5 The commandlet is run via the cmdline '-run=VirtualizationEditor.PrecachePayloads' (the VirtualizationEditor part will load the module before executing the PrecachePayloads commandlet) At the moment this is very slow to run as the VA API only allows one payload to be pulled in a single call and although we can make requests on many threads, there is a lot of overhead when requesting data from some backends and a batch pull API would be substantially quicker. This will be looked at in a future work item. There are currently 4 different ways we can iterate over the payloads and precache them, since the commandlet is still too slow to use in general production, the different ways of iterating are provided so we can experiment later. There is no way to customize what is precached, we just iterate over all the packaged in a project. We might want to improve this in the future but it would perhaps make more sense to move the package iteration options from the resave commandlet and make it something general purpose that different commandlets can also use. This would help build up a consistent way of choosing which packages are affected by a commandlet. [CL 22050364 by paul chipchase in ue5-main branch]
2022-09-16 11:35:57 -04:00
UE_LOG(LogVirtualization, Display, TEXT("Found %d packages"), PackageNames.Num());
Move some common commandlet code to a utility file so that the virtualization commandlets can share. #rb Sebastian.Nordgren #rnx #preflight 63c11303305002c641cda826 [CL 23675149 by paul chipchase in ue5-main branch]
2023-01-13 03:31:00 -05:00
TArray<FIoHash> PayloadIds = UE::Virtualization::FindVirtualizedPayloads(PackageNames);
Add a very basic way to precache all virtualized payloads in a project and store them in the cache storage backends (usually the local DDC in our internal set up) #rb Per.Larsson #jira UE-164010 #rnx #preflight 631f33368c5f7af11bc2a6b5 The commandlet is run via the cmdline '-run=VirtualizationEditor.PrecachePayloads' (the VirtualizationEditor part will load the module before executing the PrecachePayloads commandlet) At the moment this is very slow to run as the VA API only allows one payload to be pulled in a single call and although we can make requests on many threads, there is a lot of overhead when requesting data from some backends and a batch pull API would be substantially quicker. This will be looked at in a future work item. There are currently 4 different ways we can iterate over the payloads and precache them, since the commandlet is still too slow to use in general production, the different ways of iterating are provided so we can experiment later. There is no way to customize what is precached, we just iterate over all the packaged in a project. We might want to improve this in the future but it would perhaps make more sense to move the package iteration options from the resave commandlet and make it something general purpose that different commandlets can also use. This would help build up a consistent way of choosing which packages are affected by a commandlet. [CL 22050364 by paul chipchase in ue5-main branch]
2022-09-16 11:35:57 -04:00
if (PayloadIds.IsEmpty())
{
UE_LOG(LogVirtualization, Display, TEXT("No virtualized payloads found"));
return 0;
}
Add two new paths to the PreCachePayloads commandlet that take advantage of the new batch API. #rb Per.Larsson #jira UE-163093 #rnx #preflight 63528929ae33b04ec1f4d892 - Iteration method 4 requests all found payloads in one single large batch. Useful for testing but would easily OOM systems when used on real projects. - Iteration method 5 is similar to method 3, it splits the work into smaller jobs of 64 payloads at a time and spreads them out over our cores. This is the new method and in tests has managed to reach about 800mbit/s from HordeStorage. [CL 22707967 by paul chipchase in ue5-main branch]
2022-10-21 22:28:45 -04:00
Add a very basic way to precache all virtualized payloads in a project and store them in the cache storage backends (usually the local DDC in our internal set up) #rb Per.Larsson #jira UE-164010 #rnx #preflight 631f33368c5f7af11bc2a6b5 The commandlet is run via the cmdline '-run=VirtualizationEditor.PrecachePayloads' (the VirtualizationEditor part will load the module before executing the PrecachePayloads commandlet) At the moment this is very slow to run as the VA API only allows one payload to be pulled in a single call and although we can make requests on many threads, there is a lot of overhead when requesting data from some backends and a batch pull API would be substantially quicker. This will be looked at in a future work item. There are currently 4 different ways we can iterate over the payloads and precache them, since the commandlet is still too slow to use in general production, the different ways of iterating are provided so we can experiment later. There is no way to customize what is precached, we just iterate over all the packaged in a project. We might want to improve this in the future but it would perhaps make more sense to move the package iteration options from the resave commandlet and make it something general purpose that different commandlets can also use. This would help build up a consistent way of choosing which packages are affected by a commandlet. [CL 22050364 by paul chipchase in ue5-main branch]
2022-09-16 11:35:57 -04:00
UE_LOG(LogVirtualization, Display, TEXT("Found %d virtualized payloads to precache"), PayloadIds.Num());
UE_LOG(LogVirtualization, Display, TEXT("Precaching payloads..."));
Factor out the multithreaded pull system from PrecachePayloadsCommandlet to the general utilities so that other VA commandlets can make use of it. #rb trivial #jira UE-182196 #rnx #preflight 642c20f5d6069e71b1f39f80 [CL 24909424 by paul chipchase in ue5-main branch]
2023-04-04 09:16:58 -04:00
const int32 BatchSize = 64;
UE::Virtualization::PullPayloadsThreaded(PayloadIds, BatchSize, TEXT("Cached"), [](const UE::Virtualization::FPullRequest& Request)
Add a very basic way to precache all virtualized payloads in a project and store them in the cache storage backends (usually the local DDC in our internal set up) #rb Per.Larsson #jira UE-164010 #rnx #preflight 631f33368c5f7af11bc2a6b5 The commandlet is run via the cmdline '-run=VirtualizationEditor.PrecachePayloads' (the VirtualizationEditor part will load the module before executing the PrecachePayloads commandlet) At the moment this is very slow to run as the VA API only allows one payload to be pulled in a single call and although we can make requests on many threads, there is a lot of overhead when requesting data from some backends and a batch pull API would be substantially quicker. This will be looked at in a future work item. There are currently 4 different ways we can iterate over the payloads and precache them, since the commandlet is still too slow to use in general production, the different ways of iterating are provided so we can experiment later. There is no way to customize what is precached, we just iterate over all the packaged in a project. We might want to improve this in the future but it would perhaps make more sense to move the package iteration options from the resave commandlet and make it something general purpose that different commandlets can also use. This would help build up a consistent way of choosing which packages are affected by a commandlet. [CL 22050364 by paul chipchase in ue5-main branch]
2022-09-16 11:35:57 -04:00
{
Factor out the multithreaded pull system from PrecachePayloadsCommandlet to the general utilities so that other VA commandlets can make use of it. #rb trivial #jira UE-182196 #rnx #preflight 642c20f5d6069e71b1f39f80 [CL 24909424 by paul chipchase in ue5-main branch]
2023-04-04 09:16:58 -04:00
if (!Request.IsSuccess())
Add two new paths to the PreCachePayloads commandlet that take advantage of the new batch API. #rb Per.Larsson #jira UE-163093 #rnx #preflight 63528929ae33b04ec1f4d892 - Iteration method 4 requests all found payloads in one single large batch. Useful for testing but would easily OOM systems when used on real projects. - Iteration method 5 is similar to method 3, it splits the work into smaller jobs of 64 payloads at a time and spreads them out over our cores. This is the new method and in tests has managed to reach about 800mbit/s from HordeStorage. [CL 22707967 by paul chipchase in ue5-main branch]
2022-10-21 22:28:45 -04:00
{
Factor out the multithreaded pull system from PrecachePayloadsCommandlet to the general utilities so that other VA commandlets can make use of it. #rb trivial #jira UE-182196 #rnx #preflight 642c20f5d6069e71b1f39f80 [CL 24909424 by paul chipchase in ue5-main branch]
2023-04-04 09:16:58 -04:00
UE_LOG(LogVirtualization, Error, TEXT("%s: Failed to precache payload"), *LexToString(Request.GetIdentifier()));
Improve the VA PrecachePayloads commandlet #rb none #rnx #preflight 641c01e2a6092dfb8dfa1478 - The previous default iteration mode could saturdate the worker threads, which would prevent the Zen DDC backend from running clean up code resulting in very large memory spikes. - We now launch a limited number of tasks that process a limited number of payloads before completing. Once a task is finsihed we issue another until there is no more work to do. - This seems to strike a good balance of keeping the network transfer at peak speed but allowing other systems time on the cpu. - Removed all the experimental iteration modes and going all in on this solution, which makes the code easier to parse. [CL 24759860 by paul chipchase in ue5-main branch]
2023-03-23 07:02:27 -04:00
}
Factor out the multithreaded pull system from PrecachePayloadsCommandlet to the general utilities so that other VA commandlets can make use of it. #rb trivial #jira UE-182196 #rnx #preflight 642c20f5d6069e71b1f39f80 [CL 24909424 by paul chipchase in ue5-main branch]
2023-04-04 09:16:58 -04:00
});
Add a very basic way to precache all virtualized payloads in a project and store them in the cache storage backends (usually the local DDC in our internal set up) #rb Per.Larsson #jira UE-164010 #rnx #preflight 631f33368c5f7af11bc2a6b5 The commandlet is run via the cmdline '-run=VirtualizationEditor.PrecachePayloads' (the VirtualizationEditor part will load the module before executing the PrecachePayloads commandlet) At the moment this is very slow to run as the VA API only allows one payload to be pulled in a single call and although we can make requests on many threads, there is a lot of overhead when requesting data from some backends and a batch pull API would be substantially quicker. This will be looked at in a future work item. There are currently 4 different ways we can iterate over the payloads and precache them, since the commandlet is still too slow to use in general production, the different ways of iterating are provided so we can experiment later. There is no way to customize what is precached, we just iterate over all the packaged in a project. We might want to improve this in the future but it would perhaps make more sense to move the package iteration options from the resave commandlet and make it something general purpose that different commandlets can also use. This would help build up a consistent way of choosing which packages are affected by a commandlet. [CL 22050364 by paul chipchase in ue5-main branch]
2022-09-16 11:35:57 -04:00
UE_LOG(LogVirtualization, Display, TEXT("Precaching complete!"));
UE::Virtualization::IVirtualizationSystem::Get().DumpStats();
return 0;
}
Reference in New Issue Copy Permalink