izzy/UnrealEngineUWP
0
0
Fork 0
mirror of https://github.com/izzy2lost/UnrealEngineUWP.git synced 2026-03-26 18:15:20 -07:00
Code Issues Packages Projects Releases Wiki Activity
Files
e425c856daa00a2165927fbb0a67e2fb6371614c
UnrealEngineUWP/Engine/Source/Developer/Virtualization/Private/VirtualizationDDCBackend.cpp

197 lines
6.0 KiB
C++
Raw Normal View History

The DDC virtualization backend now supports the batch push API #rb trivial #rnx #preflight 629893b95e715bebb2c2a529 - The DDC2 api is already heavily threaded so we can simply keep throwing new requests onto it and wait once all requests are made and not have to do any work ourselves. - I did ask and was told that using the batch api provided by DDC2 would not improve performance so for now I am have gone with the simple change. We may still look into changing to the batch API in the future as then we would have a reason to improve the DDC2 api. [CL 20468437 by paul chipchase in ue5-main branch]
2022-06-02 09:04:45 -04:00
// Copyright Epic Games, Inc. All Rights Reserved.
#include "VirtualizationDDCBackend.h"
#include "Misc/Parse.h"
#include "DerivedDataCache.h"
#include "DerivedDataCacheRecord.h"
#include "DerivedDataRequestOwner.h"
#include "DerivedDataValue.h"
namespace UE::Virtualization
{
/** Utility function to help convert from UE::Virtualization::FIoHash to UE::DerivedData::FValueId */
static UE::DerivedData::FValueId ToDerivedDataValueId(const FIoHash& Id)
{
return UE::DerivedData::FValueId::FromHash(Id);
}
FDDCBackend::FDDCBackend(FStringView ProjectName, FStringView ConfigName, FStringView InDebugName)
Reworked the config file set up for the DDCBackend to remove the chance of Fatal errors. #rb trivial #rnx #preflight 62d96169110da0f944b69f4f - The bucket entry was already marked as optional in the code documentation but was giving a fatal error if it was not set. The value is now optional and will default to 'BulkData' if not set. - We no longer consider both the local and remote transfer policy being disabled as a fatal error, when the work UE-143676 is done this can be improved further. [CL 21200968 by paul chipchase in ue5-main branch]
2022-07-21 10:44:47 -04:00
: IVirtualizationBackend(ConfigName, InDebugName, EOperations::Push | EOperations::Pull)
, BucketName(TEXT("BulkData"))
, TransferPolicy(UE::DerivedData::ECachePolicy::None)
, QueryPolicy(UE::DerivedData::ECachePolicy::None)
The DDC virtualization backend now supports the batch push API #rb trivial #rnx #preflight 629893b95e715bebb2c2a529 - The DDC2 api is already heavily threaded so we can simply keep throwing new requests onto it and wait once all requests are made and not have to do any work ourselves. - I did ask and was told that using the batch api provided by DDC2 would not improve performance so for now I am have gone with the simple change. We may still look into changing to the batch API in the future as then we would have a reason to improve the DDC2 api. [CL 20468437 by paul chipchase in ue5-main branch]
2022-06-02 09:04:45 -04:00
{
}
bool FDDCBackend::Initialize(const FString& ConfigEntry)
{
TRACE_CPUPROFILER_EVENT_SCOPE(Initialize::Initialize);
Reworked the config file set up for the DDCBackend to remove the chance of Fatal errors. #rb trivial #rnx #preflight 62d96169110da0f944b69f4f - The bucket entry was already marked as optional in the code documentation but was giving a fatal error if it was not set. The value is now optional and will default to 'BulkData' if not set. - We no longer consider both the local and remote transfer policy being disabled as a fatal error, when the work UE-143676 is done this can be improved further. [CL 21200968 by paul chipchase in ue5-main branch]
2022-07-21 10:44:47 -04:00
FString BucketNameIniFile;
if(FParse::Value(*ConfigEntry, TEXT("Bucket="), BucketNameIniFile))
The DDC virtualization backend now supports the batch push API #rb trivial #rnx #preflight 629893b95e715bebb2c2a529 - The DDC2 api is already heavily threaded so we can simply keep throwing new requests onto it and wait once all requests are made and not have to do any work ourselves. - I did ask and was told that using the batch api provided by DDC2 would not improve performance so for now I am have gone with the simple change. We may still look into changing to the batch API in the future as then we would have a reason to improve the DDC2 api. [CL 20468437 by paul chipchase in ue5-main branch]
2022-06-02 09:04:45 -04:00
{
Reworked the config file set up for the DDCBackend to remove the chance of Fatal errors. #rb trivial #rnx #preflight 62d96169110da0f944b69f4f - The bucket entry was already marked as optional in the code documentation but was giving a fatal error if it was not set. The value is now optional and will default to 'BulkData' if not set. - We no longer consider both the local and remote transfer policy being disabled as a fatal error, when the work UE-143676 is done this can be improved further. [CL 21200968 by paul chipchase in ue5-main branch]
2022-07-21 10:44:47 -04:00
BucketName = BucketNameIniFile;
The DDC virtualization backend now supports the batch push API #rb trivial #rnx #preflight 629893b95e715bebb2c2a529 - The DDC2 api is already heavily threaded so we can simply keep throwing new requests onto it and wait once all requests are made and not have to do any work ourselves. - I did ask and was told that using the batch api provided by DDC2 would not improve performance so for now I am have gone with the simple change. We may still look into changing to the batch API in the future as then we would have a reason to improve the DDC2 api. [CL 20468437 by paul chipchase in ue5-main branch]
2022-06-02 09:04:45 -04:00
}
bool bAllowLocal = true;
Reworked the config file set up for the DDCBackend to remove the chance of Fatal errors. #rb trivial #rnx #preflight 62d96169110da0f944b69f4f - The bucket entry was already marked as optional in the code documentation but was giving a fatal error if it was not set. The value is now optional and will default to 'BulkData' if not set. - We no longer consider both the local and remote transfer policy being disabled as a fatal error, when the work UE-143676 is done this can be improved further. [CL 21200968 by paul chipchase in ue5-main branch]
2022-07-21 10:44:47 -04:00
FParse::Bool(*ConfigEntry, TEXT("LocalStorage="), bAllowLocal);
The DDC virtualization backend now supports the batch push API #rb trivial #rnx #preflight 629893b95e715bebb2c2a529 - The DDC2 api is already heavily threaded so we can simply keep throwing new requests onto it and wait once all requests are made and not have to do any work ourselves. - I did ask and was told that using the batch api provided by DDC2 would not improve performance so for now I am have gone with the simple change. We may still look into changing to the batch API in the future as then we would have a reason to improve the DDC2 api. [CL 20468437 by paul chipchase in ue5-main branch]
2022-06-02 09:04:45 -04:00
bool bAllowRemote = true;
Reworked the config file set up for the DDCBackend to remove the chance of Fatal errors. #rb trivial #rnx #preflight 62d96169110da0f944b69f4f - The bucket entry was already marked as optional in the code documentation but was giving a fatal error if it was not set. The value is now optional and will default to 'BulkData' if not set. - We no longer consider both the local and remote transfer policy being disabled as a fatal error, when the work UE-143676 is done this can be improved further. [CL 21200968 by paul chipchase in ue5-main branch]
2022-07-21 10:44:47 -04:00
FParse::Bool(*ConfigEntry, TEXT("RemoteStorage="), bAllowRemote);
UE_LOG(LogVirtualization, Log, TEXT("[%s] Bucket set to '%s"), *GetDebugName(), *BucketName);
UE_LOG(LogVirtualization, Log, TEXT("[%s] Use of local storage set to '%s"), *GetDebugName(), bAllowLocal ? TEXT("true") : TEXT("false"));
UE_LOG(LogVirtualization, Log, TEXT("[%s] Use of remote storage set to '%s"), *GetDebugName(), bAllowRemote ? TEXT("true") : TEXT("false"));
The DDC virtualization backend now supports the batch push API #rb trivial #rnx #preflight 629893b95e715bebb2c2a529 - The DDC2 api is already heavily threaded so we can simply keep throwing new requests onto it and wait once all requests are made and not have to do any work ourselves. - I did ask and was told that using the batch api provided by DDC2 would not improve performance so for now I am have gone with the simple change. We may still look into changing to the batch API in the future as then we would have a reason to improve the DDC2 api. [CL 20468437 by paul chipchase in ue5-main branch]
2022-06-02 09:04:45 -04:00
if (!bAllowLocal && !bAllowRemote)
{
Reworked the config file set up for the DDCBackend to remove the chance of Fatal errors. #rb trivial #rnx #preflight 62d96169110da0f944b69f4f - The bucket entry was already marked as optional in the code documentation but was giving a fatal error if it was not set. The value is now optional and will default to 'BulkData' if not set. - We no longer consider both the local and remote transfer policy being disabled as a fatal error, when the work UE-143676 is done this can be improved further. [CL 21200968 by paul chipchase in ue5-main branch]
2022-07-21 10:44:47 -04:00
UE_LOG(LogVirtualization, Error, TEXT("[%s] LocalStorage and RemoteStorage cannot both be disabled"), *GetDebugName());
The DDC virtualization backend now supports the batch push API #rb trivial #rnx #preflight 629893b95e715bebb2c2a529 - The DDC2 api is already heavily threaded so we can simply keep throwing new requests onto it and wait once all requests are made and not have to do any work ourselves. - I did ask and was told that using the batch api provided by DDC2 would not improve performance so for now I am have gone with the simple change. We may still look into changing to the batch API in the future as then we would have a reason to improve the DDC2 api. [CL 20468437 by paul chipchase in ue5-main branch]
2022-06-02 09:04:45 -04:00
return false;
}
if (bAllowLocal)
{
TransferPolicy |= UE::DerivedData::ECachePolicy::Local;
QueryPolicy |= UE::DerivedData::ECachePolicy::QueryLocal;
}
if (bAllowRemote)
{
TransferPolicy |= UE::DerivedData::ECachePolicy::Remote;
QueryPolicy |= UE::DerivedData::ECachePolicy::QueryRemote;
}
Bucket = UE::DerivedData::FCacheBucket(BucketName);
return true;
}
Add a number of ways for the VA backend connections to be changed to lazy initialize on first use. #rb Devin.Doucette #jira UE-161599 #rnx #preflight 6303c8d65a5d4e4624e7bf52 - There are some use cases that require the VA system to be initialized and configured correctly but would prefer that the backend connections only run if absolutely needed (usually when a payload is pulled or pushed for the first time), this change provides four different ways of doing this: -- Setting [Core.VirtualizationModule]LazyInitConnections=true in the Engine ini file -- Setting the define 'UE_VIRTUALIZATION_CONNECTION_LAZY_INIT' to 1 in a programs .target.cs -- Running with the commandline option -VA-LazyInitConnections -- Setting the cvar 'VA.LazyInitConnections' to 1 (only works if it is set before the VA system is initialized, changing it mid editor via the console does nothing) --- Note that after the config file, each setting there only opts into lazy initializing the connections, setting the cvar to 0 for example will not prevent the cmdline from opting in etc. - In the future we will allow the connection code to run async, so the latency can be hidden behind the editor loading, but for the current use case we are taking the minimal approach. -- This means we only support the backend being in 3 states. No connection has been made yet, the connection is broken and the connection is working. -- To keep things simple we only record if we have attempted to connect the backends or not. We don't check individual backends nor do we try to reconnect failed ones etc. This is all scheduled for a future work item. - If the connections are not initialized when the VA system is, we wait until the first time someone calls one of the virtualization methods that will actually use a connection: Push/Pull/Query -- We try connecting all of the backends at once, even if they won't be used in the call to keep things simple. - Only the source control backend makes use of the connection system. The horde storage (http) backend could take advantage too, but it is currently unused and most likely going to just be deleted so there seemed little point updating it. - If we try to run an operation on an unconnected backend we only log to verbose. This is to maintain existing behaviour where a failed backend would not be mounted at all. This logging will likely be revisited in a future work item. [CL 21511855 by paul chipchase in ue5-main branch]
2022-08-23 13:01:15 -04:00
IVirtualizationBackend::EConnectionStatus FDDCBackend::OnConnect()
{
return IVirtualizationBackend::EConnectionStatus::Connected;
}
Add bitflag enums to IVirtualizationBackend::PullData and IVirtualizationBackend::PushData so that it is easier to specialize behavior in the future. #rb Per.Larsson #jira UE-182205 #preflight 643932b2211b661dc4f594c6 - The enums don't contain any entries at the moment, the goal of this change is to reduce the friction that future API changes might have. - The older versions of PullData/PushData have been deprecated but this won't flag compiler warnings if people have derived their own implementations of IVirtualizationBackend, so in addition to deprecating the methods have been set to 'final' and have checkNoEntry implementations which is the best we can do at the moment. -- It is unlikely that people have actually done this as the documentation on adding new backend types will only go out in 5.3. [CL 25039959 by paul chipchase in ue5-main branch]
2023-04-14 10:13:46 -04:00
bool FDDCBackend::PushData(TArrayView<FPushRequest> Requests, EPushFlags Flags)
The DDC virtualization backend now supports the batch push API #rb trivial #rnx #preflight 629893b95e715bebb2c2a529 - The DDC2 api is already heavily threaded so we can simply keep throwing new requests onto it and wait once all requests are made and not have to do any work ourselves. - I did ask and was told that using the batch api provided by DDC2 would not improve performance so for now I am have gone with the simple change. We may still look into changing to the batch API in the future as then we would have a reason to improve the DDC2 api. [CL 20468437 by paul chipchase in ue5-main branch]
2022-06-02 09:04:45 -04:00
{
TRACE_CPUPROFILER_EVENT_SCOPE(FDDCBackend::PushData);
UE::DerivedData::ICache& Cache = UE::DerivedData::GetCache();
UE::DerivedData::FRequestOwner Owner(UE::DerivedData::EPriority::Normal);
bool bWasSuccess = true;
Add a new config option 'ForceCachingOnPull' to VA that when true will force backends to re-upload payloads when caching. #rb none #jira UE-182205 #preflight 643d2901c947f6523a2f5845 - Most backends support some form of existence check which is run before a push to prevent uploading data that is already there. Sometimes we want to ignore that check (bugs in the backend etc) and need a way to force that to happen. - The new config option [Core.VirtualizationModule]:ForceCachingOnPull (default false) will do this when set to true. - The file and DDC backends will respect the flag but for now the source control backend will continue to check to avoid accidental pointless revisions being added. [CL 25066347 by paul chipchase in ue5-main branch]
2023-04-17 07:45:19 -04:00
const bool bEnableExistenceCheck = !EnumHasAllFlags(Flags, EPushFlags::Force);
The DDC virtualization backend now supports the batch push API #rb trivial #rnx #preflight 629893b95e715bebb2c2a529 - The DDC2 api is already heavily threaded so we can simply keep throwing new requests onto it and wait once all requests are made and not have to do any work ourselves. - I did ask and was told that using the batch api provided by DDC2 would not improve performance so for now I am have gone with the simple change. We may still look into changing to the batch API in the future as then we would have a reason to improve the DDC2 api. [CL 20468437 by paul chipchase in ue5-main branch]
2022-06-02 09:04:45 -04:00
// TODO: We tend not to memory bloat too much on large batches as the requests complete quite quickly
// however we might want to consider adding better control on how much total memory we can dedicate to
// loading payloads before we wait for requests to complete?
for (FPushRequest& Request : Requests)
{
Add a new config option 'ForceCachingOnPull' to VA that when true will force backends to re-upload payloads when caching. #rb none #jira UE-182205 #preflight 643d2901c947f6523a2f5845 - Most backends support some form of existence check which is run before a push to prevent uploading data that is already there. Sometimes we want to ignore that check (bugs in the backend etc) and need a way to force that to happen. - The new config option [Core.VirtualizationModule]:ForceCachingOnPull (default false) will do this when set to true. - The file and DDC backends will respect the flag but for now the source control backend will continue to check to avoid accidental pointless revisions being added. [CL 25066347 by paul chipchase in ue5-main branch]
2023-04-17 07:45:19 -04:00
if (bEnableExistenceCheck && DoesPayloadExist(Request.GetIdentifier()))
The DDC virtualization backend now supports the batch push API #rb trivial #rnx #preflight 629893b95e715bebb2c2a529 - The DDC2 api is already heavily threaded so we can simply keep throwing new requests onto it and wait once all requests are made and not have to do any work ourselves. - I did ask and was told that using the batch api provided by DDC2 would not improve performance so for now I am have gone with the simple change. We may still look into changing to the batch API in the future as then we would have a reason to improve the DDC2 api. [CL 20468437 by paul chipchase in ue5-main branch]
2022-06-02 09:04:45 -04:00
{
Fix incorrect statistics being reported when virtualizing payloads. We no longer count a payload that was already stored in a backend as being uploaded there. #rb Per.Larsson #jira UE-160942 #rnx #preflight 6336f2ab5c2225fe5f6c08a5 - Removed the status enum from FPushRequest and replaced it with a class that represents the push result instead. -- This lets us return more context about problems. At the moment it is only being used to return the filter reason if a payload is filtered out. In the future we could return per payload error messages etc. - Now that we return more detailed info about the payload push we no longer report payloads that were already in the backend as being pushed which could cause some very misleading stats to be shown. - Changed the backends to use the new results system. -- Not all paths were correctly setting the state in the file system backend. -- The source control backend does not set the result in all paths, which will leave the results in the "pending" state. Which will still mark the payload as not uploaded and so is safe to do. - Removed the pushing of a single method from IVirtualizationBackend as it was only used in one place internally, so we might as well just change that code to call the batch push overload. [CL 22279127 by paul chipchase in ue5-main branch]
2022-09-30 15:47:04 -04:00
Request.SetResult(FPushResult::GetAsAlreadyExists());
The DDC virtualization backend now supports the batch push API #rb trivial #rnx #preflight 629893b95e715bebb2c2a529 - The DDC2 api is already heavily threaded so we can simply keep throwing new requests onto it and wait once all requests are made and not have to do any work ourselves. - I did ask and was told that using the batch api provided by DDC2 would not improve performance so for now I am have gone with the simple change. We may still look into changing to the batch API in the future as then we would have a reason to improve the DDC2 api. [CL 20468437 by paul chipchase in ue5-main branch]
2022-06-02 09:04:45 -04:00
}
else
{
UE::DerivedData::FRequestBarrier Barrier(Owner);
UE::DerivedData::FCacheKey Key;
Key.Bucket = Bucket;
Key.Hash = Request.GetIdentifier();
UE::DerivedData::FValue DerivedDataValue(Request.GetPayload());
check(DerivedDataValue.GetRawHash() == Request.GetIdentifier());
UE::DerivedData::FCacheRecordBuilder RecordBuilder(Key);
RecordBuilder.AddValue(ToDerivedDataValueId(Request.GetIdentifier()), DerivedDataValue);
UE::DerivedData::FCachePutResponse Result;
auto Callback = [&Request, &bWasSuccess](UE::DerivedData::FCachePutResponse&& Response)
{
if (Response.Status == UE::DerivedData::EStatus::Ok)
{
Fix incorrect statistics being reported when virtualizing payloads. We no longer count a payload that was already stored in a backend as being uploaded there. #rb Per.Larsson #jira UE-160942 #rnx #preflight 6336f2ab5c2225fe5f6c08a5 - Removed the status enum from FPushRequest and replaced it with a class that represents the push result instead. -- This lets us return more context about problems. At the moment it is only being used to return the filter reason if a payload is filtered out. In the future we could return per payload error messages etc. - Now that we return more detailed info about the payload push we no longer report payloads that were already in the backend as being pushed which could cause some very misleading stats to be shown. - Changed the backends to use the new results system. -- Not all paths were correctly setting the state in the file system backend. -- The source control backend does not set the result in all paths, which will leave the results in the "pending" state. Which will still mark the payload as not uploaded and so is safe to do. - Removed the pushing of a single method from IVirtualizationBackend as it was only used in one place internally, so we might as well just change that code to call the batch push overload. [CL 22279127 by paul chipchase in ue5-main branch]
2022-09-30 15:47:04 -04:00
Request.SetResult(FPushResult::GetAsPushed());
The DDC virtualization backend now supports the batch push API #rb trivial #rnx #preflight 629893b95e715bebb2c2a529 - The DDC2 api is already heavily threaded so we can simply keep throwing new requests onto it and wait once all requests are made and not have to do any work ourselves. - I did ask and was told that using the batch api provided by DDC2 would not improve performance so for now I am have gone with the simple change. We may still look into changing to the batch API in the future as then we would have a reason to improve the DDC2 api. [CL 20468437 by paul chipchase in ue5-main branch]
2022-06-02 09:04:45 -04:00
}
else
{
Fix incorrect statistics being reported when virtualizing payloads. We no longer count a payload that was already stored in a backend as being uploaded there. #rb Per.Larsson #jira UE-160942 #rnx #preflight 6336f2ab5c2225fe5f6c08a5 - Removed the status enum from FPushRequest and replaced it with a class that represents the push result instead. -- This lets us return more context about problems. At the moment it is only being used to return the filter reason if a payload is filtered out. In the future we could return per payload error messages etc. - Now that we return more detailed info about the payload push we no longer report payloads that were already in the backend as being pushed which could cause some very misleading stats to be shown. - Changed the backends to use the new results system. -- Not all paths were correctly setting the state in the file system backend. -- The source control backend does not set the result in all paths, which will leave the results in the "pending" state. Which will still mark the payload as not uploaded and so is safe to do. - Removed the pushing of a single method from IVirtualizationBackend as it was only used in one place internally, so we might as well just change that code to call the batch push overload. [CL 22279127 by paul chipchase in ue5-main branch]
2022-09-30 15:47:04 -04:00
Request.SetResult(FPushResult::GetAsError());
The DDC virtualization backend now supports the batch push API #rb trivial #rnx #preflight 629893b95e715bebb2c2a529 - The DDC2 api is already heavily threaded so we can simply keep throwing new requests onto it and wait once all requests are made and not have to do any work ourselves. - I did ask and was told that using the batch api provided by DDC2 would not improve performance so for now I am have gone with the simple change. We may still look into changing to the batch API in the future as then we would have a reason to improve the DDC2 api. [CL 20468437 by paul chipchase in ue5-main branch]
2022-06-02 09:04:45 -04:00
bWasSuccess = false;
}
};
// TODO: Improve the name when we start passing more context to this function
Cache.Put({ {{TEXT("Mirage")}, RecordBuilder.Build(), TransferPolicy} }, Owner, MoveTemp(Callback));
}
}
Owner.Wait();
return bWasSuccess;
}
Add a way for VA backends to supply additional errors for failed pull operations when calling FVirtualizationManager::OnPayloadPullError #rb none #jira UE-181418 #rnx #preflight 6447c5482804595a04d1e375 - Each pull operation now takes a FText that can be filled in with additional info about any failures that will cause the new error dialog to be displayed. -- At the moment we only have info to provide from the source control backend if the connection itself failed. -- In the future we might want to completely overhaul our error logging and display all messages to the user but our logging system is not really built for that level of redirection at the moment. - Only pass on the error if it came from a persistent backend, we don't need to worry the user about cached backends which are allowed to fail. [CL 25178752 by paul chipchase in ue5-main branch]
2023-04-25 08:47:06 -04:00
bool FDDCBackend::PullData(TArrayView<FPullRequest> Requests, EPullFlags Flags, FText& OutErrors)
The DDC virtualization backend now supports the batch push API #rb trivial #rnx #preflight 629893b95e715bebb2c2a529 - The DDC2 api is already heavily threaded so we can simply keep throwing new requests onto it and wait once all requests are made and not have to do any work ourselves. - I did ask and was told that using the batch api provided by DDC2 would not improve performance so for now I am have gone with the simple change. We may still look into changing to the batch API in the future as then we would have a reason to improve the DDC2 api. [CL 20468437 by paul chipchase in ue5-main branch]
2022-06-02 09:04:45 -04:00
{
TRACE_CPUPROFILER_EVENT_SCOPE(FDDCBackend::PullData);
Remove debug code accidently submitted in CL 25066347 #rb trivial #jira none #rnx #preflight 643d4f59930b3b84922485c2 [CL 25067672 by paul chipchase in ue5-main branch]
2023-04-17 10:15:35 -04:00
The DDC virtualization backend now supports the batch push API #rb trivial #rnx #preflight 629893b95e715bebb2c2a529 - The DDC2 api is already heavily threaded so we can simply keep throwing new requests onto it and wait once all requests are made and not have to do any work ourselves. - I did ask and was told that using the batch api provided by DDC2 would not improve performance so for now I am have gone with the simple change. We may still look into changing to the batch API in the future as then we would have a reason to improve the DDC2 api. [CL 20468437 by paul chipchase in ue5-main branch]
2022-06-02 09:04:45 -04:00
UE::DerivedData::ICache& Cache = UE::DerivedData::GetCache();
Extending the VA pulling API to allow for batch pulls #rb Per.Larsson #jira UE-163093 #rnx #preflight 63526abcae33b04ec1ee4a65 - The caller can now request more than one payload to be pulled froim the system at a time. In theory this will allow backends to pull data quicker. -- The file system backend works with the new system but makes no attempt to take advantage as the backend is not intended for serious production use. -- The DDC backend should be taking full advantage of the new batching API -- Note that although the source control API does attempt to take advantage of batching, the internal source control API implementation is not doing so. This will be addressed in a future submit. - This does make the pulling logic a bit more complicated in FVirtualizationManager as we need to deal with some payloads being found in the first backend, some in the second and some in the third etc. -- To help deal with this a new class FPullRequestCollection has been added which abstracts a lot of complexity away. [CL 22707955 by paul chipchase in ue5-main branch]
2022-10-21 22:27:40 -04:00
UE::DerivedData::FRequestOwner Owner(UE::DerivedData::EPriority::Normal);
The DDC virtualization backend now supports the batch push API #rb trivial #rnx #preflight 629893b95e715bebb2c2a529 - The DDC2 api is already heavily threaded so we can simply keep throwing new requests onto it and wait once all requests are made and not have to do any work ourselves. - I did ask and was told that using the batch api provided by DDC2 would not improve performance so for now I am have gone with the simple change. We may still look into changing to the batch API in the future as then we would have a reason to improve the DDC2 api. [CL 20468437 by paul chipchase in ue5-main branch]
2022-06-02 09:04:45 -04:00
Extending the VA pulling API to allow for batch pulls #rb Per.Larsson #jira UE-163093 #rnx #preflight 63526abcae33b04ec1ee4a65 - The caller can now request more than one payload to be pulled froim the system at a time. In theory this will allow backends to pull data quicker. -- The file system backend works with the new system but makes no attempt to take advantage as the backend is not intended for serious production use. -- The DDC backend should be taking full advantage of the new batching API -- Note that although the source control API does attempt to take advantage of batching, the internal source control API implementation is not doing so. This will be addressed in a future submit. - This does make the pulling logic a bit more complicated in FVirtualizationManager as we need to deal with some payloads being found in the first backend, some in the second and some in the third etc. -- To help deal with this a new class FPullRequestCollection has been added which abstracts a lot of complexity away. [CL 22707955 by paul chipchase in ue5-main branch]
2022-10-21 22:27:40 -04:00
for (FPullRequest& Request : Requests)
The DDC virtualization backend now supports the batch push API #rb trivial #rnx #preflight 629893b95e715bebb2c2a529 - The DDC2 api is already heavily threaded so we can simply keep throwing new requests onto it and wait once all requests are made and not have to do any work ourselves. - I did ask and was told that using the batch api provided by DDC2 would not improve performance so for now I am have gone with the simple change. We may still look into changing to the batch API in the future as then we would have a reason to improve the DDC2 api. [CL 20468437 by paul chipchase in ue5-main branch]
2022-06-02 09:04:45 -04:00
{
Extending the VA pulling API to allow for batch pulls #rb Per.Larsson #jira UE-163093 #rnx #preflight 63526abcae33b04ec1ee4a65 - The caller can now request more than one payload to be pulled froim the system at a time. In theory this will allow backends to pull data quicker. -- The file system backend works with the new system but makes no attempt to take advantage as the backend is not intended for serious production use. -- The DDC backend should be taking full advantage of the new batching API -- Note that although the source control API does attempt to take advantage of batching, the internal source control API implementation is not doing so. This will be addressed in a future submit. - This does make the pulling logic a bit more complicated in FVirtualizationManager as we need to deal with some payloads being found in the first backend, some in the second and some in the third etc. -- To help deal with this a new class FPullRequestCollection has been added which abstracts a lot of complexity away. [CL 22707955 by paul chipchase in ue5-main branch]
2022-10-21 22:27:40 -04:00
UE::DerivedData::FRequestBarrier Barrier(Owner);
The DDC virtualization backend now supports the batch push API #rb trivial #rnx #preflight 629893b95e715bebb2c2a529 - The DDC2 api is already heavily threaded so we can simply keep throwing new requests onto it and wait once all requests are made and not have to do any work ourselves. - I did ask and was told that using the batch api provided by DDC2 would not improve performance so for now I am have gone with the simple change. We may still look into changing to the batch API in the future as then we would have a reason to improve the DDC2 api. [CL 20468437 by paul chipchase in ue5-main branch]
2022-06-02 09:04:45 -04:00
Extending the VA pulling API to allow for batch pulls #rb Per.Larsson #jira UE-163093 #rnx #preflight 63526abcae33b04ec1ee4a65 - The caller can now request more than one payload to be pulled froim the system at a time. In theory this will allow backends to pull data quicker. -- The file system backend works with the new system but makes no attempt to take advantage as the backend is not intended for serious production use. -- The DDC backend should be taking full advantage of the new batching API -- Note that although the source control API does attempt to take advantage of batching, the internal source control API implementation is not doing so. This will be addressed in a future submit. - This does make the pulling logic a bit more complicated in FVirtualizationManager as we need to deal with some payloads being found in the first backend, some in the second and some in the third etc. -- To help deal with this a new class FPullRequestCollection has been added which abstracts a lot of complexity away. [CL 22707955 by paul chipchase in ue5-main branch]
2022-10-21 22:27:40 -04:00
UE::DerivedData::FCacheKey Key;
Key.Bucket = Bucket;
Key.Hash = Request.GetIdentifier();
auto Callback = [&Request](UE::DerivedData::FCacheGetResponse&& Response)
{
if (Response.Status == UE::DerivedData::EStatus::Ok)
{
UE::DerivedData::FValueId ValueId = ToDerivedDataValueId(Request.GetIdentifier());
Request.SetPayload(Response.Record.GetValue(ValueId).GetData());
}
};
// TODO: Improve the name when we start passing more context to this function
Cache.Get({ {{TEXT("Mirage")}, Key, TransferPolicy} }, Owner, MoveTemp(Callback));
}
The DDC virtualization backend now supports the batch push API #rb trivial #rnx #preflight 629893b95e715bebb2c2a529 - The DDC2 api is already heavily threaded so we can simply keep throwing new requests onto it and wait once all requests are made and not have to do any work ourselves. - I did ask and was told that using the batch api provided by DDC2 would not improve performance so for now I am have gone with the simple change. We may still look into changing to the batch API in the future as then we would have a reason to improve the DDC2 api. [CL 20468437 by paul chipchase in ue5-main branch]
2022-06-02 09:04:45 -04:00
Owner.Wait();
Remove debug code accidently submitted in CL 25066347 #rb trivial #jira none #rnx #preflight 643d4f59930b3b84922485c2 [CL 25067672 by paul chipchase in ue5-main branch]
2023-04-17 10:15:35 -04:00
Extending the VA pulling API to allow for batch pulls #rb Per.Larsson #jira UE-163093 #rnx #preflight 63526abcae33b04ec1ee4a65 - The caller can now request more than one payload to be pulled froim the system at a time. In theory this will allow backends to pull data quicker. -- The file system backend works with the new system but makes no attempt to take advantage as the backend is not intended for serious production use. -- The DDC backend should be taking full advantage of the new batching API -- Note that although the source control API does attempt to take advantage of batching, the internal source control API implementation is not doing so. This will be addressed in a future submit. - This does make the pulling logic a bit more complicated in FVirtualizationManager as we need to deal with some payloads being found in the first backend, some in the second and some in the third etc. -- To help deal with this a new class FPullRequestCollection has been added which abstracts a lot of complexity away. [CL 22707955 by paul chipchase in ue5-main branch]
2022-10-21 22:27:40 -04:00
return true;
The DDC virtualization backend now supports the batch push API #rb trivial #rnx #preflight 629893b95e715bebb2c2a529 - The DDC2 api is already heavily threaded so we can simply keep throwing new requests onto it and wait once all requests are made and not have to do any work ourselves. - I did ask and was told that using the batch api provided by DDC2 would not improve performance so for now I am have gone with the simple change. We may still look into changing to the batch API in the future as then we would have a reason to improve the DDC2 api. [CL 20468437 by paul chipchase in ue5-main branch]
2022-06-02 09:04:45 -04:00
}
bool FDDCBackend::DoesPayloadExist(const FIoHash& Id)
{
TRACE_CPUPROFILER_EVENT_SCOPE(FDDCBackend::DoesPayloadExist);
UE::DerivedData::ICache& Cache = UE::DerivedData::GetCache();
UE::DerivedData::FCacheKey Key;
Key.Bucket = Bucket;
Key.Hash = Id;
UE::DerivedData::FRequestOwner Owner(UE::DerivedData::EPriority::Blocking);
UE::DerivedData::EStatus ResultStatus;
auto Callback = [&ResultStatus](UE::DerivedData::FCacheGetResponse&& Response)
{
ResultStatus = Response.Status;
};
// TODO: Improve the name when we start passing more context to this function
Cache.Get({{{TEXT("Mirage")}, Key, QueryPolicy | UE::DerivedData::ECachePolicy::SkipData}}, Owner, MoveTemp(Callback));
Owner.Wait();
return ResultStatus == UE::DerivedData::EStatus::Ok;
}
UE_REGISTER_VIRTUALIZATION_BACKEND_FACTORY(FDDCBackend, DDCBackend);
Add a new backend to Mirage that uses DDC2 as the storage mechanism. #rb Devin.Doucette #rnx #preflight 61307f101a52e20001978a10 - This backend replaces the plans for a backend that interfaces with Zen directly. - The backend has no real context for each payload being pushed at the moment, for now we pass in a dummy string. - Currently we make blocking calls to the cache. When making the initial push to local storage this is desired as it will take place during package save. However both pulling the payload and making the submission to persistent storage should be able to be made async in the future. - The storage policy is currently exposed via the ini file for now. We might want to enforce policies for this at a later time. #ROBOMERGE-SOURCE: CL 17416948 in //UE5/Main/... #ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v865-17346139) [CL 17416956 by paul chipchase in ue5-release-engine-test branch]
2021-09-03 02:57:01 -04:00
} // namespace UE::Virtualization
Reference in New Issue Copy Permalink