You've already forked UnrealEngineUWP
mirror of
https://github.com/izzy2lost/UnrealEngineUWP.git
synced 2026-03-26 18:15:20 -07:00
21530c814b
#rb PJ.Kack #jira UE-136126 #rnx #preflight ### VirtualizationSystem - Added a new overload for Push to VirtualizationSystem that takes an array of FPushRequest, which is a new structure representing a single payload request. - Filtering by package name is currently disabled, this is because the API has been forced into changing and passing the package name in via a FString rather than FPackagePath which means we would need to be more careful. This will be done in a future submit. - The backend interface has been extended to also have a batch version of PushData, by default this will attempt to submit each request one at a time so payloads don't have to try and implement a batched version if there is no need. - The context being passed with a payload when being pushed has been changed from FPackagePath to FString due to include order issues, as the FPackagePath lives in CoreUObject and the API for virtualization lives in Core. Additionally in the future the payloads might not be owned by a package (there is nothing specifically enforcing this) so the context being a string makes more sense. - NOTE: Due to the context change we currently no longer support the filtering feature, which allows for payloads belonging to packages under specific directories to be excluded from virtualization. This is something that will be solved in a future submit. ### SourceControlBackend - Now that we can submit multiple payloads in the same submit, the CL description has been changed slightly. We will now print a list of payload identifiers -> the package trying to submit that payload. This will only tell the users which package originally caused the payload to submit. If a user submits a new package at a later date that contains the same payload we will not be updating the description. ### PackageSubmissionChecks - Converted the submission process to use the new batch push operation in VirtualizationSystem. -- This means that we do a single push and then have to update the package trailers to convert the now pushed payloads from local to virtualized. - Added new define UE_PRECHECK_PAYLOAD_STATUS that makes it easy to toggle off the checks to see which payloads need to be submitted to the persistent backend. This is useful to test if it actually helps speed up the overall operations or if it is faster to just perform the batch push operations on all payloads and check the return values. -- The hope is that over time the submission processes will become fast enough that we can remove the precheck. - Fixed up logging to not always assume more than one package or payload. ### General Notes - Errors and logging is now a bit more vague as we often not just report that X payloads failed etc rather than specific payload identifiers. This probably doesn't affect the user too much since those identifiers as fairly meaningless to them anyway. - The source control submission could be further optimized by first checking the status of the files in thge depot and only then creating/switching workspace etc. - As currently written, we need to load all of the payloads into memory, then the backends will do what they need (in the case of source control this results in the payloads being written to disk then submitted) which could create quite a large memory spike when submitting a large number of packages. -- One solution would be to change the batch push API to take a "payload provider" interface and have the payloads requested as needed rather than passing in the FCompressedBuffer directly. This would let us immediately write the payload to disk for submission then discard it from memory, preventing larger spikes. Although it could cause overhead if there are multiple backends being submitted to. Internally we are unlikely to have more than one backend per storage solution so maybe we should just make it a config option? #ROBOMERGE-AUTHOR: paul.chipchase #ROBOMERGE-SOURCE: CL 18403735 in //UE5/Release-5.0/... via CL 18403737 #ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v896-18170469) [CL 18403738 by paul chipchase in ue5-release-engine-test branch]
63 lines
2.1 KiB
C++
63 lines
2.1 KiB
C++
// Copyright Epic Games, Inc. All Rights Reserved.
|
|
|
|
#pragma once
|
|
|
|
#include "IVirtualizationBackend.h"
|
|
|
|
#include "Containers/StringView.h"
|
|
|
|
namespace UE::Virtualization
|
|
{
|
|
|
|
/**
|
|
* This backend can be used to access payloads stored in source control.
|
|
* The backend doesn't 'check out' a payload file but instead will just download the payload as
|
|
* a binary blob.
|
|
* It is assumed that the files are stored with the same path convention as the file system
|
|
* backend, found in Utils::PayloadIdToPath.
|
|
*
|
|
* Ini file setup:
|
|
* 'Name'=(Type=SourceControl, DepotRoot="//XXX/")
|
|
* Where 'Name' is the backend name in the hierarchy and 'XXX' is the path in the source control
|
|
* depot where the payload files are being stored.
|
|
*
|
|
* Optional Values:
|
|
* UsePartitionedClient: When true the temporary workspace client created to submit payloads
|
|
* from will be created as a partitioned workspace which is less overhead
|
|
* on the source control server. If your server does not support this then
|
|
* use false. [Default=True]
|
|
*/
|
|
class FSourceControlBackend final : public IVirtualizationBackend
|
|
{
|
|
public:
|
|
explicit FSourceControlBackend(FStringView ConfigName, FStringView InDebugName);
|
|
virtual ~FSourceControlBackend() = default;
|
|
|
|
private:
|
|
/* IVirtualizationBackend implementation */
|
|
|
|
virtual bool Initialize(const FString& ConfigEntry) override;
|
|
|
|
virtual EPushResult PushData(const FPayloadId& Id, const FCompressedBuffer& Payload, const FString& Context) override;
|
|
virtual bool PushData(TArrayView<FPushRequest> Requests) override;
|
|
|
|
virtual FCompressedBuffer PullData(const FPayloadId& Id) override;
|
|
|
|
virtual bool DoesPayloadExist(const FPayloadId& Id) override;
|
|
|
|
virtual bool DoPayloadsExist(TArrayView<const FPayloadId> PayloadIds, TArray<bool>& OutResults) override;
|
|
|
|
private:
|
|
|
|
void CreateDepotPath(const FPayloadId& PayloadId, FStringBuilderBase& OutPath);
|
|
|
|
/** The root where the virtualized payloads are stored in source control */
|
|
FString DepotRoot;
|
|
|
|
/** Should we try to make the temp client partitioned or not? */
|
|
bool bUsePartitionedClient = true;
|
|
|
|
};
|
|
|
|
} // namespace UE::Virtualization
|