Files
UnrealEngineUWP/Engine/Source/Developer/Virtualization/Private/VirtualizationFileBackend.h

67 lines
2.5 KiB
C
Raw Normal View History

// Copyright Epic Games, Inc. All Rights Reserved.
#pragma once
#include "IVirtualizationBackend.h"
namespace UE::Virtualization
{
/**
* A basic backend based on the file system. This can be used to access/store virtualization
* data either on a local disk or a network share. It is intended to be used as a caching system
* to speed up operations (running a local cache or a shared cache for a site) rather than as the
* proper backend solution.
*
* Ini file setup:
A quick hardening pass over the file system backend for Mirage. #rb Per.Larsson #rnx #preflight 61408a979dc6c8000148d0cf ### Misc - Updated the documentation for setting up a file system backend and added descriptions to the backend's members. - Renamed the member 'Name' to 'DebugName' to bring it in line with the other backends (still a candidate to be moved to the base class) ### Reading payloads from disk - Added the ability to retry opening file handles for read if they fail. This can commonly occur when multiple threads or processes are both pulling and pushing to the same root directory. -- By default we will retry to open the file handle up to 10 times, waiting for 100ms between each attempt. The user can configure the number of retries and the length of the pause between each attempt via the config files. -- Each failed attempt at opening the file handle will result in a warning being logged, so if the user is stalled for a long length of time while we rety over and over they are at least made aware of the cause. ### Writing payloads to disk - When attempting to write a payload to disk, we will now write to a temp file in the project save directory using a randomly generated name. Once the payload has been fully saved to disk it will then be moved to the correct location. Although not perfect this will reduce the potential for writing corrupted data to a location where it might be read in the future. -- We do attempt to clean up the temp file if the write fails but we do not print error messages if the delete fails as it is considered an optional bonus rather than an essential feature. - Previously we would only log an error if we failed to open a file handle for writing the payload to disk. Now we also log an error if the handle is opened but the actual writes fail. #ROBOMERGE-AUTHOR: paul.chipchase #ROBOMERGE-SOURCE: CL 17550126 in //UE5/Main/... #ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v870-17433530) [CL 17550133 by paul chipchase in ue5-release-engine-test branch]
2021-09-17 06:03:03 -04:00
* 'Name'=(Type=FileSystem, Path="XXX", RetryCount=X, RetryWaitTime=X)
*
* Required Values:
* 'Name': The backend name in the hierarchy.
* 'Type': The backend will be of type 'FFileSystemBackend'.
* 'Path': The root directory where the files are stored.
*
* Optional Values:
* RetryCount: How many times we should try to open a payload file for read before giving up with
* an error. Useful when many threads/processes can be pushing/pulling from the same
* path/ [Default=10]
* RetryWaitTime: The length of time the process should wait between each read attempt in milliseconds.
* Remember that the max length of time that the process can stall attempting to read
* a payload file is RetryCount * RetryWaitTime; [Default=100ms]
* EnvPathOverride: The name of the environment variable that the user can set to override 'Path'. This
* should only ever be set if you wish for the users to be able to choose where the
* payloads are being stored and do not rely on 'Path' always being used. [Default=""]
*/
class FFileSystemBackend final : public IVirtualizationBackend
{
public:
explicit FFileSystemBackend(FStringView ProjectName, FStringView ConfigName, FStringView DebugName);
virtual ~FFileSystemBackend() = default;
private:
/* IVirtualizationBackend implementation */
virtual bool Initialize(const FString& ConfigEntry) override;
Add a number of ways for the VA backend connections to be changed to lazy initialize on first use. #rb Devin.Doucette #jira UE-161599 #rnx #preflight 6303c8d65a5d4e4624e7bf52 - There are some use cases that require the VA system to be initialized and configured correctly but would prefer that the backend connections only run if absolutely needed (usually when a payload is pulled or pushed for the first time), this change provides four different ways of doing this: -- Setting [Core.VirtualizationModule]LazyInitConnections=true in the Engine ini file -- Setting the define 'UE_VIRTUALIZATION_CONNECTION_LAZY_INIT' to 1 in a programs .target.cs -- Running with the commandline option -VA-LazyInitConnections -- Setting the cvar 'VA.LazyInitConnections' to 1 (only works if it is set before the VA system is initialized, changing it mid editor via the console does nothing) --- Note that after the config file, each setting there only opts into lazy initializing the connections, setting the cvar to 0 for example will not prevent the cmdline from opting in etc. - In the future we will allow the connection code to run async, so the latency can be hidden behind the editor loading, but for the current use case we are taking the minimal approach. -- This means we only support the backend being in 3 states. No connection has been made yet, the connection is broken and the connection is working. -- To keep things simple we only record if we have attempted to connect the backends or not. We don't check individual backends nor do we try to reconnect failed ones etc. This is all scheduled for a future work item. - If the connections are not initialized when the VA system is, we wait until the first time someone calls one of the virtualization methods that will actually use a connection: Push/Pull/Query -- We try connecting all of the backends at once, even if they won't be used in the call to keep things simple. - Only the source control backend makes use of the connection system. The horde storage (http) backend could take advantage too, but it is currently unused and most likely going to just be deleted so there seemed little point updating it. - If we try to run an operation on an unconnected backend we only log to verbose. This is to maintain existing behaviour where a failed backend would not be mounted at all. This logging will likely be revisited in a future work item. [CL 21511855 by paul chipchase in ue5-main branch]
2022-08-23 13:01:15 -04:00
virtual EConnectionStatus OnConnect() override;
Refactor the VA pushing API so that we only need to implement a single path #rb Per.Larsson #jira UE-163103 #rnx #preflight 6318989c2b7fe03eb664e9f0 ### VirtualizationSystem/VirtualizationManager - Added an overload of ::Push taking just one FPUshRequest so that people don't have to keep adding MakeArrayView boiler plate when pushing a single request - Change the order of the last two parameters for the raw ::Push call as this will group all of the payload specific parameters together and leave the target storage type at the end. It is unlikely that anyone is calling the older version but it has been deprecated for safety. ### IVirtualizationBackend - The none FPushRequest overload of ::PushData is no longer virtual, it just converts the parameters to FPushRequest and calls that overload instead. In this way we now only have one pushing code path in our backends. We could probably look into removing this overload at this point (since the higher level IVirtualizationSystem will now convert all push requests into a FPushRequest form) but it is not considered worth it at the moment when the simple overload covers our needs. - Removed EPushResult in favour of just returning true/false for the overall operation.If the caller needs a more detailed breakdown then they will have to use an overload that takes an FPushRequest over raw parameters. -- At the moment FPushRequest does not contain a full breakdown of what happened, so with this submit we are effectively losing the ability to find out if the payload was already in the backends or not, however the batch version of push was already not returning this info so it is not a big loss. Fixing FPushRequest to return a better break down of what happened will be done in UE-160942 - Removed the none batch Push paths from the source control and ddc backends as they already supported batch pushing. - File backend needed to be converted to supporting batch pushing, which is pretty much the same code as before except we need to iterate over the container of FPushRequests. -- The backend does not early out on error as it tends to be quite fast. We might want to consider an official policy for the VA system, if we should early out of errors or not. [CL 21907558 by paul chipchase in ue5-main branch]
2022-09-08 19:35:36 -04:00
virtual bool PushData(TArrayView<FPushRequest> Requests) override;
virtual bool PullData(TArrayView<FPullRequest> Requests) override;
virtual bool DoesPayloadExist(const FIoHash& Id) override;
A quick hardening pass over the file system backend for Mirage. #rb Per.Larsson #rnx #preflight 61408a979dc6c8000148d0cf ### Misc - Updated the documentation for setting up a file system backend and added descriptions to the backend's members. - Renamed the member 'Name' to 'DebugName' to bring it in line with the other backends (still a candidate to be moved to the base class) ### Reading payloads from disk - Added the ability to retry opening file handles for read if they fail. This can commonly occur when multiple threads or processes are both pulling and pushing to the same root directory. -- By default we will retry to open the file handle up to 10 times, waiting for 100ms between each attempt. The user can configure the number of retries and the length of the pause between each attempt via the config files. -- Each failed attempt at opening the file handle will result in a warning being logged, so if the user is stalled for a long length of time while we rety over and over they are at least made aware of the cause. ### Writing payloads to disk - When attempting to write a payload to disk, we will now write to a temp file in the project save directory using a randomly generated name. Once the payload has been fully saved to disk it will then be moved to the correct location. Although not perfect this will reduce the potential for writing corrupted data to a location where it might be read in the future. -- We do attempt to clean up the temp file if the write fails but we do not print error messages if the delete fails as it is considered an optional bonus rather than an essential feature. - Previously we would only log an error if we failed to open a file handle for writing the payload to disk. Now we also log an error if the handle is opened but the actual writes fail. #ROBOMERGE-AUTHOR: paul.chipchase #ROBOMERGE-SOURCE: CL 17550126 in //UE5/Main/... #ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v870-17433530) [CL 17550133 by paul chipchase in ue5-release-engine-test branch]
2021-09-17 06:03:03 -04:00
private:
void CreateFilePath(const FIoHash& PayloadId, FStringBuilderBase& OutPath);
A quick hardening pass over the file system backend for Mirage. #rb Per.Larsson #rnx #preflight 61408a979dc6c8000148d0cf ### Misc - Updated the documentation for setting up a file system backend and added descriptions to the backend's members. - Renamed the member 'Name' to 'DebugName' to bring it in line with the other backends (still a candidate to be moved to the base class) ### Reading payloads from disk - Added the ability to retry opening file handles for read if they fail. This can commonly occur when multiple threads or processes are both pulling and pushing to the same root directory. -- By default we will retry to open the file handle up to 10 times, waiting for 100ms between each attempt. The user can configure the number of retries and the length of the pause between each attempt via the config files. -- Each failed attempt at opening the file handle will result in a warning being logged, so if the user is stalled for a long length of time while we rety over and over they are at least made aware of the cause. ### Writing payloads to disk - When attempting to write a payload to disk, we will now write to a temp file in the project save directory using a randomly generated name. Once the payload has been fully saved to disk it will then be moved to the correct location. Although not perfect this will reduce the potential for writing corrupted data to a location where it might be read in the future. -- We do attempt to clean up the temp file if the write fails but we do not print error messages if the delete fails as it is considered an optional bonus rather than an essential feature. - Previously we would only log an error if we failed to open a file handle for writing the payload to disk. Now we also log an error if the handle is opened but the actual writes fail. #ROBOMERGE-AUTHOR: paul.chipchase #ROBOMERGE-SOURCE: CL 17550126 in //UE5/Main/... #ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v870-17433530) [CL 17550133 by paul chipchase in ue5-release-engine-test branch]
2021-09-17 06:03:03 -04:00
TUniquePtr<FArchive> OpenFileForReading(const TCHAR* FilePath);
A quick hardening pass over the file system backend for Mirage. #rb Per.Larsson #rnx #preflight 61408a979dc6c8000148d0cf ### Misc - Updated the documentation for setting up a file system backend and added descriptions to the backend's members. - Renamed the member 'Name' to 'DebugName' to bring it in line with the other backends (still a candidate to be moved to the base class) ### Reading payloads from disk - Added the ability to retry opening file handles for read if they fail. This can commonly occur when multiple threads or processes are both pulling and pushing to the same root directory. -- By default we will retry to open the file handle up to 10 times, waiting for 100ms between each attempt. The user can configure the number of retries and the length of the pause between each attempt via the config files. -- Each failed attempt at opening the file handle will result in a warning being logged, so if the user is stalled for a long length of time while we rety over and over they are at least made aware of the cause. ### Writing payloads to disk - When attempting to write a payload to disk, we will now write to a temp file in the project save directory using a randomly generated name. Once the payload has been fully saved to disk it will then be moved to the correct location. Although not perfect this will reduce the potential for writing corrupted data to a location where it might be read in the future. -- We do attempt to clean up the temp file if the write fails but we do not print error messages if the delete fails as it is considered an optional bonus rather than an essential feature. - Previously we would only log an error if we failed to open a file handle for writing the payload to disk. Now we also log an error if the handle is opened but the actual writes fail. #ROBOMERGE-AUTHOR: paul.chipchase #ROBOMERGE-SOURCE: CL 17550126 in //UE5/Main/... #ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v870-17433530) [CL 17550133 by paul chipchase in ue5-release-engine-test branch]
2021-09-17 06:03:03 -04:00
/** The root directory where the payload files should be located. */
FString RootDirectory;
A quick hardening pass over the file system backend for Mirage. #rb Per.Larsson #rnx #preflight 61408a979dc6c8000148d0cf ### Misc - Updated the documentation for setting up a file system backend and added descriptions to the backend's members. - Renamed the member 'Name' to 'DebugName' to bring it in line with the other backends (still a candidate to be moved to the base class) ### Reading payloads from disk - Added the ability to retry opening file handles for read if they fail. This can commonly occur when multiple threads or processes are both pulling and pushing to the same root directory. -- By default we will retry to open the file handle up to 10 times, waiting for 100ms between each attempt. The user can configure the number of retries and the length of the pause between each attempt via the config files. -- Each failed attempt at opening the file handle will result in a warning being logged, so if the user is stalled for a long length of time while we rety over and over they are at least made aware of the cause. ### Writing payloads to disk - When attempting to write a payload to disk, we will now write to a temp file in the project save directory using a randomly generated name. Once the payload has been fully saved to disk it will then be moved to the correct location. Although not perfect this will reduce the potential for writing corrupted data to a location where it might be read in the future. -- We do attempt to clean up the temp file if the write fails but we do not print error messages if the delete fails as it is considered an optional bonus rather than an essential feature. - Previously we would only log an error if we failed to open a file handle for writing the payload to disk. Now we also log an error if the handle is opened but the actual writes fail. #ROBOMERGE-AUTHOR: paul.chipchase #ROBOMERGE-SOURCE: CL 17550126 in //UE5/Main/... #ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v870-17433530) [CL 17550133 by paul chipchase in ue5-release-engine-test branch]
2021-09-17 06:03:03 -04:00
/** The number of times to retry opening a payload file for read */
int32 RetryCount = 10;
/** The length of time (in milliseconds) to wait after each attempt before retrying. */
int32 RetryWaitTimeMS = 100;
};
} // namespace UE::Virtualization