Files
UnrealEngineUWP/Engine/Source/Developer/SourceControl/Private/SourceControlOperations.cpp

87 lines
3.1 KiB
C++
Raw Normal View History

Add support for downloading a file from source control directly without actually checking it out (ie p4 print) and add additional support so that the command can be called from a background thread. - Only implemented for perforce source control at the moment. - The source control operation FDownload file can be used just like any other command via ::Execute BUT it will also work with the new method ::TryToDownloadFileFromBackgroundThread which unlike ::Execute can be called from a background thread. #rb Sebastien.Lussier #rnx #preflight 60a2290322cce000014243ae * PerforceConnection - Extend ::RunCommand to take an optional FSharedBuffer reference which can be used to return text/binary data downloaded as part of a perforce command -- Ideally we would replace all of the out parameters with a single struct but that level of refactor would be better suited to its own submit. - Removed the bool parameter for FP4ClientUser and replaced it with a set of flags instead - Added a flag to allow FP4ClientUser to store data, either in binary or text format from a perforce command. Although in general this will be the result of a p4 print command. * PerforceSourceControlCommand - No longer assert if the command is not on the game thread if the operation indicates that it is safe to run on a background thread. * PerforceSourceControlModule - Register the 'DownloadFile' worker * PerforceSourceControlOperations - Add FPerforceDownloadFileWorker * PerforceSourceControlProvider - Adding implementation for ::TryToDownloadFileFromBackgroundThread. -- This method is thread safe compared to the normal ::Execute call due to: -- 1) This method does not accept any callbacks as the most common use of callbacks with ::Execute is to update the UI. -- 2) This method does not add the command to any of the internal queues nor rely on Tick to process the command. -- 3) The worker is invoked directly from the calling thread rather than as a background task. --4) This method does not allow commands that modify the cached states as this would be very difficult to manage without rewriting how all commands are processed to make sure that they are only run one at a time and the results processed in order. NOTE that async commands can be issued from the game thread and can in theory be run in any order on the p4 server depending on how the background task processing works out with no certainty as to the order the results will be processed in. It is unlikely that we will process results out of order but not impossible, but this is a pre-exisiting issue with the current API, extending it to be fully multithreaded will just make it worse. -- Note that we do not explicity check if the command supports being called from a background thread, as creating the command will check that anyway. - Calling ::OutputCommandMessages is now thread safe. When called from the game thread it will issue it's output via FMessageLog as before but when called from a background thread we will use the UE_LOG macros instead. Marshalling all output in the same way is something we will need to solve when/if we make the entire API thread safe. * PerforceControlSettings - Improve thread safety when calling ::GetConnectionInfo - If the password is stored in the UI then this is not thread safe but that feature is hidden at the moment due to being unsecure. I am unsure if we want to move the password to be stored in a member variable that can be accessed from any thread and only updated via the UI or not. * SourceControlOperations - Add new FDownloadFile operation. - This can either download the file(s) to FSharedBuffers which can be accessed by calling ::GetFileData OR directly to a directory on disk if the target directory is supplied as part of the constructor. * ISourceControlOperation - Add a new base method CanBeCalledFromBackgroundThreads (false by default) which can be overriden by derived classes and return true to indicate that the operation can be run from a background thread. This is intended for use by FDownloadFile with the perforce source control provider in lieu and making the API fully thread safe. * PerforceSourceControl.uplugin - Move the perforce plugin to load as early as possible as if Mirage is to use it then we need it to be avaliable before we start loading any potentially virtualized data in packages. [CL 16346666 by paul chipchase in ue5-main branch]
2021-05-17 05:06:44 -04:00
// Copyright Epic Games, Inc. All Rights Reserved.
#include "SourceControlOperations.h"
#include "Containers/StringView.h"
#include "HAL/FileManager.h"
Add support for downloading a file from source control directly without actually checking it out (ie p4 print) and add additional support so that the command can be called from a background thread. - Only implemented for perforce source control at the moment. - The source control operation FDownload file can be used just like any other command via ::Execute BUT it will also work with the new method ::TryToDownloadFileFromBackgroundThread which unlike ::Execute can be called from a background thread. #rb Sebastien.Lussier #rnx #preflight 60a2290322cce000014243ae * PerforceConnection - Extend ::RunCommand to take an optional FSharedBuffer reference which can be used to return text/binary data downloaded as part of a perforce command -- Ideally we would replace all of the out parameters with a single struct but that level of refactor would be better suited to its own submit. - Removed the bool parameter for FP4ClientUser and replaced it with a set of flags instead - Added a flag to allow FP4ClientUser to store data, either in binary or text format from a perforce command. Although in general this will be the result of a p4 print command. * PerforceSourceControlCommand - No longer assert if the command is not on the game thread if the operation indicates that it is safe to run on a background thread. * PerforceSourceControlModule - Register the 'DownloadFile' worker * PerforceSourceControlOperations - Add FPerforceDownloadFileWorker * PerforceSourceControlProvider - Adding implementation for ::TryToDownloadFileFromBackgroundThread. -- This method is thread safe compared to the normal ::Execute call due to: -- 1) This method does not accept any callbacks as the most common use of callbacks with ::Execute is to update the UI. -- 2) This method does not add the command to any of the internal queues nor rely on Tick to process the command. -- 3) The worker is invoked directly from the calling thread rather than as a background task. --4) This method does not allow commands that modify the cached states as this would be very difficult to manage without rewriting how all commands are processed to make sure that they are only run one at a time and the results processed in order. NOTE that async commands can be issued from the game thread and can in theory be run in any order on the p4 server depending on how the background task processing works out with no certainty as to the order the results will be processed in. It is unlikely that we will process results out of order but not impossible, but this is a pre-exisiting issue with the current API, extending it to be fully multithreaded will just make it worse. -- Note that we do not explicity check if the command supports being called from a background thread, as creating the command will check that anyway. - Calling ::OutputCommandMessages is now thread safe. When called from the game thread it will issue it's output via FMessageLog as before but when called from a background thread we will use the UE_LOG macros instead. Marshalling all output in the same way is something we will need to solve when/if we make the entire API thread safe. * PerforceControlSettings - Improve thread safety when calling ::GetConnectionInfo - If the password is stored in the UI then this is not thread safe but that feature is hidden at the moment due to being unsecure. I am unsure if we want to move the password to be stored in a member variable that can be accessed from any thread and only updated via the UI or not. * SourceControlOperations - Add new FDownloadFile operation. - This can either download the file(s) to FSharedBuffers which can be accessed by calling ::GetFileData OR directly to a directory on disk if the target directory is supplied as part of the constructor. * ISourceControlOperation - Add a new base method CanBeCalledFromBackgroundThreads (false by default) which can be overriden by derived classes and return true to indicate that the operation can be run from a background thread. This is intended for use by FDownloadFile with the perforce source control provider in lieu and making the API fully thread safe. * PerforceSourceControl.uplugin - Move the perforce plugin to load as early as possible as if Mirage is to use it then we need it to be avaliable before we start loading any potentially virtualized data in packages. [CL 16346666 by paul chipchase in ue5-main branch]
2021-05-17 05:06:44 -04:00
#include "ISourceControlModule.h"
The ability to create, delete and switch workspaces via the SourceControl API. Currently this is only implemented for the perforce source control plugin. #rb Sebastien.Lussier #rnx #preflight 611e457e2c1f1c0001292a29 ### ISourceControlProvider / PerforceSourceControlProvider - Add new method ::SwitchWorkspace allowing the caller to switch the current workspace. - Currently only implemented for perforce. - Calling this will cause the provider to close and then re-open it's connection and so will invalidate any existing established connection or cached states. Due to this the method will fail if there are outstanding commands being processed. - It is intended that this is only used for short periods of time, in order to submit or access files from locations that the user chosen workspace might be able to access. - The new workspace will not be saved to the source control settings. Although this could occur later if done. ### SourceControlOperations - Add new operation **FCreateWorkspace**. - Currently only implemented for perforce. - The name and the root of the workspace (filepath on the client machine) must be provided in the constructor. - Additional mappings can be provided by calling **AddNativeClientViewMapping** which expects the mapping to be in the native format of the target source control system. At some point in the future we may add a more generic, target agnostic, format. - Add new operation **FDeleteWorkspace**. - Currently only implemented for perforce. - Deletes the workspace of the given name although the workspace must be in a state where it can be deleted (no files open for add/edit/delete etc) or the operation will fail. ### PerforceConnection - Add a new method ::CreateWorkspace for creating workspaces - Add a new extended client 'FP4CommandWithStdInputClientUser' - We should really extend FP4ClientUser to be able to take overridden stdinput instead but that would mean adding yet another input parameter. I'd rather keep the code separate for now and then merge them together in a future refactor where the many input parameters of ::RunCommand is replaces with a struct with TOptional members. - Added a ::GetUser method, which returns the user of the current connection without the caller needing to know if the connection is in unicode format or not. [CL 17350788 by paul chipchase in ue5-main branch]
2021-08-30 08:06:38 -04:00
#include "Misc/PathViews.h"
Add support for downloading a file from source control directly without actually checking it out (ie p4 print) and add additional support so that the command can be called from a background thread. - Only implemented for perforce source control at the moment. - The source control operation FDownload file can be used just like any other command via ::Execute BUT it will also work with the new method ::TryToDownloadFileFromBackgroundThread which unlike ::Execute can be called from a background thread. #rb Sebastien.Lussier #rnx #preflight 60a2290322cce000014243ae * PerforceConnection - Extend ::RunCommand to take an optional FSharedBuffer reference which can be used to return text/binary data downloaded as part of a perforce command -- Ideally we would replace all of the out parameters with a single struct but that level of refactor would be better suited to its own submit. - Removed the bool parameter for FP4ClientUser and replaced it with a set of flags instead - Added a flag to allow FP4ClientUser to store data, either in binary or text format from a perforce command. Although in general this will be the result of a p4 print command. * PerforceSourceControlCommand - No longer assert if the command is not on the game thread if the operation indicates that it is safe to run on a background thread. * PerforceSourceControlModule - Register the 'DownloadFile' worker * PerforceSourceControlOperations - Add FPerforceDownloadFileWorker * PerforceSourceControlProvider - Adding implementation for ::TryToDownloadFileFromBackgroundThread. -- This method is thread safe compared to the normal ::Execute call due to: -- 1) This method does not accept any callbacks as the most common use of callbacks with ::Execute is to update the UI. -- 2) This method does not add the command to any of the internal queues nor rely on Tick to process the command. -- 3) The worker is invoked directly from the calling thread rather than as a background task. --4) This method does not allow commands that modify the cached states as this would be very difficult to manage without rewriting how all commands are processed to make sure that they are only run one at a time and the results processed in order. NOTE that async commands can be issued from the game thread and can in theory be run in any order on the p4 server depending on how the background task processing works out with no certainty as to the order the results will be processed in. It is unlikely that we will process results out of order but not impossible, but this is a pre-exisiting issue with the current API, extending it to be fully multithreaded will just make it worse. -- Note that we do not explicity check if the command supports being called from a background thread, as creating the command will check that anyway. - Calling ::OutputCommandMessages is now thread safe. When called from the game thread it will issue it's output via FMessageLog as before but when called from a background thread we will use the UE_LOG macros instead. Marshalling all output in the same way is something we will need to solve when/if we make the entire API thread safe. * PerforceControlSettings - Improve thread safety when calling ::GetConnectionInfo - If the password is stored in the UI then this is not thread safe but that feature is hidden at the moment due to being unsecure. I am unsure if we want to move the password to be stored in a member variable that can be accessed from any thread and only updated via the UI or not. * SourceControlOperations - Add new FDownloadFile operation. - This can either download the file(s) to FSharedBuffers which can be accessed by calling ::GetFileData OR directly to a directory on disk if the target directory is supplied as part of the constructor. * ISourceControlOperation - Add a new base method CanBeCalledFromBackgroundThreads (false by default) which can be overriden by derived classes and return true to indicate that the operation can be run from a background thread. This is intended for use by FDownloadFile with the perforce source control provider in lieu and making the API fully thread safe. * PerforceSourceControl.uplugin - Move the perforce plugin to load as early as possible as if Mirage is to use it then we need it to be avaliable before we start loading any potentially virtualized data in packages. [CL 16346666 by paul chipchase in ue5-main branch]
2021-05-17 05:06:44 -04:00
#include "Misc/Paths.h"
bool FUpdateStatus::IsDirectoryPath(const FString& Path) const
{
// Problem:
// We need to be able to work out if a path if a directory or not as some source
// control implementations will need to append wildcards to the path if it is
// a directory in order for it to function correctly.
//
// Strictly speaking the only way we can tell the difference between a file path
// and a directory path is to check with the file system.
// However some source control implementations let us reference files on the server
// and do not require us to have them on the users machine. This means checking with
// the file system does not give an accurate result and can end up being very
// slow if the file system interprets the path as a UNC path and tries to resolve it.
//
// Ideally we would let the caller clearly mark if the path is for a file or a
// directory but this would require a large overhaul of the source control API.
// We cannot change the existing behavior as it is very likely that 3rd party code is
// relying on it.
//
// Solution:
// As a 'temporary' solution, the caller can set bSetRequireDirPathEndWithSeparator
// to true. When it is true we will assume that all directory paths end with a
// separator. Although we tend to not terminate directory paths with a separator in the
// engine, this will allow us an opt in to the new behavior until more substantial API
// changes can be made.
if (bSetRequireDirPathEndWithSeparator)
{
return Path.EndsWith(TEXT("/")) || Path.EndsWith(TEXT("\\"));
}
else
{
return IFileManager::Get().DirectoryExists(*Path);
}
}
Improve error handling and logging in the file and jupiter Mirage backends. #rb Per.Larsson #rnx #preflight 60f9212d1f926d0001d41223 * VirtualizationJupiterBackend - When pulling a payload we should assume that a 400 error response when trying to GET the payload header means that the payload is not in Jupiter. -- Not being able to find the payload should not log an error, instead we can make a note of it in the verbose log (similar to the file system backend) * VirtualizationFileBackend - Moved the formatting of system errors to it's own function. - Log the system error when failing to write a payload during a push as well as a pull. - We now check that the FileArchive wrote correctly to disk and delete the output file and fail the push if it did not. -- A future piece of work will change the logic to write to a tmp file at the root of the file store and them move the file to the final location to cut down on the potential of leaving corrupted files around (similar to the process when we save packages) * Perforce - The FDownloadFile command now takes an optional parameter EVerbosity that can allow the caller to choose the level of logging output that the command will generate. - The source control backend for Mirage now opts to supress the logging of the full perforce command when we are pulling payloads as we can generate many hundreds or thousands of requests and the info is not useful to users. -- We continue to log the command when validating the depot as this is the most likely command to fail so having the info in the log may prove useful. [CL 16921815 by paul chipchase in ue5-main branch]
2021-07-22 05:43:20 -04:00
FDownloadFile::FDownloadFile(FStringView InTargetDirectory, EVerbosity InVerbosity)
: Verbosity(InVerbosity)
, TargetDirectory(InTargetDirectory)
Add support for downloading a file from source control directly without actually checking it out (ie p4 print) and add additional support so that the command can be called from a background thread. - Only implemented for perforce source control at the moment. - The source control operation FDownload file can be used just like any other command via ::Execute BUT it will also work with the new method ::TryToDownloadFileFromBackgroundThread which unlike ::Execute can be called from a background thread. #rb Sebastien.Lussier #rnx #preflight 60a2290322cce000014243ae * PerforceConnection - Extend ::RunCommand to take an optional FSharedBuffer reference which can be used to return text/binary data downloaded as part of a perforce command -- Ideally we would replace all of the out parameters with a single struct but that level of refactor would be better suited to its own submit. - Removed the bool parameter for FP4ClientUser and replaced it with a set of flags instead - Added a flag to allow FP4ClientUser to store data, either in binary or text format from a perforce command. Although in general this will be the result of a p4 print command. * PerforceSourceControlCommand - No longer assert if the command is not on the game thread if the operation indicates that it is safe to run on a background thread. * PerforceSourceControlModule - Register the 'DownloadFile' worker * PerforceSourceControlOperations - Add FPerforceDownloadFileWorker * PerforceSourceControlProvider - Adding implementation for ::TryToDownloadFileFromBackgroundThread. -- This method is thread safe compared to the normal ::Execute call due to: -- 1) This method does not accept any callbacks as the most common use of callbacks with ::Execute is to update the UI. -- 2) This method does not add the command to any of the internal queues nor rely on Tick to process the command. -- 3) The worker is invoked directly from the calling thread rather than as a background task. --4) This method does not allow commands that modify the cached states as this would be very difficult to manage without rewriting how all commands are processed to make sure that they are only run one at a time and the results processed in order. NOTE that async commands can be issued from the game thread and can in theory be run in any order on the p4 server depending on how the background task processing works out with no certainty as to the order the results will be processed in. It is unlikely that we will process results out of order but not impossible, but this is a pre-exisiting issue with the current API, extending it to be fully multithreaded will just make it worse. -- Note that we do not explicity check if the command supports being called from a background thread, as creating the command will check that anyway. - Calling ::OutputCommandMessages is now thread safe. When called from the game thread it will issue it's output via FMessageLog as before but when called from a background thread we will use the UE_LOG macros instead. Marshalling all output in the same way is something we will need to solve when/if we make the entire API thread safe. * PerforceControlSettings - Improve thread safety when calling ::GetConnectionInfo - If the password is stored in the UI then this is not thread safe but that feature is hidden at the moment due to being unsecure. I am unsure if we want to move the password to be stored in a member variable that can be accessed from any thread and only updated via the UI or not. * SourceControlOperations - Add new FDownloadFile operation. - This can either download the file(s) to FSharedBuffers which can be accessed by calling ::GetFileData OR directly to a directory on disk if the target directory is supplied as part of the constructor. * ISourceControlOperation - Add a new base method CanBeCalledFromBackgroundThreads (false by default) which can be overriden by derived classes and return true to indicate that the operation can be run from a background thread. This is intended for use by FDownloadFile with the perforce source control provider in lieu and making the API fully thread safe. * PerforceSourceControl.uplugin - Move the perforce plugin to load as early as possible as if Mirage is to use it then we need it to be avaliable before we start loading any potentially virtualized data in packages. [CL 16346666 by paul chipchase in ue5-main branch]
2021-05-17 05:06:44 -04:00
{
FPaths::NormalizeDirectoryName(TargetDirectory);
// Due to the asynchronous nature of the source control api, it might be some time before
// the TargetDirectory is actually used. So we do a validation pass on it now to try and
// give errors close to the point that they were created. The caller is still free to try
// and use the FDownloadFile but with an invalid path it probably won't work.
FText Reason;
if (!FPaths::ValidatePath(TargetDirectory, &Reason))
{
UE_LOG(LogSourceControl, Error, TEXT("Path '%s' passed to FDownloadFile is invalid due to: %s"),
*TargetDirectory, *Reason.ToString());
}
}
The ability to create, delete and switch workspaces via the SourceControl API. Currently this is only implemented for the perforce source control plugin. #rb Sebastien.Lussier #rnx #preflight 611e457e2c1f1c0001292a29 ### ISourceControlProvider / PerforceSourceControlProvider - Add new method ::SwitchWorkspace allowing the caller to switch the current workspace. - Currently only implemented for perforce. - Calling this will cause the provider to close and then re-open it's connection and so will invalidate any existing established connection or cached states. Due to this the method will fail if there are outstanding commands being processed. - It is intended that this is only used for short periods of time, in order to submit or access files from locations that the user chosen workspace might be able to access. - The new workspace will not be saved to the source control settings. Although this could occur later if done. ### SourceControlOperations - Add new operation **FCreateWorkspace**. - Currently only implemented for perforce. - The name and the root of the workspace (filepath on the client machine) must be provided in the constructor. - Additional mappings can be provided by calling **AddNativeClientViewMapping** which expects the mapping to be in the native format of the target source control system. At some point in the future we may add a more generic, target agnostic, format. - Add new operation **FDeleteWorkspace**. - Currently only implemented for perforce. - Deletes the workspace of the given name although the workspace must be in a state where it can be deleted (no files open for add/edit/delete etc) or the operation will fail. ### PerforceConnection - Add a new method ::CreateWorkspace for creating workspaces - Add a new extended client 'FP4CommandWithStdInputClientUser' - We should really extend FP4ClientUser to be able to take overridden stdinput instead but that would mean adding yet another input parameter. I'd rather keep the code separate for now and then merge them together in a future refactor where the many input parameters of ::RunCommand is replaces with a struct with TOptional members. - Added a ::GetUser method, which returns the user of the current connection without the caller needing to know if the connection is in unicode format or not. [CL 17350788 by paul chipchase in ue5-main branch]
2021-08-30 08:06:38 -04:00
FSharedBuffer FDownloadFile::GetFileData(const FStringView& Filename)
{
const uint32 Hash = GetTypeHash(Filename);
FSharedBuffer* Buffer = FileDataMap.FindByHash(Hash, Filename);
if (Buffer != nullptr)
{
return *Buffer;
}
else
{
return FSharedBuffer();
}
}
The ability to create, delete and switch workspaces via the SourceControl API. Currently this is only implemented for the perforce source control plugin. #rb Sebastien.Lussier #rnx #preflight 611e457e2c1f1c0001292a29 ### ISourceControlProvider / PerforceSourceControlProvider - Add new method ::SwitchWorkspace allowing the caller to switch the current workspace. - Currently only implemented for perforce. - Calling this will cause the provider to close and then re-open it's connection and so will invalidate any existing established connection or cached states. Due to this the method will fail if there are outstanding commands being processed. - It is intended that this is only used for short periods of time, in order to submit or access files from locations that the user chosen workspace might be able to access. - The new workspace will not be saved to the source control settings. Although this could occur later if done. ### SourceControlOperations - Add new operation **FCreateWorkspace**. - Currently only implemented for perforce. - The name and the root of the workspace (filepath on the client machine) must be provided in the constructor. - Additional mappings can be provided by calling **AddNativeClientViewMapping** which expects the mapping to be in the native format of the target source control system. At some point in the future we may add a more generic, target agnostic, format. - Add new operation **FDeleteWorkspace**. - Currently only implemented for perforce. - Deletes the workspace of the given name although the workspace must be in a state where it can be deleted (no files open for add/edit/delete etc) or the operation will fail. ### PerforceConnection - Add a new method ::CreateWorkspace for creating workspaces - Add a new extended client 'FP4CommandWithStdInputClientUser' - We should really extend FP4ClientUser to be able to take overridden stdinput instead but that would mean adding yet another input parameter. I'd rather keep the code separate for now and then merge them together in a future refactor where the many input parameters of ::RunCommand is replaces with a struct with TOptional members. - Added a ::GetUser method, which returns the user of the current connection without the caller needing to know if the connection is in unicode format or not. [CL 17350788 by paul chipchase in ue5-main branch]
2021-08-30 08:06:38 -04:00
FCreateWorkspace::FCreateWorkspace(FStringView InWorkspaceName, FStringView InWorkspaceRoot)
: WorkspaceName(InWorkspaceName)
{
TStringBuilder<512> AbsoluteWorkspaceName;
FPathViews::ToAbsolutePath(InWorkspaceRoot, AbsoluteWorkspaceName);
WorkspaceRoot = AbsoluteWorkspaceName.ToString();
}