#rb trivial
#jira none
#rnx
#preflight 64622a40cf788a25582851cd
- Virtualization/hydration can be quite slow. We should log the time taken so that it is easier for users to see if something is abnormally slow and report it.
- Small code cleanup in the hydration code to move the ::UnRegisterMountPoints call to a ON_SCOPE_EXIT
[CL 25472139 by paul chipchase in ue5-main branch]
#rb none
#jira UE-181761
#rnx
#preflight 644643bab14f1faacf8a1536
- When the tool was created we tried to avoid discovering all plugins as it was very slow. Since then it has been optimized and only costs 500ms or so, which for now is acceptable. So now we allow plugins to be discovered in the normal way. If the plugins have been set up correctly to work with the tool then they will be loaded.
- Note that the perforce plugin still needs to be mounted manually, but no longer needs the paths adding.
- Removed UE_DISABLE_PLUGIN_DISCOVERY from the Target.cs so that we scan for all plugins on startup.
- Removed adding of perforce to AdditionalPlugins in the Target.cs as it wasn't really doing anything.
[CL 25161223 by paul chipchase in ue5-main branch]
#rb none
#jira UE-151641
#rnx
#preflight 642599a738075fa13fe70c7e
- Changed the logic of TryConnectToSourceControl so that if a project is set we try to use the projects existing revision control settings for the default provider (which is the provider that will be used by the virtualization/rehydration process), if the project is not set then the only revision control operation that should occur is parsing of an existing changelist to find the files that we might want to operate on.
-- As we now also use the default provider we cannot use a single TUniquePtr as we do not own the pointer to the default provider. So there is a general pointer to be used for the current active provider and a TUniquePtr if we own the provider.
- Added calls to ::TryConnectToSourceControl if a command is setting the Checkout flag which should ensure that the packages will be able to be checked out.
[CL 24854420 by paul chipchase in ue5-main branch]
#rb none
#jira none
#rnx
#preflight 64258067b72410fc17a11f03
- The main goal was to prevent the GConfig from flushing to disk on shutdown as we do not want the tool to do that.
- We can do this by disabling the disk operations for GConfig or forcing the process to terminate, the latter of which we support in commandlets via the cmdline option '-fastexit'
[CL 24852905 by paul chipchase in ue5-main branch]
#rb none
#jira UE-151641
#rnx
#preflight 642574c0974dfaa53cae8be2
- The checkout cmdline option was not being serialized to the child process correctly.
[CL 24852626 by paul chipchase in ue5-main branch]
#rb trivial
#jira UE-151641
#rnx
#preflight 64103cb1c41a0a2a770c96a4
- Although we now relaunch the tool with the correct project path (if not supplied) this was not enough to allow us to load the projects config files and so set up the DDC and source control etc properly. My original testing was picking up hacks in the programs config file that was allowing it to work.
- FGenericPlatformMisc::ProjectDir has a bespoke path for standalone programs which means even if the uproject is on the cmdline it is ignored.
- We probably should look into a way for a project to mark itself as being "project aware" but for now we can make use of the project path override system to make sure it is set correctly, which will fix the problem for now.
[CL 24631824 by paul chipchase in ue5-main branch]
#rb trivial
#jira UE-177413
#preflight 63ef542fe832ea950a6d936c
- This means by default all log messages with 'Display' verbosity will be shown, we will only demote them to 'Log' verbosity when -minimallogging is passed in.
- UVT was originally intended to be called as a custom tool in P4V, at which point it made sense to hide a lot of the logging from the user so that the output would be easier for them to follow and if they did encounter a problem they could send the full log file in the bug report.
- In the past few months there has been an increase in people trying to use UVT to verify that they have setup VA on their project correctly rather than using the editor. In many of these cases it was not obvious that they'd need to pass in -verbose to see full output.
- By inverting the cmdline meaning we can display all logging to people by default (i.e. calling from the command line) and then supress the logging when called via P4V by using the new cmdline.
[CL 24279361 by paul chipchase in ue5-main branch]
#rb trivial
#jira none
#rnx
#preflight 63ee4321bd6bb956b9174a6e
- Improve some formatting and make it clear that the virtualize and rehydrate modes should take the project path too.
[CL 24257706 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-151641
#preflight 63ea068cde74ffbae5dc3a7f
### Problem
- A lot of systems rely on the engine to know which project is being loaded on start up so that the correct config settings can be applied however so far the UVT has not required the user to provide the desired project. This is because in theory, the tool could be given a list of packages or a changelist to process that contain packages from more than one project and if that happens we need to handle it correctly.
- The tool currently groups the provided packages by project, then runs the requested command on each project one at a time, initializing the VA system with that projects config files, then shutting it done once the project packages are processed. This allows us to pick up the correct settings for VA but other systems that we rely on, such as the DDC do not allow this sort of dynamic initialization and so may not have the correct settings.
### Solution
- Before the projects packages are processed we check to see if the project applied to the process during initialization (if any) matches the project to be processed. If not we launch a new version of the tool with the correct project settings and have the child process perform the main processing. The output of each child process (or any projects we were able to run in process) are then provided to the command for a final "post projects" phase.
- This means we can support multiple packages from multiple processes AND not require the user type out the project name when using the tool from the cmdline.
- Although launching a child process only adds a few seconds of overhead (per project) any tool or process that make use of UVT should be updated to provide the project where possible.
### Changes
### FUnrealVirtualizationToolApp
- As explained in the problem section, we now need to support launching a new instance of the tool as a child process.
- We communicate with the child process by providing an input file via the cmdline, which is then parsed and used to recreate the original command and project settings. Once the command has run in the child process it will write its output to an output file which the calling process can then read back in.
- So our new logic is to check the project that the tool was started with (in case the user provided a project path in the cmdline) against each project that we need to process. If the paths match then we can just process the project, if they do not match we launch a child process with the correct path and retrieve the output once done.
-- If a child process detects an incorrect path we just error out to avoid endless recursive process launching.
- Output from the child process is clearly logged to the user as coming from a child process to make debugging easier.
### FCommand
- Added ToJson/FromJson methods to FCommand, which simulates the same interface as though it was using the json serialization macros, however we want more control over things so do the serialization manually via FCommand::Serialize
-- Serialization is only used if the command is sent to a child process (if the current project is wrong), in which case the command should serialize out all info that it derived during setup that it will need to process correctly. Usually this consists of the options that were parsed from the command line. Then when the child process is run we will recreate the command via the same serialize method.
- Added a new class FProcessPipes, which acts as a wrapper around the pipes we can create by calling FPlatformProcess::CreatePipe which are returned as raw pointers. This ensures that the pipes are cleaned up once the FProcessPipes object goes out of scope.
- The ::Run method has been split into two, ::ProcessProject and ::ProcessOutput
-- ::ProcessProject is called once per project and allows the command to return output via the Output parameter. This can be call in process or via a child process if the project needs to be set.
-- ::ProcessOutput is called once all projects have been processed and provides an array which is a collection of the output from each call to ::ProcessProject. This is always called in process.
-- The output is passed around by the base type FCommandOutput. It is expected that commands derive their own type and as we do not mix command types, they should know how to cast the output back to the correct type when ::ProcessOutput is called. This is a bit messy but to go further risks over engineering given it is unlikely that more commands will be added to the tool. If we do expand the tool in the future then this can be revisited.
### FCommandOutput
- A struct used to represent the output of a command.
- If the command is run as a child process then this struct will be serialized out to disk (via json) then loaded back in by the calling process so that the output of each child process can be combined.
- Commands are expected to derive their own output structure and deal with the json serialization. To make this easier we provide a JSON_SERIALIZE_PARENT macro to allow for serialization of inheritance chains.
- This system does require that commands know which type of output to cast FCommandOutput to. At the moment we do not provide any type safety.
- Add json serialization to FProject and it's child struct FPlugin.
- FProject::GetProjectFilePath now returns as a reference to a FString rather than a FStringView.
[CL 24257111 by paul chipchase in ue5-main branch]
#rb trivial
#jira UE-151641
#rnx
#preflight 63d94cd3ba4fadeef0930cca
- As ISourceControlProvide::Init does not return if a connection was established or not we need to also call ISourceControlProvide::IsAvailable afterwards to check if we actually have a connection.
- This lets us fail early with a more specific error message rather than waiting for the first command to fail especially as our commands rarely print useful errors if the connection failed.
[CL 23931768 by paul chipchase in ue5-main branch]
#rb trivial
#jira none
#rnx
#preflight none (accidently submitted instead of launching preflight)
- This makes calling it without a valid client spec alittle cleaner as we no longer need to pass in a dummy FStringView/String
[CL 23926635 by paul chipchase in ue5-main branch]
#rb Sebastian.Nordgren
#jira UE-175587
#rnx
#preflight 63d8ec967a39a18021c168bf
- The DDC module can now invoke the 'DesktopPlatform' module when trying to set up the interactive ddc login. At the moment the 'DerivedDataCache' module only links to the includes with no dependency on the module itself so as a short term fix I am adding a dependency from the tool itself.
- The interactive login process relies on our basic OS message dialogs. On windows this requires that the .rc file in the 'Launch' module be included otherwise the message dialog will not be shown. We don't want a dependency on that module directly as it is super heavy weight but we can add our own .rc file for the tool and inside that include the .rc file of the 'Launch' module.
[CL 23926364 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-169626
#rnx
#preflight 63bd670a71079a8d1c0e837b
- Since the API was forcing the caller to pass in a results structure to be filled in, we might as well make it the return value.
- Added a ::WasSuccessful method to the results structures that can be used instead of checking if the result had errors or not.
- Remove the reset method from FVirtualizationResult/FRehydrationResult as they no longer need it.
- The older deprecated methods still use the results enum, so we cannot easily deprecate those enums yet.
[CL 23626072 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-169626
#rnx
#preflight 63bd0ebfd862fdd347bce1fe
### VA System
- This allows us to provide the user with more ways to customize the rehydration and return more detailed info about it if the calling code wishes to log additional info. In both cases we can extend the options and the data returned without changing the API.
- At the moment the only flag we support is 'Checkout', which requests that the rehydration process checkout any package that it needs to modify rather than giving an error. This means that the user does not need to check packages out from revision control before running the rehydration process.
-- We still check if packages can be modified and warn the user if they can't, as package files could be locked in other ways.
- The rehydration process will now long the time taken if verbose logging is set for the category 'LogVirtualization'
### UnrealVirtualizationTool
- The virtualize command now reports how many packages were checked out if the flag was set.
- The rehydration command now supports a '-Checkout' commandline flag, which when enabled will use the new api to checkout the packages that need to be checked out when rehydrated.
[CL 23625132 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-169626
#rnx
#preflight 639c4112012902cb8db43e13
- This allows us to provide the user with more ways to customize the virtualization and return more detailed info about it if the calling code wishes to log additional info. In both cases we can extend the options and the data returned without changing the API.
- Previously if we virtualized a package that was not checked out in revision control we would warn the user and then skip updating the package file on disk. This means the payloads would be uploaded but the user would be left with no local changes. Since sometimes we know we don't need to check out any package (virtualizing the packages in a change list for example) we don't want to always incur the cost of polling reivision control to see which packages do need checking out. This is why we now allow the caller to request package files be checked out via the new options enum EVirtualizationOptions.
-- If the EVirtualizationOptions::Checkout flag is provided we will poll the revision control status of all package files and then check out those which need it.
-- We still check if packages can be modified and warn the user if they can't, as package files could be locked in other ways.
- Added a new utility function to SourceControlUtilties to make it easier to check out packages. There is similar functionality elsewhere in the code base but the virtualization module is too low level to make use of it.
- Updated existing code that calls ::TryVirtualizePackages and add cases of ''using namespace UE::Virtualization' where required to improve readability.
- The UnrealVirtualizationTool now supports a new cmdline option "-checkout" that can be used when virtualizing packages. This will checkout any package that was actually virtualized so the result can be saved back out to the workspace domain. This means we no longer require the caller to have checked out the packages before running the tool.
[CL 23536832 by paul chipchase in ue5-main branch]
#rb trivial
#jira none
#rnx
#preflight 639877140a6715255053caad
- Moved the shutdown code to scope exit so that they will continue to trigger on any early out paths.
[CL 23490478 by paul chipchase in ue5-main branch]
