#rb Juan.Legaz
#jira UE-222213
#rnx
- At the moment most projects allow submits with failed virtualization attempts to continue uninterrupted but that means we rely on users reporting issues to us. By raising an ensure we will get a report sent to our crash reporting systems which we can use to dig into the problems. This system will be opt in via the cmdline arg.
- The cmdline arg will be passed onto any spawned child processes so that they will report too.
-- Note that we also now pass '-fastexit' onto the child processes which was not done before.
- We only raise an ensure if the problem occurred in the current process, it is expected that child processes will raise their own.
-- We use a new enum EProcessResult to pass this info around.
- The log filtering has been extended to filter out warning/error messages normally logged when an ensure fires as these will concern the end users, we don't want the reporting to not interrupt their day to day work either.
[CL 35695506 by paul chipchase in ue5-main branch]
#rb Juan.Legaz
#jira UE-221396
- When invoking a child process UVT will pass in the path of the target .uproject file as well as a path to a .inputfile containing the details of the operation to run. If either of these paths contained whitespace then they would fail to parse correctly and either the target project would not be correctly set or the input file would not be readable.
- We now properly escape the args so that the whole paths are parsed, even when containing whitespace.
[CL 35393883 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-215823
#rnx
- When the UVT needs to launch a child process we send it info on the command to run via a .input file and in turn the child process will write out a .output file to return info to the calling process.
- This change will now attempt to delete these files after we are done with them.
- If the delete fails we will raise a warning rather than an error as it is not essential that the files be deleted.
[CL 33923415 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
- The problem with the process set up is described properly in UE-208628, but the quick summary is that a lot of our code is creating a single pipe, then using the input as stdin and the output as stdout when launching a child process so if that process ever did read from stdin it would end up using it's own output. Then we are copy/pasting the same bug around the code base,
- Fixed the problem by adding a second pipe to FProcessPipes.
[CL 32465578 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#rnx
- Do not log if '-MinimalLogging' is set in the cmdline. If it is set then the tool is supposed to only display non-technical info and the user probably doesn't care about the flag. If we need to check the state of it, the log file will contain the processes cmdline anyway.
- Add a missing tab to the logging in FUnrealVirtualizationToolApp::TrySortFilesByProject to keep it aligned.
[CL 28833497 by paul chipchase in ue5-main branch]
#rb none
#jira UE-181761
#rnx
#preflight 644643bab14f1faacf8a1536
- When the tool was created we tried to avoid discovering all plugins as it was very slow. Since then it has been optimized and only costs 500ms or so, which for now is acceptable. So now we allow plugins to be discovered in the normal way. If the plugins have been set up correctly to work with the tool then they will be loaded.
- Note that the perforce plugin still needs to be mounted manually, but no longer needs the paths adding.
- Removed UE_DISABLE_PLUGIN_DISCOVERY from the Target.cs so that we scan for all plugins on startup.
- Removed adding of perforce to AdditionalPlugins in the Target.cs as it wasn't really doing anything.
[CL 25161223 by paul chipchase in ue5-main branch]
#rb trivial
#jira UE-177413
#preflight 63ef542fe832ea950a6d936c
- This means by default all log messages with 'Display' verbosity will be shown, we will only demote them to 'Log' verbosity when -minimallogging is passed in.
- UVT was originally intended to be called as a custom tool in P4V, at which point it made sense to hide a lot of the logging from the user so that the output would be easier for them to follow and if they did encounter a problem they could send the full log file in the bug report.
- In the past few months there has been an increase in people trying to use UVT to verify that they have setup VA on their project correctly rather than using the editor. In many of these cases it was not obvious that they'd need to pass in -verbose to see full output.
- By inverting the cmdline meaning we can display all logging to people by default (i.e. calling from the command line) and then supress the logging when called via P4V by using the new cmdline.
[CL 24279361 by paul chipchase in ue5-main branch]
#rb trivial
#jira none
#rnx
#preflight 63ee4321bd6bb956b9174a6e
- Improve some formatting and make it clear that the virtualize and rehydrate modes should take the project path too.
[CL 24257706 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-151641
#preflight 63ea068cde74ffbae5dc3a7f
### Problem
- A lot of systems rely on the engine to know which project is being loaded on start up so that the correct config settings can be applied however so far the UVT has not required the user to provide the desired project. This is because in theory, the tool could be given a list of packages or a changelist to process that contain packages from more than one project and if that happens we need to handle it correctly.
- The tool currently groups the provided packages by project, then runs the requested command on each project one at a time, initializing the VA system with that projects config files, then shutting it done once the project packages are processed. This allows us to pick up the correct settings for VA but other systems that we rely on, such as the DDC do not allow this sort of dynamic initialization and so may not have the correct settings.
### Solution
- Before the projects packages are processed we check to see if the project applied to the process during initialization (if any) matches the project to be processed. If not we launch a new version of the tool with the correct project settings and have the child process perform the main processing. The output of each child process (or any projects we were able to run in process) are then provided to the command for a final "post projects" phase.
- This means we can support multiple packages from multiple processes AND not require the user type out the project name when using the tool from the cmdline.
- Although launching a child process only adds a few seconds of overhead (per project) any tool or process that make use of UVT should be updated to provide the project where possible.
### Changes
### FUnrealVirtualizationToolApp
- As explained in the problem section, we now need to support launching a new instance of the tool as a child process.
- We communicate with the child process by providing an input file via the cmdline, which is then parsed and used to recreate the original command and project settings. Once the command has run in the child process it will write its output to an output file which the calling process can then read back in.
- So our new logic is to check the project that the tool was started with (in case the user provided a project path in the cmdline) against each project that we need to process. If the paths match then we can just process the project, if they do not match we launch a child process with the correct path and retrieve the output once done.
-- If a child process detects an incorrect path we just error out to avoid endless recursive process launching.
- Output from the child process is clearly logged to the user as coming from a child process to make debugging easier.
### FCommand
- Added ToJson/FromJson methods to FCommand, which simulates the same interface as though it was using the json serialization macros, however we want more control over things so do the serialization manually via FCommand::Serialize
-- Serialization is only used if the command is sent to a child process (if the current project is wrong), in which case the command should serialize out all info that it derived during setup that it will need to process correctly. Usually this consists of the options that were parsed from the command line. Then when the child process is run we will recreate the command via the same serialize method.
- Added a new class FProcessPipes, which acts as a wrapper around the pipes we can create by calling FPlatformProcess::CreatePipe which are returned as raw pointers. This ensures that the pipes are cleaned up once the FProcessPipes object goes out of scope.
- The ::Run method has been split into two, ::ProcessProject and ::ProcessOutput
-- ::ProcessProject is called once per project and allows the command to return output via the Output parameter. This can be call in process or via a child process if the project needs to be set.
-- ::ProcessOutput is called once all projects have been processed and provides an array which is a collection of the output from each call to ::ProcessProject. This is always called in process.
-- The output is passed around by the base type FCommandOutput. It is expected that commands derive their own type and as we do not mix command types, they should know how to cast the output back to the correct type when ::ProcessOutput is called. This is a bit messy but to go further risks over engineering given it is unlikely that more commands will be added to the tool. If we do expand the tool in the future then this can be revisited.
### FCommandOutput
- A struct used to represent the output of a command.
- If the command is run as a child process then this struct will be serialized out to disk (via json) then loaded back in by the calling process so that the output of each child process can be combined.
- Commands are expected to derive their own output structure and deal with the json serialization. To make this easier we provide a JSON_SERIALIZE_PARENT macro to allow for serialization of inheritance chains.
- This system does require that commands know which type of output to cast FCommandOutput to. At the moment we do not provide any type safety.
- Add json serialization to FProject and it's child struct FPlugin.
- FProject::GetProjectFilePath now returns as a reference to a FString rather than a FStringView.
[CL 24257111 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-170657, UE-160396
#rnx
#preflight 637c9232fa348e8480bdc7e2
- When the rehydration functionality was added it started to look like more functionality would be added to the tool than originally thought so I started to add the new functionality via a command system. This change now moves the older legacy functionality to the coommand system as well.
- Added a new FVirtualizeCommand which can accept package paths, directory paths, packagelist files or changelists as the input.
-- Unlike the legacy commands, virtualizing via changelist does not require the client spec to be provided on the commandline, although doing so will avoid several perforce commands and speed up the call. It is expected that people calling the tool on the commandline will probably opt to not supply a clientspec and let the tool workout which workspace a changelist is under, where as calls from other tools (such as P4VUtils) can provide it if already known to speed things up.
-- FVirtualizeLegacyChangeListCommand replicates the functionality of the old -Mode=Changelist command.
-- FVirtualizeLegacyPackageListCommand replicates the functionality of the old -Mode=Packagelist command.
-- The new command will only try to submit the results if a changelist was provided as the input and even then it will not do so by default. The tool now requires people to opt into submitting the changelist via the command line option -submit. NOTE: Other versions of the command maybe allow submission in the future but the virtualization process needs to be improved so that it can check out package files before this really makes sense.
- The rehydration command no longer requires a clientspec on the command line, it wasn't using the value anyway.
- Moved the source control code from the app code files to CommandBase. If we add more commands in the future we might want to factor this out to its own base class so commands can opt into source control functionality.
- In the future we should probably move the package -> project sorting code (FUnrealVirtualizationToolApp::TrySortFilesByProject) into the command base code as well.
[CL 23233456 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-161296
#rnx
#preflight 62fe3ce73d3fb466b229bcc0
- There are some usecases that require the VA system to initialize the first time it is accessed (usually the first time we attempt to pull a virtualized payload) rather than be initialized in the program start up. This change provides three different methods to achieve this:
-- Setting the define 'UE_VIRTUALIZATION_SYSTEM_LAZY_INIT' to 1 in a programs .target.cs
-- Setting [Core.ContentVirtualization]LazyInit=true in the Engine ini file
-- Running with the commandline option -VA-LazyInit
- If we detect that the source control backend is being initialized on a background thread we do not try to run the FConnect operation. The backend will still work but this does reduce the potential error checking on initialization. This is done because the FConnect operation currently only works on the main thread and to change this would be a bigger work item than we can schedule at the moment.
- UE::Virtualization::Initialize functions now take a EInitializationFlags enum as a parameter. This enum allows the call to ignore all lazy init settings and force the initialization immediately. This is useful for programs like the Virtualization standalone tool which just needs to start the system when needed.
-- The call to ::Initialize in LaunchEngineLoop passes in None and does not ignore lazy initialization.
-- Calls to ::Initialize in the UnrealVirtualizationTool however all use EInitializationFlags::ForceInitialize and ignore lazy initialization settings.
- Fixed an odd bug in UE::Virtualization::Initialize where the error path (if the config file cannot be found) was using a different start up code path.
- Add asserts when assigning to GVirtualizationSystem to make sure that it is null. This is not 100% safe but should catch some potential threading issues, if any.
- Add an assert after lazy initialization (IVirtualizationSystem::Get) to make sure that GVirtualizationSystem was assigned a valid object.
- Improve how we check for legacy values in [Core.ContentVirtualization]. We now support multiple allowed values.
- Added a way to poll if a VA system has been initialize yet or not, this allows us to avoid initializing a VA system if one has not yet been created and we try to:
-- Dump VA profiling stats after cooking
-- Send VA stats to studio analytics
- Note that currently using lazy init loading will probably cause the VA statistics panel not to work, this will be fixed in future work where we will allow the panel to register for a callback when the system is initialized.
[CL 21467510 by paul chipchase in ue5-main branch]
#rb Sebastian.Nordgren
#rnx
#jira UE-156436
#preflight 62c287f9a3568e30664eb94f
### VA Standalone Tool
- We now plan to add much more functionality to the tool than just virtualizing and submitting changelists, so to make this easier I am moving the tool towards a design where it should be fairly easy to add new functionality.
- Added FCommand, which is a base class for adding new functionality, simple derive from FCommand and hook it up at the appropriate locations.
-- In the future it should be possible for new command types to automatically register themselves to be initiated from the command line. There should be no need to edit UnrealVirtualizationToolApp to add a new command but this will be done as an additional work item.
-- At the moment FCommand comes with a number of utility methods to call that cover some common source control commands.
-- The original functionality has not yet been moved to the command system and so the code is a little bit weird at the moment. Updating older code to the new system will be done as an additional work item.
- FProject/FPlugin have been moved to their own code files.
### Rehydrate Command
- The rehydrate command will take a number of packages, check them out of source control and then attempt to virtualize them.
- At the moment the chekout logic is fairly basic, we just check out every package supplied, we don't check if the package is virtualized or not yet. This can be improved in additional work items. Ideally by the end of command the only packages that we have checked out should also be rehydrated.
- At the moment the command can either take a path of a specific package, a path of a directory to find packages in, or a changelist containing packages that should be rehydrated.
- A cleint spec (workspace) can optionally be provided, but if not supplied we will attempt to find a client spec for which to check out the packages.
- Currently we will check out the packages to the default change list.
### Rehydrate process
- Added the rehydration process in it's own code files in the virtualization module. Like the virtualization process this is exposed in a public header file and no via the Core interface which means it is very specific to our module/implementation.
- The process expects that the caller will have checked out any required packages from source control. It will treat being unable to update a package file as an error.
- Added PackageUtils.h/.cpp and moved some of the generic code from the virtualization process code there so that it can be shared by the rehydration process.
### Misc
Moving away from the using things like FPackagePath as that requires that the correct mount points have been registered for a project and at the moment (with the flakiness of FConfig*) it seems that the best idea would be to prefer absolute file paths where possible.
[CL 20982284 by paul chipchase in ue5-main branch]
#rb trivial
#rnx
#preflight 628ccc455665463c21036888
- The main change is that if we do not find the project file relative to the first /content/ directory we will keep looking up the path to see if we can find additional /content/ directories.
- All log messages have been changed to warnings for consistency.
[CL 20346906 by paul chipchase in ue5-main branch]
#rb trivial
#rnx
#preflight 626b99f95c220f34acd748f6
- We were printing out the packagelist file path both when it was found as a command line arg and when parsing it, we now only print it when it is found as an arg
- Any log section that indicates it could be slow has had "..." appended to it to show the user that this is expected
-- Any log message in a section like this is now tab indented to make the association easy for the user to see.
[CL 19976012 by paul chipchase in ue5-main branch]
#rb trivial
#rnx
#preflight 626b928953253f874b90a23a
- We now connect to perforce if either a) we need to parse a given changelist to find the package files b) we need to submit the given changelist
- This means we won't connect to perforce at all when running with the cmdline '-Mode=PackageList' saving a few seconds (although if the virtualization system is using the source control backend, then a connection will be made there anyway)
[CL 19975945 by paul chipchase in ue5-main branch]
#rb trivial
#rnx
#preflight 626a99f3a5009ff191af09db
- The log message stating that the tool is now trying to submit a changelist was accidently set to 'Error' when it should be 'Display'
[CL 19961053 by paul chipchase in ue5-main branch]
#rb trivial
#rnx
#preflight 626a5913631e64c0b9f23de3
- Some of our local structures, like FPLugin can get confused by the debugger with the version in the engine. Wrapping the tools code in the virtualization namespace helps keep the separation clear and stops the debugger from getting confused.
- Plus even though this is an UE program, I probably should've stuck to the new namespace rules anyway.
#ushell-cherrypick of 19955706 by paul.chipchase
[CL 19956237 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#jira UE-143675
#rnx
#preflight 624c6a1814634fba5c7e3d71
### Rationale
- The new mode being added allows us to write a list of package file paths to be virtualized to a file on disk and then the file to be passed to the tool so that it knows what to virtualize. This new mode will then be used by P4VUtils.dll which is how we will run the virtualization process via p4v custom tools.
- We need this second mode because:
-- A workspace could have multiple projects in it, which means a user could try to virtualize a changelist with packages from more than one project.
-- Projects can be associated with different engine installations so the user could try to virtualize a changelist with packages using different engine installations, which in turn means that they need to be virtualized by different versions of the stand-alone tool.
-- This means we cannot rely on passing the CL number to the tool, install P4VUtils.dll will need to sort the package paths by engine installation, then pass those paths to the correct version of the tool.
-- The paths need to be written to a text file to be passed to the tool as the total length of path data can easily exceed the commandline max length limit if someone were to try and virtualize an entire project in one go for example.
- Note that the new mode does NOT submit the package files since there might be multiple calls to different versions of the stand-alone tool. In theory if all the packages are under one engine install (which is almost certainly going to be 99.999% of all submits) we could allow the tool to do the submit but that means we need to maintain two different code paths for the same operation.
- The existing mode (virtualize and submit a changelist) remains unchanged. It is not likely that an end user will ever use this mode, but it will make for a useful debugging tool in the future.
### Changes
- When the tool first starts we now print the log path to the user so that they can more easily find it. This might be something that we want to remove after a while, but during the initial release of the tool it should make getting good debug info back from users with problems somewhat easier.
- Some logging and error messages have had small updates made, to make the final output to the user more readable (based on the many many times this tool was run while testing). In some places additional logging has been added where I felt it was not clear to the user as to what was going on.
- Connecting to source control has been moved from the changelist parsing code to it's own method ::TryConnectToSourceControl since all modes will require it.
- The cmdline help output has been redone to support the multiple command line options we now have.
- Although we could infer the mode to run based on the commandline options, we require the users to explicitly state the mode via -Mode=XXX. In general the tool will be invoked from a gui so adding extra commands doesn't really affect the usability of the system and it makes things a bit clearer.
[CL 19665307 by paul chipchase in ue5-main branch]
