2022-11-22 07:23:08 -05:00
|
|
|
// Copyright Epic Games, Inc. All Rights Reserved.
|
|
|
|
|
|
|
|
|
|
#include "Commands/VirtualizeCommand.h"
|
|
|
|
|
|
|
|
|
|
#include "HAL/FileManager.h"
|
|
|
|
|
#include "Misc/FileHelper.h"
|
|
|
|
|
#include "Misc/ScopeExit.h"
|
|
|
|
|
#include "ProjectFiles.h"
|
|
|
|
|
#include "UnrealVirtualizationTool.h"
|
|
|
|
|
#include "Virtualization/VirtualizationSystem.h"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
#include "ISourceControlModule.h"
|
|
|
|
|
#include "ISourceControlProvider.h"
|
|
|
|
|
#include "SourceControlOperations.h"
|
|
|
|
|
|
[UVT] Improve the robustness of the tool wrt ensuring that the correct project config settings are loaded.
#rb Per.Larsson
#jira UE-151641
#preflight 63ea068cde74ffbae5dc3a7f
### Problem
- A lot of systems rely on the engine to know which project is being loaded on start up so that the correct config settings can be applied however so far the UVT has not required the user to provide the desired project. This is because in theory, the tool could be given a list of packages or a changelist to process that contain packages from more than one project and if that happens we need to handle it correctly.
- The tool currently groups the provided packages by project, then runs the requested command on each project one at a time, initializing the VA system with that projects config files, then shutting it done once the project packages are processed. This allows us to pick up the correct settings for VA but other systems that we rely on, such as the DDC do not allow this sort of dynamic initialization and so may not have the correct settings.
### Solution
- Before the projects packages are processed we check to see if the project applied to the process during initialization (if any) matches the project to be processed. If not we launch a new version of the tool with the correct project settings and have the child process perform the main processing. The output of each child process (or any projects we were able to run in process) are then provided to the command for a final "post projects" phase.
- This means we can support multiple packages from multiple processes AND not require the user type out the project name when using the tool from the cmdline.
- Although launching a child process only adds a few seconds of overhead (per project) any tool or process that make use of UVT should be updated to provide the project where possible.
### Changes
### FUnrealVirtualizationToolApp
- As explained in the problem section, we now need to support launching a new instance of the tool as a child process.
- We communicate with the child process by providing an input file via the cmdline, which is then parsed and used to recreate the original command and project settings. Once the command has run in the child process it will write its output to an output file which the calling process can then read back in.
- So our new logic is to check the project that the tool was started with (in case the user provided a project path in the cmdline) against each project that we need to process. If the paths match then we can just process the project, if they do not match we launch a child process with the correct path and retrieve the output once done.
-- If a child process detects an incorrect path we just error out to avoid endless recursive process launching.
- Output from the child process is clearly logged to the user as coming from a child process to make debugging easier.
### FCommand
- Added ToJson/FromJson methods to FCommand, which simulates the same interface as though it was using the json serialization macros, however we want more control over things so do the serialization manually via FCommand::Serialize
-- Serialization is only used if the command is sent to a child process (if the current project is wrong), in which case the command should serialize out all info that it derived during setup that it will need to process correctly. Usually this consists of the options that were parsed from the command line. Then when the child process is run we will recreate the command via the same serialize method.
- Added a new class FProcessPipes, which acts as a wrapper around the pipes we can create by calling FPlatformProcess::CreatePipe which are returned as raw pointers. This ensures that the pipes are cleaned up once the FProcessPipes object goes out of scope.
- The ::Run method has been split into two, ::ProcessProject and ::ProcessOutput
-- ::ProcessProject is called once per project and allows the command to return output via the Output parameter. This can be call in process or via a child process if the project needs to be set.
-- ::ProcessOutput is called once all projects have been processed and provides an array which is a collection of the output from each call to ::ProcessProject. This is always called in process.
-- The output is passed around by the base type FCommandOutput. It is expected that commands derive their own type and as we do not mix command types, they should know how to cast the output back to the correct type when ::ProcessOutput is called. This is a bit messy but to go further risks over engineering given it is unlikely that more commands will be added to the tool. If we do expand the tool in the future then this can be revisited.
### FCommandOutput
- A struct used to represent the output of a command.
- If the command is run as a child process then this struct will be serialized out to disk (via json) then loaded back in by the calling process so that the output of each child process can be combined.
- Commands are expected to derive their own output structure and deal with the json serialization. To make this easier we provide a JSON_SERIALIZE_PARENT macro to allow for serialization of inheritance chains.
- This system does require that commands know which type of output to cast FCommandOutput to. At the moment we do not provide any type safety.
- Add json serialization to FProject and it's child struct FPlugin.
- FProject::GetProjectFilePath now returns as a reference to a FString rather than a FStringView.
[CL 24257111 by paul chipchase in ue5-main branch]
2023-02-16 09:18:15 -05:00
|
|
|
namespace UE::Virtualization
|
2022-11-22 07:23:08 -05:00
|
|
|
{
|
|
|
|
|
|
|
|
|
|
/** Utility */
|
[UVT] Improve the robustness of the tool wrt ensuring that the correct project config settings are loaded.
#rb Per.Larsson
#jira UE-151641
#preflight 63ea068cde74ffbae5dc3a7f
### Problem
- A lot of systems rely on the engine to know which project is being loaded on start up so that the correct config settings can be applied however so far the UVT has not required the user to provide the desired project. This is because in theory, the tool could be given a list of packages or a changelist to process that contain packages from more than one project and if that happens we need to handle it correctly.
- The tool currently groups the provided packages by project, then runs the requested command on each project one at a time, initializing the VA system with that projects config files, then shutting it done once the project packages are processed. This allows us to pick up the correct settings for VA but other systems that we rely on, such as the DDC do not allow this sort of dynamic initialization and so may not have the correct settings.
### Solution
- Before the projects packages are processed we check to see if the project applied to the process during initialization (if any) matches the project to be processed. If not we launch a new version of the tool with the correct project settings and have the child process perform the main processing. The output of each child process (or any projects we were able to run in process) are then provided to the command for a final "post projects" phase.
- This means we can support multiple packages from multiple processes AND not require the user type out the project name when using the tool from the cmdline.
- Although launching a child process only adds a few seconds of overhead (per project) any tool or process that make use of UVT should be updated to provide the project where possible.
### Changes
### FUnrealVirtualizationToolApp
- As explained in the problem section, we now need to support launching a new instance of the tool as a child process.
- We communicate with the child process by providing an input file via the cmdline, which is then parsed and used to recreate the original command and project settings. Once the command has run in the child process it will write its output to an output file which the calling process can then read back in.
- So our new logic is to check the project that the tool was started with (in case the user provided a project path in the cmdline) against each project that we need to process. If the paths match then we can just process the project, if they do not match we launch a child process with the correct path and retrieve the output once done.
-- If a child process detects an incorrect path we just error out to avoid endless recursive process launching.
- Output from the child process is clearly logged to the user as coming from a child process to make debugging easier.
### FCommand
- Added ToJson/FromJson methods to FCommand, which simulates the same interface as though it was using the json serialization macros, however we want more control over things so do the serialization manually via FCommand::Serialize
-- Serialization is only used if the command is sent to a child process (if the current project is wrong), in which case the command should serialize out all info that it derived during setup that it will need to process correctly. Usually this consists of the options that were parsed from the command line. Then when the child process is run we will recreate the command via the same serialize method.
- Added a new class FProcessPipes, which acts as a wrapper around the pipes we can create by calling FPlatformProcess::CreatePipe which are returned as raw pointers. This ensures that the pipes are cleaned up once the FProcessPipes object goes out of scope.
- The ::Run method has been split into two, ::ProcessProject and ::ProcessOutput
-- ::ProcessProject is called once per project and allows the command to return output via the Output parameter. This can be call in process or via a child process if the project needs to be set.
-- ::ProcessOutput is called once all projects have been processed and provides an array which is a collection of the output from each call to ::ProcessProject. This is always called in process.
-- The output is passed around by the base type FCommandOutput. It is expected that commands derive their own type and as we do not mix command types, they should know how to cast the output back to the correct type when ::ProcessOutput is called. This is a bit messy but to go further risks over engineering given it is unlikely that more commands will be added to the tool. If we do expand the tool in the future then this can be revisited.
### FCommandOutput
- A struct used to represent the output of a command.
- If the command is run as a child process then this struct will be serialized out to disk (via json) then loaded back in by the calling process so that the output of each child process can be combined.
- Commands are expected to derive their own output structure and deal with the json serialization. To make this easier we provide a JSON_SERIALIZE_PARENT macro to allow for serialization of inheritance chains.
- This system does require that commands know which type of output to cast FCommandOutput to. At the moment we do not provide any type safety.
- Add json serialization to FProject and it's child struct FPlugin.
- FProject::GetProjectFilePath now returns as a reference to a FString rather than a FStringView.
[CL 24257111 by paul chipchase in ue5-main branch]
2023-02-16 09:18:15 -05:00
|
|
|
TArray<FString> BuildFinalTagDescriptions(const TArray<TUniquePtr<FCommandOutput>>& OutputArray)
|
2022-11-22 07:23:08 -05:00
|
|
|
{
|
|
|
|
|
TArray<FString> CleanedDescriptions;
|
[UVT] Improve the robustness of the tool wrt ensuring that the correct project config settings are loaded.
#rb Per.Larsson
#jira UE-151641
#preflight 63ea068cde74ffbae5dc3a7f
### Problem
- A lot of systems rely on the engine to know which project is being loaded on start up so that the correct config settings can be applied however so far the UVT has not required the user to provide the desired project. This is because in theory, the tool could be given a list of packages or a changelist to process that contain packages from more than one project and if that happens we need to handle it correctly.
- The tool currently groups the provided packages by project, then runs the requested command on each project one at a time, initializing the VA system with that projects config files, then shutting it done once the project packages are processed. This allows us to pick up the correct settings for VA but other systems that we rely on, such as the DDC do not allow this sort of dynamic initialization and so may not have the correct settings.
### Solution
- Before the projects packages are processed we check to see if the project applied to the process during initialization (if any) matches the project to be processed. If not we launch a new version of the tool with the correct project settings and have the child process perform the main processing. The output of each child process (or any projects we were able to run in process) are then provided to the command for a final "post projects" phase.
- This means we can support multiple packages from multiple processes AND not require the user type out the project name when using the tool from the cmdline.
- Although launching a child process only adds a few seconds of overhead (per project) any tool or process that make use of UVT should be updated to provide the project where possible.
### Changes
### FUnrealVirtualizationToolApp
- As explained in the problem section, we now need to support launching a new instance of the tool as a child process.
- We communicate with the child process by providing an input file via the cmdline, which is then parsed and used to recreate the original command and project settings. Once the command has run in the child process it will write its output to an output file which the calling process can then read back in.
- So our new logic is to check the project that the tool was started with (in case the user provided a project path in the cmdline) against each project that we need to process. If the paths match then we can just process the project, if they do not match we launch a child process with the correct path and retrieve the output once done.
-- If a child process detects an incorrect path we just error out to avoid endless recursive process launching.
- Output from the child process is clearly logged to the user as coming from a child process to make debugging easier.
### FCommand
- Added ToJson/FromJson methods to FCommand, which simulates the same interface as though it was using the json serialization macros, however we want more control over things so do the serialization manually via FCommand::Serialize
-- Serialization is only used if the command is sent to a child process (if the current project is wrong), in which case the command should serialize out all info that it derived during setup that it will need to process correctly. Usually this consists of the options that were parsed from the command line. Then when the child process is run we will recreate the command via the same serialize method.
- Added a new class FProcessPipes, which acts as a wrapper around the pipes we can create by calling FPlatformProcess::CreatePipe which are returned as raw pointers. This ensures that the pipes are cleaned up once the FProcessPipes object goes out of scope.
- The ::Run method has been split into two, ::ProcessProject and ::ProcessOutput
-- ::ProcessProject is called once per project and allows the command to return output via the Output parameter. This can be call in process or via a child process if the project needs to be set.
-- ::ProcessOutput is called once all projects have been processed and provides an array which is a collection of the output from each call to ::ProcessProject. This is always called in process.
-- The output is passed around by the base type FCommandOutput. It is expected that commands derive their own type and as we do not mix command types, they should know how to cast the output back to the correct type when ::ProcessOutput is called. This is a bit messy but to go further risks over engineering given it is unlikely that more commands will be added to the tool. If we do expand the tool in the future then this can be revisited.
### FCommandOutput
- A struct used to represent the output of a command.
- If the command is run as a child process then this struct will be serialized out to disk (via json) then loaded back in by the calling process so that the output of each child process can be combined.
- Commands are expected to derive their own output structure and deal with the json serialization. To make this easier we provide a JSON_SERIALIZE_PARENT macro to allow for serialization of inheritance chains.
- This system does require that commands know which type of output to cast FCommandOutput to. At the moment we do not provide any type safety.
- Add json serialization to FProject and it's child struct FPlugin.
- FProject::GetProjectFilePath now returns as a reference to a FString rather than a FStringView.
[CL 24257111 by paul chipchase in ue5-main branch]
2023-02-16 09:18:15 -05:00
|
|
|
|
|
|
|
|
for (const TUniquePtr<FCommandOutput>& Output : OutputArray)
|
2022-11-22 07:23:08 -05:00
|
|
|
{
|
[UVT] Improve the robustness of the tool wrt ensuring that the correct project config settings are loaded.
#rb Per.Larsson
#jira UE-151641
#preflight 63ea068cde74ffbae5dc3a7f
### Problem
- A lot of systems rely on the engine to know which project is being loaded on start up so that the correct config settings can be applied however so far the UVT has not required the user to provide the desired project. This is because in theory, the tool could be given a list of packages or a changelist to process that contain packages from more than one project and if that happens we need to handle it correctly.
- The tool currently groups the provided packages by project, then runs the requested command on each project one at a time, initializing the VA system with that projects config files, then shutting it done once the project packages are processed. This allows us to pick up the correct settings for VA but other systems that we rely on, such as the DDC do not allow this sort of dynamic initialization and so may not have the correct settings.
### Solution
- Before the projects packages are processed we check to see if the project applied to the process during initialization (if any) matches the project to be processed. If not we launch a new version of the tool with the correct project settings and have the child process perform the main processing. The output of each child process (or any projects we were able to run in process) are then provided to the command for a final "post projects" phase.
- This means we can support multiple packages from multiple processes AND not require the user type out the project name when using the tool from the cmdline.
- Although launching a child process only adds a few seconds of overhead (per project) any tool or process that make use of UVT should be updated to provide the project where possible.
### Changes
### FUnrealVirtualizationToolApp
- As explained in the problem section, we now need to support launching a new instance of the tool as a child process.
- We communicate with the child process by providing an input file via the cmdline, which is then parsed and used to recreate the original command and project settings. Once the command has run in the child process it will write its output to an output file which the calling process can then read back in.
- So our new logic is to check the project that the tool was started with (in case the user provided a project path in the cmdline) against each project that we need to process. If the paths match then we can just process the project, if they do not match we launch a child process with the correct path and retrieve the output once done.
-- If a child process detects an incorrect path we just error out to avoid endless recursive process launching.
- Output from the child process is clearly logged to the user as coming from a child process to make debugging easier.
### FCommand
- Added ToJson/FromJson methods to FCommand, which simulates the same interface as though it was using the json serialization macros, however we want more control over things so do the serialization manually via FCommand::Serialize
-- Serialization is only used if the command is sent to a child process (if the current project is wrong), in which case the command should serialize out all info that it derived during setup that it will need to process correctly. Usually this consists of the options that were parsed from the command line. Then when the child process is run we will recreate the command via the same serialize method.
- Added a new class FProcessPipes, which acts as a wrapper around the pipes we can create by calling FPlatformProcess::CreatePipe which are returned as raw pointers. This ensures that the pipes are cleaned up once the FProcessPipes object goes out of scope.
- The ::Run method has been split into two, ::ProcessProject and ::ProcessOutput
-- ::ProcessProject is called once per project and allows the command to return output via the Output parameter. This can be call in process or via a child process if the project needs to be set.
-- ::ProcessOutput is called once all projects have been processed and provides an array which is a collection of the output from each call to ::ProcessProject. This is always called in process.
-- The output is passed around by the base type FCommandOutput. It is expected that commands derive their own type and as we do not mix command types, they should know how to cast the output back to the correct type when ::ProcessOutput is called. This is a bit messy but to go further risks over engineering given it is unlikely that more commands will be added to the tool. If we do expand the tool in the future then this can be revisited.
### FCommandOutput
- A struct used to represent the output of a command.
- If the command is run as a child process then this struct will be serialized out to disk (via json) then loaded back in by the calling process so that the output of each child process can be combined.
- Commands are expected to derive their own output structure and deal with the json serialization. To make this easier we provide a JSON_SERIALIZE_PARENT macro to allow for serialization of inheritance chains.
- This system does require that commands know which type of output to cast FCommandOutput to. At the moment we do not provide any type safety.
- Add json serialization to FProject and it's child struct FPlugin.
- FProject::GetProjectFilePath now returns as a reference to a FString rather than a FStringView.
[CL 24257111 by paul chipchase in ue5-main branch]
2023-02-16 09:18:15 -05:00
|
|
|
if (Output)
|
|
|
|
|
{
|
|
|
|
|
const FVirtualizeCommandOutput* CmdOutput = (const FVirtualizeCommandOutput*)Output.Get();
|
|
|
|
|
|
|
|
|
|
for (const FString& Tag : CmdOutput->DescriptionTags)
|
|
|
|
|
{
|
|
|
|
|
CleanedDescriptions.AddUnique(Tag);
|
|
|
|
|
}
|
|
|
|
|
}
|
2022-11-22 07:23:08 -05:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
return CleanedDescriptions;
|
|
|
|
|
}
|
|
|
|
|
|
[UVT] Improve the robustness of the tool wrt ensuring that the correct project config settings are loaded.
#rb Per.Larsson
#jira UE-151641
#preflight 63ea068cde74ffbae5dc3a7f
### Problem
- A lot of systems rely on the engine to know which project is being loaded on start up so that the correct config settings can be applied however so far the UVT has not required the user to provide the desired project. This is because in theory, the tool could be given a list of packages or a changelist to process that contain packages from more than one project and if that happens we need to handle it correctly.
- The tool currently groups the provided packages by project, then runs the requested command on each project one at a time, initializing the VA system with that projects config files, then shutting it done once the project packages are processed. This allows us to pick up the correct settings for VA but other systems that we rely on, such as the DDC do not allow this sort of dynamic initialization and so may not have the correct settings.
### Solution
- Before the projects packages are processed we check to see if the project applied to the process during initialization (if any) matches the project to be processed. If not we launch a new version of the tool with the correct project settings and have the child process perform the main processing. The output of each child process (or any projects we were able to run in process) are then provided to the command for a final "post projects" phase.
- This means we can support multiple packages from multiple processes AND not require the user type out the project name when using the tool from the cmdline.
- Although launching a child process only adds a few seconds of overhead (per project) any tool or process that make use of UVT should be updated to provide the project where possible.
### Changes
### FUnrealVirtualizationToolApp
- As explained in the problem section, we now need to support launching a new instance of the tool as a child process.
- We communicate with the child process by providing an input file via the cmdline, which is then parsed and used to recreate the original command and project settings. Once the command has run in the child process it will write its output to an output file which the calling process can then read back in.
- So our new logic is to check the project that the tool was started with (in case the user provided a project path in the cmdline) against each project that we need to process. If the paths match then we can just process the project, if they do not match we launch a child process with the correct path and retrieve the output once done.
-- If a child process detects an incorrect path we just error out to avoid endless recursive process launching.
- Output from the child process is clearly logged to the user as coming from a child process to make debugging easier.
### FCommand
- Added ToJson/FromJson methods to FCommand, which simulates the same interface as though it was using the json serialization macros, however we want more control over things so do the serialization manually via FCommand::Serialize
-- Serialization is only used if the command is sent to a child process (if the current project is wrong), in which case the command should serialize out all info that it derived during setup that it will need to process correctly. Usually this consists of the options that were parsed from the command line. Then when the child process is run we will recreate the command via the same serialize method.
- Added a new class FProcessPipes, which acts as a wrapper around the pipes we can create by calling FPlatformProcess::CreatePipe which are returned as raw pointers. This ensures that the pipes are cleaned up once the FProcessPipes object goes out of scope.
- The ::Run method has been split into two, ::ProcessProject and ::ProcessOutput
-- ::ProcessProject is called once per project and allows the command to return output via the Output parameter. This can be call in process or via a child process if the project needs to be set.
-- ::ProcessOutput is called once all projects have been processed and provides an array which is a collection of the output from each call to ::ProcessProject. This is always called in process.
-- The output is passed around by the base type FCommandOutput. It is expected that commands derive their own type and as we do not mix command types, they should know how to cast the output back to the correct type when ::ProcessOutput is called. This is a bit messy but to go further risks over engineering given it is unlikely that more commands will be added to the tool. If we do expand the tool in the future then this can be revisited.
### FCommandOutput
- A struct used to represent the output of a command.
- If the command is run as a child process then this struct will be serialized out to disk (via json) then loaded back in by the calling process so that the output of each child process can be combined.
- Commands are expected to derive their own output structure and deal with the json serialization. To make this easier we provide a JSON_SERIALIZE_PARENT macro to allow for serialization of inheritance chains.
- This system does require that commands know which type of output to cast FCommandOutput to. At the moment we do not provide any type safety.
- Add json serialization to FProject and it's child struct FPlugin.
- FProject::GetProjectFilePath now returns as a reference to a FString rather than a FStringView.
[CL 24257111 by paul chipchase in ue5-main branch]
2023-02-16 09:18:15 -05:00
|
|
|
FVirtualizeCommandOutput::FVirtualizeCommandOutput(FStringView InProjectName, const TArray<FText>& InDescriptionTags)
|
|
|
|
|
: FCommandOutput(InProjectName)
|
2022-11-22 07:23:08 -05:00
|
|
|
{
|
[UVT] Improve the robustness of the tool wrt ensuring that the correct project config settings are loaded.
#rb Per.Larsson
#jira UE-151641
#preflight 63ea068cde74ffbae5dc3a7f
### Problem
- A lot of systems rely on the engine to know which project is being loaded on start up so that the correct config settings can be applied however so far the UVT has not required the user to provide the desired project. This is because in theory, the tool could be given a list of packages or a changelist to process that contain packages from more than one project and if that happens we need to handle it correctly.
- The tool currently groups the provided packages by project, then runs the requested command on each project one at a time, initializing the VA system with that projects config files, then shutting it done once the project packages are processed. This allows us to pick up the correct settings for VA but other systems that we rely on, such as the DDC do not allow this sort of dynamic initialization and so may not have the correct settings.
### Solution
- Before the projects packages are processed we check to see if the project applied to the process during initialization (if any) matches the project to be processed. If not we launch a new version of the tool with the correct project settings and have the child process perform the main processing. The output of each child process (or any projects we were able to run in process) are then provided to the command for a final "post projects" phase.
- This means we can support multiple packages from multiple processes AND not require the user type out the project name when using the tool from the cmdline.
- Although launching a child process only adds a few seconds of overhead (per project) any tool or process that make use of UVT should be updated to provide the project where possible.
### Changes
### FUnrealVirtualizationToolApp
- As explained in the problem section, we now need to support launching a new instance of the tool as a child process.
- We communicate with the child process by providing an input file via the cmdline, which is then parsed and used to recreate the original command and project settings. Once the command has run in the child process it will write its output to an output file which the calling process can then read back in.
- So our new logic is to check the project that the tool was started with (in case the user provided a project path in the cmdline) against each project that we need to process. If the paths match then we can just process the project, if they do not match we launch a child process with the correct path and retrieve the output once done.
-- If a child process detects an incorrect path we just error out to avoid endless recursive process launching.
- Output from the child process is clearly logged to the user as coming from a child process to make debugging easier.
### FCommand
- Added ToJson/FromJson methods to FCommand, which simulates the same interface as though it was using the json serialization macros, however we want more control over things so do the serialization manually via FCommand::Serialize
-- Serialization is only used if the command is sent to a child process (if the current project is wrong), in which case the command should serialize out all info that it derived during setup that it will need to process correctly. Usually this consists of the options that were parsed from the command line. Then when the child process is run we will recreate the command via the same serialize method.
- Added a new class FProcessPipes, which acts as a wrapper around the pipes we can create by calling FPlatformProcess::CreatePipe which are returned as raw pointers. This ensures that the pipes are cleaned up once the FProcessPipes object goes out of scope.
- The ::Run method has been split into two, ::ProcessProject and ::ProcessOutput
-- ::ProcessProject is called once per project and allows the command to return output via the Output parameter. This can be call in process or via a child process if the project needs to be set.
-- ::ProcessOutput is called once all projects have been processed and provides an array which is a collection of the output from each call to ::ProcessProject. This is always called in process.
-- The output is passed around by the base type FCommandOutput. It is expected that commands derive their own type and as we do not mix command types, they should know how to cast the output back to the correct type when ::ProcessOutput is called. This is a bit messy but to go further risks over engineering given it is unlikely that more commands will be added to the tool. If we do expand the tool in the future then this can be revisited.
### FCommandOutput
- A struct used to represent the output of a command.
- If the command is run as a child process then this struct will be serialized out to disk (via json) then loaded back in by the calling process so that the output of each child process can be combined.
- Commands are expected to derive their own output structure and deal with the json serialization. To make this easier we provide a JSON_SERIALIZE_PARENT macro to allow for serialization of inheritance chains.
- This system does require that commands know which type of output to cast FCommandOutput to. At the moment we do not provide any type safety.
- Add json serialization to FProject and it's child struct FPlugin.
- FProject::GetProjectFilePath now returns as a reference to a FString rather than a FStringView.
[CL 24257111 by paul chipchase in ue5-main branch]
2023-02-16 09:18:15 -05:00
|
|
|
DescriptionTags.Reserve(InDescriptionTags.Num());
|
|
|
|
|
|
|
|
|
|
for (const FText& Description : InDescriptionTags)
|
|
|
|
|
{
|
|
|
|
|
DescriptionTags.Add(Description.ToString());
|
|
|
|
|
}
|
|
|
|
|
}
|
2022-11-22 07:23:08 -05:00
|
|
|
|
|
|
|
|
FVirtualizeCommand::FVirtualizeCommand(FStringView CommandName)
|
|
|
|
|
: FCommand(CommandName)
|
|
|
|
|
{
|
|
|
|
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
void FVirtualizeCommand::PrintCmdLineHelp()
|
|
|
|
|
{
|
2023-02-16 10:21:04 -05:00
|
|
|
UE_LOG(LogVirtualizationTool, Display, TEXT("<ProjectFilePath> -Mode=Virtualize -Changelist=<number> -Submit [optional]"));
|
|
|
|
|
UE_LOG(LogVirtualizationTool, Display, TEXT("<ProjectFilePath> -Mode=Virtualize -Path=<string>"));
|
2022-11-22 07:23:08 -05:00
|
|
|
UE_LOG(LogVirtualizationTool, Display, TEXT(""));
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
bool FVirtualizeCommand::Initialize(const TCHAR* CmdLine)
|
|
|
|
|
{
|
|
|
|
|
TRACE_CPUPROFILER_EVENT_SCOPE(FVirtualizeCommand::Initialize);
|
|
|
|
|
|
|
|
|
|
TArray<FString> Tokens;
|
|
|
|
|
TArray<FString> Switches;
|
|
|
|
|
|
|
|
|
|
ParseCommandLine(CmdLine, Tokens, Switches);
|
|
|
|
|
|
|
|
|
|
FString SwitchValue;
|
|
|
|
|
for (const FString& Switch : Switches)
|
|
|
|
|
{
|
|
|
|
|
EPathResult Result = ParseSwitchForPaths(Switch, AllPackages);
|
|
|
|
|
if (Result == EPathResult::Error)
|
|
|
|
|
{
|
|
|
|
|
return false;
|
|
|
|
|
}
|
|
|
|
|
else if (Result == EPathResult::Success)
|
|
|
|
|
{
|
|
|
|
|
continue; // If we already matched the switch we don't need to check against any others
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if (FParse::Value(*Switch, TEXT("Changelist="), SwitchValue))
|
|
|
|
|
{
|
|
|
|
|
SourceChangelistNumber = SwitchValue;
|
|
|
|
|
}
|
|
|
|
|
else if (Switch == TEXT("Submit"))
|
|
|
|
|
{
|
|
|
|
|
bShouldSubmitChangelist = true;
|
|
|
|
|
}
|
2022-12-16 06:25:07 -05:00
|
|
|
else if (Switch == TEXT("Checkout"))
|
|
|
|
|
{
|
|
|
|
|
bShouldCheckout = true;
|
|
|
|
|
}
|
2022-11-22 07:23:08 -05:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Process the provided changelist if one was found
|
|
|
|
|
if (!SourceChangelistNumber.IsEmpty() )
|
|
|
|
|
{
|
|
|
|
|
// If no client spec was provided we need to find it for the changelist
|
|
|
|
|
// In theory this duplicates a lot of the work found in ::TryParseChangelist
|
|
|
|
|
// but at the moment the FGetChangelistDetails operation is not compatible
|
|
|
|
|
// with the FSourceControlChangelistStateRef/FSourceControlStateRef API
|
|
|
|
|
// so we are stuck with duplication of work.
|
|
|
|
|
if (ClientSpecName.IsEmpty())
|
|
|
|
|
{
|
|
|
|
|
ClientSpecName = FindClientSpecForChangelist(SourceChangelistNumber);
|
|
|
|
|
if (!ClientSpecName.IsEmpty())
|
|
|
|
|
{
|
|
|
|
|
FSourceControlResultInfo Info;
|
|
|
|
|
if (SCCProvider->SwitchWorkspace(ClientSpecName, Info, nullptr) != ECommandResult::Succeeded)
|
|
|
|
|
{
|
|
|
|
|
UE_LOG(LogVirtualization, Error, TEXT("Failed to switch to workspace '%s'"), *ClientSpecName);
|
|
|
|
|
return false;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
else
|
|
|
|
|
{
|
|
|
|
|
UE_LOG(LogVirtualization, Error, TEXT("Count not find a valid workspace for the changelist '%s'"), *SourceChangelistNumber);
|
|
|
|
|
return false;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if (!TryParseChangelist(ClientSpecName, SourceChangelistNumber, AllPackages, &SourceChangelist))
|
|
|
|
|
{
|
|
|
|
|
UE_LOG(LogVirtualization, Error, TEXT("Failed to find the files in the changelist '%s'"), *SourceChangelistNumber);
|
|
|
|
|
return false;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
return true;
|
|
|
|
|
}
|
|
|
|
|
|
[UVT] Improve the robustness of the tool wrt ensuring that the correct project config settings are loaded.
#rb Per.Larsson
#jira UE-151641
#preflight 63ea068cde74ffbae5dc3a7f
### Problem
- A lot of systems rely on the engine to know which project is being loaded on start up so that the correct config settings can be applied however so far the UVT has not required the user to provide the desired project. This is because in theory, the tool could be given a list of packages or a changelist to process that contain packages from more than one project and if that happens we need to handle it correctly.
- The tool currently groups the provided packages by project, then runs the requested command on each project one at a time, initializing the VA system with that projects config files, then shutting it done once the project packages are processed. This allows us to pick up the correct settings for VA but other systems that we rely on, such as the DDC do not allow this sort of dynamic initialization and so may not have the correct settings.
### Solution
- Before the projects packages are processed we check to see if the project applied to the process during initialization (if any) matches the project to be processed. If not we launch a new version of the tool with the correct project settings and have the child process perform the main processing. The output of each child process (or any projects we were able to run in process) are then provided to the command for a final "post projects" phase.
- This means we can support multiple packages from multiple processes AND not require the user type out the project name when using the tool from the cmdline.
- Although launching a child process only adds a few seconds of overhead (per project) any tool or process that make use of UVT should be updated to provide the project where possible.
### Changes
### FUnrealVirtualizationToolApp
- As explained in the problem section, we now need to support launching a new instance of the tool as a child process.
- We communicate with the child process by providing an input file via the cmdline, which is then parsed and used to recreate the original command and project settings. Once the command has run in the child process it will write its output to an output file which the calling process can then read back in.
- So our new logic is to check the project that the tool was started with (in case the user provided a project path in the cmdline) against each project that we need to process. If the paths match then we can just process the project, if they do not match we launch a child process with the correct path and retrieve the output once done.
-- If a child process detects an incorrect path we just error out to avoid endless recursive process launching.
- Output from the child process is clearly logged to the user as coming from a child process to make debugging easier.
### FCommand
- Added ToJson/FromJson methods to FCommand, which simulates the same interface as though it was using the json serialization macros, however we want more control over things so do the serialization manually via FCommand::Serialize
-- Serialization is only used if the command is sent to a child process (if the current project is wrong), in which case the command should serialize out all info that it derived during setup that it will need to process correctly. Usually this consists of the options that were parsed from the command line. Then when the child process is run we will recreate the command via the same serialize method.
- Added a new class FProcessPipes, which acts as a wrapper around the pipes we can create by calling FPlatformProcess::CreatePipe which are returned as raw pointers. This ensures that the pipes are cleaned up once the FProcessPipes object goes out of scope.
- The ::Run method has been split into two, ::ProcessProject and ::ProcessOutput
-- ::ProcessProject is called once per project and allows the command to return output via the Output parameter. This can be call in process or via a child process if the project needs to be set.
-- ::ProcessOutput is called once all projects have been processed and provides an array which is a collection of the output from each call to ::ProcessProject. This is always called in process.
-- The output is passed around by the base type FCommandOutput. It is expected that commands derive their own type and as we do not mix command types, they should know how to cast the output back to the correct type when ::ProcessOutput is called. This is a bit messy but to go further risks over engineering given it is unlikely that more commands will be added to the tool. If we do expand the tool in the future then this can be revisited.
### FCommandOutput
- A struct used to represent the output of a command.
- If the command is run as a child process then this struct will be serialized out to disk (via json) then loaded back in by the calling process so that the output of each child process can be combined.
- Commands are expected to derive their own output structure and deal with the json serialization. To make this easier we provide a JSON_SERIALIZE_PARENT macro to allow for serialization of inheritance chains.
- This system does require that commands know which type of output to cast FCommandOutput to. At the moment we do not provide any type safety.
- Add json serialization to FProject and it's child struct FPlugin.
- FProject::GetProjectFilePath now returns as a reference to a FString rather than a FStringView.
[CL 24257111 by paul chipchase in ue5-main branch]
2023-02-16 09:18:15 -05:00
|
|
|
void FVirtualizeCommand::Serialize(FJsonSerializerBase& Serializer)
|
2022-11-22 07:23:08 -05:00
|
|
|
{
|
[UVT] Improve the robustness of the tool wrt ensuring that the correct project config settings are loaded.
#rb Per.Larsson
#jira UE-151641
#preflight 63ea068cde74ffbae5dc3a7f
### Problem
- A lot of systems rely on the engine to know which project is being loaded on start up so that the correct config settings can be applied however so far the UVT has not required the user to provide the desired project. This is because in theory, the tool could be given a list of packages or a changelist to process that contain packages from more than one project and if that happens we need to handle it correctly.
- The tool currently groups the provided packages by project, then runs the requested command on each project one at a time, initializing the VA system with that projects config files, then shutting it done once the project packages are processed. This allows us to pick up the correct settings for VA but other systems that we rely on, such as the DDC do not allow this sort of dynamic initialization and so may not have the correct settings.
### Solution
- Before the projects packages are processed we check to see if the project applied to the process during initialization (if any) matches the project to be processed. If not we launch a new version of the tool with the correct project settings and have the child process perform the main processing. The output of each child process (or any projects we were able to run in process) are then provided to the command for a final "post projects" phase.
- This means we can support multiple packages from multiple processes AND not require the user type out the project name when using the tool from the cmdline.
- Although launching a child process only adds a few seconds of overhead (per project) any tool or process that make use of UVT should be updated to provide the project where possible.
### Changes
### FUnrealVirtualizationToolApp
- As explained in the problem section, we now need to support launching a new instance of the tool as a child process.
- We communicate with the child process by providing an input file via the cmdline, which is then parsed and used to recreate the original command and project settings. Once the command has run in the child process it will write its output to an output file which the calling process can then read back in.
- So our new logic is to check the project that the tool was started with (in case the user provided a project path in the cmdline) against each project that we need to process. If the paths match then we can just process the project, if they do not match we launch a child process with the correct path and retrieve the output once done.
-- If a child process detects an incorrect path we just error out to avoid endless recursive process launching.
- Output from the child process is clearly logged to the user as coming from a child process to make debugging easier.
### FCommand
- Added ToJson/FromJson methods to FCommand, which simulates the same interface as though it was using the json serialization macros, however we want more control over things so do the serialization manually via FCommand::Serialize
-- Serialization is only used if the command is sent to a child process (if the current project is wrong), in which case the command should serialize out all info that it derived during setup that it will need to process correctly. Usually this consists of the options that were parsed from the command line. Then when the child process is run we will recreate the command via the same serialize method.
- Added a new class FProcessPipes, which acts as a wrapper around the pipes we can create by calling FPlatformProcess::CreatePipe which are returned as raw pointers. This ensures that the pipes are cleaned up once the FProcessPipes object goes out of scope.
- The ::Run method has been split into two, ::ProcessProject and ::ProcessOutput
-- ::ProcessProject is called once per project and allows the command to return output via the Output parameter. This can be call in process or via a child process if the project needs to be set.
-- ::ProcessOutput is called once all projects have been processed and provides an array which is a collection of the output from each call to ::ProcessProject. This is always called in process.
-- The output is passed around by the base type FCommandOutput. It is expected that commands derive their own type and as we do not mix command types, they should know how to cast the output back to the correct type when ::ProcessOutput is called. This is a bit messy but to go further risks over engineering given it is unlikely that more commands will be added to the tool. If we do expand the tool in the future then this can be revisited.
### FCommandOutput
- A struct used to represent the output of a command.
- If the command is run as a child process then this struct will be serialized out to disk (via json) then loaded back in by the calling process so that the output of each child process can be combined.
- Commands are expected to derive their own output structure and deal with the json serialization. To make this easier we provide a JSON_SERIALIZE_PARENT macro to allow for serialization of inheritance chains.
- This system does require that commands know which type of output to cast FCommandOutput to. At the moment we do not provide any type safety.
- Add json serialization to FProject and it's child struct FPlugin.
- FProject::GetProjectFilePath now returns as a reference to a FString rather than a FStringView.
[CL 24257111 by paul chipchase in ue5-main branch]
2023-02-16 09:18:15 -05:00
|
|
|
TRACE_CPUPROFILER_EVENT_SCOPE(FVirtualizeCommand::Serialize);
|
|
|
|
|
|
|
|
|
|
Serializer.Serialize(TEXT("ShouldCheckout"), bShouldCheckout);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
bool FVirtualizeCommand::ProcessProject(const FProject& Project, TUniquePtr<FCommandOutput>& Output)
|
|
|
|
|
{
|
|
|
|
|
TRACE_CPUPROFILER_EVENT_SCOPE(FVirtualizeCommand::ProcessProject);
|
2022-11-22 07:23:08 -05:00
|
|
|
|
|
|
|
|
UE_LOG(LogVirtualizationTool, Display, TEXT("Running the virtualization process..."));
|
|
|
|
|
|
[UVT] Improve the robustness of the tool wrt ensuring that the correct project config settings are loaded.
#rb Per.Larsson
#jira UE-151641
#preflight 63ea068cde74ffbae5dc3a7f
### Problem
- A lot of systems rely on the engine to know which project is being loaded on start up so that the correct config settings can be applied however so far the UVT has not required the user to provide the desired project. This is because in theory, the tool could be given a list of packages or a changelist to process that contain packages from more than one project and if that happens we need to handle it correctly.
- The tool currently groups the provided packages by project, then runs the requested command on each project one at a time, initializing the VA system with that projects config files, then shutting it done once the project packages are processed. This allows us to pick up the correct settings for VA but other systems that we rely on, such as the DDC do not allow this sort of dynamic initialization and so may not have the correct settings.
### Solution
- Before the projects packages are processed we check to see if the project applied to the process during initialization (if any) matches the project to be processed. If not we launch a new version of the tool with the correct project settings and have the child process perform the main processing. The output of each child process (or any projects we were able to run in process) are then provided to the command for a final "post projects" phase.
- This means we can support multiple packages from multiple processes AND not require the user type out the project name when using the tool from the cmdline.
- Although launching a child process only adds a few seconds of overhead (per project) any tool or process that make use of UVT should be updated to provide the project where possible.
### Changes
### FUnrealVirtualizationToolApp
- As explained in the problem section, we now need to support launching a new instance of the tool as a child process.
- We communicate with the child process by providing an input file via the cmdline, which is then parsed and used to recreate the original command and project settings. Once the command has run in the child process it will write its output to an output file which the calling process can then read back in.
- So our new logic is to check the project that the tool was started with (in case the user provided a project path in the cmdline) against each project that we need to process. If the paths match then we can just process the project, if they do not match we launch a child process with the correct path and retrieve the output once done.
-- If a child process detects an incorrect path we just error out to avoid endless recursive process launching.
- Output from the child process is clearly logged to the user as coming from a child process to make debugging easier.
### FCommand
- Added ToJson/FromJson methods to FCommand, which simulates the same interface as though it was using the json serialization macros, however we want more control over things so do the serialization manually via FCommand::Serialize
-- Serialization is only used if the command is sent to a child process (if the current project is wrong), in which case the command should serialize out all info that it derived during setup that it will need to process correctly. Usually this consists of the options that were parsed from the command line. Then when the child process is run we will recreate the command via the same serialize method.
- Added a new class FProcessPipes, which acts as a wrapper around the pipes we can create by calling FPlatformProcess::CreatePipe which are returned as raw pointers. This ensures that the pipes are cleaned up once the FProcessPipes object goes out of scope.
- The ::Run method has been split into two, ::ProcessProject and ::ProcessOutput
-- ::ProcessProject is called once per project and allows the command to return output via the Output parameter. This can be call in process or via a child process if the project needs to be set.
-- ::ProcessOutput is called once all projects have been processed and provides an array which is a collection of the output from each call to ::ProcessProject. This is always called in process.
-- The output is passed around by the base type FCommandOutput. It is expected that commands derive their own type and as we do not mix command types, they should know how to cast the output back to the correct type when ::ProcessOutput is called. This is a bit messy but to go further risks over engineering given it is unlikely that more commands will be added to the tool. If we do expand the tool in the future then this can be revisited.
### FCommandOutput
- A struct used to represent the output of a command.
- If the command is run as a child process then this struct will be serialized out to disk (via json) then loaded back in by the calling process so that the output of each child process can be combined.
- Commands are expected to derive their own output structure and deal with the json serialization. To make this easier we provide a JSON_SERIALIZE_PARENT macro to allow for serialization of inheritance chains.
- This system does require that commands know which type of output to cast FCommandOutput to. At the moment we do not provide any type safety.
- Add json serialization to FProject and it's child struct FPlugin.
- FProject::GetProjectFilePath now returns as a reference to a FString rather than a FStringView.
[CL 24257111 by paul chipchase in ue5-main branch]
2023-02-16 09:18:15 -05:00
|
|
|
TStringBuilder<128> ProjectName;
|
|
|
|
|
ProjectName << Project.GetProjectName();
|
|
|
|
|
|
|
|
|
|
UE_LOG(LogVirtualizationTool, Display, TEXT("\tChecking package(s) for the project '%s'..."), ProjectName.ToString());
|
|
|
|
|
|
|
|
|
|
FConfigFile EngineConfigWithProject;
|
|
|
|
|
if (!Project.TryLoadConfig(EngineConfigWithProject))
|
2022-11-22 07:23:08 -05:00
|
|
|
{
|
[UVT] Improve the robustness of the tool wrt ensuring that the correct project config settings are loaded.
#rb Per.Larsson
#jira UE-151641
#preflight 63ea068cde74ffbae5dc3a7f
### Problem
- A lot of systems rely on the engine to know which project is being loaded on start up so that the correct config settings can be applied however so far the UVT has not required the user to provide the desired project. This is because in theory, the tool could be given a list of packages or a changelist to process that contain packages from more than one project and if that happens we need to handle it correctly.
- The tool currently groups the provided packages by project, then runs the requested command on each project one at a time, initializing the VA system with that projects config files, then shutting it done once the project packages are processed. This allows us to pick up the correct settings for VA but other systems that we rely on, such as the DDC do not allow this sort of dynamic initialization and so may not have the correct settings.
### Solution
- Before the projects packages are processed we check to see if the project applied to the process during initialization (if any) matches the project to be processed. If not we launch a new version of the tool with the correct project settings and have the child process perform the main processing. The output of each child process (or any projects we were able to run in process) are then provided to the command for a final "post projects" phase.
- This means we can support multiple packages from multiple processes AND not require the user type out the project name when using the tool from the cmdline.
- Although launching a child process only adds a few seconds of overhead (per project) any tool or process that make use of UVT should be updated to provide the project where possible.
### Changes
### FUnrealVirtualizationToolApp
- As explained in the problem section, we now need to support launching a new instance of the tool as a child process.
- We communicate with the child process by providing an input file via the cmdline, which is then parsed and used to recreate the original command and project settings. Once the command has run in the child process it will write its output to an output file which the calling process can then read back in.
- So our new logic is to check the project that the tool was started with (in case the user provided a project path in the cmdline) against each project that we need to process. If the paths match then we can just process the project, if they do not match we launch a child process with the correct path and retrieve the output once done.
-- If a child process detects an incorrect path we just error out to avoid endless recursive process launching.
- Output from the child process is clearly logged to the user as coming from a child process to make debugging easier.
### FCommand
- Added ToJson/FromJson methods to FCommand, which simulates the same interface as though it was using the json serialization macros, however we want more control over things so do the serialization manually via FCommand::Serialize
-- Serialization is only used if the command is sent to a child process (if the current project is wrong), in which case the command should serialize out all info that it derived during setup that it will need to process correctly. Usually this consists of the options that were parsed from the command line. Then when the child process is run we will recreate the command via the same serialize method.
- Added a new class FProcessPipes, which acts as a wrapper around the pipes we can create by calling FPlatformProcess::CreatePipe which are returned as raw pointers. This ensures that the pipes are cleaned up once the FProcessPipes object goes out of scope.
- The ::Run method has been split into two, ::ProcessProject and ::ProcessOutput
-- ::ProcessProject is called once per project and allows the command to return output via the Output parameter. This can be call in process or via a child process if the project needs to be set.
-- ::ProcessOutput is called once all projects have been processed and provides an array which is a collection of the output from each call to ::ProcessProject. This is always called in process.
-- The output is passed around by the base type FCommandOutput. It is expected that commands derive their own type and as we do not mix command types, they should know how to cast the output back to the correct type when ::ProcessOutput is called. This is a bit messy but to go further risks over engineering given it is unlikely that more commands will be added to the tool. If we do expand the tool in the future then this can be revisited.
### FCommandOutput
- A struct used to represent the output of a command.
- If the command is run as a child process then this struct will be serialized out to disk (via json) then loaded back in by the calling process so that the output of each child process can be combined.
- Commands are expected to derive their own output structure and deal with the json serialization. To make this easier we provide a JSON_SERIALIZE_PARENT macro to allow for serialization of inheritance chains.
- This system does require that commands know which type of output to cast FCommandOutput to. At the moment we do not provide any type safety.
- Add json serialization to FProject and it's child struct FPlugin.
- FProject::GetProjectFilePath now returns as a reference to a FString rather than a FStringView.
[CL 24257111 by paul chipchase in ue5-main branch]
2023-02-16 09:18:15 -05:00
|
|
|
return false;
|
2022-11-22 07:23:08 -05:00
|
|
|
}
|
|
|
|
|
|
[UVT] Improve the robustness of the tool wrt ensuring that the correct project config settings are loaded.
#rb Per.Larsson
#jira UE-151641
#preflight 63ea068cde74ffbae5dc3a7f
### Problem
- A lot of systems rely on the engine to know which project is being loaded on start up so that the correct config settings can be applied however so far the UVT has not required the user to provide the desired project. This is because in theory, the tool could be given a list of packages or a changelist to process that contain packages from more than one project and if that happens we need to handle it correctly.
- The tool currently groups the provided packages by project, then runs the requested command on each project one at a time, initializing the VA system with that projects config files, then shutting it done once the project packages are processed. This allows us to pick up the correct settings for VA but other systems that we rely on, such as the DDC do not allow this sort of dynamic initialization and so may not have the correct settings.
### Solution
- Before the projects packages are processed we check to see if the project applied to the process during initialization (if any) matches the project to be processed. If not we launch a new version of the tool with the correct project settings and have the child process perform the main processing. The output of each child process (or any projects we were able to run in process) are then provided to the command for a final "post projects" phase.
- This means we can support multiple packages from multiple processes AND not require the user type out the project name when using the tool from the cmdline.
- Although launching a child process only adds a few seconds of overhead (per project) any tool or process that make use of UVT should be updated to provide the project where possible.
### Changes
### FUnrealVirtualizationToolApp
- As explained in the problem section, we now need to support launching a new instance of the tool as a child process.
- We communicate with the child process by providing an input file via the cmdline, which is then parsed and used to recreate the original command and project settings. Once the command has run in the child process it will write its output to an output file which the calling process can then read back in.
- So our new logic is to check the project that the tool was started with (in case the user provided a project path in the cmdline) against each project that we need to process. If the paths match then we can just process the project, if they do not match we launch a child process with the correct path and retrieve the output once done.
-- If a child process detects an incorrect path we just error out to avoid endless recursive process launching.
- Output from the child process is clearly logged to the user as coming from a child process to make debugging easier.
### FCommand
- Added ToJson/FromJson methods to FCommand, which simulates the same interface as though it was using the json serialization macros, however we want more control over things so do the serialization manually via FCommand::Serialize
-- Serialization is only used if the command is sent to a child process (if the current project is wrong), in which case the command should serialize out all info that it derived during setup that it will need to process correctly. Usually this consists of the options that were parsed from the command line. Then when the child process is run we will recreate the command via the same serialize method.
- Added a new class FProcessPipes, which acts as a wrapper around the pipes we can create by calling FPlatformProcess::CreatePipe which are returned as raw pointers. This ensures that the pipes are cleaned up once the FProcessPipes object goes out of scope.
- The ::Run method has been split into two, ::ProcessProject and ::ProcessOutput
-- ::ProcessProject is called once per project and allows the command to return output via the Output parameter. This can be call in process or via a child process if the project needs to be set.
-- ::ProcessOutput is called once all projects have been processed and provides an array which is a collection of the output from each call to ::ProcessProject. This is always called in process.
-- The output is passed around by the base type FCommandOutput. It is expected that commands derive their own type and as we do not mix command types, they should know how to cast the output back to the correct type when ::ProcessOutput is called. This is a bit messy but to go further risks over engineering given it is unlikely that more commands will be added to the tool. If we do expand the tool in the future then this can be revisited.
### FCommandOutput
- A struct used to represent the output of a command.
- If the command is run as a child process then this struct will be serialized out to disk (via json) then loaded back in by the calling process so that the output of each child process can be combined.
- Commands are expected to derive their own output structure and deal with the json serialization. To make this easier we provide a JSON_SERIALIZE_PARENT macro to allow for serialization of inheritance chains.
- This system does require that commands know which type of output to cast FCommandOutput to. At the moment we do not provide any type safety.
- Add json serialization to FProject and it's child struct FPlugin.
- FProject::GetProjectFilePath now returns as a reference to a FString rather than a FStringView.
[CL 24257111 by paul chipchase in ue5-main branch]
2023-02-16 09:18:15 -05:00
|
|
|
Project.RegisterMountPoints();
|
|
|
|
|
|
|
|
|
|
ON_SCOPE_EXIT
|
|
|
|
|
{
|
|
|
|
|
Project.UnRegisterMountPoints();
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
UE::Virtualization::FInitParams InitParams(ProjectName, EngineConfigWithProject);
|
|
|
|
|
UE::Virtualization::Initialize(InitParams, UE::Virtualization::EInitializationFlags::ForceInitialize);
|
|
|
|
|
|
|
|
|
|
ON_SCOPE_EXIT
|
|
|
|
|
{
|
|
|
|
|
UE::Virtualization::Shutdown();
|
|
|
|
|
};
|
|
|
|
|
|
2023-10-17 04:02:28 -04:00
|
|
|
if (!UE::Virtualization::IVirtualizationSystem::Get().IsEnabled())
|
|
|
|
|
{
|
|
|
|
|
UE_LOG(LogVirtualizationTool, Display, TEXT("\tVirtualization is not enabled for this project"));
|
|
|
|
|
Output = MakeUnique<FVirtualizeCommandOutput>(Project.GetProjectName(), TArray<FText>());
|
|
|
|
|
return true;
|
|
|
|
|
}
|
|
|
|
|
|
[UVT] Improve the robustness of the tool wrt ensuring that the correct project config settings are loaded.
#rb Per.Larsson
#jira UE-151641
#preflight 63ea068cde74ffbae5dc3a7f
### Problem
- A lot of systems rely on the engine to know which project is being loaded on start up so that the correct config settings can be applied however so far the UVT has not required the user to provide the desired project. This is because in theory, the tool could be given a list of packages or a changelist to process that contain packages from more than one project and if that happens we need to handle it correctly.
- The tool currently groups the provided packages by project, then runs the requested command on each project one at a time, initializing the VA system with that projects config files, then shutting it done once the project packages are processed. This allows us to pick up the correct settings for VA but other systems that we rely on, such as the DDC do not allow this sort of dynamic initialization and so may not have the correct settings.
### Solution
- Before the projects packages are processed we check to see if the project applied to the process during initialization (if any) matches the project to be processed. If not we launch a new version of the tool with the correct project settings and have the child process perform the main processing. The output of each child process (or any projects we were able to run in process) are then provided to the command for a final "post projects" phase.
- This means we can support multiple packages from multiple processes AND not require the user type out the project name when using the tool from the cmdline.
- Although launching a child process only adds a few seconds of overhead (per project) any tool or process that make use of UVT should be updated to provide the project where possible.
### Changes
### FUnrealVirtualizationToolApp
- As explained in the problem section, we now need to support launching a new instance of the tool as a child process.
- We communicate with the child process by providing an input file via the cmdline, which is then parsed and used to recreate the original command and project settings. Once the command has run in the child process it will write its output to an output file which the calling process can then read back in.
- So our new logic is to check the project that the tool was started with (in case the user provided a project path in the cmdline) against each project that we need to process. If the paths match then we can just process the project, if they do not match we launch a child process with the correct path and retrieve the output once done.
-- If a child process detects an incorrect path we just error out to avoid endless recursive process launching.
- Output from the child process is clearly logged to the user as coming from a child process to make debugging easier.
### FCommand
- Added ToJson/FromJson methods to FCommand, which simulates the same interface as though it was using the json serialization macros, however we want more control over things so do the serialization manually via FCommand::Serialize
-- Serialization is only used if the command is sent to a child process (if the current project is wrong), in which case the command should serialize out all info that it derived during setup that it will need to process correctly. Usually this consists of the options that were parsed from the command line. Then when the child process is run we will recreate the command via the same serialize method.
- Added a new class FProcessPipes, which acts as a wrapper around the pipes we can create by calling FPlatformProcess::CreatePipe which are returned as raw pointers. This ensures that the pipes are cleaned up once the FProcessPipes object goes out of scope.
- The ::Run method has been split into two, ::ProcessProject and ::ProcessOutput
-- ::ProcessProject is called once per project and allows the command to return output via the Output parameter. This can be call in process or via a child process if the project needs to be set.
-- ::ProcessOutput is called once all projects have been processed and provides an array which is a collection of the output from each call to ::ProcessProject. This is always called in process.
-- The output is passed around by the base type FCommandOutput. It is expected that commands derive their own type and as we do not mix command types, they should know how to cast the output back to the correct type when ::ProcessOutput is called. This is a bit messy but to go further risks over engineering given it is unlikely that more commands will be added to the tool. If we do expand the tool in the future then this can be revisited.
### FCommandOutput
- A struct used to represent the output of a command.
- If the command is run as a child process then this struct will be serialized out to disk (via json) then loaded back in by the calling process so that the output of each child process can be combined.
- Commands are expected to derive their own output structure and deal with the json serialization. To make this easier we provide a JSON_SERIALIZE_PARENT macro to allow for serialization of inheritance chains.
- This system does require that commands know which type of output to cast FCommandOutput to. At the moment we do not provide any type safety.
- Add json serialization to FProject and it's child struct FPlugin.
- FProject::GetProjectFilePath now returns as a reference to a FString rather than a FStringView.
[CL 24257111 by paul chipchase in ue5-main branch]
2023-02-16 09:18:15 -05:00
|
|
|
TArray<FString> ProjectPackages = Project.GetAllPackages();
|
|
|
|
|
|
|
|
|
|
EVirtualizationOptions Options = EVirtualizationOptions::None;
|
|
|
|
|
if (bShouldCheckout)
|
|
|
|
|
{
|
|
|
|
|
Options |= EVirtualizationOptions::Checkout;
|
2023-03-30 11:05:15 -04:00
|
|
|
|
|
|
|
|
// Make sure that we have a valid source control connection if we might try to checkout packages
|
|
|
|
|
TryConnectToSourceControl();
|
[UVT] Improve the robustness of the tool wrt ensuring that the correct project config settings are loaded.
#rb Per.Larsson
#jira UE-151641
#preflight 63ea068cde74ffbae5dc3a7f
### Problem
- A lot of systems rely on the engine to know which project is being loaded on start up so that the correct config settings can be applied however so far the UVT has not required the user to provide the desired project. This is because in theory, the tool could be given a list of packages or a changelist to process that contain packages from more than one project and if that happens we need to handle it correctly.
- The tool currently groups the provided packages by project, then runs the requested command on each project one at a time, initializing the VA system with that projects config files, then shutting it done once the project packages are processed. This allows us to pick up the correct settings for VA but other systems that we rely on, such as the DDC do not allow this sort of dynamic initialization and so may not have the correct settings.
### Solution
- Before the projects packages are processed we check to see if the project applied to the process during initialization (if any) matches the project to be processed. If not we launch a new version of the tool with the correct project settings and have the child process perform the main processing. The output of each child process (or any projects we were able to run in process) are then provided to the command for a final "post projects" phase.
- This means we can support multiple packages from multiple processes AND not require the user type out the project name when using the tool from the cmdline.
- Although launching a child process only adds a few seconds of overhead (per project) any tool or process that make use of UVT should be updated to provide the project where possible.
### Changes
### FUnrealVirtualizationToolApp
- As explained in the problem section, we now need to support launching a new instance of the tool as a child process.
- We communicate with the child process by providing an input file via the cmdline, which is then parsed and used to recreate the original command and project settings. Once the command has run in the child process it will write its output to an output file which the calling process can then read back in.
- So our new logic is to check the project that the tool was started with (in case the user provided a project path in the cmdline) against each project that we need to process. If the paths match then we can just process the project, if they do not match we launch a child process with the correct path and retrieve the output once done.
-- If a child process detects an incorrect path we just error out to avoid endless recursive process launching.
- Output from the child process is clearly logged to the user as coming from a child process to make debugging easier.
### FCommand
- Added ToJson/FromJson methods to FCommand, which simulates the same interface as though it was using the json serialization macros, however we want more control over things so do the serialization manually via FCommand::Serialize
-- Serialization is only used if the command is sent to a child process (if the current project is wrong), in which case the command should serialize out all info that it derived during setup that it will need to process correctly. Usually this consists of the options that were parsed from the command line. Then when the child process is run we will recreate the command via the same serialize method.
- Added a new class FProcessPipes, which acts as a wrapper around the pipes we can create by calling FPlatformProcess::CreatePipe which are returned as raw pointers. This ensures that the pipes are cleaned up once the FProcessPipes object goes out of scope.
- The ::Run method has been split into two, ::ProcessProject and ::ProcessOutput
-- ::ProcessProject is called once per project and allows the command to return output via the Output parameter. This can be call in process or via a child process if the project needs to be set.
-- ::ProcessOutput is called once all projects have been processed and provides an array which is a collection of the output from each call to ::ProcessProject. This is always called in process.
-- The output is passed around by the base type FCommandOutput. It is expected that commands derive their own type and as we do not mix command types, they should know how to cast the output back to the correct type when ::ProcessOutput is called. This is a bit messy but to go further risks over engineering given it is unlikely that more commands will be added to the tool. If we do expand the tool in the future then this can be revisited.
### FCommandOutput
- A struct used to represent the output of a command.
- If the command is run as a child process then this struct will be serialized out to disk (via json) then loaded back in by the calling process so that the output of each child process can be combined.
- Commands are expected to derive their own output structure and deal with the json serialization. To make this easier we provide a JSON_SERIALIZE_PARENT macro to allow for serialization of inheritance chains.
- This system does require that commands know which type of output to cast FCommandOutput to. At the moment we do not provide any type safety.
- Add json serialization to FProject and it's child struct FPlugin.
- FProject::GetProjectFilePath now returns as a reference to a FString rather than a FStringView.
[CL 24257111 by paul chipchase in ue5-main branch]
2023-02-16 09:18:15 -05:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
FVirtualizationResult Result = UE::Virtualization::IVirtualizationSystem::Get().TryVirtualizePackages(ProjectPackages, Options);
|
|
|
|
|
|
|
|
|
|
if (!Result.WasSuccessful())
|
|
|
|
|
{
|
|
|
|
|
UE_LOG(LogVirtualizationTool, Error, TEXT("The virtualization process failed with the following errors:"));
|
|
|
|
|
for (const FText& Error : Result.Errors)
|
|
|
|
|
{
|
|
|
|
|
UE_LOG(LogVirtualizationTool, Error, TEXT("\t%s"), *Error.ToString());
|
|
|
|
|
}
|
|
|
|
|
return false;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if (bShouldCheckout)
|
|
|
|
|
{
|
|
|
|
|
UE_LOG(LogVirtualizationTool, Display, TEXT("\t\t%d packages were checked out of revision control"), Result.CheckedOutPackages.Num());
|
|
|
|
|
}
|
|
|
|
|
|
2023-05-15 08:59:03 -04:00
|
|
|
UE_LOG(LogVirtualizationTool, Display, TEXT("\t\tTime taken %.2f(s)"), Result.TimeTaken);
|
[UVT] Improve the robustness of the tool wrt ensuring that the correct project config settings are loaded.
#rb Per.Larsson
#jira UE-151641
#preflight 63ea068cde74ffbae5dc3a7f
### Problem
- A lot of systems rely on the engine to know which project is being loaded on start up so that the correct config settings can be applied however so far the UVT has not required the user to provide the desired project. This is because in theory, the tool could be given a list of packages or a changelist to process that contain packages from more than one project and if that happens we need to handle it correctly.
- The tool currently groups the provided packages by project, then runs the requested command on each project one at a time, initializing the VA system with that projects config files, then shutting it done once the project packages are processed. This allows us to pick up the correct settings for VA but other systems that we rely on, such as the DDC do not allow this sort of dynamic initialization and so may not have the correct settings.
### Solution
- Before the projects packages are processed we check to see if the project applied to the process during initialization (if any) matches the project to be processed. If not we launch a new version of the tool with the correct project settings and have the child process perform the main processing. The output of each child process (or any projects we were able to run in process) are then provided to the command for a final "post projects" phase.
- This means we can support multiple packages from multiple processes AND not require the user type out the project name when using the tool from the cmdline.
- Although launching a child process only adds a few seconds of overhead (per project) any tool or process that make use of UVT should be updated to provide the project where possible.
### Changes
### FUnrealVirtualizationToolApp
- As explained in the problem section, we now need to support launching a new instance of the tool as a child process.
- We communicate with the child process by providing an input file via the cmdline, which is then parsed and used to recreate the original command and project settings. Once the command has run in the child process it will write its output to an output file which the calling process can then read back in.
- So our new logic is to check the project that the tool was started with (in case the user provided a project path in the cmdline) against each project that we need to process. If the paths match then we can just process the project, if they do not match we launch a child process with the correct path and retrieve the output once done.
-- If a child process detects an incorrect path we just error out to avoid endless recursive process launching.
- Output from the child process is clearly logged to the user as coming from a child process to make debugging easier.
### FCommand
- Added ToJson/FromJson methods to FCommand, which simulates the same interface as though it was using the json serialization macros, however we want more control over things so do the serialization manually via FCommand::Serialize
-- Serialization is only used if the command is sent to a child process (if the current project is wrong), in which case the command should serialize out all info that it derived during setup that it will need to process correctly. Usually this consists of the options that were parsed from the command line. Then when the child process is run we will recreate the command via the same serialize method.
- Added a new class FProcessPipes, which acts as a wrapper around the pipes we can create by calling FPlatformProcess::CreatePipe which are returned as raw pointers. This ensures that the pipes are cleaned up once the FProcessPipes object goes out of scope.
- The ::Run method has been split into two, ::ProcessProject and ::ProcessOutput
-- ::ProcessProject is called once per project and allows the command to return output via the Output parameter. This can be call in process or via a child process if the project needs to be set.
-- ::ProcessOutput is called once all projects have been processed and provides an array which is a collection of the output from each call to ::ProcessProject. This is always called in process.
-- The output is passed around by the base type FCommandOutput. It is expected that commands derive their own type and as we do not mix command types, they should know how to cast the output back to the correct type when ::ProcessOutput is called. This is a bit messy but to go further risks over engineering given it is unlikely that more commands will be added to the tool. If we do expand the tool in the future then this can be revisited.
### FCommandOutput
- A struct used to represent the output of a command.
- If the command is run as a child process then this struct will be serialized out to disk (via json) then loaded back in by the calling process so that the output of each child process can be combined.
- Commands are expected to derive their own output structure and deal with the json serialization. To make this easier we provide a JSON_SERIALIZE_PARENT macro to allow for serialization of inheritance chains.
- This system does require that commands know which type of output to cast FCommandOutput to. At the moment we do not provide any type safety.
- Add json serialization to FProject and it's child struct FPlugin.
- FProject::GetProjectFilePath now returns as a reference to a FString rather than a FStringView.
[CL 24257111 by paul chipchase in ue5-main branch]
2023-02-16 09:18:15 -05:00
|
|
|
UE_LOG(LogVirtualizationTool, Display, TEXT("\t\tVirtualization of project packages complete!"), ProjectName.ToString());
|
|
|
|
|
|
|
|
|
|
Output = MakeUnique<FVirtualizeCommandOutput>(Project.GetProjectName(), Result.DescriptionTags);
|
|
|
|
|
|
|
|
|
|
return true;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
bool FVirtualizeCommand::ProcessOutput(const TArray<TUniquePtr<FCommandOutput>>& CmdOutputArray)
|
|
|
|
|
{
|
|
|
|
|
TRACE_CPUPROFILER_EVENT_SCOPE(FVirtualizeCommand::ProcessOutput);
|
|
|
|
|
|
2022-11-22 07:23:08 -05:00
|
|
|
if (bShouldSubmitChangelist)
|
|
|
|
|
{
|
[UVT] Improve the robustness of the tool wrt ensuring that the correct project config settings are loaded.
#rb Per.Larsson
#jira UE-151641
#preflight 63ea068cde74ffbae5dc3a7f
### Problem
- A lot of systems rely on the engine to know which project is being loaded on start up so that the correct config settings can be applied however so far the UVT has not required the user to provide the desired project. This is because in theory, the tool could be given a list of packages or a changelist to process that contain packages from more than one project and if that happens we need to handle it correctly.
- The tool currently groups the provided packages by project, then runs the requested command on each project one at a time, initializing the VA system with that projects config files, then shutting it done once the project packages are processed. This allows us to pick up the correct settings for VA but other systems that we rely on, such as the DDC do not allow this sort of dynamic initialization and so may not have the correct settings.
### Solution
- Before the projects packages are processed we check to see if the project applied to the process during initialization (if any) matches the project to be processed. If not we launch a new version of the tool with the correct project settings and have the child process perform the main processing. The output of each child process (or any projects we were able to run in process) are then provided to the command for a final "post projects" phase.
- This means we can support multiple packages from multiple processes AND not require the user type out the project name when using the tool from the cmdline.
- Although launching a child process only adds a few seconds of overhead (per project) any tool or process that make use of UVT should be updated to provide the project where possible.
### Changes
### FUnrealVirtualizationToolApp
- As explained in the problem section, we now need to support launching a new instance of the tool as a child process.
- We communicate with the child process by providing an input file via the cmdline, which is then parsed and used to recreate the original command and project settings. Once the command has run in the child process it will write its output to an output file which the calling process can then read back in.
- So our new logic is to check the project that the tool was started with (in case the user provided a project path in the cmdline) against each project that we need to process. If the paths match then we can just process the project, if they do not match we launch a child process with the correct path and retrieve the output once done.
-- If a child process detects an incorrect path we just error out to avoid endless recursive process launching.
- Output from the child process is clearly logged to the user as coming from a child process to make debugging easier.
### FCommand
- Added ToJson/FromJson methods to FCommand, which simulates the same interface as though it was using the json serialization macros, however we want more control over things so do the serialization manually via FCommand::Serialize
-- Serialization is only used if the command is sent to a child process (if the current project is wrong), in which case the command should serialize out all info that it derived during setup that it will need to process correctly. Usually this consists of the options that were parsed from the command line. Then when the child process is run we will recreate the command via the same serialize method.
- Added a new class FProcessPipes, which acts as a wrapper around the pipes we can create by calling FPlatformProcess::CreatePipe which are returned as raw pointers. This ensures that the pipes are cleaned up once the FProcessPipes object goes out of scope.
- The ::Run method has been split into two, ::ProcessProject and ::ProcessOutput
-- ::ProcessProject is called once per project and allows the command to return output via the Output parameter. This can be call in process or via a child process if the project needs to be set.
-- ::ProcessOutput is called once all projects have been processed and provides an array which is a collection of the output from each call to ::ProcessProject. This is always called in process.
-- The output is passed around by the base type FCommandOutput. It is expected that commands derive their own type and as we do not mix command types, they should know how to cast the output back to the correct type when ::ProcessOutput is called. This is a bit messy but to go further risks over engineering given it is unlikely that more commands will be added to the tool. If we do expand the tool in the future then this can be revisited.
### FCommandOutput
- A struct used to represent the output of a command.
- If the command is run as a child process then this struct will be serialized out to disk (via json) then loaded back in by the calling process so that the output of each child process can be combined.
- Commands are expected to derive their own output structure and deal with the json serialization. To make this easier we provide a JSON_SERIALIZE_PARENT macro to allow for serialization of inheritance chains.
- This system does require that commands know which type of output to cast FCommandOutput to. At the moment we do not provide any type safety.
- Add json serialization to FProject and it's child struct FPlugin.
- FProject::GetProjectFilePath now returns as a reference to a FString rather than a FStringView.
[CL 24257111 by paul chipchase in ue5-main branch]
2023-02-16 09:18:15 -05:00
|
|
|
TArray<FString> FinalDescriptionTags = BuildFinalTagDescriptions(CmdOutputArray);
|
2022-11-22 07:23:08 -05:00
|
|
|
|
|
|
|
|
if (!TrySubmitChangelist(SourceChangelist, FinalDescriptionTags))
|
|
|
|
|
{
|
|
|
|
|
return false;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
return true;
|
|
|
|
|
}
|
|
|
|
|
|
[UVT] Improve the robustness of the tool wrt ensuring that the correct project config settings are loaded.
#rb Per.Larsson
#jira UE-151641
#preflight 63ea068cde74ffbae5dc3a7f
### Problem
- A lot of systems rely on the engine to know which project is being loaded on start up so that the correct config settings can be applied however so far the UVT has not required the user to provide the desired project. This is because in theory, the tool could be given a list of packages or a changelist to process that contain packages from more than one project and if that happens we need to handle it correctly.
- The tool currently groups the provided packages by project, then runs the requested command on each project one at a time, initializing the VA system with that projects config files, then shutting it done once the project packages are processed. This allows us to pick up the correct settings for VA but other systems that we rely on, such as the DDC do not allow this sort of dynamic initialization and so may not have the correct settings.
### Solution
- Before the projects packages are processed we check to see if the project applied to the process during initialization (if any) matches the project to be processed. If not we launch a new version of the tool with the correct project settings and have the child process perform the main processing. The output of each child process (or any projects we were able to run in process) are then provided to the command for a final "post projects" phase.
- This means we can support multiple packages from multiple processes AND not require the user type out the project name when using the tool from the cmdline.
- Although launching a child process only adds a few seconds of overhead (per project) any tool or process that make use of UVT should be updated to provide the project where possible.
### Changes
### FUnrealVirtualizationToolApp
- As explained in the problem section, we now need to support launching a new instance of the tool as a child process.
- We communicate with the child process by providing an input file via the cmdline, which is then parsed and used to recreate the original command and project settings. Once the command has run in the child process it will write its output to an output file which the calling process can then read back in.
- So our new logic is to check the project that the tool was started with (in case the user provided a project path in the cmdline) against each project that we need to process. If the paths match then we can just process the project, if they do not match we launch a child process with the correct path and retrieve the output once done.
-- If a child process detects an incorrect path we just error out to avoid endless recursive process launching.
- Output from the child process is clearly logged to the user as coming from a child process to make debugging easier.
### FCommand
- Added ToJson/FromJson methods to FCommand, which simulates the same interface as though it was using the json serialization macros, however we want more control over things so do the serialization manually via FCommand::Serialize
-- Serialization is only used if the command is sent to a child process (if the current project is wrong), in which case the command should serialize out all info that it derived during setup that it will need to process correctly. Usually this consists of the options that were parsed from the command line. Then when the child process is run we will recreate the command via the same serialize method.
- Added a new class FProcessPipes, which acts as a wrapper around the pipes we can create by calling FPlatformProcess::CreatePipe which are returned as raw pointers. This ensures that the pipes are cleaned up once the FProcessPipes object goes out of scope.
- The ::Run method has been split into two, ::ProcessProject and ::ProcessOutput
-- ::ProcessProject is called once per project and allows the command to return output via the Output parameter. This can be call in process or via a child process if the project needs to be set.
-- ::ProcessOutput is called once all projects have been processed and provides an array which is a collection of the output from each call to ::ProcessProject. This is always called in process.
-- The output is passed around by the base type FCommandOutput. It is expected that commands derive their own type and as we do not mix command types, they should know how to cast the output back to the correct type when ::ProcessOutput is called. This is a bit messy but to go further risks over engineering given it is unlikely that more commands will be added to the tool. If we do expand the tool in the future then this can be revisited.
### FCommandOutput
- A struct used to represent the output of a command.
- If the command is run as a child process then this struct will be serialized out to disk (via json) then loaded back in by the calling process so that the output of each child process can be combined.
- Commands are expected to derive their own output structure and deal with the json serialization. To make this easier we provide a JSON_SERIALIZE_PARENT macro to allow for serialization of inheritance chains.
- This system does require that commands know which type of output to cast FCommandOutput to. At the moment we do not provide any type safety.
- Add json serialization to FProject and it's child struct FPlugin.
- FProject::GetProjectFilePath now returns as a reference to a FString rather than a FStringView.
[CL 24257111 by paul chipchase in ue5-main branch]
2023-02-16 09:18:15 -05:00
|
|
|
TUniquePtr<FCommandOutput> FVirtualizeCommand::CreateOutputObject() const
|
|
|
|
|
{
|
|
|
|
|
return MakeUnique<FVirtualizeCommandOutput>();
|
|
|
|
|
}
|
|
|
|
|
|
2022-11-22 07:23:08 -05:00
|
|
|
const TArray<FString>& FVirtualizeCommand::GetPackages() const
|
|
|
|
|
{
|
|
|
|
|
return AllPackages;
|
|
|
|
|
}
|
|
|
|
|
|
[UVT] Improve the robustness of the tool wrt ensuring that the correct project config settings are loaded.
#rb Per.Larsson
#jira UE-151641
#preflight 63ea068cde74ffbae5dc3a7f
### Problem
- A lot of systems rely on the engine to know which project is being loaded on start up so that the correct config settings can be applied however so far the UVT has not required the user to provide the desired project. This is because in theory, the tool could be given a list of packages or a changelist to process that contain packages from more than one project and if that happens we need to handle it correctly.
- The tool currently groups the provided packages by project, then runs the requested command on each project one at a time, initializing the VA system with that projects config files, then shutting it done once the project packages are processed. This allows us to pick up the correct settings for VA but other systems that we rely on, such as the DDC do not allow this sort of dynamic initialization and so may not have the correct settings.
### Solution
- Before the projects packages are processed we check to see if the project applied to the process during initialization (if any) matches the project to be processed. If not we launch a new version of the tool with the correct project settings and have the child process perform the main processing. The output of each child process (or any projects we were able to run in process) are then provided to the command for a final "post projects" phase.
- This means we can support multiple packages from multiple processes AND not require the user type out the project name when using the tool from the cmdline.
- Although launching a child process only adds a few seconds of overhead (per project) any tool or process that make use of UVT should be updated to provide the project where possible.
### Changes
### FUnrealVirtualizationToolApp
- As explained in the problem section, we now need to support launching a new instance of the tool as a child process.
- We communicate with the child process by providing an input file via the cmdline, which is then parsed and used to recreate the original command and project settings. Once the command has run in the child process it will write its output to an output file which the calling process can then read back in.
- So our new logic is to check the project that the tool was started with (in case the user provided a project path in the cmdline) against each project that we need to process. If the paths match then we can just process the project, if they do not match we launch a child process with the correct path and retrieve the output once done.
-- If a child process detects an incorrect path we just error out to avoid endless recursive process launching.
- Output from the child process is clearly logged to the user as coming from a child process to make debugging easier.
### FCommand
- Added ToJson/FromJson methods to FCommand, which simulates the same interface as though it was using the json serialization macros, however we want more control over things so do the serialization manually via FCommand::Serialize
-- Serialization is only used if the command is sent to a child process (if the current project is wrong), in which case the command should serialize out all info that it derived during setup that it will need to process correctly. Usually this consists of the options that were parsed from the command line. Then when the child process is run we will recreate the command via the same serialize method.
- Added a new class FProcessPipes, which acts as a wrapper around the pipes we can create by calling FPlatformProcess::CreatePipe which are returned as raw pointers. This ensures that the pipes are cleaned up once the FProcessPipes object goes out of scope.
- The ::Run method has been split into two, ::ProcessProject and ::ProcessOutput
-- ::ProcessProject is called once per project and allows the command to return output via the Output parameter. This can be call in process or via a child process if the project needs to be set.
-- ::ProcessOutput is called once all projects have been processed and provides an array which is a collection of the output from each call to ::ProcessProject. This is always called in process.
-- The output is passed around by the base type FCommandOutput. It is expected that commands derive their own type and as we do not mix command types, they should know how to cast the output back to the correct type when ::ProcessOutput is called. This is a bit messy but to go further risks over engineering given it is unlikely that more commands will be added to the tool. If we do expand the tool in the future then this can be revisited.
### FCommandOutput
- A struct used to represent the output of a command.
- If the command is run as a child process then this struct will be serialized out to disk (via json) then loaded back in by the calling process so that the output of each child process can be combined.
- Commands are expected to derive their own output structure and deal with the json serialization. To make this easier we provide a JSON_SERIALIZE_PARENT macro to allow for serialization of inheritance chains.
- This system does require that commands know which type of output to cast FCommandOutput to. At the moment we do not provide any type safety.
- Add json serialization to FProject and it's child struct FPlugin.
- FProject::GetProjectFilePath now returns as a reference to a FString rather than a FStringView.
[CL 24257111 by paul chipchase in ue5-main branch]
2023-02-16 09:18:15 -05:00
|
|
|
bool FVirtualizeCommand::TrySubmitChangelist(const FSourceControlChangelistPtr& ChangelistToSubmit, const TArray<FString>& InDescriptionTags)
|
2022-11-22 07:23:08 -05:00
|
|
|
{
|
|
|
|
|
if (!ChangelistToSubmit.IsValid())
|
|
|
|
|
{
|
|
|
|
|
return true;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if (!TryConnectToSourceControl(ClientSpecName))
|
|
|
|
|
{
|
|
|
|
|
UE_LOG(LogVirtualizationTool, Error, TEXT("Submit failed, cannot find a valid source control provider"));
|
|
|
|
|
return false;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
TRACE_CPUPROFILER_EVENT_SCOPE(FVirtualizeCommand::TrySubmitChangelist);
|
|
|
|
|
|
|
|
|
|
FSourceControlChangelistRef Changelist = ChangelistToSubmit.ToSharedRef();
|
|
|
|
|
FSourceControlChangelistStatePtr ChangelistState = SCCProvider->GetState(Changelist, EStateCacheUsage::Use);
|
|
|
|
|
|
|
|
|
|
if (!ChangelistState.IsValid())
|
|
|
|
|
{
|
|
|
|
|
UE_LOG(LogVirtualizationTool, Error, TEXT("Submit failed, failed to find the state for the changelist"));
|
|
|
|
|
return false;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
const FString ChangelistNumber = ChangelistState->GetDisplayText().ToString();
|
|
|
|
|
|
|
|
|
|
UE_LOG(LogVirtualizationTool, Display, TEXT("Attempting to submit the changelist '%s'"), *ChangelistNumber);
|
|
|
|
|
|
|
|
|
|
TSharedRef<FCheckIn, ESPMode::ThreadSafe> CheckInOperation = ISourceControlOperation::Create<FCheckIn>();
|
|
|
|
|
|
|
|
|
|
// Grab the original changelist description then append our tags afterwards
|
|
|
|
|
TStringBuilder<512> Description;
|
|
|
|
|
Description << ChangelistState->GetDescriptionText().ToString();
|
|
|
|
|
|
[UVT] Improve the robustness of the tool wrt ensuring that the correct project config settings are loaded.
#rb Per.Larsson
#jira UE-151641
#preflight 63ea068cde74ffbae5dc3a7f
### Problem
- A lot of systems rely on the engine to know which project is being loaded on start up so that the correct config settings can be applied however so far the UVT has not required the user to provide the desired project. This is because in theory, the tool could be given a list of packages or a changelist to process that contain packages from more than one project and if that happens we need to handle it correctly.
- The tool currently groups the provided packages by project, then runs the requested command on each project one at a time, initializing the VA system with that projects config files, then shutting it done once the project packages are processed. This allows us to pick up the correct settings for VA but other systems that we rely on, such as the DDC do not allow this sort of dynamic initialization and so may not have the correct settings.
### Solution
- Before the projects packages are processed we check to see if the project applied to the process during initialization (if any) matches the project to be processed. If not we launch a new version of the tool with the correct project settings and have the child process perform the main processing. The output of each child process (or any projects we were able to run in process) are then provided to the command for a final "post projects" phase.
- This means we can support multiple packages from multiple processes AND not require the user type out the project name when using the tool from the cmdline.
- Although launching a child process only adds a few seconds of overhead (per project) any tool or process that make use of UVT should be updated to provide the project where possible.
### Changes
### FUnrealVirtualizationToolApp
- As explained in the problem section, we now need to support launching a new instance of the tool as a child process.
- We communicate with the child process by providing an input file via the cmdline, which is then parsed and used to recreate the original command and project settings. Once the command has run in the child process it will write its output to an output file which the calling process can then read back in.
- So our new logic is to check the project that the tool was started with (in case the user provided a project path in the cmdline) against each project that we need to process. If the paths match then we can just process the project, if they do not match we launch a child process with the correct path and retrieve the output once done.
-- If a child process detects an incorrect path we just error out to avoid endless recursive process launching.
- Output from the child process is clearly logged to the user as coming from a child process to make debugging easier.
### FCommand
- Added ToJson/FromJson methods to FCommand, which simulates the same interface as though it was using the json serialization macros, however we want more control over things so do the serialization manually via FCommand::Serialize
-- Serialization is only used if the command is sent to a child process (if the current project is wrong), in which case the command should serialize out all info that it derived during setup that it will need to process correctly. Usually this consists of the options that were parsed from the command line. Then when the child process is run we will recreate the command via the same serialize method.
- Added a new class FProcessPipes, which acts as a wrapper around the pipes we can create by calling FPlatformProcess::CreatePipe which are returned as raw pointers. This ensures that the pipes are cleaned up once the FProcessPipes object goes out of scope.
- The ::Run method has been split into two, ::ProcessProject and ::ProcessOutput
-- ::ProcessProject is called once per project and allows the command to return output via the Output parameter. This can be call in process or via a child process if the project needs to be set.
-- ::ProcessOutput is called once all projects have been processed and provides an array which is a collection of the output from each call to ::ProcessProject. This is always called in process.
-- The output is passed around by the base type FCommandOutput. It is expected that commands derive their own type and as we do not mix command types, they should know how to cast the output back to the correct type when ::ProcessOutput is called. This is a bit messy but to go further risks over engineering given it is unlikely that more commands will be added to the tool. If we do expand the tool in the future then this can be revisited.
### FCommandOutput
- A struct used to represent the output of a command.
- If the command is run as a child process then this struct will be serialized out to disk (via json) then loaded back in by the calling process so that the output of each child process can be combined.
- Commands are expected to derive their own output structure and deal with the json serialization. To make this easier we provide a JSON_SERIALIZE_PARENT macro to allow for serialization of inheritance chains.
- This system does require that commands know which type of output to cast FCommandOutput to. At the moment we do not provide any type safety.
- Add json serialization to FProject and it's child struct FPlugin.
- FProject::GetProjectFilePath now returns as a reference to a FString rather than a FStringView.
[CL 24257111 by paul chipchase in ue5-main branch]
2023-02-16 09:18:15 -05:00
|
|
|
for (const FString& Tag : InDescriptionTags)
|
2022-11-22 07:23:08 -05:00
|
|
|
{
|
|
|
|
|
Description << TEXT("\n") << Tag;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
CheckInOperation->SetDescription(FText::FromString(Description.ToString()));
|
|
|
|
|
|
|
|
|
|
if (SCCProvider->Execute(CheckInOperation, Changelist) == ECommandResult::Succeeded)
|
|
|
|
|
{
|
|
|
|
|
UE_LOG(LogVirtualizationTool, Display, TEXT("%s"), *CheckInOperation->GetSuccessMessage().ToString());
|
|
|
|
|
return true;
|
|
|
|
|
}
|
|
|
|
|
else
|
|
|
|
|
{
|
|
|
|
|
// Even when log suppression is active we still show errors to the users and as the source control
|
|
|
|
|
// operation should have logged the problem as an error the user will see it. This means we don't
|
|
|
|
|
// have to extract it from CheckInOperation .
|
|
|
|
|
UE_LOG(LogVirtualizationTool, Error, TEXT("Submit failed, please check the log!"));
|
|
|
|
|
|
|
|
|
|
return false;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
bool FVirtualizeCommand::TryParsePackageList(const FString& PackageListPath, TArray<FString>& OutPackages)
|
|
|
|
|
{
|
|
|
|
|
TRACE_CPUPROFILER_EVENT_SCOPE(FVirtualizeCommand::TryParsePackageList);
|
|
|
|
|
|
|
|
|
|
UE_LOG(LogVirtualizationTool, Display, TEXT("Parsing the package list '%s'..."), *PackageListPath);
|
|
|
|
|
|
|
|
|
|
if (!IFileManager::Get().FileExists(*PackageListPath))
|
|
|
|
|
{
|
|
|
|
|
UE_LOG(LogVirtualizationTool, Error, TEXT("\tThe package list '%s' does not exist"), *PackageListPath);
|
|
|
|
|
return false;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if (FFileHelper::LoadFileToStringArray(OutPackages, *PackageListPath))
|
|
|
|
|
{
|
|
|
|
|
// We don't have control over how the package list was generated so make sure that the paths
|
|
|
|
|
// are in the format that we want.
|
|
|
|
|
for (FString& PackagePath : OutPackages)
|
|
|
|
|
{
|
|
|
|
|
FPaths::NormalizeFilename(PackagePath);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
return true;
|
|
|
|
|
}
|
|
|
|
|
else
|
|
|
|
|
{
|
|
|
|
|
UE_LOG(LogVirtualizationTool, Error, TEXT("\tFailed to parse the package list '%s'"), *PackageListPath);
|
|
|
|
|
return false;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
FVirtualizeLegacyChangeListCommand::FVirtualizeLegacyChangeListCommand(FStringView CommandName)
|
|
|
|
|
: FVirtualizeCommand(CommandName)
|
|
|
|
|
{
|
|
|
|
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
bool FVirtualizeLegacyChangeListCommand::Initialize(const TCHAR* CmdLine)
|
|
|
|
|
{
|
|
|
|
|
TRACE_CPUPROFILER_EVENT_SCOPE(FVirtualizeLegacyChangeListCommand::Initialize);
|
|
|
|
|
|
|
|
|
|
UE_LOG(LogVirtualizationTool, Warning, TEXT("Using legacy -Mode=Changelist command, use '-Mode=Virtualization -Changelist=123' instead!"));
|
|
|
|
|
|
|
|
|
|
if (!FParse::Value(CmdLine, TEXT("-ClientSpecName="), ClientSpecName))
|
|
|
|
|
{
|
|
|
|
|
UE_LOG(LogVirtualizationTool, Error, TEXT("Failed to find cmdline switch 'ClientSpecName', this is a required parameter!"));
|
|
|
|
|
return false;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if (!FParse::Value(CmdLine, TEXT("-Changelist="), SourceChangelistNumber))
|
|
|
|
|
{
|
|
|
|
|
UE_LOG(LogVirtualizationTool, Error, TEXT("Failed to find cmdline switch 'Changelist', this is a required parameter!"));
|
|
|
|
|
return false;
|
|
|
|
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if (!TryParseChangelist(ClientSpecName, SourceChangelistNumber, AllPackages, &SourceChangelist))
|
|
|
|
|
{
|
|
|
|
|
return false;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if (FParse::Param(CmdLine, TEXT("NoSubmit")))
|
|
|
|
|
{
|
|
|
|
|
UE_LOG(LogVirtualizationTool, Display, TEXT("Cmdline parameter '-NoSubmit' found, the changelist will be virtualized but not submitted!"));
|
|
|
|
|
bShouldSubmitChangelist = false;
|
|
|
|
|
}
|
|
|
|
|
else
|
|
|
|
|
{
|
|
|
|
|
// The legacy command was opt out when it came to submitting the changelist, we need to maintain those defaults
|
|
|
|
|
bShouldSubmitChangelist = true;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
return true;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
FVirtualizeLegacyPackageListCommand::FVirtualizeLegacyPackageListCommand(FStringView CommandName)
|
|
|
|
|
: FVirtualizeCommand(CommandName)
|
|
|
|
|
{
|
|
|
|
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
bool FVirtualizeLegacyPackageListCommand::Initialize(const TCHAR* CmdLine)
|
|
|
|
|
{
|
|
|
|
|
TRACE_CPUPROFILER_EVENT_SCOPE(FVirtualizeLegacyPackageListCommand::Initialize);
|
|
|
|
|
|
|
|
|
|
UE_LOG(LogVirtualizationTool, Warning, TEXT("Using legacy -Mode=Packagelist command, use '-Mode=Virtualization -Path=PathToFile' instead!"));
|
|
|
|
|
|
|
|
|
|
FString PackageListPath;
|
|
|
|
|
if (FParse::Value(CmdLine, TEXT("-Path="), PackageListPath))
|
|
|
|
|
{
|
|
|
|
|
return TryParsePackageList(PackageListPath, AllPackages);
|
|
|
|
|
}
|
|
|
|
|
else
|
|
|
|
|
{
|
|
|
|
|
|
|
|
|
|
return false;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
} // namespace UE::Virtualization
|