Add a rehydration command to the stand alone virtualization tool, making it easier to reverse the effects of asset virtualization. Unlike previous processes, this one does not require that we load the package and will just manipulate the data storaged in the package trailer.
#rb Sebastian.Nordgren
#rnx
#jira UE-156436
#preflight 62c287f9a3568e30664eb94f
### VA Standalone Tool
- We now plan to add much more functionality to the tool than just virtualizing and submitting changelists, so to make this easier I am moving the tool towards a design where it should be fairly easy to add new functionality.
- Added FCommand, which is a base class for adding new functionality, simple derive from FCommand and hook it up at the appropriate locations.
-- In the future it should be possible for new command types to automatically register themselves to be initiated from the command line. There should be no need to edit UnrealVirtualizationToolApp to add a new command but this will be done as an additional work item.
-- At the moment FCommand comes with a number of utility methods to call that cover some common source control commands.
-- The original functionality has not yet been moved to the command system and so the code is a little bit weird at the moment. Updating older code to the new system will be done as an additional work item.
- FProject/FPlugin have been moved to their own code files.
### Rehydrate Command
- The rehydrate command will take a number of packages, check them out of source control and then attempt to virtualize them.
- At the moment the chekout logic is fairly basic, we just check out every package supplied, we don't check if the package is virtualized or not yet. This can be improved in additional work items. Ideally by the end of command the only packages that we have checked out should also be rehydrated.
- At the moment the command can either take a path of a specific package, a path of a directory to find packages in, or a changelist containing packages that should be rehydrated.
- A cleint spec (workspace) can optionally be provided, but if not supplied we will attempt to find a client spec for which to check out the packages.
- Currently we will check out the packages to the default change list.
### Rehydrate process
- Added the rehydration process in it's own code files in the virtualization module. Like the virtualization process this is exposed in a public header file and no via the Core interface which means it is very specific to our module/implementation.
- The process expects that the caller will have checked out any required packages from source control. It will treat being unable to update a package file as an error.
- Added PackageUtils.h/.cpp and moved some of the generic code from the virtualization process code there so that it can be shared by the rehydration process.
### Misc
Moving away from the using things like FPackagePath as that requires that the correct mount points have been registered for a project and at the moment (with the flakiness of FConfig*) it seems that the best idea would be to prefer absolute file paths where possible.
[CL 20982284 by paul chipchase in ue5-main branch]
2022-07-07 06:54:33 -04:00
// Copyright Epic Games, Inc. All Rights Reserved.
# include "ProjectFiles.h"
# include "Containers/StringView.h"
# include "Misc/ConfigCacheIni.h"
# include "Misc/PackageName.h"
# include "Misc/Paths.h"
# include "Misc/PathViews.h"
# include "UnrealVirtualizationTool.h"
# include "UObject/Class.h"
# include "UObject/UObjectHash.h"
namespace
{
/** Utility to find the two string values we need for a mount point based on the project file path */
void ConvertToMountPoint ( const FString & ProjectFilePath , FString & OutRootPath , FString & OutContentPath )
{
FStringView BaseFilename = FPathViews : : GetBaseFilename ( ProjectFilePath ) ;
OutRootPath = * WriteToString < 260 > ( TEXT ( " / " ) , BaseFilename , TEXT ( " / " ) ) ;
OutContentPath = FPaths : : GetPath ( ProjectFilePath ) / TEXT ( " Content " ) ;
}
/**
* Utility taken from UGameFeatureData : : ReloadConfigs that allows us to apply changes to the ini files ( after
* loading them from game feature plugins for example ) and have the changes applied to UObjects .
* For our use case we need this so that optin / optout settings for UVirtualizationFilterSettings are applied .
*
* This is required because we perform filtering at payload submission time . If we change filtering to be
* applied when a package is saved ( i . e . when the package trailer is created ) then we can remove this .
* If we opt to keep the current strategy then this code should be moved to a location where it can be shared
* by both this tool and the game feature plugin system rather than maintaining two copies .
*/
void ReloadConfigs ( FConfigFile & PluginConfig )
{
// Reload configs so objects get the changes
2023-09-15 13:39:02 -04:00
for ( const auto & ConfigEntry : AsConst ( PluginConfig ) )
Add a rehydration command to the stand alone virtualization tool, making it easier to reverse the effects of asset virtualization. Unlike previous processes, this one does not require that we load the package and will just manipulate the data storaged in the package trailer.
#rb Sebastian.Nordgren
#rnx
#jira UE-156436
#preflight 62c287f9a3568e30664eb94f
### VA Standalone Tool
- We now plan to add much more functionality to the tool than just virtualizing and submitting changelists, so to make this easier I am moving the tool towards a design where it should be fairly easy to add new functionality.
- Added FCommand, which is a base class for adding new functionality, simple derive from FCommand and hook it up at the appropriate locations.
-- In the future it should be possible for new command types to automatically register themselves to be initiated from the command line. There should be no need to edit UnrealVirtualizationToolApp to add a new command but this will be done as an additional work item.
-- At the moment FCommand comes with a number of utility methods to call that cover some common source control commands.
-- The original functionality has not yet been moved to the command system and so the code is a little bit weird at the moment. Updating older code to the new system will be done as an additional work item.
- FProject/FPlugin have been moved to their own code files.
### Rehydrate Command
- The rehydrate command will take a number of packages, check them out of source control and then attempt to virtualize them.
- At the moment the chekout logic is fairly basic, we just check out every package supplied, we don't check if the package is virtualized or not yet. This can be improved in additional work items. Ideally by the end of command the only packages that we have checked out should also be rehydrated.
- At the moment the command can either take a path of a specific package, a path of a directory to find packages in, or a changelist containing packages that should be rehydrated.
- A cleint spec (workspace) can optionally be provided, but if not supplied we will attempt to find a client spec for which to check out the packages.
- Currently we will check out the packages to the default change list.
### Rehydrate process
- Added the rehydration process in it's own code files in the virtualization module. Like the virtualization process this is exposed in a public header file and no via the Core interface which means it is very specific to our module/implementation.
- The process expects that the caller will have checked out any required packages from source control. It will treat being unable to update a package file as an error.
- Added PackageUtils.h/.cpp and moved some of the generic code from the virtualization process code there so that it can be shared by the rehydration process.
### Misc
Moving away from the using things like FPackagePath as that requires that the correct mount points have been registered for a project and at the moment (with the flakiness of FConfig*) it seems that the best idea would be to prefer absolute file paths where possible.
[CL 20982284 by paul chipchase in ue5-main branch]
2022-07-07 06:54:33 -04:00
{
// Skip out if someone put a config section in the INI without any actual data
if ( ConfigEntry . Value . Num ( ) = = 0 )
{
continue ;
}
const FString & SectionName = ConfigEntry . Key ;
// @todo: This entire overarching process is very similar in its goals as that of UOnlineHotfixManager::HotfixIniFile.
// Could consider a combined refactor of the hotfix manager, the base config cache system, etc. to expose an easier way to support this pattern
// INI files might be handling per-object config items, so need to handle them specifically
const int32 PerObjConfigDelimIdx = SectionName . Find ( " " ) ;
if ( PerObjConfigDelimIdx ! = INDEX_NONE )
{
const FString ObjectName = SectionName . Left ( PerObjConfigDelimIdx ) ;
const FString ClassName = SectionName . Mid ( PerObjConfigDelimIdx + 1 ) ;
// Try to find the class specified by the per-object config
UClass * ObjClass = UClass : : TryFindTypeSlow < UClass > ( * ClassName , EFindFirstObjectOptions : : NativeFirst | EFindFirstObjectOptions : : EnsureIfAmbiguous ) ;
if ( ObjClass )
{
// Now try to actually find the object it's referencing specifically and update it
// @note: Choosing not to warn on not finding it for now, as Fortnite has transient uses instantiated at run-time (might not be constructed yet)
UObject * PerObjConfigObj = StaticFindFirstObject ( ObjClass , * ObjectName , EFindFirstObjectOptions : : ExactClass , ELogVerbosity : : Warning , TEXT ( " UGameFeatureData::ReloadConfigs " ) ) ;
if ( PerObjConfigObj )
{
// Intentionally using LoadConfig instead of ReloadConfig, since we do not want to call modify/preeditchange/posteditchange on the objects changed when GIsEditor
PerObjConfigObj - > LoadConfig ( nullptr , nullptr , UE : : LCPF_ReloadingConfigData | UE : : LCPF_ReadParentSections , nullptr ) ;
}
}
else
{
PLATFORM_BREAK ( ) ;
// UE_LOG(LogGameFeatures, Warning, TEXT("[GameFeatureData %s]: Couldn't find PerObjectConfig class %s for %s while processing %s, config changes won't be reloaded."), *GetPathNameSafe(this), *ClassName, *ObjectName, *PluginConfig.Name.ToString());
}
}
// Standard INI section case
else
{
// Find the affected class and push updates to all instances of it, including children
// @note: Intentionally not using the propagation flags inherent in ReloadConfig to handle this, as it utilizes a naive complete object iterator
// and tanks performance pretty badly
UClass * ObjClass = FindFirstObject < UClass > ( * SectionName , EFindFirstObjectOptions : : ExactClass | EFindFirstObjectOptions : : EnsureIfAmbiguous | EFindFirstObjectOptions : : NativeFirst ) ;
if ( ObjClass )
{
TArray < UObject * > FoundObjects ;
GetObjectsOfClass ( ObjClass , FoundObjects , true , RF_NoFlags ) ;
for ( UObject * CurFoundObj : FoundObjects )
{
if ( IsValid ( CurFoundObj ) )
{
// Intentionally using LoadConfig instead of ReloadConfig, since we do not want to call modify/preeditchange/posteditchange on the objects changed when GIsEditor
CurFoundObj - > LoadConfig ( nullptr , nullptr , UE : : LCPF_ReloadingConfigData | UE : : LCPF_ReadParentSections , nullptr ) ;
}
}
}
}
}
}
} // namespace
namespace UE : : Virtualization
{
FProject : : FProject ( FString & & InProjectFilePath )
: ProjectFilePath ( MoveTemp ( InProjectFilePath ) )
{
}
void FProject : : AddFile ( const FString & PackagePath )
{
PackagePaths . Add ( PackagePath ) ;
}
void FProject : : AddPluginFile ( const FString & PackagePath , FString & & PluginFilePath )
{
FPlugin * Plugin = Plugins . FindByPredicate ( [ & PluginFilePath ] ( const FPlugin & Plugin ) - > bool
{
return Plugin . PluginFilePath = = PluginFilePath ;
} ) ;
if ( Plugin = = nullptr )
{
Plugin = & Plugins . AddDefaulted_GetRef ( ) ;
Plugin - > PluginFilePath = MoveTemp ( PluginFilePath ) ;
}
check ( Plugin ! = nullptr ) ;
Plugin - > PackagePaths . Add ( PackagePath ) ;
} ;
[UVT] Improve the robustness of the tool wrt ensuring that the correct project config settings are loaded.
#rb Per.Larsson
#jira UE-151641
#preflight 63ea068cde74ffbae5dc3a7f
### Problem
- A lot of systems rely on the engine to know which project is being loaded on start up so that the correct config settings can be applied however so far the UVT has not required the user to provide the desired project. This is because in theory, the tool could be given a list of packages or a changelist to process that contain packages from more than one project and if that happens we need to handle it correctly.
- The tool currently groups the provided packages by project, then runs the requested command on each project one at a time, initializing the VA system with that projects config files, then shutting it done once the project packages are processed. This allows us to pick up the correct settings for VA but other systems that we rely on, such as the DDC do not allow this sort of dynamic initialization and so may not have the correct settings.
### Solution
- Before the projects packages are processed we check to see if the project applied to the process during initialization (if any) matches the project to be processed. If not we launch a new version of the tool with the correct project settings and have the child process perform the main processing. The output of each child process (or any projects we were able to run in process) are then provided to the command for a final "post projects" phase.
- This means we can support multiple packages from multiple processes AND not require the user type out the project name when using the tool from the cmdline.
- Although launching a child process only adds a few seconds of overhead (per project) any tool or process that make use of UVT should be updated to provide the project where possible.
### Changes
### FUnrealVirtualizationToolApp
- As explained in the problem section, we now need to support launching a new instance of the tool as a child process.
- We communicate with the child process by providing an input file via the cmdline, which is then parsed and used to recreate the original command and project settings. Once the command has run in the child process it will write its output to an output file which the calling process can then read back in.
- So our new logic is to check the project that the tool was started with (in case the user provided a project path in the cmdline) against each project that we need to process. If the paths match then we can just process the project, if they do not match we launch a child process with the correct path and retrieve the output once done.
-- If a child process detects an incorrect path we just error out to avoid endless recursive process launching.
- Output from the child process is clearly logged to the user as coming from a child process to make debugging easier.
### FCommand
- Added ToJson/FromJson methods to FCommand, which simulates the same interface as though it was using the json serialization macros, however we want more control over things so do the serialization manually via FCommand::Serialize
-- Serialization is only used if the command is sent to a child process (if the current project is wrong), in which case the command should serialize out all info that it derived during setup that it will need to process correctly. Usually this consists of the options that were parsed from the command line. Then when the child process is run we will recreate the command via the same serialize method.
- Added a new class FProcessPipes, which acts as a wrapper around the pipes we can create by calling FPlatformProcess::CreatePipe which are returned as raw pointers. This ensures that the pipes are cleaned up once the FProcessPipes object goes out of scope.
- The ::Run method has been split into two, ::ProcessProject and ::ProcessOutput
-- ::ProcessProject is called once per project and allows the command to return output via the Output parameter. This can be call in process or via a child process if the project needs to be set.
-- ::ProcessOutput is called once all projects have been processed and provides an array which is a collection of the output from each call to ::ProcessProject. This is always called in process.
-- The output is passed around by the base type FCommandOutput. It is expected that commands derive their own type and as we do not mix command types, they should know how to cast the output back to the correct type when ::ProcessOutput is called. This is a bit messy but to go further risks over engineering given it is unlikely that more commands will be added to the tool. If we do expand the tool in the future then this can be revisited.
### FCommandOutput
- A struct used to represent the output of a command.
- If the command is run as a child process then this struct will be serialized out to disk (via json) then loaded back in by the calling process so that the output of each child process can be combined.
- Commands are expected to derive their own output structure and deal with the json serialization. To make this easier we provide a JSON_SERIALIZE_PARENT macro to allow for serialization of inheritance chains.
- This system does require that commands know which type of output to cast FCommandOutput to. At the moment we do not provide any type safety.
- Add json serialization to FProject and it's child struct FPlugin.
- FProject::GetProjectFilePath now returns as a reference to a FString rather than a FStringView.
[CL 24257111 by paul chipchase in ue5-main branch]
2023-02-16 09:18:15 -05:00
const FString & FProject : : GetProjectFilePath ( ) const
Add a rehydration command to the stand alone virtualization tool, making it easier to reverse the effects of asset virtualization. Unlike previous processes, this one does not require that we load the package and will just manipulate the data storaged in the package trailer.
#rb Sebastian.Nordgren
#rnx
#jira UE-156436
#preflight 62c287f9a3568e30664eb94f
### VA Standalone Tool
- We now plan to add much more functionality to the tool than just virtualizing and submitting changelists, so to make this easier I am moving the tool towards a design where it should be fairly easy to add new functionality.
- Added FCommand, which is a base class for adding new functionality, simple derive from FCommand and hook it up at the appropriate locations.
-- In the future it should be possible for new command types to automatically register themselves to be initiated from the command line. There should be no need to edit UnrealVirtualizationToolApp to add a new command but this will be done as an additional work item.
-- At the moment FCommand comes with a number of utility methods to call that cover some common source control commands.
-- The original functionality has not yet been moved to the command system and so the code is a little bit weird at the moment. Updating older code to the new system will be done as an additional work item.
- FProject/FPlugin have been moved to their own code files.
### Rehydrate Command
- The rehydrate command will take a number of packages, check them out of source control and then attempt to virtualize them.
- At the moment the chekout logic is fairly basic, we just check out every package supplied, we don't check if the package is virtualized or not yet. This can be improved in additional work items. Ideally by the end of command the only packages that we have checked out should also be rehydrated.
- At the moment the command can either take a path of a specific package, a path of a directory to find packages in, or a changelist containing packages that should be rehydrated.
- A cleint spec (workspace) can optionally be provided, but if not supplied we will attempt to find a client spec for which to check out the packages.
- Currently we will check out the packages to the default change list.
### Rehydrate process
- Added the rehydration process in it's own code files in the virtualization module. Like the virtualization process this is exposed in a public header file and no via the Core interface which means it is very specific to our module/implementation.
- The process expects that the caller will have checked out any required packages from source control. It will treat being unable to update a package file as an error.
- Added PackageUtils.h/.cpp and moved some of the generic code from the virtualization process code there so that it can be shared by the rehydration process.
### Misc
Moving away from the using things like FPackagePath as that requires that the correct mount points have been registered for a project and at the moment (with the flakiness of FConfig*) it seems that the best idea would be to prefer absolute file paths where possible.
[CL 20982284 by paul chipchase in ue5-main branch]
2022-07-07 06:54:33 -04:00
{
return ProjectFilePath ;
}
FStringView FProject : : GetProjectName ( ) const
{
return FPathViews : : GetBaseFilename ( ProjectFilePath ) ;
}
FStringView FProject : : GetProjectRoot ( ) const
{
return FPathViews : : GetPath ( ProjectFilePath ) ;
}
TArray < FString > FProject : : GetAllPackages ( ) const
{
TArray < FString > Packages = PackagePaths ;
for ( const FPlugin & Plugin : Plugins )
{
Packages . Append ( Plugin . PackagePaths ) ;
}
return Packages ;
}
void FProject : : RegisterMountPoints ( ) const
{
TRACE_CPUPROFILER_EVENT_SCOPE ( FProject : : RegisterMountPoints ) ;
FString ProjectRootPath ;
FString ProjectContentPath ;
ConvertToMountPoint ( ProjectFilePath , ProjectRootPath , ProjectContentPath ) ;
FPackageName : : RegisterMountPoint ( ProjectRootPath , ProjectContentPath ) ;
for ( const FPlugin & Plugin : Plugins )
{
FString PluginRootPath ;
FString PluginContentPath ;
ConvertToMountPoint ( Plugin . PluginFilePath , PluginRootPath , PluginContentPath ) ;
FPackageName : : RegisterMountPoint ( PluginRootPath , PluginContentPath ) ;
}
}
void FProject : : UnRegisterMountPoints ( ) const
{
TRACE_CPUPROFILER_EVENT_SCOPE ( FProject : : UnRegisterMountPoints ) ;
for ( const FPlugin & Plugin : Plugins )
{
FString PluginRootPath ;
FString PluginContentPath ;
ConvertToMountPoint ( Plugin . PluginFilePath , PluginRootPath , PluginContentPath ) ;
FPackageName : : UnRegisterMountPoint ( PluginRootPath , PluginContentPath ) ;
}
FString ProjectRootPath ;
FString ProjectContentPath ;
ConvertToMountPoint ( ProjectFilePath , ProjectRootPath , ProjectContentPath ) ;
FPackageName : : UnRegisterMountPoint ( ProjectRootPath , ProjectContentPath ) ;
}
bool FProject : : TryLoadConfig ( FConfigFile & OutConfig ) const
{
TRACE_CPUPROFILER_EVENT_SCOPE ( FProject : : TryLoadConfig ) ;
const FString ProjectPath = FPaths : : GetPath ( ProjectFilePath ) ;
const FString EngineConfigPath = FPaths : : Combine ( FPaths : : EngineDir ( ) , TEXT ( " Config/ " ) ) ;
const FString ProjectConfigPath = FPaths : : Combine ( ProjectPath , TEXT ( " Config/ " ) ) ;
OutConfig . Reset ( ) ;
if ( ! FConfigCacheIni : : LoadExternalIniFile ( OutConfig , TEXT ( " Engine " ) , * EngineConfigPath , * ProjectConfigPath , true ) )
{
UE_LOG ( LogVirtualizationTool , Error , TEXT ( " Failed to load config files for the project '%s " ) , * ProjectFilePath ) ;
return false ;
}
// Note that the following is taken from UGameFeatureData::InitializeHierarchicalPluginIniFiles as with
// ReloadConfigs if we decide to keep filtering at submission time, rather than save time then we should
// probably move this code to a shared location rather than the copy/paste.
for ( const FPlugin & Plugin : Plugins )
{
const FString PluginName = FPaths : : GetBaseFilename ( Plugin . PluginFilePath ) ;
const FString PluginIniName = PluginName + TEXT ( " Engine " ) ;
const FString PluginPath = FPaths : : GetPath ( Plugin . PluginFilePath ) ;
const FString PluginConfigPath = FPaths : : Combine ( PluginPath , TEXT ( " Config/ " ) ) ;
FConfigFile PluginConfig ;
if ( FConfigCacheIni : : LoadExternalIniFile ( PluginConfig , * PluginIniName , * EngineConfigPath , * PluginConfigPath , false ) & & ( PluginConfig . Num ( ) > 0 ) )
{
const FString IniFile = GConfig - > GetConfigFilename ( TEXT ( " Engine " ) ) ;
if ( FConfigFile * ExistingConfig = GConfig - > FindConfigFile ( IniFile ) )
{
const FString PluginIniPath = FString : : Printf ( TEXT ( " %s%s.ini " ) , * PluginConfigPath , * PluginIniName ) ;
if ( ExistingConfig - > Combine ( PluginIniPath ) )
{
ReloadConfigs ( PluginConfig ) ;
}
}
}
}
return true ;
}
} // namespace UE::Virtualization