Files
UnrealEngineUWP/Engine/Source/Programs/UnrealVirtualizationTool/Private/UnrealVirtualizationToolApp.cpp

793 lines
24 KiB
C++
Raw Normal View History

// Copyright Epic Games, Inc. All Rights Reserved.
#include "UnrealVirtualizationToolApp.h"
Add a rehydration command to the stand alone virtualization tool, making it easier to reverse the effects of asset virtualization. Unlike previous processes, this one does not require that we load the package and will just manipulate the data storaged in the package trailer. #rb Sebastian.Nordgren #rnx #jira UE-156436 #preflight 62c287f9a3568e30664eb94f ### VA Standalone Tool - We now plan to add much more functionality to the tool than just virtualizing and submitting changelists, so to make this easier I am moving the tool towards a design where it should be fairly easy to add new functionality. - Added FCommand, which is a base class for adding new functionality, simple derive from FCommand and hook it up at the appropriate locations. -- In the future it should be possible for new command types to automatically register themselves to be initiated from the command line. There should be no need to edit UnrealVirtualizationToolApp to add a new command but this will be done as an additional work item. -- At the moment FCommand comes with a number of utility methods to call that cover some common source control commands. -- The original functionality has not yet been moved to the command system and so the code is a little bit weird at the moment. Updating older code to the new system will be done as an additional work item. - FProject/FPlugin have been moved to their own code files. ### Rehydrate Command - The rehydrate command will take a number of packages, check them out of source control and then attempt to virtualize them. - At the moment the chekout logic is fairly basic, we just check out every package supplied, we don't check if the package is virtualized or not yet. This can be improved in additional work items. Ideally by the end of command the only packages that we have checked out should also be rehydrated. - At the moment the command can either take a path of a specific package, a path of a directory to find packages in, or a changelist containing packages that should be rehydrated. - A cleint spec (workspace) can optionally be provided, but if not supplied we will attempt to find a client spec for which to check out the packages. - Currently we will check out the packages to the default change list. ### Rehydrate process - Added the rehydration process in it's own code files in the virtualization module. Like the virtualization process this is exposed in a public header file and no via the Core interface which means it is very specific to our module/implementation. - The process expects that the caller will have checked out any required packages from source control. It will treat being unable to update a package file as an error. - Added PackageUtils.h/.cpp and moved some of the generic code from the virtualization process code there so that it can be shared by the rehydration process. ### Misc Moving away from the using things like FPackagePath as that requires that the correct mount points have been registered for a project and at the moment (with the flakiness of FConfig*) it seems that the best idea would be to prefer absolute file paths where possible. [CL 20982284 by paul chipchase in ue5-main branch]
2022-07-07 06:54:33 -04:00
#include "Commands/CommandBase.h"
#include "Commands/RehydrateCommand.h"
Extend the VA stand-alone tool to work on both changelists (the existing functionality) and lists of packages stored in a text file (the new functionality) #rb PJ.Kack #jira UE-143675 #rnx #preflight 624c6a1814634fba5c7e3d71 ### Rationale - The new mode being added allows us to write a list of package file paths to be virtualized to a file on disk and then the file to be passed to the tool so that it knows what to virtualize. This new mode will then be used by P4VUtils.dll which is how we will run the virtualization process via p4v custom tools. - We need this second mode because: -- A workspace could have multiple projects in it, which means a user could try to virtualize a changelist with packages from more than one project. -- Projects can be associated with different engine installations so the user could try to virtualize a changelist with packages using different engine installations, which in turn means that they need to be virtualized by different versions of the stand-alone tool. -- This means we cannot rely on passing the CL number to the tool, install P4VUtils.dll will need to sort the package paths by engine installation, then pass those paths to the correct version of the tool. -- The paths need to be written to a text file to be passed to the tool as the total length of path data can easily exceed the commandline max length limit if someone were to try and virtualize an entire project in one go for example. - Note that the new mode does NOT submit the package files since there might be multiple calls to different versions of the stand-alone tool. In theory if all the packages are under one engine install (which is almost certainly going to be 99.999% of all submits) we could allow the tool to do the submit but that means we need to maintain two different code paths for the same operation. - The existing mode (virtualize and submit a changelist) remains unchanged. It is not likely that an end user will ever use this mode, but it will make for a useful debugging tool in the future. ### Changes - When the tool first starts we now print the log path to the user so that they can more easily find it. This might be something that we want to remove after a while, but during the initial release of the tool it should make getting good debug info back from users with problems somewhat easier. - Some logging and error messages have had small updates made, to make the final output to the user more readable (based on the many many times this tool was run while testing). In some places additional logging has been added where I felt it was not clear to the user as to what was going on. - Connecting to source control has been moved from the changelist parsing code to it's own method ::TryConnectToSourceControl since all modes will require it. - The cmdline help output has been redone to support the multiple command line options we now have. - Although we could infer the mode to run based on the commandline options, we require the users to explicitly state the mode via -Mode=XXX. In general the tool will be invoked from a gui so adding extra commands doesn't really affect the usability of the system and it makes things a bit clearer. [CL 19665307 by paul chipchase in ue5-main branch]
2022-04-07 02:08:03 -04:00
#include "GenericPlatform/GenericPlatformOutputDevices.h"
#include "HAL/FeedbackContextAnsi.h"
#include "HAL/FileManager.h"
#include "Interfaces/IPluginManager.h"
#include "ISourceControlModule.h"
#include "ISourceControlProvider.h"
#include "Misc/CommandLine.h"
#include "Misc/ConfigCacheIni.h"
Extend the VA stand-alone tool to work on both changelists (the existing functionality) and lists of packages stored in a text file (the new functionality) #rb PJ.Kack #jira UE-143675 #rnx #preflight 624c6a1814634fba5c7e3d71 ### Rationale - The new mode being added allows us to write a list of package file paths to be virtualized to a file on disk and then the file to be passed to the tool so that it knows what to virtualize. This new mode will then be used by P4VUtils.dll which is how we will run the virtualization process via p4v custom tools. - We need this second mode because: -- A workspace could have multiple projects in it, which means a user could try to virtualize a changelist with packages from more than one project. -- Projects can be associated with different engine installations so the user could try to virtualize a changelist with packages using different engine installations, which in turn means that they need to be virtualized by different versions of the stand-alone tool. -- This means we cannot rely on passing the CL number to the tool, install P4VUtils.dll will need to sort the package paths by engine installation, then pass those paths to the correct version of the tool. -- The paths need to be written to a text file to be passed to the tool as the total length of path data can easily exceed the commandline max length limit if someone were to try and virtualize an entire project in one go for example. - Note that the new mode does NOT submit the package files since there might be multiple calls to different versions of the stand-alone tool. In theory if all the packages are under one engine install (which is almost certainly going to be 99.999% of all submits) we could allow the tool to do the submit but that means we need to maintain two different code paths for the same operation. - The existing mode (virtualize and submit a changelist) remains unchanged. It is not likely that an end user will ever use this mode, but it will make for a useful debugging tool in the future. ### Changes - When the tool first starts we now print the log path to the user so that they can more easily find it. This might be something that we want to remove after a while, but during the initial release of the tool it should make getting good debug info back from users with problems somewhat easier. - Some logging and error messages have had small updates made, to make the final output to the user more readable (based on the many many times this tool was run while testing). In some places additional logging has been added where I felt it was not clear to the user as to what was going on. - Connecting to source control has been moved from the changelist parsing code to it's own method ::TryConnectToSourceControl since all modes will require it. - The cmdline help output has been redone to support the multiple command line options we now have. - Although we could infer the mode to run based on the commandline options, we require the users to explicitly state the mode via -Mode=XXX. In general the tool will be invoked from a gui so adding extra commands doesn't really affect the usability of the system and it makes things a bit clearer. [CL 19665307 by paul chipchase in ue5-main branch]
2022-04-07 02:08:03 -04:00
#include "Misc/FileHelper.h"
#include "Misc/Parse.h"
#include "Misc/Paths.h"
#include "Misc/PathViews.h"
#include "SourceControlInitSettings.h"
#include "SourceControlOperations.h"
#include "UnrealVirtualizationTool.h"
#include "Virtualization/VirtualizationSystem.h"
namespace
{
/** Utility for testing if a file path resolves to a valid package file or not */
bool IsPackageFile(const FString FilePath)
{
// ::IsPackageExtension requires a TCHAR so we cannot use FPathViews here
const FString Extension = FPaths::GetExtension(FilePath);
// Currently we don't virtualize text based assets so no call to FPackageName::IsTextPackageExtension
return FPackageName::IsPackageExtension(*Extension);
}
/**
* Utility to clean up the tags we got from the virtualization system. Convert the FText to FString and
* discard any duplicate entries.
*/
TArray<FString> BuildFinalTagDescriptions(TArray<FText>& DescriptionTags)
{
TArray<FString> CleanedDescriptions;
CleanedDescriptions.Reserve(DescriptionTags.Num());
for (const FText& Tag : DescriptionTags)
{
CleanedDescriptions.AddUnique(Tag.ToString());
}
return CleanedDescriptions;
}
Extend the VA stand-alone tool to work on both changelists (the existing functionality) and lists of packages stored in a text file (the new functionality) #rb PJ.Kack #jira UE-143675 #rnx #preflight 624c6a1814634fba5c7e3d71 ### Rationale - The new mode being added allows us to write a list of package file paths to be virtualized to a file on disk and then the file to be passed to the tool so that it knows what to virtualize. This new mode will then be used by P4VUtils.dll which is how we will run the virtualization process via p4v custom tools. - We need this second mode because: -- A workspace could have multiple projects in it, which means a user could try to virtualize a changelist with packages from more than one project. -- Projects can be associated with different engine installations so the user could try to virtualize a changelist with packages using different engine installations, which in turn means that they need to be virtualized by different versions of the stand-alone tool. -- This means we cannot rely on passing the CL number to the tool, install P4VUtils.dll will need to sort the package paths by engine installation, then pass those paths to the correct version of the tool. -- The paths need to be written to a text file to be passed to the tool as the total length of path data can easily exceed the commandline max length limit if someone were to try and virtualize an entire project in one go for example. - Note that the new mode does NOT submit the package files since there might be multiple calls to different versions of the stand-alone tool. In theory if all the packages are under one engine install (which is almost certainly going to be 99.999% of all submits) we could allow the tool to do the submit but that means we need to maintain two different code paths for the same operation. - The existing mode (virtualize and submit a changelist) remains unchanged. It is not likely that an end user will ever use this mode, but it will make for a useful debugging tool in the future. ### Changes - When the tool first starts we now print the log path to the user so that they can more easily find it. This might be something that we want to remove after a while, but during the initial release of the tool it should make getting good debug info back from users with problems somewhat easier. - Some logging and error messages have had small updates made, to make the final output to the user more readable (based on the many many times this tool was run while testing). In some places additional logging has been added where I felt it was not clear to the user as to what was going on. - Connecting to source control has been moved from the changelist parsing code to it's own method ::TryConnectToSourceControl since all modes will require it. - The cmdline help output has been redone to support the multiple command line options we now have. - Although we could infer the mode to run based on the commandline options, we require the users to explicitly state the mode via -Mode=XXX. In general the tool will be invoked from a gui so adding extra commands doesn't really affect the usability of the system and it makes things a bit clearer. [CL 19665307 by paul chipchase in ue5-main branch]
2022-04-07 02:08:03 -04:00
/** Utility to get EMode from a string */
void LexFromString(UE::Virtualization::EMode& OutValue, const FStringView& InString)
Extend the VA stand-alone tool to work on both changelists (the existing functionality) and lists of packages stored in a text file (the new functionality) #rb PJ.Kack #jira UE-143675 #rnx #preflight 624c6a1814634fba5c7e3d71 ### Rationale - The new mode being added allows us to write a list of package file paths to be virtualized to a file on disk and then the file to be passed to the tool so that it knows what to virtualize. This new mode will then be used by P4VUtils.dll which is how we will run the virtualization process via p4v custom tools. - We need this second mode because: -- A workspace could have multiple projects in it, which means a user could try to virtualize a changelist with packages from more than one project. -- Projects can be associated with different engine installations so the user could try to virtualize a changelist with packages using different engine installations, which in turn means that they need to be virtualized by different versions of the stand-alone tool. -- This means we cannot rely on passing the CL number to the tool, install P4VUtils.dll will need to sort the package paths by engine installation, then pass those paths to the correct version of the tool. -- The paths need to be written to a text file to be passed to the tool as the total length of path data can easily exceed the commandline max length limit if someone were to try and virtualize an entire project in one go for example. - Note that the new mode does NOT submit the package files since there might be multiple calls to different versions of the stand-alone tool. In theory if all the packages are under one engine install (which is almost certainly going to be 99.999% of all submits) we could allow the tool to do the submit but that means we need to maintain two different code paths for the same operation. - The existing mode (virtualize and submit a changelist) remains unchanged. It is not likely that an end user will ever use this mode, but it will make for a useful debugging tool in the future. ### Changes - When the tool first starts we now print the log path to the user so that they can more easily find it. This might be something that we want to remove after a while, but during the initial release of the tool it should make getting good debug info back from users with problems somewhat easier. - Some logging and error messages have had small updates made, to make the final output to the user more readable (based on the many many times this tool was run while testing). In some places additional logging has been added where I felt it was not clear to the user as to what was going on. - Connecting to source control has been moved from the changelist parsing code to it's own method ::TryConnectToSourceControl since all modes will require it. - The cmdline help output has been redone to support the multiple command line options we now have. - Although we could infer the mode to run based on the commandline options, we require the users to explicitly state the mode via -Mode=XXX. In general the tool will be invoked from a gui so adding extra commands doesn't really affect the usability of the system and it makes things a bit clearer. [CL 19665307 by paul chipchase in ue5-main branch]
2022-04-07 02:08:03 -04:00
{
if (InString == TEXT("Changelist"))
{
OutValue = UE::Virtualization::EMode::Changelist;
Extend the VA stand-alone tool to work on both changelists (the existing functionality) and lists of packages stored in a text file (the new functionality) #rb PJ.Kack #jira UE-143675 #rnx #preflight 624c6a1814634fba5c7e3d71 ### Rationale - The new mode being added allows us to write a list of package file paths to be virtualized to a file on disk and then the file to be passed to the tool so that it knows what to virtualize. This new mode will then be used by P4VUtils.dll which is how we will run the virtualization process via p4v custom tools. - We need this second mode because: -- A workspace could have multiple projects in it, which means a user could try to virtualize a changelist with packages from more than one project. -- Projects can be associated with different engine installations so the user could try to virtualize a changelist with packages using different engine installations, which in turn means that they need to be virtualized by different versions of the stand-alone tool. -- This means we cannot rely on passing the CL number to the tool, install P4VUtils.dll will need to sort the package paths by engine installation, then pass those paths to the correct version of the tool. -- The paths need to be written to a text file to be passed to the tool as the total length of path data can easily exceed the commandline max length limit if someone were to try and virtualize an entire project in one go for example. - Note that the new mode does NOT submit the package files since there might be multiple calls to different versions of the stand-alone tool. In theory if all the packages are under one engine install (which is almost certainly going to be 99.999% of all submits) we could allow the tool to do the submit but that means we need to maintain two different code paths for the same operation. - The existing mode (virtualize and submit a changelist) remains unchanged. It is not likely that an end user will ever use this mode, but it will make for a useful debugging tool in the future. ### Changes - When the tool first starts we now print the log path to the user so that they can more easily find it. This might be something that we want to remove after a while, but during the initial release of the tool it should make getting good debug info back from users with problems somewhat easier. - Some logging and error messages have had small updates made, to make the final output to the user more readable (based on the many many times this tool was run while testing). In some places additional logging has been added where I felt it was not clear to the user as to what was going on. - Connecting to source control has been moved from the changelist parsing code to it's own method ::TryConnectToSourceControl since all modes will require it. - The cmdline help output has been redone to support the multiple command line options we now have. - Although we could infer the mode to run based on the commandline options, we require the users to explicitly state the mode via -Mode=XXX. In general the tool will be invoked from a gui so adding extra commands doesn't really affect the usability of the system and it makes things a bit clearer. [CL 19665307 by paul chipchase in ue5-main branch]
2022-04-07 02:08:03 -04:00
}
else if (InString == TEXT("PackageList"))
{
OutValue = UE::Virtualization::EMode::PackageList;
Extend the VA stand-alone tool to work on both changelists (the existing functionality) and lists of packages stored in a text file (the new functionality) #rb PJ.Kack #jira UE-143675 #rnx #preflight 624c6a1814634fba5c7e3d71 ### Rationale - The new mode being added allows us to write a list of package file paths to be virtualized to a file on disk and then the file to be passed to the tool so that it knows what to virtualize. This new mode will then be used by P4VUtils.dll which is how we will run the virtualization process via p4v custom tools. - We need this second mode because: -- A workspace could have multiple projects in it, which means a user could try to virtualize a changelist with packages from more than one project. -- Projects can be associated with different engine installations so the user could try to virtualize a changelist with packages using different engine installations, which in turn means that they need to be virtualized by different versions of the stand-alone tool. -- This means we cannot rely on passing the CL number to the tool, install P4VUtils.dll will need to sort the package paths by engine installation, then pass those paths to the correct version of the tool. -- The paths need to be written to a text file to be passed to the tool as the total length of path data can easily exceed the commandline max length limit if someone were to try and virtualize an entire project in one go for example. - Note that the new mode does NOT submit the package files since there might be multiple calls to different versions of the stand-alone tool. In theory if all the packages are under one engine install (which is almost certainly going to be 99.999% of all submits) we could allow the tool to do the submit but that means we need to maintain two different code paths for the same operation. - The existing mode (virtualize and submit a changelist) remains unchanged. It is not likely that an end user will ever use this mode, but it will make for a useful debugging tool in the future. ### Changes - When the tool first starts we now print the log path to the user so that they can more easily find it. This might be something that we want to remove after a while, but during the initial release of the tool it should make getting good debug info back from users with problems somewhat easier. - Some logging and error messages have had small updates made, to make the final output to the user more readable (based on the many many times this tool was run while testing). In some places additional logging has been added where I felt it was not clear to the user as to what was going on. - Connecting to source control has been moved from the changelist parsing code to it's own method ::TryConnectToSourceControl since all modes will require it. - The cmdline help output has been redone to support the multiple command line options we now have. - Although we could infer the mode to run based on the commandline options, we require the users to explicitly state the mode via -Mode=XXX. In general the tool will be invoked from a gui so adding extra commands doesn't really affect the usability of the system and it makes things a bit clearer. [CL 19665307 by paul chipchase in ue5-main branch]
2022-04-07 02:08:03 -04:00
}
Add a rehydration command to the stand alone virtualization tool, making it easier to reverse the effects of asset virtualization. Unlike previous processes, this one does not require that we load the package and will just manipulate the data storaged in the package trailer. #rb Sebastian.Nordgren #rnx #jira UE-156436 #preflight 62c287f9a3568e30664eb94f ### VA Standalone Tool - We now plan to add much more functionality to the tool than just virtualizing and submitting changelists, so to make this easier I am moving the tool towards a design where it should be fairly easy to add new functionality. - Added FCommand, which is a base class for adding new functionality, simple derive from FCommand and hook it up at the appropriate locations. -- In the future it should be possible for new command types to automatically register themselves to be initiated from the command line. There should be no need to edit UnrealVirtualizationToolApp to add a new command but this will be done as an additional work item. -- At the moment FCommand comes with a number of utility methods to call that cover some common source control commands. -- The original functionality has not yet been moved to the command system and so the code is a little bit weird at the moment. Updating older code to the new system will be done as an additional work item. - FProject/FPlugin have been moved to their own code files. ### Rehydrate Command - The rehydrate command will take a number of packages, check them out of source control and then attempt to virtualize them. - At the moment the chekout logic is fairly basic, we just check out every package supplied, we don't check if the package is virtualized or not yet. This can be improved in additional work items. Ideally by the end of command the only packages that we have checked out should also be rehydrated. - At the moment the command can either take a path of a specific package, a path of a directory to find packages in, or a changelist containing packages that should be rehydrated. - A cleint spec (workspace) can optionally be provided, but if not supplied we will attempt to find a client spec for which to check out the packages. - Currently we will check out the packages to the default change list. ### Rehydrate process - Added the rehydration process in it's own code files in the virtualization module. Like the virtualization process this is exposed in a public header file and no via the Core interface which means it is very specific to our module/implementation. - The process expects that the caller will have checked out any required packages from source control. It will treat being unable to update a package file as an error. - Added PackageUtils.h/.cpp and moved some of the generic code from the virtualization process code there so that it can be shared by the rehydration process. ### Misc Moving away from the using things like FPackagePath as that requires that the correct mount points have been registered for a project and at the moment (with the flakiness of FConfig*) it seems that the best idea would be to prefer absolute file paths where possible. [CL 20982284 by paul chipchase in ue5-main branch]
2022-07-07 06:54:33 -04:00
else if (InString == TEXT("Rehydrate"))
{
OutValue = UE::Virtualization::EMode::Rehydrate;
Add a rehydration command to the stand alone virtualization tool, making it easier to reverse the effects of asset virtualization. Unlike previous processes, this one does not require that we load the package and will just manipulate the data storaged in the package trailer. #rb Sebastian.Nordgren #rnx #jira UE-156436 #preflight 62c287f9a3568e30664eb94f ### VA Standalone Tool - We now plan to add much more functionality to the tool than just virtualizing and submitting changelists, so to make this easier I am moving the tool towards a design where it should be fairly easy to add new functionality. - Added FCommand, which is a base class for adding new functionality, simple derive from FCommand and hook it up at the appropriate locations. -- In the future it should be possible for new command types to automatically register themselves to be initiated from the command line. There should be no need to edit UnrealVirtualizationToolApp to add a new command but this will be done as an additional work item. -- At the moment FCommand comes with a number of utility methods to call that cover some common source control commands. -- The original functionality has not yet been moved to the command system and so the code is a little bit weird at the moment. Updating older code to the new system will be done as an additional work item. - FProject/FPlugin have been moved to their own code files. ### Rehydrate Command - The rehydrate command will take a number of packages, check them out of source control and then attempt to virtualize them. - At the moment the chekout logic is fairly basic, we just check out every package supplied, we don't check if the package is virtualized or not yet. This can be improved in additional work items. Ideally by the end of command the only packages that we have checked out should also be rehydrated. - At the moment the command can either take a path of a specific package, a path of a directory to find packages in, or a changelist containing packages that should be rehydrated. - A cleint spec (workspace) can optionally be provided, but if not supplied we will attempt to find a client spec for which to check out the packages. - Currently we will check out the packages to the default change list. ### Rehydrate process - Added the rehydration process in it's own code files in the virtualization module. Like the virtualization process this is exposed in a public header file and no via the Core interface which means it is very specific to our module/implementation. - The process expects that the caller will have checked out any required packages from source control. It will treat being unable to update a package file as an error. - Added PackageUtils.h/.cpp and moved some of the generic code from the virtualization process code there so that it can be shared by the rehydration process. ### Misc Moving away from the using things like FPackagePath as that requires that the correct mount points have been registered for a project and at the moment (with the flakiness of FConfig*) it seems that the best idea would be to prefer absolute file paths where possible. [CL 20982284 by paul chipchase in ue5-main branch]
2022-07-07 06:54:33 -04:00
}
Extend the VA stand-alone tool to work on both changelists (the existing functionality) and lists of packages stored in a text file (the new functionality) #rb PJ.Kack #jira UE-143675 #rnx #preflight 624c6a1814634fba5c7e3d71 ### Rationale - The new mode being added allows us to write a list of package file paths to be virtualized to a file on disk and then the file to be passed to the tool so that it knows what to virtualize. This new mode will then be used by P4VUtils.dll which is how we will run the virtualization process via p4v custom tools. - We need this second mode because: -- A workspace could have multiple projects in it, which means a user could try to virtualize a changelist with packages from more than one project. -- Projects can be associated with different engine installations so the user could try to virtualize a changelist with packages using different engine installations, which in turn means that they need to be virtualized by different versions of the stand-alone tool. -- This means we cannot rely on passing the CL number to the tool, install P4VUtils.dll will need to sort the package paths by engine installation, then pass those paths to the correct version of the tool. -- The paths need to be written to a text file to be passed to the tool as the total length of path data can easily exceed the commandline max length limit if someone were to try and virtualize an entire project in one go for example. - Note that the new mode does NOT submit the package files since there might be multiple calls to different versions of the stand-alone tool. In theory if all the packages are under one engine install (which is almost certainly going to be 99.999% of all submits) we could allow the tool to do the submit but that means we need to maintain two different code paths for the same operation. - The existing mode (virtualize and submit a changelist) remains unchanged. It is not likely that an end user will ever use this mode, but it will make for a useful debugging tool in the future. ### Changes - When the tool first starts we now print the log path to the user so that they can more easily find it. This might be something that we want to remove after a while, but during the initial release of the tool it should make getting good debug info back from users with problems somewhat easier. - Some logging and error messages have had small updates made, to make the final output to the user more readable (based on the many many times this tool was run while testing). In some places additional logging has been added where I felt it was not clear to the user as to what was going on. - Connecting to source control has been moved from the changelist parsing code to it's own method ::TryConnectToSourceControl since all modes will require it. - The cmdline help output has been redone to support the multiple command line options we now have. - Although we could infer the mode to run based on the commandline options, we require the users to explicitly state the mode via -Mode=XXX. In general the tool will be invoked from a gui so adding extra commands doesn't really affect the usability of the system and it makes things a bit clearer. [CL 19665307 by paul chipchase in ue5-main branch]
2022-04-07 02:08:03 -04:00
else
{
OutValue = UE::Virtualization::EMode::Unknown;
Extend the VA stand-alone tool to work on both changelists (the existing functionality) and lists of packages stored in a text file (the new functionality) #rb PJ.Kack #jira UE-143675 #rnx #preflight 624c6a1814634fba5c7e3d71 ### Rationale - The new mode being added allows us to write a list of package file paths to be virtualized to a file on disk and then the file to be passed to the tool so that it knows what to virtualize. This new mode will then be used by P4VUtils.dll which is how we will run the virtualization process via p4v custom tools. - We need this second mode because: -- A workspace could have multiple projects in it, which means a user could try to virtualize a changelist with packages from more than one project. -- Projects can be associated with different engine installations so the user could try to virtualize a changelist with packages using different engine installations, which in turn means that they need to be virtualized by different versions of the stand-alone tool. -- This means we cannot rely on passing the CL number to the tool, install P4VUtils.dll will need to sort the package paths by engine installation, then pass those paths to the correct version of the tool. -- The paths need to be written to a text file to be passed to the tool as the total length of path data can easily exceed the commandline max length limit if someone were to try and virtualize an entire project in one go for example. - Note that the new mode does NOT submit the package files since there might be multiple calls to different versions of the stand-alone tool. In theory if all the packages are under one engine install (which is almost certainly going to be 99.999% of all submits) we could allow the tool to do the submit but that means we need to maintain two different code paths for the same operation. - The existing mode (virtualize and submit a changelist) remains unchanged. It is not likely that an end user will ever use this mode, but it will make for a useful debugging tool in the future. ### Changes - When the tool first starts we now print the log path to the user so that they can more easily find it. This might be something that we want to remove after a while, but during the initial release of the tool it should make getting good debug info back from users with problems somewhat easier. - Some logging and error messages have had small updates made, to make the final output to the user more readable (based on the many many times this tool was run while testing). In some places additional logging has been added where I felt it was not clear to the user as to what was going on. - Connecting to source control has been moved from the changelist parsing code to it's own method ::TryConnectToSourceControl since all modes will require it. - The cmdline help output has been redone to support the multiple command line options we now have. - Although we could infer the mode to run based on the commandline options, we require the users to explicitly state the mode via -Mode=XXX. In general the tool will be invoked from a gui so adding extra commands doesn't really affect the usability of the system and it makes things a bit clearer. [CL 19665307 by paul chipchase in ue5-main branch]
2022-04-07 02:08:03 -04:00
}
}
Add a rehydration command to the stand alone virtualization tool, making it easier to reverse the effects of asset virtualization. Unlike previous processes, this one does not require that we load the package and will just manipulate the data storaged in the package trailer. #rb Sebastian.Nordgren #rnx #jira UE-156436 #preflight 62c287f9a3568e30664eb94f ### VA Standalone Tool - We now plan to add much more functionality to the tool than just virtualizing and submitting changelists, so to make this easier I am moving the tool towards a design where it should be fairly easy to add new functionality. - Added FCommand, which is a base class for adding new functionality, simple derive from FCommand and hook it up at the appropriate locations. -- In the future it should be possible for new command types to automatically register themselves to be initiated from the command line. There should be no need to edit UnrealVirtualizationToolApp to add a new command but this will be done as an additional work item. -- At the moment FCommand comes with a number of utility methods to call that cover some common source control commands. -- The original functionality has not yet been moved to the command system and so the code is a little bit weird at the moment. Updating older code to the new system will be done as an additional work item. - FProject/FPlugin have been moved to their own code files. ### Rehydrate Command - The rehydrate command will take a number of packages, check them out of source control and then attempt to virtualize them. - At the moment the chekout logic is fairly basic, we just check out every package supplied, we don't check if the package is virtualized or not yet. This can be improved in additional work items. Ideally by the end of command the only packages that we have checked out should also be rehydrated. - At the moment the command can either take a path of a specific package, a path of a directory to find packages in, or a changelist containing packages that should be rehydrated. - A cleint spec (workspace) can optionally be provided, but if not supplied we will attempt to find a client spec for which to check out the packages. - Currently we will check out the packages to the default change list. ### Rehydrate process - Added the rehydration process in it's own code files in the virtualization module. Like the virtualization process this is exposed in a public header file and no via the Core interface which means it is very specific to our module/implementation. - The process expects that the caller will have checked out any required packages from source control. It will treat being unable to update a package file as an error. - Added PackageUtils.h/.cpp and moved some of the generic code from the virtualization process code there so that it can be shared by the rehydration process. ### Misc Moving away from the using things like FPackagePath as that requires that the correct mount points have been registered for a project and at the moment (with the flakiness of FConfig*) it seems that the best idea would be to prefer absolute file paths where possible. [CL 20982284 by paul chipchase in ue5-main branch]
2022-07-07 06:54:33 -04:00
/** Utility for creating a new command */
template<typename CommandType>
TUniquePtr<UE::Virtualization::FCommand> CreateCommand(const FString& ModeName, const TCHAR* CmdLine)
Add a rehydration command to the stand alone virtualization tool, making it easier to reverse the effects of asset virtualization. Unlike previous processes, this one does not require that we load the package and will just manipulate the data storaged in the package trailer. #rb Sebastian.Nordgren #rnx #jira UE-156436 #preflight 62c287f9a3568e30664eb94f ### VA Standalone Tool - We now plan to add much more functionality to the tool than just virtualizing and submitting changelists, so to make this easier I am moving the tool towards a design where it should be fairly easy to add new functionality. - Added FCommand, which is a base class for adding new functionality, simple derive from FCommand and hook it up at the appropriate locations. -- In the future it should be possible for new command types to automatically register themselves to be initiated from the command line. There should be no need to edit UnrealVirtualizationToolApp to add a new command but this will be done as an additional work item. -- At the moment FCommand comes with a number of utility methods to call that cover some common source control commands. -- The original functionality has not yet been moved to the command system and so the code is a little bit weird at the moment. Updating older code to the new system will be done as an additional work item. - FProject/FPlugin have been moved to their own code files. ### Rehydrate Command - The rehydrate command will take a number of packages, check them out of source control and then attempt to virtualize them. - At the moment the chekout logic is fairly basic, we just check out every package supplied, we don't check if the package is virtualized or not yet. This can be improved in additional work items. Ideally by the end of command the only packages that we have checked out should also be rehydrated. - At the moment the command can either take a path of a specific package, a path of a directory to find packages in, or a changelist containing packages that should be rehydrated. - A cleint spec (workspace) can optionally be provided, but if not supplied we will attempt to find a client spec for which to check out the packages. - Currently we will check out the packages to the default change list. ### Rehydrate process - Added the rehydration process in it's own code files in the virtualization module. Like the virtualization process this is exposed in a public header file and no via the Core interface which means it is very specific to our module/implementation. - The process expects that the caller will have checked out any required packages from source control. It will treat being unable to update a package file as an error. - Added PackageUtils.h/.cpp and moved some of the generic code from the virtualization process code there so that it can be shared by the rehydration process. ### Misc Moving away from the using things like FPackagePath as that requires that the correct mount points have been registered for a project and at the moment (with the flakiness of FConfig*) it seems that the best idea would be to prefer absolute file paths where possible. [CL 20982284 by paul chipchase in ue5-main branch]
2022-07-07 06:54:33 -04:00
{
UE_LOG(LogVirtualizationTool, Display, TEXT("Attempting to initialize command '%s'..."), *ModeName);
TUniquePtr<UE::Virtualization::FCommand> Command = MakeUnique<CommandType>(ModeName);
Add a rehydration command to the stand alone virtualization tool, making it easier to reverse the effects of asset virtualization. Unlike previous processes, this one does not require that we load the package and will just manipulate the data storaged in the package trailer. #rb Sebastian.Nordgren #rnx #jira UE-156436 #preflight 62c287f9a3568e30664eb94f ### VA Standalone Tool - We now plan to add much more functionality to the tool than just virtualizing and submitting changelists, so to make this easier I am moving the tool towards a design where it should be fairly easy to add new functionality. - Added FCommand, which is a base class for adding new functionality, simple derive from FCommand and hook it up at the appropriate locations. -- In the future it should be possible for new command types to automatically register themselves to be initiated from the command line. There should be no need to edit UnrealVirtualizationToolApp to add a new command but this will be done as an additional work item. -- At the moment FCommand comes with a number of utility methods to call that cover some common source control commands. -- The original functionality has not yet been moved to the command system and so the code is a little bit weird at the moment. Updating older code to the new system will be done as an additional work item. - FProject/FPlugin have been moved to their own code files. ### Rehydrate Command - The rehydrate command will take a number of packages, check them out of source control and then attempt to virtualize them. - At the moment the chekout logic is fairly basic, we just check out every package supplied, we don't check if the package is virtualized or not yet. This can be improved in additional work items. Ideally by the end of command the only packages that we have checked out should also be rehydrated. - At the moment the command can either take a path of a specific package, a path of a directory to find packages in, or a changelist containing packages that should be rehydrated. - A cleint spec (workspace) can optionally be provided, but if not supplied we will attempt to find a client spec for which to check out the packages. - Currently we will check out the packages to the default change list. ### Rehydrate process - Added the rehydration process in it's own code files in the virtualization module. Like the virtualization process this is exposed in a public header file and no via the Core interface which means it is very specific to our module/implementation. - The process expects that the caller will have checked out any required packages from source control. It will treat being unable to update a package file as an error. - Added PackageUtils.h/.cpp and moved some of the generic code from the virtualization process code there so that it can be shared by the rehydration process. ### Misc Moving away from the using things like FPackagePath as that requires that the correct mount points have been registered for a project and at the moment (with the flakiness of FConfig*) it seems that the best idea would be to prefer absolute file paths where possible. [CL 20982284 by paul chipchase in ue5-main branch]
2022-07-07 06:54:33 -04:00
if (Command->Initialize(CmdLine))
{
return Command;
}
else
{
return TUniquePtr<UE::Virtualization::FCommand>();
Add a rehydration command to the stand alone virtualization tool, making it easier to reverse the effects of asset virtualization. Unlike previous processes, this one does not require that we load the package and will just manipulate the data storaged in the package trailer. #rb Sebastian.Nordgren #rnx #jira UE-156436 #preflight 62c287f9a3568e30664eb94f ### VA Standalone Tool - We now plan to add much more functionality to the tool than just virtualizing and submitting changelists, so to make this easier I am moving the tool towards a design where it should be fairly easy to add new functionality. - Added FCommand, which is a base class for adding new functionality, simple derive from FCommand and hook it up at the appropriate locations. -- In the future it should be possible for new command types to automatically register themselves to be initiated from the command line. There should be no need to edit UnrealVirtualizationToolApp to add a new command but this will be done as an additional work item. -- At the moment FCommand comes with a number of utility methods to call that cover some common source control commands. -- The original functionality has not yet been moved to the command system and so the code is a little bit weird at the moment. Updating older code to the new system will be done as an additional work item. - FProject/FPlugin have been moved to their own code files. ### Rehydrate Command - The rehydrate command will take a number of packages, check them out of source control and then attempt to virtualize them. - At the moment the chekout logic is fairly basic, we just check out every package supplied, we don't check if the package is virtualized or not yet. This can be improved in additional work items. Ideally by the end of command the only packages that we have checked out should also be rehydrated. - At the moment the command can either take a path of a specific package, a path of a directory to find packages in, or a changelist containing packages that should be rehydrated. - A cleint spec (workspace) can optionally be provided, but if not supplied we will attempt to find a client spec for which to check out the packages. - Currently we will check out the packages to the default change list. ### Rehydrate process - Added the rehydration process in it's own code files in the virtualization module. Like the virtualization process this is exposed in a public header file and no via the Core interface which means it is very specific to our module/implementation. - The process expects that the caller will have checked out any required packages from source control. It will treat being unable to update a package file as an error. - Added PackageUtils.h/.cpp and moved some of the generic code from the virtualization process code there so that it can be shared by the rehydration process. ### Misc Moving away from the using things like FPackagePath as that requires that the correct mount points have been registered for a project and at the moment (with the flakiness of FConfig*) it seems that the best idea would be to prefer absolute file paths where possible. [CL 20982284 by paul chipchase in ue5-main branch]
2022-07-07 06:54:33 -04:00
}
}
/**
* This class can be used to prevent log messages from other systems being logged with the Display verbosity.
* In practical terms this means as long as the class is alive, only LogVirtualizationTool messages will
* be logged to the display meaning the user will have less information to deal with.
*/
class FOverrideOutputDevice final : public FFeedbackContextAnsi
{
public:
FOverrideOutputDevice()
{
OriginalLog = GWarn;
GWarn = this;
}
virtual ~FOverrideOutputDevice()
{
GWarn = OriginalLog;
}
virtual void Serialize(const TCHAR* V, ELogVerbosity::Type Verbosity, const class FName& Category) override
{
if (Verbosity == ELogVerbosity::Display && Category != LogVirtualizationTool.GetCategoryName())
{
Verbosity = ELogVerbosity::Log;
}
FFeedbackContextAnsi::Serialize(V, Verbosity, Category);
}
private:
FFeedbackContext* OriginalLog;
};
} // namespace
namespace UE::Virtualization
{
FUnrealVirtualizationToolApp::FUnrealVirtualizationToolApp()
{
}
FUnrealVirtualizationToolApp::~FUnrealVirtualizationToolApp()
{
}
EInitResult FUnrealVirtualizationToolApp::Initialize()
{
TRACE_CPUPROFILER_EVENT_SCOPE(Initialize);
UE_LOG(LogVirtualizationTool, Display, TEXT("Initializing..."));
Extend the VA stand-alone tool to work on both changelists (the existing functionality) and lists of packages stored in a text file (the new functionality) #rb PJ.Kack #jira UE-143675 #rnx #preflight 624c6a1814634fba5c7e3d71 ### Rationale - The new mode being added allows us to write a list of package file paths to be virtualized to a file on disk and then the file to be passed to the tool so that it knows what to virtualize. This new mode will then be used by P4VUtils.dll which is how we will run the virtualization process via p4v custom tools. - We need this second mode because: -- A workspace could have multiple projects in it, which means a user could try to virtualize a changelist with packages from more than one project. -- Projects can be associated with different engine installations so the user could try to virtualize a changelist with packages using different engine installations, which in turn means that they need to be virtualized by different versions of the stand-alone tool. -- This means we cannot rely on passing the CL number to the tool, install P4VUtils.dll will need to sort the package paths by engine installation, then pass those paths to the correct version of the tool. -- The paths need to be written to a text file to be passed to the tool as the total length of path data can easily exceed the commandline max length limit if someone were to try and virtualize an entire project in one go for example. - Note that the new mode does NOT submit the package files since there might be multiple calls to different versions of the stand-alone tool. In theory if all the packages are under one engine install (which is almost certainly going to be 99.999% of all submits) we could allow the tool to do the submit but that means we need to maintain two different code paths for the same operation. - The existing mode (virtualize and submit a changelist) remains unchanged. It is not likely that an end user will ever use this mode, but it will make for a useful debugging tool in the future. ### Changes - When the tool first starts we now print the log path to the user so that they can more easily find it. This might be something that we want to remove after a while, but during the initial release of the tool it should make getting good debug info back from users with problems somewhat easier. - Some logging and error messages have had small updates made, to make the final output to the user more readable (based on the many many times this tool was run while testing). In some places additional logging has been added where I felt it was not clear to the user as to what was going on. - Connecting to source control has been moved from the changelist parsing code to it's own method ::TryConnectToSourceControl since all modes will require it. - The cmdline help output has been redone to support the multiple command line options we now have. - Although we could infer the mode to run based on the commandline options, we require the users to explicitly state the mode via -Mode=XXX. In general the tool will be invoked from a gui so adding extra commands doesn't really affect the usability of the system and it makes things a bit clearer. [CL 19665307 by paul chipchase in ue5-main branch]
2022-04-07 02:08:03 -04:00
// Display the log path to the user so that they can more easily find it
// Note that ::GetAbsoluteLogFilename does not always return an absolute filename
FString LogFilePath = FGenericPlatformOutputDevices::GetAbsoluteLogFilename();
LogFilePath = IFileManager::Get().ConvertToAbsolutePathForExternalAppForRead(*LogFilePath);
UE_LOG(LogVirtualizationTool, Display, TEXT("Logging process to '%s'"), *LogFilePath);
if (!TryLoadModules())
{
return EInitResult::Error;
}
if (!TryInitEnginePlugins())
{
return EInitResult::Error;
}
Add a rehydration command to the stand alone virtualization tool, making it easier to reverse the effects of asset virtualization. Unlike previous processes, this one does not require that we load the package and will just manipulate the data storaged in the package trailer. #rb Sebastian.Nordgren #rnx #jira UE-156436 #preflight 62c287f9a3568e30664eb94f ### VA Standalone Tool - We now plan to add much more functionality to the tool than just virtualizing and submitting changelists, so to make this easier I am moving the tool towards a design where it should be fairly easy to add new functionality. - Added FCommand, which is a base class for adding new functionality, simple derive from FCommand and hook it up at the appropriate locations. -- In the future it should be possible for new command types to automatically register themselves to be initiated from the command line. There should be no need to edit UnrealVirtualizationToolApp to add a new command but this will be done as an additional work item. -- At the moment FCommand comes with a number of utility methods to call that cover some common source control commands. -- The original functionality has not yet been moved to the command system and so the code is a little bit weird at the moment. Updating older code to the new system will be done as an additional work item. - FProject/FPlugin have been moved to their own code files. ### Rehydrate Command - The rehydrate command will take a number of packages, check them out of source control and then attempt to virtualize them. - At the moment the chekout logic is fairly basic, we just check out every package supplied, we don't check if the package is virtualized or not yet. This can be improved in additional work items. Ideally by the end of command the only packages that we have checked out should also be rehydrated. - At the moment the command can either take a path of a specific package, a path of a directory to find packages in, or a changelist containing packages that should be rehydrated. - A cleint spec (workspace) can optionally be provided, but if not supplied we will attempt to find a client spec for which to check out the packages. - Currently we will check out the packages to the default change list. ### Rehydrate process - Added the rehydration process in it's own code files in the virtualization module. Like the virtualization process this is exposed in a public header file and no via the Core interface which means it is very specific to our module/implementation. - The process expects that the caller will have checked out any required packages from source control. It will treat being unable to update a package file as an error. - Added PackageUtils.h/.cpp and moved some of the generic code from the virtualization process code there so that it can be shared by the rehydration process. ### Misc Moving away from the using things like FPackagePath as that requires that the correct mount points have been registered for a project and at the moment (with the flakiness of FConfig*) it seems that the best idea would be to prefer absolute file paths where possible. [CL 20982284 by paul chipchase in ue5-main branch]
2022-07-07 06:54:33 -04:00
EInitResult CmdLineResult = TryParseCmdLine();
if (CmdLineResult != EInitResult::Success)
Extend the VA stand-alone tool to work on both changelists (the existing functionality) and lists of packages stored in a text file (the new functionality) #rb PJ.Kack #jira UE-143675 #rnx #preflight 624c6a1814634fba5c7e3d71 ### Rationale - The new mode being added allows us to write a list of package file paths to be virtualized to a file on disk and then the file to be passed to the tool so that it knows what to virtualize. This new mode will then be used by P4VUtils.dll which is how we will run the virtualization process via p4v custom tools. - We need this second mode because: -- A workspace could have multiple projects in it, which means a user could try to virtualize a changelist with packages from more than one project. -- Projects can be associated with different engine installations so the user could try to virtualize a changelist with packages using different engine installations, which in turn means that they need to be virtualized by different versions of the stand-alone tool. -- This means we cannot rely on passing the CL number to the tool, install P4VUtils.dll will need to sort the package paths by engine installation, then pass those paths to the correct version of the tool. -- The paths need to be written to a text file to be passed to the tool as the total length of path data can easily exceed the commandline max length limit if someone were to try and virtualize an entire project in one go for example. - Note that the new mode does NOT submit the package files since there might be multiple calls to different versions of the stand-alone tool. In theory if all the packages are under one engine install (which is almost certainly going to be 99.999% of all submits) we could allow the tool to do the submit but that means we need to maintain two different code paths for the same operation. - The existing mode (virtualize and submit a changelist) remains unchanged. It is not likely that an end user will ever use this mode, but it will make for a useful debugging tool in the future. ### Changes - When the tool first starts we now print the log path to the user so that they can more easily find it. This might be something that we want to remove after a while, but during the initial release of the tool it should make getting good debug info back from users with problems somewhat easier. - Some logging and error messages have had small updates made, to make the final output to the user more readable (based on the many many times this tool was run while testing). In some places additional logging has been added where I felt it was not clear to the user as to what was going on. - Connecting to source control has been moved from the changelist parsing code to it's own method ::TryConnectToSourceControl since all modes will require it. - The cmdline help output has been redone to support the multiple command line options we now have. - Although we could infer the mode to run based on the commandline options, we require the users to explicitly state the mode via -Mode=XXX. In general the tool will be invoked from a gui so adding extra commands doesn't really affect the usability of the system and it makes things a bit clearer. [CL 19665307 by paul chipchase in ue5-main branch]
2022-04-07 02:08:03 -04:00
{
Add a rehydration command to the stand alone virtualization tool, making it easier to reverse the effects of asset virtualization. Unlike previous processes, this one does not require that we load the package and will just manipulate the data storaged in the package trailer. #rb Sebastian.Nordgren #rnx #jira UE-156436 #preflight 62c287f9a3568e30664eb94f ### VA Standalone Tool - We now plan to add much more functionality to the tool than just virtualizing and submitting changelists, so to make this easier I am moving the tool towards a design where it should be fairly easy to add new functionality. - Added FCommand, which is a base class for adding new functionality, simple derive from FCommand and hook it up at the appropriate locations. -- In the future it should be possible for new command types to automatically register themselves to be initiated from the command line. There should be no need to edit UnrealVirtualizationToolApp to add a new command but this will be done as an additional work item. -- At the moment FCommand comes with a number of utility methods to call that cover some common source control commands. -- The original functionality has not yet been moved to the command system and so the code is a little bit weird at the moment. Updating older code to the new system will be done as an additional work item. - FProject/FPlugin have been moved to their own code files. ### Rehydrate Command - The rehydrate command will take a number of packages, check them out of source control and then attempt to virtualize them. - At the moment the chekout logic is fairly basic, we just check out every package supplied, we don't check if the package is virtualized or not yet. This can be improved in additional work items. Ideally by the end of command the only packages that we have checked out should also be rehydrated. - At the moment the command can either take a path of a specific package, a path of a directory to find packages in, or a changelist containing packages that should be rehydrated. - A cleint spec (workspace) can optionally be provided, but if not supplied we will attempt to find a client spec for which to check out the packages. - Currently we will check out the packages to the default change list. ### Rehydrate process - Added the rehydration process in it's own code files in the virtualization module. Like the virtualization process this is exposed in a public header file and no via the Core interface which means it is very specific to our module/implementation. - The process expects that the caller will have checked out any required packages from source control. It will treat being unable to update a package file as an error. - Added PackageUtils.h/.cpp and moved some of the generic code from the virtualization process code there so that it can be shared by the rehydration process. ### Misc Moving away from the using things like FPackagePath as that requires that the correct mount points have been registered for a project and at the moment (with the flakiness of FConfig*) it seems that the best idea would be to prefer absolute file paths where possible. [CL 20982284 by paul chipchase in ue5-main branch]
2022-07-07 06:54:33 -04:00
return CmdLineResult;
Extend the VA stand-alone tool to work on both changelists (the existing functionality) and lists of packages stored in a text file (the new functionality) #rb PJ.Kack #jira UE-143675 #rnx #preflight 624c6a1814634fba5c7e3d71 ### Rationale - The new mode being added allows us to write a list of package file paths to be virtualized to a file on disk and then the file to be passed to the tool so that it knows what to virtualize. This new mode will then be used by P4VUtils.dll which is how we will run the virtualization process via p4v custom tools. - We need this second mode because: -- A workspace could have multiple projects in it, which means a user could try to virtualize a changelist with packages from more than one project. -- Projects can be associated with different engine installations so the user could try to virtualize a changelist with packages using different engine installations, which in turn means that they need to be virtualized by different versions of the stand-alone tool. -- This means we cannot rely on passing the CL number to the tool, install P4VUtils.dll will need to sort the package paths by engine installation, then pass those paths to the correct version of the tool. -- The paths need to be written to a text file to be passed to the tool as the total length of path data can easily exceed the commandline max length limit if someone were to try and virtualize an entire project in one go for example. - Note that the new mode does NOT submit the package files since there might be multiple calls to different versions of the stand-alone tool. In theory if all the packages are under one engine install (which is almost certainly going to be 99.999% of all submits) we could allow the tool to do the submit but that means we need to maintain two different code paths for the same operation. - The existing mode (virtualize and submit a changelist) remains unchanged. It is not likely that an end user will ever use this mode, but it will make for a useful debugging tool in the future. ### Changes - When the tool first starts we now print the log path to the user so that they can more easily find it. This might be something that we want to remove after a while, but during the initial release of the tool it should make getting good debug info back from users with problems somewhat easier. - Some logging and error messages have had small updates made, to make the final output to the user more readable (based on the many many times this tool was run while testing). In some places additional logging has been added where I felt it was not clear to the user as to what was going on. - Connecting to source control has been moved from the changelist parsing code to it's own method ::TryConnectToSourceControl since all modes will require it. - The cmdline help output has been redone to support the multiple command line options we now have. - Although we could infer the mode to run based on the commandline options, we require the users to explicitly state the mode via -Mode=XXX. In general the tool will be invoked from a gui so adding extra commands doesn't really affect the usability of the system and it makes things a bit clearer. [CL 19665307 by paul chipchase in ue5-main branch]
2022-04-07 02:08:03 -04:00
}
Add a rehydration command to the stand alone virtualization tool, making it easier to reverse the effects of asset virtualization. Unlike previous processes, this one does not require that we load the package and will just manipulate the data storaged in the package trailer. #rb Sebastian.Nordgren #rnx #jira UE-156436 #preflight 62c287f9a3568e30664eb94f ### VA Standalone Tool - We now plan to add much more functionality to the tool than just virtualizing and submitting changelists, so to make this easier I am moving the tool towards a design where it should be fairly easy to add new functionality. - Added FCommand, which is a base class for adding new functionality, simple derive from FCommand and hook it up at the appropriate locations. -- In the future it should be possible for new command types to automatically register themselves to be initiated from the command line. There should be no need to edit UnrealVirtualizationToolApp to add a new command but this will be done as an additional work item. -- At the moment FCommand comes with a number of utility methods to call that cover some common source control commands. -- The original functionality has not yet been moved to the command system and so the code is a little bit weird at the moment. Updating older code to the new system will be done as an additional work item. - FProject/FPlugin have been moved to their own code files. ### Rehydrate Command - The rehydrate command will take a number of packages, check them out of source control and then attempt to virtualize them. - At the moment the chekout logic is fairly basic, we just check out every package supplied, we don't check if the package is virtualized or not yet. This can be improved in additional work items. Ideally by the end of command the only packages that we have checked out should also be rehydrated. - At the moment the command can either take a path of a specific package, a path of a directory to find packages in, or a changelist containing packages that should be rehydrated. - A cleint spec (workspace) can optionally be provided, but if not supplied we will attempt to find a client spec for which to check out the packages. - Currently we will check out the packages to the default change list. ### Rehydrate process - Added the rehydration process in it's own code files in the virtualization module. Like the virtualization process this is exposed in a public header file and no via the Core interface which means it is very specific to our module/implementation. - The process expects that the caller will have checked out any required packages from source control. It will treat being unable to update a package file as an error. - Added PackageUtils.h/.cpp and moved some of the generic code from the virtualization process code there so that it can be shared by the rehydration process. ### Misc Moving away from the using things like FPackagePath as that requires that the correct mount points have been registered for a project and at the moment (with the flakiness of FConfig*) it seems that the best idea would be to prefer absolute file paths where possible. [CL 20982284 by paul chipchase in ue5-main branch]
2022-07-07 06:54:33 -04:00
TArray<FString> Packages;
if (!CurrentCommand.IsValid())
{
// Legacy code path for virtualization (which has not yet been ported to a command)
switch (Mode)
{
case EMode::Changelist:
if (!TryParseChangelist(Packages))
{
return EInitResult::Error;
}
break;
case EMode::PackageList:
if (!TryParsePackageList(Packages))
{
return EInitResult::Error;
}
break;
default:
UE_LOG(LogVirtualizationTool, Display, TEXT("Unknown mode, cannot find packages!"));
return EInitResult::Error;
break;
}
}
else
{
Packages = CurrentCommand->GetPackages();
}
UE_LOG(LogVirtualizationTool, Display, TEXT("\tFound '%d' package file(s)"), Packages.Num());
Extend the VA stand-alone tool to work on both changelists (the existing functionality) and lists of packages stored in a text file (the new functionality) #rb PJ.Kack #jira UE-143675 #rnx #preflight 624c6a1814634fba5c7e3d71 ### Rationale - The new mode being added allows us to write a list of package file paths to be virtualized to a file on disk and then the file to be passed to the tool so that it knows what to virtualize. This new mode will then be used by P4VUtils.dll which is how we will run the virtualization process via p4v custom tools. - We need this second mode because: -- A workspace could have multiple projects in it, which means a user could try to virtualize a changelist with packages from more than one project. -- Projects can be associated with different engine installations so the user could try to virtualize a changelist with packages using different engine installations, which in turn means that they need to be virtualized by different versions of the stand-alone tool. -- This means we cannot rely on passing the CL number to the tool, install P4VUtils.dll will need to sort the package paths by engine installation, then pass those paths to the correct version of the tool. -- The paths need to be written to a text file to be passed to the tool as the total length of path data can easily exceed the commandline max length limit if someone were to try and virtualize an entire project in one go for example. - Note that the new mode does NOT submit the package files since there might be multiple calls to different versions of the stand-alone tool. In theory if all the packages are under one engine install (which is almost certainly going to be 99.999% of all submits) we could allow the tool to do the submit but that means we need to maintain two different code paths for the same operation. - The existing mode (virtualize and submit a changelist) remains unchanged. It is not likely that an end user will ever use this mode, but it will make for a useful debugging tool in the future. ### Changes - When the tool first starts we now print the log path to the user so that they can more easily find it. This might be something that we want to remove after a while, but during the initial release of the tool it should make getting good debug info back from users with problems somewhat easier. - Some logging and error messages have had small updates made, to make the final output to the user more readable (based on the many many times this tool was run while testing). In some places additional logging has been added where I felt it was not clear to the user as to what was going on. - Connecting to source control has been moved from the changelist parsing code to it's own method ::TryConnectToSourceControl since all modes will require it. - The cmdline help output has been redone to support the multiple command line options we now have. - Although we could infer the mode to run based on the commandline options, we require the users to explicitly state the mode via -Mode=XXX. In general the tool will be invoked from a gui so adding extra commands doesn't really affect the usability of the system and it makes things a bit clearer. [CL 19665307 by paul chipchase in ue5-main branch]
2022-04-07 02:08:03 -04:00
if (!TrySortFilesByProject(Packages))
{
return EInitResult::Error;
}
UE_LOG(LogVirtualizationTool, Display, TEXT("Initialization complete!"));
return EInitResult::Success;
}
bool FUnrealVirtualizationToolApp::Run()
{
TRACE_CPUPROFILER_EVENT_SCOPE(Run);
Add a rehydration command to the stand alone virtualization tool, making it easier to reverse the effects of asset virtualization. Unlike previous processes, this one does not require that we load the package and will just manipulate the data storaged in the package trailer. #rb Sebastian.Nordgren #rnx #jira UE-156436 #preflight 62c287f9a3568e30664eb94f ### VA Standalone Tool - We now plan to add much more functionality to the tool than just virtualizing and submitting changelists, so to make this easier I am moving the tool towards a design where it should be fairly easy to add new functionality. - Added FCommand, which is a base class for adding new functionality, simple derive from FCommand and hook it up at the appropriate locations. -- In the future it should be possible for new command types to automatically register themselves to be initiated from the command line. There should be no need to edit UnrealVirtualizationToolApp to add a new command but this will be done as an additional work item. -- At the moment FCommand comes with a number of utility methods to call that cover some common source control commands. -- The original functionality has not yet been moved to the command system and so the code is a little bit weird at the moment. Updating older code to the new system will be done as an additional work item. - FProject/FPlugin have been moved to their own code files. ### Rehydrate Command - The rehydrate command will take a number of packages, check them out of source control and then attempt to virtualize them. - At the moment the chekout logic is fairly basic, we just check out every package supplied, we don't check if the package is virtualized or not yet. This can be improved in additional work items. Ideally by the end of command the only packages that we have checked out should also be rehydrated. - At the moment the command can either take a path of a specific package, a path of a directory to find packages in, or a changelist containing packages that should be rehydrated. - A cleint spec (workspace) can optionally be provided, but if not supplied we will attempt to find a client spec for which to check out the packages. - Currently we will check out the packages to the default change list. ### Rehydrate process - Added the rehydration process in it's own code files in the virtualization module. Like the virtualization process this is exposed in a public header file and no via the Core interface which means it is very specific to our module/implementation. - The process expects that the caller will have checked out any required packages from source control. It will treat being unable to update a package file as an error. - Added PackageUtils.h/.cpp and moved some of the generic code from the virtualization process code there so that it can be shared by the rehydration process. ### Misc Moving away from the using things like FPackagePath as that requires that the correct mount points have been registered for a project and at the moment (with the flakiness of FConfig*) it seems that the best idea would be to prefer absolute file paths where possible. [CL 20982284 by paul chipchase in ue5-main branch]
2022-07-07 06:54:33 -04:00
if (!CurrentCommand.IsValid())
{
Add a rehydration command to the stand alone virtualization tool, making it easier to reverse the effects of asset virtualization. Unlike previous processes, this one does not require that we load the package and will just manipulate the data storaged in the package trailer. #rb Sebastian.Nordgren #rnx #jira UE-156436 #preflight 62c287f9a3568e30664eb94f ### VA Standalone Tool - We now plan to add much more functionality to the tool than just virtualizing and submitting changelists, so to make this easier I am moving the tool towards a design where it should be fairly easy to add new functionality. - Added FCommand, which is a base class for adding new functionality, simple derive from FCommand and hook it up at the appropriate locations. -- In the future it should be possible for new command types to automatically register themselves to be initiated from the command line. There should be no need to edit UnrealVirtualizationToolApp to add a new command but this will be done as an additional work item. -- At the moment FCommand comes with a number of utility methods to call that cover some common source control commands. -- The original functionality has not yet been moved to the command system and so the code is a little bit weird at the moment. Updating older code to the new system will be done as an additional work item. - FProject/FPlugin have been moved to their own code files. ### Rehydrate Command - The rehydrate command will take a number of packages, check them out of source control and then attempt to virtualize them. - At the moment the chekout logic is fairly basic, we just check out every package supplied, we don't check if the package is virtualized or not yet. This can be improved in additional work items. Ideally by the end of command the only packages that we have checked out should also be rehydrated. - At the moment the command can either take a path of a specific package, a path of a directory to find packages in, or a changelist containing packages that should be rehydrated. - A cleint spec (workspace) can optionally be provided, but if not supplied we will attempt to find a client spec for which to check out the packages. - Currently we will check out the packages to the default change list. ### Rehydrate process - Added the rehydration process in it's own code files in the virtualization module. Like the virtualization process this is exposed in a public header file and no via the Core interface which means it is very specific to our module/implementation. - The process expects that the caller will have checked out any required packages from source control. It will treat being unable to update a package file as an error. - Added PackageUtils.h/.cpp and moved some of the generic code from the virtualization process code there so that it can be shared by the rehydration process. ### Misc Moving away from the using things like FPackagePath as that requires that the correct mount points have been registered for a project and at the moment (with the flakiness of FConfig*) it seems that the best idea would be to prefer absolute file paths where possible. [CL 20982284 by paul chipchase in ue5-main branch]
2022-07-07 06:54:33 -04:00
// Legacy code path for virtualization (which has not yet been ported to a command)
TArray<FString> FinalDescriptionTags;
Add a rehydration command to the stand alone virtualization tool, making it easier to reverse the effects of asset virtualization. Unlike previous processes, this one does not require that we load the package and will just manipulate the data storaged in the package trailer. #rb Sebastian.Nordgren #rnx #jira UE-156436 #preflight 62c287f9a3568e30664eb94f ### VA Standalone Tool - We now plan to add much more functionality to the tool than just virtualizing and submitting changelists, so to make this easier I am moving the tool towards a design where it should be fairly easy to add new functionality. - Added FCommand, which is a base class for adding new functionality, simple derive from FCommand and hook it up at the appropriate locations. -- In the future it should be possible for new command types to automatically register themselves to be initiated from the command line. There should be no need to edit UnrealVirtualizationToolApp to add a new command but this will be done as an additional work item. -- At the moment FCommand comes with a number of utility methods to call that cover some common source control commands. -- The original functionality has not yet been moved to the command system and so the code is a little bit weird at the moment. Updating older code to the new system will be done as an additional work item. - FProject/FPlugin have been moved to their own code files. ### Rehydrate Command - The rehydrate command will take a number of packages, check them out of source control and then attempt to virtualize them. - At the moment the chekout logic is fairly basic, we just check out every package supplied, we don't check if the package is virtualized or not yet. This can be improved in additional work items. Ideally by the end of command the only packages that we have checked out should also be rehydrated. - At the moment the command can either take a path of a specific package, a path of a directory to find packages in, or a changelist containing packages that should be rehydrated. - A cleint spec (workspace) can optionally be provided, but if not supplied we will attempt to find a client spec for which to check out the packages. - Currently we will check out the packages to the default change list. ### Rehydrate process - Added the rehydration process in it's own code files in the virtualization module. Like the virtualization process this is exposed in a public header file and no via the Core interface which means it is very specific to our module/implementation. - The process expects that the caller will have checked out any required packages from source control. It will treat being unable to update a package file as an error. - Added PackageUtils.h/.cpp and moved some of the generic code from the virtualization process code there so that it can be shared by the rehydration process. ### Misc Moving away from the using things like FPackagePath as that requires that the correct mount points have been registered for a project and at the moment (with the flakiness of FConfig*) it seems that the best idea would be to prefer absolute file paths where possible. [CL 20982284 by paul chipchase in ue5-main branch]
2022-07-07 06:54:33 -04:00
if (EnumHasAllFlags(ProcessOptions, EProcessOptions::Virtualize))
{
Add a rehydration command to the stand alone virtualization tool, making it easier to reverse the effects of asset virtualization. Unlike previous processes, this one does not require that we load the package and will just manipulate the data storaged in the package trailer. #rb Sebastian.Nordgren #rnx #jira UE-156436 #preflight 62c287f9a3568e30664eb94f ### VA Standalone Tool - We now plan to add much more functionality to the tool than just virtualizing and submitting changelists, so to make this easier I am moving the tool towards a design where it should be fairly easy to add new functionality. - Added FCommand, which is a base class for adding new functionality, simple derive from FCommand and hook it up at the appropriate locations. -- In the future it should be possible for new command types to automatically register themselves to be initiated from the command line. There should be no need to edit UnrealVirtualizationToolApp to add a new command but this will be done as an additional work item. -- At the moment FCommand comes with a number of utility methods to call that cover some common source control commands. -- The original functionality has not yet been moved to the command system and so the code is a little bit weird at the moment. Updating older code to the new system will be done as an additional work item. - FProject/FPlugin have been moved to their own code files. ### Rehydrate Command - The rehydrate command will take a number of packages, check them out of source control and then attempt to virtualize them. - At the moment the chekout logic is fairly basic, we just check out every package supplied, we don't check if the package is virtualized or not yet. This can be improved in additional work items. Ideally by the end of command the only packages that we have checked out should also be rehydrated. - At the moment the command can either take a path of a specific package, a path of a directory to find packages in, or a changelist containing packages that should be rehydrated. - A cleint spec (workspace) can optionally be provided, but if not supplied we will attempt to find a client spec for which to check out the packages. - Currently we will check out the packages to the default change list. ### Rehydrate process - Added the rehydration process in it's own code files in the virtualization module. Like the virtualization process this is exposed in a public header file and no via the Core interface which means it is very specific to our module/implementation. - The process expects that the caller will have checked out any required packages from source control. It will treat being unable to update a package file as an error. - Added PackageUtils.h/.cpp and moved some of the generic code from the virtualization process code there so that it can be shared by the rehydration process. ### Misc Moving away from the using things like FPackagePath as that requires that the correct mount points have been registered for a project and at the moment (with the flakiness of FConfig*) it seems that the best idea would be to prefer absolute file paths where possible. [CL 20982284 by paul chipchase in ue5-main branch]
2022-07-07 06:54:33 -04:00
UE_LOG(LogVirtualizationTool, Display, TEXT("Running the virtualization process..."));
Extend the VA stand-alone tool to work on both changelists (the existing functionality) and lists of packages stored in a text file (the new functionality) #rb PJ.Kack #jira UE-143675 #rnx #preflight 624c6a1814634fba5c7e3d71 ### Rationale - The new mode being added allows us to write a list of package file paths to be virtualized to a file on disk and then the file to be passed to the tool so that it knows what to virtualize. This new mode will then be used by P4VUtils.dll which is how we will run the virtualization process via p4v custom tools. - We need this second mode because: -- A workspace could have multiple projects in it, which means a user could try to virtualize a changelist with packages from more than one project. -- Projects can be associated with different engine installations so the user could try to virtualize a changelist with packages using different engine installations, which in turn means that they need to be virtualized by different versions of the stand-alone tool. -- This means we cannot rely on passing the CL number to the tool, install P4VUtils.dll will need to sort the package paths by engine installation, then pass those paths to the correct version of the tool. -- The paths need to be written to a text file to be passed to the tool as the total length of path data can easily exceed the commandline max length limit if someone were to try and virtualize an entire project in one go for example. - Note that the new mode does NOT submit the package files since there might be multiple calls to different versions of the stand-alone tool. In theory if all the packages are under one engine install (which is almost certainly going to be 99.999% of all submits) we could allow the tool to do the submit but that means we need to maintain two different code paths for the same operation. - The existing mode (virtualize and submit a changelist) remains unchanged. It is not likely that an end user will ever use this mode, but it will make for a useful debugging tool in the future. ### Changes - When the tool first starts we now print the log path to the user so that they can more easily find it. This might be something that we want to remove after a while, but during the initial release of the tool it should make getting good debug info back from users with problems somewhat easier. - Some logging and error messages have had small updates made, to make the final output to the user more readable (based on the many many times this tool was run while testing). In some places additional logging has been added where I felt it was not clear to the user as to what was going on. - Connecting to source control has been moved from the changelist parsing code to it's own method ::TryConnectToSourceControl since all modes will require it. - The cmdline help output has been redone to support the multiple command line options we now have. - Although we could infer the mode to run based on the commandline options, we require the users to explicitly state the mode via -Mode=XXX. In general the tool will be invoked from a gui so adding extra commands doesn't really affect the usability of the system and it makes things a bit clearer. [CL 19665307 by paul chipchase in ue5-main branch]
2022-04-07 02:08:03 -04:00
Add a rehydration command to the stand alone virtualization tool, making it easier to reverse the effects of asset virtualization. Unlike previous processes, this one does not require that we load the package and will just manipulate the data storaged in the package trailer. #rb Sebastian.Nordgren #rnx #jira UE-156436 #preflight 62c287f9a3568e30664eb94f ### VA Standalone Tool - We now plan to add much more functionality to the tool than just virtualizing and submitting changelists, so to make this easier I am moving the tool towards a design where it should be fairly easy to add new functionality. - Added FCommand, which is a base class for adding new functionality, simple derive from FCommand and hook it up at the appropriate locations. -- In the future it should be possible for new command types to automatically register themselves to be initiated from the command line. There should be no need to edit UnrealVirtualizationToolApp to add a new command but this will be done as an additional work item. -- At the moment FCommand comes with a number of utility methods to call that cover some common source control commands. -- The original functionality has not yet been moved to the command system and so the code is a little bit weird at the moment. Updating older code to the new system will be done as an additional work item. - FProject/FPlugin have been moved to their own code files. ### Rehydrate Command - The rehydrate command will take a number of packages, check them out of source control and then attempt to virtualize them. - At the moment the chekout logic is fairly basic, we just check out every package supplied, we don't check if the package is virtualized or not yet. This can be improved in additional work items. Ideally by the end of command the only packages that we have checked out should also be rehydrated. - At the moment the command can either take a path of a specific package, a path of a directory to find packages in, or a changelist containing packages that should be rehydrated. - A cleint spec (workspace) can optionally be provided, but if not supplied we will attempt to find a client spec for which to check out the packages. - Currently we will check out the packages to the default change list. ### Rehydrate process - Added the rehydration process in it's own code files in the virtualization module. Like the virtualization process this is exposed in a public header file and no via the Core interface which means it is very specific to our module/implementation. - The process expects that the caller will have checked out any required packages from source control. It will treat being unable to update a package file as an error. - Added PackageUtils.h/.cpp and moved some of the generic code from the virtualization process code there so that it can be shared by the rehydration process. ### Misc Moving away from the using things like FPackagePath as that requires that the correct mount points have been registered for a project and at the moment (with the flakiness of FConfig*) it seems that the best idea would be to prefer absolute file paths where possible. [CL 20982284 by paul chipchase in ue5-main branch]
2022-07-07 06:54:33 -04:00
TArray<FText> DescriptionTags;
Extend the VA stand-alone tool to work on both changelists (the existing functionality) and lists of packages stored in a text file (the new functionality) #rb PJ.Kack #jira UE-143675 #rnx #preflight 624c6a1814634fba5c7e3d71 ### Rationale - The new mode being added allows us to write a list of package file paths to be virtualized to a file on disk and then the file to be passed to the tool so that it knows what to virtualize. This new mode will then be used by P4VUtils.dll which is how we will run the virtualization process via p4v custom tools. - We need this second mode because: -- A workspace could have multiple projects in it, which means a user could try to virtualize a changelist with packages from more than one project. -- Projects can be associated with different engine installations so the user could try to virtualize a changelist with packages using different engine installations, which in turn means that they need to be virtualized by different versions of the stand-alone tool. -- This means we cannot rely on passing the CL number to the tool, install P4VUtils.dll will need to sort the package paths by engine installation, then pass those paths to the correct version of the tool. -- The paths need to be written to a text file to be passed to the tool as the total length of path data can easily exceed the commandline max length limit if someone were to try and virtualize an entire project in one go for example. - Note that the new mode does NOT submit the package files since there might be multiple calls to different versions of the stand-alone tool. In theory if all the packages are under one engine install (which is almost certainly going to be 99.999% of all submits) we could allow the tool to do the submit but that means we need to maintain two different code paths for the same operation. - The existing mode (virtualize and submit a changelist) remains unchanged. It is not likely that an end user will ever use this mode, but it will make for a useful debugging tool in the future. ### Changes - When the tool first starts we now print the log path to the user so that they can more easily find it. This might be something that we want to remove after a while, but during the initial release of the tool it should make getting good debug info back from users with problems somewhat easier. - Some logging and error messages have had small updates made, to make the final output to the user more readable (based on the many many times this tool was run while testing). In some places additional logging has been added where I felt it was not clear to the user as to what was going on. - Connecting to source control has been moved from the changelist parsing code to it's own method ::TryConnectToSourceControl since all modes will require it. - The cmdline help output has been redone to support the multiple command line options we now have. - Although we could infer the mode to run based on the commandline options, we require the users to explicitly state the mode via -Mode=XXX. In general the tool will be invoked from a gui so adding extra commands doesn't really affect the usability of the system and it makes things a bit clearer. [CL 19665307 by paul chipchase in ue5-main branch]
2022-04-07 02:08:03 -04:00
Add a rehydration command to the stand alone virtualization tool, making it easier to reverse the effects of asset virtualization. Unlike previous processes, this one does not require that we load the package and will just manipulate the data storaged in the package trailer. #rb Sebastian.Nordgren #rnx #jira UE-156436 #preflight 62c287f9a3568e30664eb94f ### VA Standalone Tool - We now plan to add much more functionality to the tool than just virtualizing and submitting changelists, so to make this easier I am moving the tool towards a design where it should be fairly easy to add new functionality. - Added FCommand, which is a base class for adding new functionality, simple derive from FCommand and hook it up at the appropriate locations. -- In the future it should be possible for new command types to automatically register themselves to be initiated from the command line. There should be no need to edit UnrealVirtualizationToolApp to add a new command but this will be done as an additional work item. -- At the moment FCommand comes with a number of utility methods to call that cover some common source control commands. -- The original functionality has not yet been moved to the command system and so the code is a little bit weird at the moment. Updating older code to the new system will be done as an additional work item. - FProject/FPlugin have been moved to their own code files. ### Rehydrate Command - The rehydrate command will take a number of packages, check them out of source control and then attempt to virtualize them. - At the moment the chekout logic is fairly basic, we just check out every package supplied, we don't check if the package is virtualized or not yet. This can be improved in additional work items. Ideally by the end of command the only packages that we have checked out should also be rehydrated. - At the moment the command can either take a path of a specific package, a path of a directory to find packages in, or a changelist containing packages that should be rehydrated. - A cleint spec (workspace) can optionally be provided, but if not supplied we will attempt to find a client spec for which to check out the packages. - Currently we will check out the packages to the default change list. ### Rehydrate process - Added the rehydration process in it's own code files in the virtualization module. Like the virtualization process this is exposed in a public header file and no via the Core interface which means it is very specific to our module/implementation. - The process expects that the caller will have checked out any required packages from source control. It will treat being unable to update a package file as an error. - Added PackageUtils.h/.cpp and moved some of the generic code from the virtualization process code there so that it can be shared by the rehydration process. ### Misc Moving away from the using things like FPackagePath as that requires that the correct mount points have been registered for a project and at the moment (with the flakiness of FConfig*) it seems that the best idea would be to prefer absolute file paths where possible. [CL 20982284 by paul chipchase in ue5-main branch]
2022-07-07 06:54:33 -04:00
for (const FProject& Project : Projects)
{
Add a rehydration command to the stand alone virtualization tool, making it easier to reverse the effects of asset virtualization. Unlike previous processes, this one does not require that we load the package and will just manipulate the data storaged in the package trailer. #rb Sebastian.Nordgren #rnx #jira UE-156436 #preflight 62c287f9a3568e30664eb94f ### VA Standalone Tool - We now plan to add much more functionality to the tool than just virtualizing and submitting changelists, so to make this easier I am moving the tool towards a design where it should be fairly easy to add new functionality. - Added FCommand, which is a base class for adding new functionality, simple derive from FCommand and hook it up at the appropriate locations. -- In the future it should be possible for new command types to automatically register themselves to be initiated from the command line. There should be no need to edit UnrealVirtualizationToolApp to add a new command but this will be done as an additional work item. -- At the moment FCommand comes with a number of utility methods to call that cover some common source control commands. -- The original functionality has not yet been moved to the command system and so the code is a little bit weird at the moment. Updating older code to the new system will be done as an additional work item. - FProject/FPlugin have been moved to their own code files. ### Rehydrate Command - The rehydrate command will take a number of packages, check them out of source control and then attempt to virtualize them. - At the moment the chekout logic is fairly basic, we just check out every package supplied, we don't check if the package is virtualized or not yet. This can be improved in additional work items. Ideally by the end of command the only packages that we have checked out should also be rehydrated. - At the moment the command can either take a path of a specific package, a path of a directory to find packages in, or a changelist containing packages that should be rehydrated. - A cleint spec (workspace) can optionally be provided, but if not supplied we will attempt to find a client spec for which to check out the packages. - Currently we will check out the packages to the default change list. ### Rehydrate process - Added the rehydration process in it's own code files in the virtualization module. Like the virtualization process this is exposed in a public header file and no via the Core interface which means it is very specific to our module/implementation. - The process expects that the caller will have checked out any required packages from source control. It will treat being unable to update a package file as an error. - Added PackageUtils.h/.cpp and moved some of the generic code from the virtualization process code there so that it can be shared by the rehydration process. ### Misc Moving away from the using things like FPackagePath as that requires that the correct mount points have been registered for a project and at the moment (with the flakiness of FConfig*) it seems that the best idea would be to prefer absolute file paths where possible. [CL 20982284 by paul chipchase in ue5-main branch]
2022-07-07 06:54:33 -04:00
TStringBuilder<128> ProjectName;
ProjectName << Project.GetProjectName();
Add a rehydration command to the stand alone virtualization tool, making it easier to reverse the effects of asset virtualization. Unlike previous processes, this one does not require that we load the package and will just manipulate the data storaged in the package trailer. #rb Sebastian.Nordgren #rnx #jira UE-156436 #preflight 62c287f9a3568e30664eb94f ### VA Standalone Tool - We now plan to add much more functionality to the tool than just virtualizing and submitting changelists, so to make this easier I am moving the tool towards a design where it should be fairly easy to add new functionality. - Added FCommand, which is a base class for adding new functionality, simple derive from FCommand and hook it up at the appropriate locations. -- In the future it should be possible for new command types to automatically register themselves to be initiated from the command line. There should be no need to edit UnrealVirtualizationToolApp to add a new command but this will be done as an additional work item. -- At the moment FCommand comes with a number of utility methods to call that cover some common source control commands. -- The original functionality has not yet been moved to the command system and so the code is a little bit weird at the moment. Updating older code to the new system will be done as an additional work item. - FProject/FPlugin have been moved to their own code files. ### Rehydrate Command - The rehydrate command will take a number of packages, check them out of source control and then attempt to virtualize them. - At the moment the chekout logic is fairly basic, we just check out every package supplied, we don't check if the package is virtualized or not yet. This can be improved in additional work items. Ideally by the end of command the only packages that we have checked out should also be rehydrated. - At the moment the command can either take a path of a specific package, a path of a directory to find packages in, or a changelist containing packages that should be rehydrated. - A cleint spec (workspace) can optionally be provided, but if not supplied we will attempt to find a client spec for which to check out the packages. - Currently we will check out the packages to the default change list. ### Rehydrate process - Added the rehydration process in it's own code files in the virtualization module. Like the virtualization process this is exposed in a public header file and no via the Core interface which means it is very specific to our module/implementation. - The process expects that the caller will have checked out any required packages from source control. It will treat being unable to update a package file as an error. - Added PackageUtils.h/.cpp and moved some of the generic code from the virtualization process code there so that it can be shared by the rehydration process. ### Misc Moving away from the using things like FPackagePath as that requires that the correct mount points have been registered for a project and at the moment (with the flakiness of FConfig*) it seems that the best idea would be to prefer absolute file paths where possible. [CL 20982284 by paul chipchase in ue5-main branch]
2022-07-07 06:54:33 -04:00
UE_LOG(LogVirtualizationTool, Display, TEXT("\tChecking package(s) for the project '%s'..."), ProjectName.ToString());
Add a rehydration command to the stand alone virtualization tool, making it easier to reverse the effects of asset virtualization. Unlike previous processes, this one does not require that we load the package and will just manipulate the data storaged in the package trailer. #rb Sebastian.Nordgren #rnx #jira UE-156436 #preflight 62c287f9a3568e30664eb94f ### VA Standalone Tool - We now plan to add much more functionality to the tool than just virtualizing and submitting changelists, so to make this easier I am moving the tool towards a design where it should be fairly easy to add new functionality. - Added FCommand, which is a base class for adding new functionality, simple derive from FCommand and hook it up at the appropriate locations. -- In the future it should be possible for new command types to automatically register themselves to be initiated from the command line. There should be no need to edit UnrealVirtualizationToolApp to add a new command but this will be done as an additional work item. -- At the moment FCommand comes with a number of utility methods to call that cover some common source control commands. -- The original functionality has not yet been moved to the command system and so the code is a little bit weird at the moment. Updating older code to the new system will be done as an additional work item. - FProject/FPlugin have been moved to their own code files. ### Rehydrate Command - The rehydrate command will take a number of packages, check them out of source control and then attempt to virtualize them. - At the moment the chekout logic is fairly basic, we just check out every package supplied, we don't check if the package is virtualized or not yet. This can be improved in additional work items. Ideally by the end of command the only packages that we have checked out should also be rehydrated. - At the moment the command can either take a path of a specific package, a path of a directory to find packages in, or a changelist containing packages that should be rehydrated. - A cleint spec (workspace) can optionally be provided, but if not supplied we will attempt to find a client spec for which to check out the packages. - Currently we will check out the packages to the default change list. ### Rehydrate process - Added the rehydration process in it's own code files in the virtualization module. Like the virtualization process this is exposed in a public header file and no via the Core interface which means it is very specific to our module/implementation. - The process expects that the caller will have checked out any required packages from source control. It will treat being unable to update a package file as an error. - Added PackageUtils.h/.cpp and moved some of the generic code from the virtualization process code there so that it can be shared by the rehydration process. ### Misc Moving away from the using things like FPackagePath as that requires that the correct mount points have been registered for a project and at the moment (with the flakiness of FConfig*) it seems that the best idea would be to prefer absolute file paths where possible. [CL 20982284 by paul chipchase in ue5-main branch]
2022-07-07 06:54:33 -04:00
FConfigFile EngineConfigWithProject;
if (!Project.TryLoadConfig(EngineConfigWithProject))
{
Add a rehydration command to the stand alone virtualization tool, making it easier to reverse the effects of asset virtualization. Unlike previous processes, this one does not require that we load the package and will just manipulate the data storaged in the package trailer. #rb Sebastian.Nordgren #rnx #jira UE-156436 #preflight 62c287f9a3568e30664eb94f ### VA Standalone Tool - We now plan to add much more functionality to the tool than just virtualizing and submitting changelists, so to make this easier I am moving the tool towards a design where it should be fairly easy to add new functionality. - Added FCommand, which is a base class for adding new functionality, simple derive from FCommand and hook it up at the appropriate locations. -- In the future it should be possible for new command types to automatically register themselves to be initiated from the command line. There should be no need to edit UnrealVirtualizationToolApp to add a new command but this will be done as an additional work item. -- At the moment FCommand comes with a number of utility methods to call that cover some common source control commands. -- The original functionality has not yet been moved to the command system and so the code is a little bit weird at the moment. Updating older code to the new system will be done as an additional work item. - FProject/FPlugin have been moved to their own code files. ### Rehydrate Command - The rehydrate command will take a number of packages, check them out of source control and then attempt to virtualize them. - At the moment the chekout logic is fairly basic, we just check out every package supplied, we don't check if the package is virtualized or not yet. This can be improved in additional work items. Ideally by the end of command the only packages that we have checked out should also be rehydrated. - At the moment the command can either take a path of a specific package, a path of a directory to find packages in, or a changelist containing packages that should be rehydrated. - A cleint spec (workspace) can optionally be provided, but if not supplied we will attempt to find a client spec for which to check out the packages. - Currently we will check out the packages to the default change list. ### Rehydrate process - Added the rehydration process in it's own code files in the virtualization module. Like the virtualization process this is exposed in a public header file and no via the Core interface which means it is very specific to our module/implementation. - The process expects that the caller will have checked out any required packages from source control. It will treat being unable to update a package file as an error. - Added PackageUtils.h/.cpp and moved some of the generic code from the virtualization process code there so that it can be shared by the rehydration process. ### Misc Moving away from the using things like FPackagePath as that requires that the correct mount points have been registered for a project and at the moment (with the flakiness of FConfig*) it seems that the best idea would be to prefer absolute file paths where possible. [CL 20982284 by paul chipchase in ue5-main branch]
2022-07-07 06:54:33 -04:00
return false;
}
Add a rehydration command to the stand alone virtualization tool, making it easier to reverse the effects of asset virtualization. Unlike previous processes, this one does not require that we load the package and will just manipulate the data storaged in the package trailer. #rb Sebastian.Nordgren #rnx #jira UE-156436 #preflight 62c287f9a3568e30664eb94f ### VA Standalone Tool - We now plan to add much more functionality to the tool than just virtualizing and submitting changelists, so to make this easier I am moving the tool towards a design where it should be fairly easy to add new functionality. - Added FCommand, which is a base class for adding new functionality, simple derive from FCommand and hook it up at the appropriate locations. -- In the future it should be possible for new command types to automatically register themselves to be initiated from the command line. There should be no need to edit UnrealVirtualizationToolApp to add a new command but this will be done as an additional work item. -- At the moment FCommand comes with a number of utility methods to call that cover some common source control commands. -- The original functionality has not yet been moved to the command system and so the code is a little bit weird at the moment. Updating older code to the new system will be done as an additional work item. - FProject/FPlugin have been moved to their own code files. ### Rehydrate Command - The rehydrate command will take a number of packages, check them out of source control and then attempt to virtualize them. - At the moment the chekout logic is fairly basic, we just check out every package supplied, we don't check if the package is virtualized or not yet. This can be improved in additional work items. Ideally by the end of command the only packages that we have checked out should also be rehydrated. - At the moment the command can either take a path of a specific package, a path of a directory to find packages in, or a changelist containing packages that should be rehydrated. - A cleint spec (workspace) can optionally be provided, but if not supplied we will attempt to find a client spec for which to check out the packages. - Currently we will check out the packages to the default change list. ### Rehydrate process - Added the rehydration process in it's own code files in the virtualization module. Like the virtualization process this is exposed in a public header file and no via the Core interface which means it is very specific to our module/implementation. - The process expects that the caller will have checked out any required packages from source control. It will treat being unable to update a package file as an error. - Added PackageUtils.h/.cpp and moved some of the generic code from the virtualization process code there so that it can be shared by the rehydration process. ### Misc Moving away from the using things like FPackagePath as that requires that the correct mount points have been registered for a project and at the moment (with the flakiness of FConfig*) it seems that the best idea would be to prefer absolute file paths where possible. [CL 20982284 by paul chipchase in ue5-main branch]
2022-07-07 06:54:33 -04:00
Project.RegisterMountPoints();
UE::Virtualization::FInitParams InitParams(ProjectName, EngineConfigWithProject);
UE::Virtualization::Initialize(InitParams);
TArray<FString> Packages = Project.GetAllPackages();
TArray<FText> Errors;
UE::Virtualization::IVirtualizationSystem::Get().TryVirtualizePackages(Packages, DescriptionTags, Errors);
if (!Errors.IsEmpty())
{
UE_LOG(LogVirtualizationTool, Error, TEXT("The virtualization process failed with the following errors:"));
for (const FText& Error : Errors)
{
UE_LOG(LogVirtualizationTool, Error, TEXT("\t%s"), *Error.ToString());
}
return false;
}
UE_LOG(LogVirtualizationTool, Display, TEXT("\tCheck complete"));
UE::Virtualization::Shutdown();
Project.UnRegisterMountPoints();
}
Add a rehydration command to the stand alone virtualization tool, making it easier to reverse the effects of asset virtualization. Unlike previous processes, this one does not require that we load the package and will just manipulate the data storaged in the package trailer. #rb Sebastian.Nordgren #rnx #jira UE-156436 #preflight 62c287f9a3568e30664eb94f ### VA Standalone Tool - We now plan to add much more functionality to the tool than just virtualizing and submitting changelists, so to make this easier I am moving the tool towards a design where it should be fairly easy to add new functionality. - Added FCommand, which is a base class for adding new functionality, simple derive from FCommand and hook it up at the appropriate locations. -- In the future it should be possible for new command types to automatically register themselves to be initiated from the command line. There should be no need to edit UnrealVirtualizationToolApp to add a new command but this will be done as an additional work item. -- At the moment FCommand comes with a number of utility methods to call that cover some common source control commands. -- The original functionality has not yet been moved to the command system and so the code is a little bit weird at the moment. Updating older code to the new system will be done as an additional work item. - FProject/FPlugin have been moved to their own code files. ### Rehydrate Command - The rehydrate command will take a number of packages, check them out of source control and then attempt to virtualize them. - At the moment the chekout logic is fairly basic, we just check out every package supplied, we don't check if the package is virtualized or not yet. This can be improved in additional work items. Ideally by the end of command the only packages that we have checked out should also be rehydrated. - At the moment the command can either take a path of a specific package, a path of a directory to find packages in, or a changelist containing packages that should be rehydrated. - A cleint spec (workspace) can optionally be provided, but if not supplied we will attempt to find a client spec for which to check out the packages. - Currently we will check out the packages to the default change list. ### Rehydrate process - Added the rehydration process in it's own code files in the virtualization module. Like the virtualization process this is exposed in a public header file and no via the Core interface which means it is very specific to our module/implementation. - The process expects that the caller will have checked out any required packages from source control. It will treat being unable to update a package file as an error. - Added PackageUtils.h/.cpp and moved some of the generic code from the virtualization process code there so that it can be shared by the rehydration process. ### Misc Moving away from the using things like FPackagePath as that requires that the correct mount points have been registered for a project and at the moment (with the flakiness of FConfig*) it seems that the best idea would be to prefer absolute file paths where possible. [CL 20982284 by paul chipchase in ue5-main branch]
2022-07-07 06:54:33 -04:00
FinalDescriptionTags = BuildFinalTagDescriptions(DescriptionTags);
}
else
{
UE_LOG(LogVirtualizationTool, Display, TEXT("Skipping the virtualization process"));
}
Add a rehydration command to the stand alone virtualization tool, making it easier to reverse the effects of asset virtualization. Unlike previous processes, this one does not require that we load the package and will just manipulate the data storaged in the package trailer. #rb Sebastian.Nordgren #rnx #jira UE-156436 #preflight 62c287f9a3568e30664eb94f ### VA Standalone Tool - We now plan to add much more functionality to the tool than just virtualizing and submitting changelists, so to make this easier I am moving the tool towards a design where it should be fairly easy to add new functionality. - Added FCommand, which is a base class for adding new functionality, simple derive from FCommand and hook it up at the appropriate locations. -- In the future it should be possible for new command types to automatically register themselves to be initiated from the command line. There should be no need to edit UnrealVirtualizationToolApp to add a new command but this will be done as an additional work item. -- At the moment FCommand comes with a number of utility methods to call that cover some common source control commands. -- The original functionality has not yet been moved to the command system and so the code is a little bit weird at the moment. Updating older code to the new system will be done as an additional work item. - FProject/FPlugin have been moved to their own code files. ### Rehydrate Command - The rehydrate command will take a number of packages, check them out of source control and then attempt to virtualize them. - At the moment the chekout logic is fairly basic, we just check out every package supplied, we don't check if the package is virtualized or not yet. This can be improved in additional work items. Ideally by the end of command the only packages that we have checked out should also be rehydrated. - At the moment the command can either take a path of a specific package, a path of a directory to find packages in, or a changelist containing packages that should be rehydrated. - A cleint spec (workspace) can optionally be provided, but if not supplied we will attempt to find a client spec for which to check out the packages. - Currently we will check out the packages to the default change list. ### Rehydrate process - Added the rehydration process in it's own code files in the virtualization module. Like the virtualization process this is exposed in a public header file and no via the Core interface which means it is very specific to our module/implementation. - The process expects that the caller will have checked out any required packages from source control. It will treat being unable to update a package file as an error. - Added PackageUtils.h/.cpp and moved some of the generic code from the virtualization process code there so that it can be shared by the rehydration process. ### Misc Moving away from the using things like FPackagePath as that requires that the correct mount points have been registered for a project and at the moment (with the flakiness of FConfig*) it seems that the best idea would be to prefer absolute file paths where possible. [CL 20982284 by paul chipchase in ue5-main branch]
2022-07-07 06:54:33 -04:00
if (Mode == EMode::Changelist)
{
if (!TrySubmitChangelist(FinalDescriptionTags))
{
return false;
}
}
return true;
}
else
{
Add a rehydration command to the stand alone virtualization tool, making it easier to reverse the effects of asset virtualization. Unlike previous processes, this one does not require that we load the package and will just manipulate the data storaged in the package trailer. #rb Sebastian.Nordgren #rnx #jira UE-156436 #preflight 62c287f9a3568e30664eb94f ### VA Standalone Tool - We now plan to add much more functionality to the tool than just virtualizing and submitting changelists, so to make this easier I am moving the tool towards a design where it should be fairly easy to add new functionality. - Added FCommand, which is a base class for adding new functionality, simple derive from FCommand and hook it up at the appropriate locations. -- In the future it should be possible for new command types to automatically register themselves to be initiated from the command line. There should be no need to edit UnrealVirtualizationToolApp to add a new command but this will be done as an additional work item. -- At the moment FCommand comes with a number of utility methods to call that cover some common source control commands. -- The original functionality has not yet been moved to the command system and so the code is a little bit weird at the moment. Updating older code to the new system will be done as an additional work item. - FProject/FPlugin have been moved to their own code files. ### Rehydrate Command - The rehydrate command will take a number of packages, check them out of source control and then attempt to virtualize them. - At the moment the chekout logic is fairly basic, we just check out every package supplied, we don't check if the package is virtualized or not yet. This can be improved in additional work items. Ideally by the end of command the only packages that we have checked out should also be rehydrated. - At the moment the command can either take a path of a specific package, a path of a directory to find packages in, or a changelist containing packages that should be rehydrated. - A cleint spec (workspace) can optionally be provided, but if not supplied we will attempt to find a client spec for which to check out the packages. - Currently we will check out the packages to the default change list. ### Rehydrate process - Added the rehydration process in it's own code files in the virtualization module. Like the virtualization process this is exposed in a public header file and no via the Core interface which means it is very specific to our module/implementation. - The process expects that the caller will have checked out any required packages from source control. It will treat being unable to update a package file as an error. - Added PackageUtils.h/.cpp and moved some of the generic code from the virtualization process code there so that it can be shared by the rehydration process. ### Misc Moving away from the using things like FPackagePath as that requires that the correct mount points have been registered for a project and at the moment (with the flakiness of FConfig*) it seems that the best idea would be to prefer absolute file paths where possible. [CL 20982284 by paul chipchase in ue5-main branch]
2022-07-07 06:54:33 -04:00
UE_LOG(LogVirtualizationTool, Display, TEXT("Running the '%s' command..."), *CurrentCommand->GetName());
Add a rehydration command to the stand alone virtualization tool, making it easier to reverse the effects of asset virtualization. Unlike previous processes, this one does not require that we load the package and will just manipulate the data storaged in the package trailer. #rb Sebastian.Nordgren #rnx #jira UE-156436 #preflight 62c287f9a3568e30664eb94f ### VA Standalone Tool - We now plan to add much more functionality to the tool than just virtualizing and submitting changelists, so to make this easier I am moving the tool towards a design where it should be fairly easy to add new functionality. - Added FCommand, which is a base class for adding new functionality, simple derive from FCommand and hook it up at the appropriate locations. -- In the future it should be possible for new command types to automatically register themselves to be initiated from the command line. There should be no need to edit UnrealVirtualizationToolApp to add a new command but this will be done as an additional work item. -- At the moment FCommand comes with a number of utility methods to call that cover some common source control commands. -- The original functionality has not yet been moved to the command system and so the code is a little bit weird at the moment. Updating older code to the new system will be done as an additional work item. - FProject/FPlugin have been moved to their own code files. ### Rehydrate Command - The rehydrate command will take a number of packages, check them out of source control and then attempt to virtualize them. - At the moment the chekout logic is fairly basic, we just check out every package supplied, we don't check if the package is virtualized or not yet. This can be improved in additional work items. Ideally by the end of command the only packages that we have checked out should also be rehydrated. - At the moment the command can either take a path of a specific package, a path of a directory to find packages in, or a changelist containing packages that should be rehydrated. - A cleint spec (workspace) can optionally be provided, but if not supplied we will attempt to find a client spec for which to check out the packages. - Currently we will check out the packages to the default change list. ### Rehydrate process - Added the rehydration process in it's own code files in the virtualization module. Like the virtualization process this is exposed in a public header file and no via the Core interface which means it is very specific to our module/implementation. - The process expects that the caller will have checked out any required packages from source control. It will treat being unable to update a package file as an error. - Added PackageUtils.h/.cpp and moved some of the generic code from the virtualization process code there so that it can be shared by the rehydration process. ### Misc Moving away from the using things like FPackagePath as that requires that the correct mount points have been registered for a project and at the moment (with the flakiness of FConfig*) it seems that the best idea would be to prefer absolute file paths where possible. [CL 20982284 by paul chipchase in ue5-main branch]
2022-07-07 06:54:33 -04:00
if (CurrentCommand->Run(Projects))
Extend the VA stand-alone tool to work on both changelists (the existing functionality) and lists of packages stored in a text file (the new functionality) #rb PJ.Kack #jira UE-143675 #rnx #preflight 624c6a1814634fba5c7e3d71 ### Rationale - The new mode being added allows us to write a list of package file paths to be virtualized to a file on disk and then the file to be passed to the tool so that it knows what to virtualize. This new mode will then be used by P4VUtils.dll which is how we will run the virtualization process via p4v custom tools. - We need this second mode because: -- A workspace could have multiple projects in it, which means a user could try to virtualize a changelist with packages from more than one project. -- Projects can be associated with different engine installations so the user could try to virtualize a changelist with packages using different engine installations, which in turn means that they need to be virtualized by different versions of the stand-alone tool. -- This means we cannot rely on passing the CL number to the tool, install P4VUtils.dll will need to sort the package paths by engine installation, then pass those paths to the correct version of the tool. -- The paths need to be written to a text file to be passed to the tool as the total length of path data can easily exceed the commandline max length limit if someone were to try and virtualize an entire project in one go for example. - Note that the new mode does NOT submit the package files since there might be multiple calls to different versions of the stand-alone tool. In theory if all the packages are under one engine install (which is almost certainly going to be 99.999% of all submits) we could allow the tool to do the submit but that means we need to maintain two different code paths for the same operation. - The existing mode (virtualize and submit a changelist) remains unchanged. It is not likely that an end user will ever use this mode, but it will make for a useful debugging tool in the future. ### Changes - When the tool first starts we now print the log path to the user so that they can more easily find it. This might be something that we want to remove after a while, but during the initial release of the tool it should make getting good debug info back from users with problems somewhat easier. - Some logging and error messages have had small updates made, to make the final output to the user more readable (based on the many many times this tool was run while testing). In some places additional logging has been added where I felt it was not clear to the user as to what was going on. - Connecting to source control has been moved from the changelist parsing code to it's own method ::TryConnectToSourceControl since all modes will require it. - The cmdline help output has been redone to support the multiple command line options we now have. - Although we could infer the mode to run based on the commandline options, we require the users to explicitly state the mode via -Mode=XXX. In general the tool will be invoked from a gui so adding extra commands doesn't really affect the usability of the system and it makes things a bit clearer. [CL 19665307 by paul chipchase in ue5-main branch]
2022-04-07 02:08:03 -04:00
{
Add a rehydration command to the stand alone virtualization tool, making it easier to reverse the effects of asset virtualization. Unlike previous processes, this one does not require that we load the package and will just manipulate the data storaged in the package trailer. #rb Sebastian.Nordgren #rnx #jira UE-156436 #preflight 62c287f9a3568e30664eb94f ### VA Standalone Tool - We now plan to add much more functionality to the tool than just virtualizing and submitting changelists, so to make this easier I am moving the tool towards a design where it should be fairly easy to add new functionality. - Added FCommand, which is a base class for adding new functionality, simple derive from FCommand and hook it up at the appropriate locations. -- In the future it should be possible for new command types to automatically register themselves to be initiated from the command line. There should be no need to edit UnrealVirtualizationToolApp to add a new command but this will be done as an additional work item. -- At the moment FCommand comes with a number of utility methods to call that cover some common source control commands. -- The original functionality has not yet been moved to the command system and so the code is a little bit weird at the moment. Updating older code to the new system will be done as an additional work item. - FProject/FPlugin have been moved to their own code files. ### Rehydrate Command - The rehydrate command will take a number of packages, check them out of source control and then attempt to virtualize them. - At the moment the chekout logic is fairly basic, we just check out every package supplied, we don't check if the package is virtualized or not yet. This can be improved in additional work items. Ideally by the end of command the only packages that we have checked out should also be rehydrated. - At the moment the command can either take a path of a specific package, a path of a directory to find packages in, or a changelist containing packages that should be rehydrated. - A cleint spec (workspace) can optionally be provided, but if not supplied we will attempt to find a client spec for which to check out the packages. - Currently we will check out the packages to the default change list. ### Rehydrate process - Added the rehydration process in it's own code files in the virtualization module. Like the virtualization process this is exposed in a public header file and no via the Core interface which means it is very specific to our module/implementation. - The process expects that the caller will have checked out any required packages from source control. It will treat being unable to update a package file as an error. - Added PackageUtils.h/.cpp and moved some of the generic code from the virtualization process code there so that it can be shared by the rehydration process. ### Misc Moving away from the using things like FPackagePath as that requires that the correct mount points have been registered for a project and at the moment (with the flakiness of FConfig*) it seems that the best idea would be to prefer absolute file paths where possible. [CL 20982284 by paul chipchase in ue5-main branch]
2022-07-07 06:54:33 -04:00
UE_LOG(LogVirtualizationTool, Display, TEXT("Command '%s' suceeeded!"), *CurrentCommand->GetName());
return true;
}
else
{
UE_LOG(LogVirtualizationTool, Error, TEXT("Command '%s' failed!"), *CurrentCommand->GetName());
Extend the VA stand-alone tool to work on both changelists (the existing functionality) and lists of packages stored in a text file (the new functionality) #rb PJ.Kack #jira UE-143675 #rnx #preflight 624c6a1814634fba5c7e3d71 ### Rationale - The new mode being added allows us to write a list of package file paths to be virtualized to a file on disk and then the file to be passed to the tool so that it knows what to virtualize. This new mode will then be used by P4VUtils.dll which is how we will run the virtualization process via p4v custom tools. - We need this second mode because: -- A workspace could have multiple projects in it, which means a user could try to virtualize a changelist with packages from more than one project. -- Projects can be associated with different engine installations so the user could try to virtualize a changelist with packages using different engine installations, which in turn means that they need to be virtualized by different versions of the stand-alone tool. -- This means we cannot rely on passing the CL number to the tool, install P4VUtils.dll will need to sort the package paths by engine installation, then pass those paths to the correct version of the tool. -- The paths need to be written to a text file to be passed to the tool as the total length of path data can easily exceed the commandline max length limit if someone were to try and virtualize an entire project in one go for example. - Note that the new mode does NOT submit the package files since there might be multiple calls to different versions of the stand-alone tool. In theory if all the packages are under one engine install (which is almost certainly going to be 99.999% of all submits) we could allow the tool to do the submit but that means we need to maintain two different code paths for the same operation. - The existing mode (virtualize and submit a changelist) remains unchanged. It is not likely that an end user will ever use this mode, but it will make for a useful debugging tool in the future. ### Changes - When the tool first starts we now print the log path to the user so that they can more easily find it. This might be something that we want to remove after a while, but during the initial release of the tool it should make getting good debug info back from users with problems somewhat easier. - Some logging and error messages have had small updates made, to make the final output to the user more readable (based on the many many times this tool was run while testing). In some places additional logging has been added where I felt it was not clear to the user as to what was going on. - Connecting to source control has been moved from the changelist parsing code to it's own method ::TryConnectToSourceControl since all modes will require it. - The cmdline help output has been redone to support the multiple command line options we now have. - Although we could infer the mode to run based on the commandline options, we require the users to explicitly state the mode via -Mode=XXX. In general the tool will be invoked from a gui so adding extra commands doesn't really affect the usability of the system and it makes things a bit clearer. [CL 19665307 by paul chipchase in ue5-main branch]
2022-04-07 02:08:03 -04:00
return false;
}
}
}
void FUnrealVirtualizationToolApp::PrintCmdLineHelp() const
{
UE_LOG(LogVirtualizationTool, Display, TEXT("Usage:"));
Extend the VA stand-alone tool to work on both changelists (the existing functionality) and lists of packages stored in a text file (the new functionality) #rb PJ.Kack #jira UE-143675 #rnx #preflight 624c6a1814634fba5c7e3d71 ### Rationale - The new mode being added allows us to write a list of package file paths to be virtualized to a file on disk and then the file to be passed to the tool so that it knows what to virtualize. This new mode will then be used by P4VUtils.dll which is how we will run the virtualization process via p4v custom tools. - We need this second mode because: -- A workspace could have multiple projects in it, which means a user could try to virtualize a changelist with packages from more than one project. -- Projects can be associated with different engine installations so the user could try to virtualize a changelist with packages using different engine installations, which in turn means that they need to be virtualized by different versions of the stand-alone tool. -- This means we cannot rely on passing the CL number to the tool, install P4VUtils.dll will need to sort the package paths by engine installation, then pass those paths to the correct version of the tool. -- The paths need to be written to a text file to be passed to the tool as the total length of path data can easily exceed the commandline max length limit if someone were to try and virtualize an entire project in one go for example. - Note that the new mode does NOT submit the package files since there might be multiple calls to different versions of the stand-alone tool. In theory if all the packages are under one engine install (which is almost certainly going to be 99.999% of all submits) we could allow the tool to do the submit but that means we need to maintain two different code paths for the same operation. - The existing mode (virtualize and submit a changelist) remains unchanged. It is not likely that an end user will ever use this mode, but it will make for a useful debugging tool in the future. ### Changes - When the tool first starts we now print the log path to the user so that they can more easily find it. This might be something that we want to remove after a while, but during the initial release of the tool it should make getting good debug info back from users with problems somewhat easier. - Some logging and error messages have had small updates made, to make the final output to the user more readable (based on the many many times this tool was run while testing). In some places additional logging has been added where I felt it was not clear to the user as to what was going on. - Connecting to source control has been moved from the changelist parsing code to it's own method ::TryConnectToSourceControl since all modes will require it. - The cmdline help output has been redone to support the multiple command line options we now have. - Although we could infer the mode to run based on the commandline options, we require the users to explicitly state the mode via -Mode=XXX. In general the tool will be invoked from a gui so adding extra commands doesn't really affect the usability of the system and it makes things a bit clearer. [CL 19665307 by paul chipchase in ue5-main branch]
2022-04-07 02:08:03 -04:00
UE_LOG(LogVirtualizationTool, Display, TEXT("UnrealVirtualizationTool -ClientSpecName=<name> -Mode=Changelist -Changelist=<number> [-nosubmit] [global options]"));
UE_LOG(LogVirtualizationTool, Display, TEXT("\t[optional]-nosubmit (the changelist will be virtualized but not submitted)"));
UE_LOG(LogVirtualizationTool, Display, TEXT("UnrealVirtualizationTool -ClientSpecName=<name> -Mode=PackageList -Path=<string> [global options]"));
Add a rehydration command to the stand alone virtualization tool, making it easier to reverse the effects of asset virtualization. Unlike previous processes, this one does not require that we load the package and will just manipulate the data storaged in the package trailer. #rb Sebastian.Nordgren #rnx #jira UE-156436 #preflight 62c287f9a3568e30664eb94f ### VA Standalone Tool - We now plan to add much more functionality to the tool than just virtualizing and submitting changelists, so to make this easier I am moving the tool towards a design where it should be fairly easy to add new functionality. - Added FCommand, which is a base class for adding new functionality, simple derive from FCommand and hook it up at the appropriate locations. -- In the future it should be possible for new command types to automatically register themselves to be initiated from the command line. There should be no need to edit UnrealVirtualizationToolApp to add a new command but this will be done as an additional work item. -- At the moment FCommand comes with a number of utility methods to call that cover some common source control commands. -- The original functionality has not yet been moved to the command system and so the code is a little bit weird at the moment. Updating older code to the new system will be done as an additional work item. - FProject/FPlugin have been moved to their own code files. ### Rehydrate Command - The rehydrate command will take a number of packages, check them out of source control and then attempt to virtualize them. - At the moment the chekout logic is fairly basic, we just check out every package supplied, we don't check if the package is virtualized or not yet. This can be improved in additional work items. Ideally by the end of command the only packages that we have checked out should also be rehydrated. - At the moment the command can either take a path of a specific package, a path of a directory to find packages in, or a changelist containing packages that should be rehydrated. - A cleint spec (workspace) can optionally be provided, but if not supplied we will attempt to find a client spec for which to check out the packages. - Currently we will check out the packages to the default change list. ### Rehydrate process - Added the rehydration process in it's own code files in the virtualization module. Like the virtualization process this is exposed in a public header file and no via the Core interface which means it is very specific to our module/implementation. - The process expects that the caller will have checked out any required packages from source control. It will treat being unable to update a package file as an error. - Added PackageUtils.h/.cpp and moved some of the generic code from the virtualization process code there so that it can be shared by the rehydration process. ### Misc Moving away from the using things like FPackagePath as that requires that the correct mount points have been registered for a project and at the moment (with the flakiness of FConfig*) it seems that the best idea would be to prefer absolute file paths where possible. [CL 20982284 by paul chipchase in ue5-main branch]
2022-07-07 06:54:33 -04:00
// TODO: If the commands were registered in some way we could automate this
FRehydrateCommand::PrintCmdLineHelp();
Extend the VA stand-alone tool to work on both changelists (the existing functionality) and lists of packages stored in a text file (the new functionality) #rb PJ.Kack #jira UE-143675 #rnx #preflight 624c6a1814634fba5c7e3d71 ### Rationale - The new mode being added allows us to write a list of package file paths to be virtualized to a file on disk and then the file to be passed to the tool so that it knows what to virtualize. This new mode will then be used by P4VUtils.dll which is how we will run the virtualization process via p4v custom tools. - We need this second mode because: -- A workspace could have multiple projects in it, which means a user could try to virtualize a changelist with packages from more than one project. -- Projects can be associated with different engine installations so the user could try to virtualize a changelist with packages using different engine installations, which in turn means that they need to be virtualized by different versions of the stand-alone tool. -- This means we cannot rely on passing the CL number to the tool, install P4VUtils.dll will need to sort the package paths by engine installation, then pass those paths to the correct version of the tool. -- The paths need to be written to a text file to be passed to the tool as the total length of path data can easily exceed the commandline max length limit if someone were to try and virtualize an entire project in one go for example. - Note that the new mode does NOT submit the package files since there might be multiple calls to different versions of the stand-alone tool. In theory if all the packages are under one engine install (which is almost certainly going to be 99.999% of all submits) we could allow the tool to do the submit but that means we need to maintain two different code paths for the same operation. - The existing mode (virtualize and submit a changelist) remains unchanged. It is not likely that an end user will ever use this mode, but it will make for a useful debugging tool in the future. ### Changes - When the tool first starts we now print the log path to the user so that they can more easily find it. This might be something that we want to remove after a while, but during the initial release of the tool it should make getting good debug info back from users with problems somewhat easier. - Some logging and error messages have had small updates made, to make the final output to the user more readable (based on the many many times this tool was run while testing). In some places additional logging has been added where I felt it was not clear to the user as to what was going on. - Connecting to source control has been moved from the changelist parsing code to it's own method ::TryConnectToSourceControl since all modes will require it. - The cmdline help output has been redone to support the multiple command line options we now have. - Although we could infer the mode to run based on the commandline options, we require the users to explicitly state the mode via -Mode=XXX. In general the tool will be invoked from a gui so adding extra commands doesn't really affect the usability of the system and it makes things a bit clearer. [CL 19665307 by paul chipchase in ue5-main branch]
2022-04-07 02:08:03 -04:00
UE_LOG(LogVirtualizationTool, Display, TEXT("Global Options:"));
UE_LOG(LogVirtualizationTool, Display, TEXT("\t-verbose (all log messages with display verbosity will be displayed, not just LogVirtualizationTool)"));
}
bool FUnrealVirtualizationToolApp::TrySubmitChangelist(const TArray<FString>& DescriptionTags)
{
if (!EnumHasAllFlags(ProcessOptions, EProcessOptions::Submit))
{
UE_LOG(LogVirtualizationTool, Display, TEXT("Skipping submit of changelist '%s' due to cmdline options"), *ChangelistNumber);
return true;
}
UE_LOG(LogVirtualizationTool, Display, TEXT("Attempting to submit the changelist '%s'"), *ChangelistNumber);
Add a rehydration command to the stand alone virtualization tool, making it easier to reverse the effects of asset virtualization. Unlike previous processes, this one does not require that we load the package and will just manipulate the data storaged in the package trailer. #rb Sebastian.Nordgren #rnx #jira UE-156436 #preflight 62c287f9a3568e30664eb94f ### VA Standalone Tool - We now plan to add much more functionality to the tool than just virtualizing and submitting changelists, so to make this easier I am moving the tool towards a design where it should be fairly easy to add new functionality. - Added FCommand, which is a base class for adding new functionality, simple derive from FCommand and hook it up at the appropriate locations. -- In the future it should be possible for new command types to automatically register themselves to be initiated from the command line. There should be no need to edit UnrealVirtualizationToolApp to add a new command but this will be done as an additional work item. -- At the moment FCommand comes with a number of utility methods to call that cover some common source control commands. -- The original functionality has not yet been moved to the command system and so the code is a little bit weird at the moment. Updating older code to the new system will be done as an additional work item. - FProject/FPlugin have been moved to their own code files. ### Rehydrate Command - The rehydrate command will take a number of packages, check them out of source control and then attempt to virtualize them. - At the moment the chekout logic is fairly basic, we just check out every package supplied, we don't check if the package is virtualized or not yet. This can be improved in additional work items. Ideally by the end of command the only packages that we have checked out should also be rehydrated. - At the moment the command can either take a path of a specific package, a path of a directory to find packages in, or a changelist containing packages that should be rehydrated. - A cleint spec (workspace) can optionally be provided, but if not supplied we will attempt to find a client spec for which to check out the packages. - Currently we will check out the packages to the default change list. ### Rehydrate process - Added the rehydration process in it's own code files in the virtualization module. Like the virtualization process this is exposed in a public header file and no via the Core interface which means it is very specific to our module/implementation. - The process expects that the caller will have checked out any required packages from source control. It will treat being unable to update a package file as an error. - Added PackageUtils.h/.cpp and moved some of the generic code from the virtualization process code there so that it can be shared by the rehydration process. ### Misc Moving away from the using things like FPackagePath as that requires that the correct mount points have been registered for a project and at the moment (with the flakiness of FConfig*) it seems that the best idea would be to prefer absolute file paths where possible. [CL 20982284 by paul chipchase in ue5-main branch]
2022-07-07 06:54:33 -04:00
if (!TryConnectToSourceControl())
{
Add a rehydration command to the stand alone virtualization tool, making it easier to reverse the effects of asset virtualization. Unlike previous processes, this one does not require that we load the package and will just manipulate the data storaged in the package trailer. #rb Sebastian.Nordgren #rnx #jira UE-156436 #preflight 62c287f9a3568e30664eb94f ### VA Standalone Tool - We now plan to add much more functionality to the tool than just virtualizing and submitting changelists, so to make this easier I am moving the tool towards a design where it should be fairly easy to add new functionality. - Added FCommand, which is a base class for adding new functionality, simple derive from FCommand and hook it up at the appropriate locations. -- In the future it should be possible for new command types to automatically register themselves to be initiated from the command line. There should be no need to edit UnrealVirtualizationToolApp to add a new command but this will be done as an additional work item. -- At the moment FCommand comes with a number of utility methods to call that cover some common source control commands. -- The original functionality has not yet been moved to the command system and so the code is a little bit weird at the moment. Updating older code to the new system will be done as an additional work item. - FProject/FPlugin have been moved to their own code files. ### Rehydrate Command - The rehydrate command will take a number of packages, check them out of source control and then attempt to virtualize them. - At the moment the chekout logic is fairly basic, we just check out every package supplied, we don't check if the package is virtualized or not yet. This can be improved in additional work items. Ideally by the end of command the only packages that we have checked out should also be rehydrated. - At the moment the command can either take a path of a specific package, a path of a directory to find packages in, or a changelist containing packages that should be rehydrated. - A cleint spec (workspace) can optionally be provided, but if not supplied we will attempt to find a client spec for which to check out the packages. - Currently we will check out the packages to the default change list. ### Rehydrate process - Added the rehydration process in it's own code files in the virtualization module. Like the virtualization process this is exposed in a public header file and no via the Core interface which means it is very specific to our module/implementation. - The process expects that the caller will have checked out any required packages from source control. It will treat being unable to update a package file as an error. - Added PackageUtils.h/.cpp and moved some of the generic code from the virtualization process code there so that it can be shared by the rehydration process. ### Misc Moving away from the using things like FPackagePath as that requires that the correct mount points have been registered for a project and at the moment (with the flakiness of FConfig*) it seems that the best idea would be to prefer absolute file paths where possible. [CL 20982284 by paul chipchase in ue5-main branch]
2022-07-07 06:54:33 -04:00
UE_LOG(LogVirtualizationTool, Error, TEXT("Submit failed, cannot find a valid source control provider"));
return false;
}
if (!ChangelistToSubmit.IsValid())
{
// This should not be possible, the check and error message is to guard against potential future problems only.
UE_LOG(LogVirtualizationTool, Error, TEXT("Submit failed, could not find the changelist"));
return false;
}
FSourceControlChangelistRef Changelist = ChangelistToSubmit.ToSharedRef();
FSourceControlChangelistStatePtr ChangelistState = SCCProvider->GetState(Changelist, EStateCacheUsage::Use);
if (!ChangelistState.IsValid())
{
UE_LOG(LogVirtualizationTool, Error, TEXT("Submit failed, failed to find the state for the changelist"));
return false;
}
TSharedRef<FCheckIn, ESPMode::ThreadSafe> CheckInOperation = ISourceControlOperation::Create<FCheckIn>();
// Grab the original changelist description then append our tags afterwards
TStringBuilder<512> Description;
Description << ChangelistState->GetDescriptionText().ToString();
for (const FString& Tag : DescriptionTags)
{
Description << TEXT("\n") << Tag;
}
CheckInOperation->SetDescription(FText::FromString(Description.ToString()));
if (SCCProvider->Execute(CheckInOperation, Changelist) == ECommandResult::Succeeded)
{
UE_LOG(LogVirtualizationTool, Display, TEXT("%s"), *CheckInOperation->GetSuccessMessage().ToString());
return true;
}
else
{
Extend the VA stand-alone tool to work on both changelists (the existing functionality) and lists of packages stored in a text file (the new functionality) #rb PJ.Kack #jira UE-143675 #rnx #preflight 624c6a1814634fba5c7e3d71 ### Rationale - The new mode being added allows us to write a list of package file paths to be virtualized to a file on disk and then the file to be passed to the tool so that it knows what to virtualize. This new mode will then be used by P4VUtils.dll which is how we will run the virtualization process via p4v custom tools. - We need this second mode because: -- A workspace could have multiple projects in it, which means a user could try to virtualize a changelist with packages from more than one project. -- Projects can be associated with different engine installations so the user could try to virtualize a changelist with packages using different engine installations, which in turn means that they need to be virtualized by different versions of the stand-alone tool. -- This means we cannot rely on passing the CL number to the tool, install P4VUtils.dll will need to sort the package paths by engine installation, then pass those paths to the correct version of the tool. -- The paths need to be written to a text file to be passed to the tool as the total length of path data can easily exceed the commandline max length limit if someone were to try and virtualize an entire project in one go for example. - Note that the new mode does NOT submit the package files since there might be multiple calls to different versions of the stand-alone tool. In theory if all the packages are under one engine install (which is almost certainly going to be 99.999% of all submits) we could allow the tool to do the submit but that means we need to maintain two different code paths for the same operation. - The existing mode (virtualize and submit a changelist) remains unchanged. It is not likely that an end user will ever use this mode, but it will make for a useful debugging tool in the future. ### Changes - When the tool first starts we now print the log path to the user so that they can more easily find it. This might be something that we want to remove after a while, but during the initial release of the tool it should make getting good debug info back from users with problems somewhat easier. - Some logging and error messages have had small updates made, to make the final output to the user more readable (based on the many many times this tool was run while testing). In some places additional logging has been added where I felt it was not clear to the user as to what was going on. - Connecting to source control has been moved from the changelist parsing code to it's own method ::TryConnectToSourceControl since all modes will require it. - The cmdline help output has been redone to support the multiple command line options we now have. - Although we could infer the mode to run based on the commandline options, we require the users to explicitly state the mode via -Mode=XXX. In general the tool will be invoked from a gui so adding extra commands doesn't really affect the usability of the system and it makes things a bit clearer. [CL 19665307 by paul chipchase in ue5-main branch]
2022-04-07 02:08:03 -04:00
// Even when log suppression is active we still show errors to the users and as the source control
// operation should have logged the problem as an error the user will see it. This means we don't
// have to extract it from CheckInOperation .
UE_LOG(LogVirtualizationTool, Error, TEXT("Submit failed, please check the log!"));
return false;
}
}
bool FUnrealVirtualizationToolApp::TryLoadModules()
{
if (FModuleManager::Get().LoadModule(TEXT("Virtualization"), ELoadModuleFlags::LogFailures) == nullptr)
{
UE_LOG(LogVirtualizationTool, Error, TEXT("Failed to load the 'Virtualization' module"));
}
return true;
}
bool FUnrealVirtualizationToolApp::TryInitEnginePlugins()
{
TRACE_CPUPROFILER_EVENT_SCOPE(TryInitEnginePlugins);
UE_LOG(LogVirtualizationTool, Log, TEXT("Loading Engine Plugins"));
IPluginManager& PluginMgr = IPluginManager::Get();
const FString PerforcePluginPath = FPaths::EnginePluginsDir() / TEXT("Developer/PerforceSourceControl/PerforceSourceControl.uplugin");
FText ErrorMsg;
if (!PluginMgr.AddToPluginsList(PerforcePluginPath, &ErrorMsg))
{
UE_LOG(LogVirtualizationTool, Error, TEXT("Failed to find 'PerforceSourceControl' plugin due to: %s"), *ErrorMsg.ToString());
return false;
}
PluginMgr.MountNewlyCreatedPlugin(TEXT("PerforceSourceControl"));
TSharedPtr<IPlugin> Plugin = PluginMgr.FindPlugin(TEXT("PerforceSourceControl"));
if (Plugin == nullptr || !Plugin->IsEnabled())
{
UE_LOG(LogVirtualizationTool, Error, TEXT("The 'PerforceSourceControl' plugin is disabled."));
return false;
}
return true;
}
Extend the VA stand-alone tool to work on both changelists (the existing functionality) and lists of packages stored in a text file (the new functionality) #rb PJ.Kack #jira UE-143675 #rnx #preflight 624c6a1814634fba5c7e3d71 ### Rationale - The new mode being added allows us to write a list of package file paths to be virtualized to a file on disk and then the file to be passed to the tool so that it knows what to virtualize. This new mode will then be used by P4VUtils.dll which is how we will run the virtualization process via p4v custom tools. - We need this second mode because: -- A workspace could have multiple projects in it, which means a user could try to virtualize a changelist with packages from more than one project. -- Projects can be associated with different engine installations so the user could try to virtualize a changelist with packages using different engine installations, which in turn means that they need to be virtualized by different versions of the stand-alone tool. -- This means we cannot rely on passing the CL number to the tool, install P4VUtils.dll will need to sort the package paths by engine installation, then pass those paths to the correct version of the tool. -- The paths need to be written to a text file to be passed to the tool as the total length of path data can easily exceed the commandline max length limit if someone were to try and virtualize an entire project in one go for example. - Note that the new mode does NOT submit the package files since there might be multiple calls to different versions of the stand-alone tool. In theory if all the packages are under one engine install (which is almost certainly going to be 99.999% of all submits) we could allow the tool to do the submit but that means we need to maintain two different code paths for the same operation. - The existing mode (virtualize and submit a changelist) remains unchanged. It is not likely that an end user will ever use this mode, but it will make for a useful debugging tool in the future. ### Changes - When the tool first starts we now print the log path to the user so that they can more easily find it. This might be something that we want to remove after a while, but during the initial release of the tool it should make getting good debug info back from users with problems somewhat easier. - Some logging and error messages have had small updates made, to make the final output to the user more readable (based on the many many times this tool was run while testing). In some places additional logging has been added where I felt it was not clear to the user as to what was going on. - Connecting to source control has been moved from the changelist parsing code to it's own method ::TryConnectToSourceControl since all modes will require it. - The cmdline help output has been redone to support the multiple command line options we now have. - Although we could infer the mode to run based on the commandline options, we require the users to explicitly state the mode via -Mode=XXX. In general the tool will be invoked from a gui so adding extra commands doesn't really affect the usability of the system and it makes things a bit clearer. [CL 19665307 by paul chipchase in ue5-main branch]
2022-04-07 02:08:03 -04:00
bool FUnrealVirtualizationToolApp::TryConnectToSourceControl()
{
TRACE_CPUPROFILER_EVENT_SCOPE(TryConnectToSourceControl);
Add a rehydration command to the stand alone virtualization tool, making it easier to reverse the effects of asset virtualization. Unlike previous processes, this one does not require that we load the package and will just manipulate the data storaged in the package trailer. #rb Sebastian.Nordgren #rnx #jira UE-156436 #preflight 62c287f9a3568e30664eb94f ### VA Standalone Tool - We now plan to add much more functionality to the tool than just virtualizing and submitting changelists, so to make this easier I am moving the tool towards a design where it should be fairly easy to add new functionality. - Added FCommand, which is a base class for adding new functionality, simple derive from FCommand and hook it up at the appropriate locations. -- In the future it should be possible for new command types to automatically register themselves to be initiated from the command line. There should be no need to edit UnrealVirtualizationToolApp to add a new command but this will be done as an additional work item. -- At the moment FCommand comes with a number of utility methods to call that cover some common source control commands. -- The original functionality has not yet been moved to the command system and so the code is a little bit weird at the moment. Updating older code to the new system will be done as an additional work item. - FProject/FPlugin have been moved to their own code files. ### Rehydrate Command - The rehydrate command will take a number of packages, check them out of source control and then attempt to virtualize them. - At the moment the chekout logic is fairly basic, we just check out every package supplied, we don't check if the package is virtualized or not yet. This can be improved in additional work items. Ideally by the end of command the only packages that we have checked out should also be rehydrated. - At the moment the command can either take a path of a specific package, a path of a directory to find packages in, or a changelist containing packages that should be rehydrated. - A cleint spec (workspace) can optionally be provided, but if not supplied we will attempt to find a client spec for which to check out the packages. - Currently we will check out the packages to the default change list. ### Rehydrate process - Added the rehydration process in it's own code files in the virtualization module. Like the virtualization process this is exposed in a public header file and no via the Core interface which means it is very specific to our module/implementation. - The process expects that the caller will have checked out any required packages from source control. It will treat being unable to update a package file as an error. - Added PackageUtils.h/.cpp and moved some of the generic code from the virtualization process code there so that it can be shared by the rehydration process. ### Misc Moving away from the using things like FPackagePath as that requires that the correct mount points have been registered for a project and at the moment (with the flakiness of FConfig*) it seems that the best idea would be to prefer absolute file paths where possible. [CL 20982284 by paul chipchase in ue5-main branch]
2022-07-07 06:54:33 -04:00
if (SCCProvider.IsValid())
{
// Already connected so just return
return true;
}
Extend the VA stand-alone tool to work on both changelists (the existing functionality) and lists of packages stored in a text file (the new functionality) #rb PJ.Kack #jira UE-143675 #rnx #preflight 624c6a1814634fba5c7e3d71 ### Rationale - The new mode being added allows us to write a list of package file paths to be virtualized to a file on disk and then the file to be passed to the tool so that it knows what to virtualize. This new mode will then be used by P4VUtils.dll which is how we will run the virtualization process via p4v custom tools. - We need this second mode because: -- A workspace could have multiple projects in it, which means a user could try to virtualize a changelist with packages from more than one project. -- Projects can be associated with different engine installations so the user could try to virtualize a changelist with packages using different engine installations, which in turn means that they need to be virtualized by different versions of the stand-alone tool. -- This means we cannot rely on passing the CL number to the tool, install P4VUtils.dll will need to sort the package paths by engine installation, then pass those paths to the correct version of the tool. -- The paths need to be written to a text file to be passed to the tool as the total length of path data can easily exceed the commandline max length limit if someone were to try and virtualize an entire project in one go for example. - Note that the new mode does NOT submit the package files since there might be multiple calls to different versions of the stand-alone tool. In theory if all the packages are under one engine install (which is almost certainly going to be 99.999% of all submits) we could allow the tool to do the submit but that means we need to maintain two different code paths for the same operation. - The existing mode (virtualize and submit a changelist) remains unchanged. It is not likely that an end user will ever use this mode, but it will make for a useful debugging tool in the future. ### Changes - When the tool first starts we now print the log path to the user so that they can more easily find it. This might be something that we want to remove after a while, but during the initial release of the tool it should make getting good debug info back from users with problems somewhat easier. - Some logging and error messages have had small updates made, to make the final output to the user more readable (based on the many many times this tool was run while testing). In some places additional logging has been added where I felt it was not clear to the user as to what was going on. - Connecting to source control has been moved from the changelist parsing code to it's own method ::TryConnectToSourceControl since all modes will require it. - The cmdline help output has been redone to support the multiple command line options we now have. - Although we could infer the mode to run based on the commandline options, we require the users to explicitly state the mode via -Mode=XXX. In general the tool will be invoked from a gui so adding extra commands doesn't really affect the usability of the system and it makes things a bit clearer. [CL 19665307 by paul chipchase in ue5-main branch]
2022-04-07 02:08:03 -04:00
UE_LOG(LogVirtualizationTool, Log, TEXT("Trying to connect to source control..."));
FSourceControlInitSettings SCCSettings(FSourceControlInitSettings::EBehavior::OverrideAll);
SCCSettings.AddSetting(TEXT("P4Client"), ClientSpecName);
SCCProvider = ISourceControlModule::Get().CreateProvider(FName("Perforce"), TEXT("UnrealVirtualizationTool"), SCCSettings);
if (SCCProvider.IsValid())
{
return true;
}
else
{
UE_LOG(LogVirtualizationTool, Error, TEXT("Failed to create a perforce connection"));
return false;
}
}
EInitResult FUnrealVirtualizationToolApp::TryParseCmdLine()
{
TRACE_CPUPROFILER_EVENT_SCOPE(TryParseCmdLine);
UE_LOG(LogVirtualizationTool, Log, TEXT("Parsing the commandline"));
const TCHAR* CmdLine = FCommandLine::Get();
if (CmdLine == nullptr || CmdLine[0] == TEXT('\0'))
{
UE_LOG(LogVirtualizationTool, Error, TEXT("No commandline parameters found!"));
PrintCmdLineHelp();
return EInitResult::Error;
}
if (FParse::Param(CmdLine, TEXT("Help")) || FParse::Param(CmdLine, TEXT("?")))
{
UE_LOG(LogVirtualizationTool, Display, TEXT("Commandline help requested"));
PrintCmdLineHelp();
return EInitResult::EarlyOut;
}
Extend the VA stand-alone tool to work on both changelists (the existing functionality) and lists of packages stored in a text file (the new functionality) #rb PJ.Kack #jira UE-143675 #rnx #preflight 624c6a1814634fba5c7e3d71 ### Rationale - The new mode being added allows us to write a list of package file paths to be virtualized to a file on disk and then the file to be passed to the tool so that it knows what to virtualize. This new mode will then be used by P4VUtils.dll which is how we will run the virtualization process via p4v custom tools. - We need this second mode because: -- A workspace could have multiple projects in it, which means a user could try to virtualize a changelist with packages from more than one project. -- Projects can be associated with different engine installations so the user could try to virtualize a changelist with packages using different engine installations, which in turn means that they need to be virtualized by different versions of the stand-alone tool. -- This means we cannot rely on passing the CL number to the tool, install P4VUtils.dll will need to sort the package paths by engine installation, then pass those paths to the correct version of the tool. -- The paths need to be written to a text file to be passed to the tool as the total length of path data can easily exceed the commandline max length limit if someone were to try and virtualize an entire project in one go for example. - Note that the new mode does NOT submit the package files since there might be multiple calls to different versions of the stand-alone tool. In theory if all the packages are under one engine install (which is almost certainly going to be 99.999% of all submits) we could allow the tool to do the submit but that means we need to maintain two different code paths for the same operation. - The existing mode (virtualize and submit a changelist) remains unchanged. It is not likely that an end user will ever use this mode, but it will make for a useful debugging tool in the future. ### Changes - When the tool first starts we now print the log path to the user so that they can more easily find it. This might be something that we want to remove after a while, but during the initial release of the tool it should make getting good debug info back from users with problems somewhat easier. - Some logging and error messages have had small updates made, to make the final output to the user more readable (based on the many many times this tool was run while testing). In some places additional logging has been added where I felt it was not clear to the user as to what was going on. - Connecting to source control has been moved from the changelist parsing code to it's own method ::TryConnectToSourceControl since all modes will require it. - The cmdline help output has been redone to support the multiple command line options we now have. - Although we could infer the mode to run based on the commandline options, we require the users to explicitly state the mode via -Mode=XXX. In general the tool will be invoked from a gui so adding extra commands doesn't really affect the usability of the system and it makes things a bit clearer. [CL 19665307 by paul chipchase in ue5-main branch]
2022-04-07 02:08:03 -04:00
// First parse the command line options that can apply to all modes
if (!FParse::Value(CmdLine, TEXT("-ClientSpecName="), ClientSpecName))
{
Extend the VA stand-alone tool to work on both changelists (the existing functionality) and lists of packages stored in a text file (the new functionality) #rb PJ.Kack #jira UE-143675 #rnx #preflight 624c6a1814634fba5c7e3d71 ### Rationale - The new mode being added allows us to write a list of package file paths to be virtualized to a file on disk and then the file to be passed to the tool so that it knows what to virtualize. This new mode will then be used by P4VUtils.dll which is how we will run the virtualization process via p4v custom tools. - We need this second mode because: -- A workspace could have multiple projects in it, which means a user could try to virtualize a changelist with packages from more than one project. -- Projects can be associated with different engine installations so the user could try to virtualize a changelist with packages using different engine installations, which in turn means that they need to be virtualized by different versions of the stand-alone tool. -- This means we cannot rely on passing the CL number to the tool, install P4VUtils.dll will need to sort the package paths by engine installation, then pass those paths to the correct version of the tool. -- The paths need to be written to a text file to be passed to the tool as the total length of path data can easily exceed the commandline max length limit if someone were to try and virtualize an entire project in one go for example. - Note that the new mode does NOT submit the package files since there might be multiple calls to different versions of the stand-alone tool. In theory if all the packages are under one engine install (which is almost certainly going to be 99.999% of all submits) we could allow the tool to do the submit but that means we need to maintain two different code paths for the same operation. - The existing mode (virtualize and submit a changelist) remains unchanged. It is not likely that an end user will ever use this mode, but it will make for a useful debugging tool in the future. ### Changes - When the tool first starts we now print the log path to the user so that they can more easily find it. This might be something that we want to remove after a while, but during the initial release of the tool it should make getting good debug info back from users with problems somewhat easier. - Some logging and error messages have had small updates made, to make the final output to the user more readable (based on the many many times this tool was run while testing). In some places additional logging has been added where I felt it was not clear to the user as to what was going on. - Connecting to source control has been moved from the changelist parsing code to it's own method ::TryConnectToSourceControl since all modes will require it. - The cmdline help output has been redone to support the multiple command line options we now have. - Although we could infer the mode to run based on the commandline options, we require the users to explicitly state the mode via -Mode=XXX. In general the tool will be invoked from a gui so adding extra commands doesn't really affect the usability of the system and it makes things a bit clearer. [CL 19665307 by paul chipchase in ue5-main branch]
2022-04-07 02:08:03 -04:00
UE_LOG(LogVirtualizationTool, Error, TEXT("Failed to find cmdline switch 'ClientSpecName', this is a required parameter!"));
PrintCmdLineHelp();
return EInitResult::Error;
}
if (FParse::Param(CmdLine, TEXT("Verbose")))
{
UE_LOG(LogVirtualizationTool, Display, TEXT("Cmdline parameter '-Verbose' found, no longer supressing Display log messages!"));
Add a rehydration command to the stand alone virtualization tool, making it easier to reverse the effects of asset virtualization. Unlike previous processes, this one does not require that we load the package and will just manipulate the data storaged in the package trailer. #rb Sebastian.Nordgren #rnx #jira UE-156436 #preflight 62c287f9a3568e30664eb94f ### VA Standalone Tool - We now plan to add much more functionality to the tool than just virtualizing and submitting changelists, so to make this easier I am moving the tool towards a design where it should be fairly easy to add new functionality. - Added FCommand, which is a base class for adding new functionality, simple derive from FCommand and hook it up at the appropriate locations. -- In the future it should be possible for new command types to automatically register themselves to be initiated from the command line. There should be no need to edit UnrealVirtualizationToolApp to add a new command but this will be done as an additional work item. -- At the moment FCommand comes with a number of utility methods to call that cover some common source control commands. -- The original functionality has not yet been moved to the command system and so the code is a little bit weird at the moment. Updating older code to the new system will be done as an additional work item. - FProject/FPlugin have been moved to their own code files. ### Rehydrate Command - The rehydrate command will take a number of packages, check them out of source control and then attempt to virtualize them. - At the moment the chekout logic is fairly basic, we just check out every package supplied, we don't check if the package is virtualized or not yet. This can be improved in additional work items. Ideally by the end of command the only packages that we have checked out should also be rehydrated. - At the moment the command can either take a path of a specific package, a path of a directory to find packages in, or a changelist containing packages that should be rehydrated. - A cleint spec (workspace) can optionally be provided, but if not supplied we will attempt to find a client spec for which to check out the packages. - Currently we will check out the packages to the default change list. ### Rehydrate process - Added the rehydration process in it's own code files in the virtualization module. Like the virtualization process this is exposed in a public header file and no via the Core interface which means it is very specific to our module/implementation. - The process expects that the caller will have checked out any required packages from source control. It will treat being unable to update a package file as an error. - Added PackageUtils.h/.cpp and moved some of the generic code from the virtualization process code there so that it can be shared by the rehydration process. ### Misc Moving away from the using things like FPackagePath as that requires that the correct mount points have been registered for a project and at the moment (with the flakiness of FConfig*) it seems that the best idea would be to prefer absolute file paths where possible. [CL 20982284 by paul chipchase in ue5-main branch]
2022-07-07 06:54:33 -04:00
OutputDeviceOverride.Reset();
}
else
{
OutputDeviceOverride = MakeUnique<FOverrideOutputDevice>();
}
Extend the VA stand-alone tool to work on both changelists (the existing functionality) and lists of packages stored in a text file (the new functionality) #rb PJ.Kack #jira UE-143675 #rnx #preflight 624c6a1814634fba5c7e3d71 ### Rationale - The new mode being added allows us to write a list of package file paths to be virtualized to a file on disk and then the file to be passed to the tool so that it knows what to virtualize. This new mode will then be used by P4VUtils.dll which is how we will run the virtualization process via p4v custom tools. - We need this second mode because: -- A workspace could have multiple projects in it, which means a user could try to virtualize a changelist with packages from more than one project. -- Projects can be associated with different engine installations so the user could try to virtualize a changelist with packages using different engine installations, which in turn means that they need to be virtualized by different versions of the stand-alone tool. -- This means we cannot rely on passing the CL number to the tool, install P4VUtils.dll will need to sort the package paths by engine installation, then pass those paths to the correct version of the tool. -- The paths need to be written to a text file to be passed to the tool as the total length of path data can easily exceed the commandline max length limit if someone were to try and virtualize an entire project in one go for example. - Note that the new mode does NOT submit the package files since there might be multiple calls to different versions of the stand-alone tool. In theory if all the packages are under one engine install (which is almost certainly going to be 99.999% of all submits) we could allow the tool to do the submit but that means we need to maintain two different code paths for the same operation. - The existing mode (virtualize and submit a changelist) remains unchanged. It is not likely that an end user will ever use this mode, but it will make for a useful debugging tool in the future. ### Changes - When the tool first starts we now print the log path to the user so that they can more easily find it. This might be something that we want to remove after a while, but during the initial release of the tool it should make getting good debug info back from users with problems somewhat easier. - Some logging and error messages have had small updates made, to make the final output to the user more readable (based on the many many times this tool was run while testing). In some places additional logging has been added where I felt it was not clear to the user as to what was going on. - Connecting to source control has been moved from the changelist parsing code to it's own method ::TryConnectToSourceControl since all modes will require it. - The cmdline help output has been redone to support the multiple command line options we now have. - Although we could infer the mode to run based on the commandline options, we require the users to explicitly state the mode via -Mode=XXX. In general the tool will be invoked from a gui so adding extra commands doesn't really affect the usability of the system and it makes things a bit clearer. [CL 19665307 by paul chipchase in ue5-main branch]
2022-04-07 02:08:03 -04:00
// Now parse the mode specific command line options
FString ModeAsString;
if (!FParse::Value(CmdLine, TEXT("-Mode="), ModeAsString))
{
UE_LOG(LogVirtualizationTool, Error, TEXT("Failed to find cmdline switch 'Mode', this is a required parameter!"));
PrintCmdLineHelp();
return EInitResult::Error;
}
LexFromString(Mode, ModeAsString);
switch (Mode)
{
case EMode::Changelist:
return TryParseChangelistCmdLine(CmdLine);
break;
case EMode::PackageList:
return TryParsePackageListCmdLine(CmdLine);
break;
Add a rehydration command to the stand alone virtualization tool, making it easier to reverse the effects of asset virtualization. Unlike previous processes, this one does not require that we load the package and will just manipulate the data storaged in the package trailer. #rb Sebastian.Nordgren #rnx #jira UE-156436 #preflight 62c287f9a3568e30664eb94f ### VA Standalone Tool - We now plan to add much more functionality to the tool than just virtualizing and submitting changelists, so to make this easier I am moving the tool towards a design where it should be fairly easy to add new functionality. - Added FCommand, which is a base class for adding new functionality, simple derive from FCommand and hook it up at the appropriate locations. -- In the future it should be possible for new command types to automatically register themselves to be initiated from the command line. There should be no need to edit UnrealVirtualizationToolApp to add a new command but this will be done as an additional work item. -- At the moment FCommand comes with a number of utility methods to call that cover some common source control commands. -- The original functionality has not yet been moved to the command system and so the code is a little bit weird at the moment. Updating older code to the new system will be done as an additional work item. - FProject/FPlugin have been moved to their own code files. ### Rehydrate Command - The rehydrate command will take a number of packages, check them out of source control and then attempt to virtualize them. - At the moment the chekout logic is fairly basic, we just check out every package supplied, we don't check if the package is virtualized or not yet. This can be improved in additional work items. Ideally by the end of command the only packages that we have checked out should also be rehydrated. - At the moment the command can either take a path of a specific package, a path of a directory to find packages in, or a changelist containing packages that should be rehydrated. - A cleint spec (workspace) can optionally be provided, but if not supplied we will attempt to find a client spec for which to check out the packages. - Currently we will check out the packages to the default change list. ### Rehydrate process - Added the rehydration process in it's own code files in the virtualization module. Like the virtualization process this is exposed in a public header file and no via the Core interface which means it is very specific to our module/implementation. - The process expects that the caller will have checked out any required packages from source control. It will treat being unable to update a package file as an error. - Added PackageUtils.h/.cpp and moved some of the generic code from the virtualization process code there so that it can be shared by the rehydration process. ### Misc Moving away from the using things like FPackagePath as that requires that the correct mount points have been registered for a project and at the moment (with the flakiness of FConfig*) it seems that the best idea would be to prefer absolute file paths where possible. [CL 20982284 by paul chipchase in ue5-main branch]
2022-07-07 06:54:33 -04:00
case EMode::Rehydrate:
CurrentCommand = CreateCommand<FRehydrateCommand>(ModeAsString, CmdLine);
return CurrentCommand.IsValid() ? EInitResult::Success : EInitResult::Error;
break;
Extend the VA stand-alone tool to work on both changelists (the existing functionality) and lists of packages stored in a text file (the new functionality) #rb PJ.Kack #jira UE-143675 #rnx #preflight 624c6a1814634fba5c7e3d71 ### Rationale - The new mode being added allows us to write a list of package file paths to be virtualized to a file on disk and then the file to be passed to the tool so that it knows what to virtualize. This new mode will then be used by P4VUtils.dll which is how we will run the virtualization process via p4v custom tools. - We need this second mode because: -- A workspace could have multiple projects in it, which means a user could try to virtualize a changelist with packages from more than one project. -- Projects can be associated with different engine installations so the user could try to virtualize a changelist with packages using different engine installations, which in turn means that they need to be virtualized by different versions of the stand-alone tool. -- This means we cannot rely on passing the CL number to the tool, install P4VUtils.dll will need to sort the package paths by engine installation, then pass those paths to the correct version of the tool. -- The paths need to be written to a text file to be passed to the tool as the total length of path data can easily exceed the commandline max length limit if someone were to try and virtualize an entire project in one go for example. - Note that the new mode does NOT submit the package files since there might be multiple calls to different versions of the stand-alone tool. In theory if all the packages are under one engine install (which is almost certainly going to be 99.999% of all submits) we could allow the tool to do the submit but that means we need to maintain two different code paths for the same operation. - The existing mode (virtualize and submit a changelist) remains unchanged. It is not likely that an end user will ever use this mode, but it will make for a useful debugging tool in the future. ### Changes - When the tool first starts we now print the log path to the user so that they can more easily find it. This might be something that we want to remove after a while, but during the initial release of the tool it should make getting good debug info back from users with problems somewhat easier. - Some logging and error messages have had small updates made, to make the final output to the user more readable (based on the many many times this tool was run while testing). In some places additional logging has been added where I felt it was not clear to the user as to what was going on. - Connecting to source control has been moved from the changelist parsing code to it's own method ::TryConnectToSourceControl since all modes will require it. - The cmdline help output has been redone to support the multiple command line options we now have. - Although we could infer the mode to run based on the commandline options, we require the users to explicitly state the mode via -Mode=XXX. In general the tool will be invoked from a gui so adding extra commands doesn't really affect the usability of the system and it makes things a bit clearer. [CL 19665307 by paul chipchase in ue5-main branch]
2022-04-07 02:08:03 -04:00
case EMode::Unknown:
default:
UE_LOG(LogVirtualizationTool, Error, TEXT("Unexpected value for the cmdline switch 'Mode', this is a required parameter!"));
PrintCmdLineHelp();
return EInitResult::Error;
break;
}
}
Extend the VA stand-alone tool to work on both changelists (the existing functionality) and lists of packages stored in a text file (the new functionality) #rb PJ.Kack #jira UE-143675 #rnx #preflight 624c6a1814634fba5c7e3d71 ### Rationale - The new mode being added allows us to write a list of package file paths to be virtualized to a file on disk and then the file to be passed to the tool so that it knows what to virtualize. This new mode will then be used by P4VUtils.dll which is how we will run the virtualization process via p4v custom tools. - We need this second mode because: -- A workspace could have multiple projects in it, which means a user could try to virtualize a changelist with packages from more than one project. -- Projects can be associated with different engine installations so the user could try to virtualize a changelist with packages using different engine installations, which in turn means that they need to be virtualized by different versions of the stand-alone tool. -- This means we cannot rely on passing the CL number to the tool, install P4VUtils.dll will need to sort the package paths by engine installation, then pass those paths to the correct version of the tool. -- The paths need to be written to a text file to be passed to the tool as the total length of path data can easily exceed the commandline max length limit if someone were to try and virtualize an entire project in one go for example. - Note that the new mode does NOT submit the package files since there might be multiple calls to different versions of the stand-alone tool. In theory if all the packages are under one engine install (which is almost certainly going to be 99.999% of all submits) we could allow the tool to do the submit but that means we need to maintain two different code paths for the same operation. - The existing mode (virtualize and submit a changelist) remains unchanged. It is not likely that an end user will ever use this mode, but it will make for a useful debugging tool in the future. ### Changes - When the tool first starts we now print the log path to the user so that they can more easily find it. This might be something that we want to remove after a while, but during the initial release of the tool it should make getting good debug info back from users with problems somewhat easier. - Some logging and error messages have had small updates made, to make the final output to the user more readable (based on the many many times this tool was run while testing). In some places additional logging has been added where I felt it was not clear to the user as to what was going on. - Connecting to source control has been moved from the changelist parsing code to it's own method ::TryConnectToSourceControl since all modes will require it. - The cmdline help output has been redone to support the multiple command line options we now have. - Although we could infer the mode to run based on the commandline options, we require the users to explicitly state the mode via -Mode=XXX. In general the tool will be invoked from a gui so adding extra commands doesn't really affect the usability of the system and it makes things a bit clearer. [CL 19665307 by paul chipchase in ue5-main branch]
2022-04-07 02:08:03 -04:00
EInitResult FUnrealVirtualizationToolApp::TryParseChangelistCmdLine(const TCHAR* CmdLine)
{
if (FParse::Value(CmdLine, TEXT("-Changelist="), ChangelistNumber))
{
// Optional switches
if (FParse::Param(CmdLine, TEXT("NoSubmit")))
{
UE_LOG(LogVirtualizationTool, Display, TEXT("Cmdline parameter '-NoSubmit' found, the changelist will be virtualized but not submitted!"));
}
else
{
ProcessOptions |= EProcessOptions::Submit;
}
UE_LOG(LogVirtualizationTool, Display, TEXT("Attempting to virtualize changelist '%s'"), *ChangelistNumber);
return EInitResult::Success;
}
else
{
UE_LOG(LogVirtualizationTool, Error, TEXT("Failed to find cmdline switch 'Changelist', this is a required parameter for the 'Changelist' mode!"));
PrintCmdLineHelp();
return EInitResult::Error;
}
}
EInitResult FUnrealVirtualizationToolApp::TryParsePackageListCmdLine(const TCHAR* CmdLine)
{
if (FParse::Value(CmdLine, TEXT("-Path="), PackageListPath))
{
UE_LOG(LogVirtualizationTool, Display, TEXT("Virtualizing packages found in package list: '%s'"), *PackageListPath);
Extend the VA stand-alone tool to work on both changelists (the existing functionality) and lists of packages stored in a text file (the new functionality) #rb PJ.Kack #jira UE-143675 #rnx #preflight 624c6a1814634fba5c7e3d71 ### Rationale - The new mode being added allows us to write a list of package file paths to be virtualized to a file on disk and then the file to be passed to the tool so that it knows what to virtualize. This new mode will then be used by P4VUtils.dll which is how we will run the virtualization process via p4v custom tools. - We need this second mode because: -- A workspace could have multiple projects in it, which means a user could try to virtualize a changelist with packages from more than one project. -- Projects can be associated with different engine installations so the user could try to virtualize a changelist with packages using different engine installations, which in turn means that they need to be virtualized by different versions of the stand-alone tool. -- This means we cannot rely on passing the CL number to the tool, install P4VUtils.dll will need to sort the package paths by engine installation, then pass those paths to the correct version of the tool. -- The paths need to be written to a text file to be passed to the tool as the total length of path data can easily exceed the commandline max length limit if someone were to try and virtualize an entire project in one go for example. - Note that the new mode does NOT submit the package files since there might be multiple calls to different versions of the stand-alone tool. In theory if all the packages are under one engine install (which is almost certainly going to be 99.999% of all submits) we could allow the tool to do the submit but that means we need to maintain two different code paths for the same operation. - The existing mode (virtualize and submit a changelist) remains unchanged. It is not likely that an end user will ever use this mode, but it will make for a useful debugging tool in the future. ### Changes - When the tool first starts we now print the log path to the user so that they can more easily find it. This might be something that we want to remove after a while, but during the initial release of the tool it should make getting good debug info back from users with problems somewhat easier. - Some logging and error messages have had small updates made, to make the final output to the user more readable (based on the many many times this tool was run while testing). In some places additional logging has been added where I felt it was not clear to the user as to what was going on. - Connecting to source control has been moved from the changelist parsing code to it's own method ::TryConnectToSourceControl since all modes will require it. - The cmdline help output has been redone to support the multiple command line options we now have. - Although we could infer the mode to run based on the commandline options, we require the users to explicitly state the mode via -Mode=XXX. In general the tool will be invoked from a gui so adding extra commands doesn't really affect the usability of the system and it makes things a bit clearer. [CL 19665307 by paul chipchase in ue5-main branch]
2022-04-07 02:08:03 -04:00
return EInitResult::Success;
}
else
{
UE_LOG(LogVirtualizationTool, Error, TEXT("Failed to find cmdline switch 'Path', this is a required parameter for the 'PackageList mode!"));
PrintCmdLineHelp();
return EInitResult::Error;
}
}
bool FUnrealVirtualizationToolApp::TryParseChangelist(TArray<FString>& OutPackages)
{
TRACE_CPUPROFILER_EVENT_SCOPE(TryParseChangelist);
if (!TryConnectToSourceControl())
{
return false;
}
UE_LOG(LogVirtualizationTool, Display, TEXT("Attempting to parse changelist '%s' in workspace '%s'"), *ChangelistNumber, *ClientSpecName);
if (!SCCProvider.IsValid())
{
Extend the VA stand-alone tool to work on both changelists (the existing functionality) and lists of packages stored in a text file (the new functionality) #rb PJ.Kack #jira UE-143675 #rnx #preflight 624c6a1814634fba5c7e3d71 ### Rationale - The new mode being added allows us to write a list of package file paths to be virtualized to a file on disk and then the file to be passed to the tool so that it knows what to virtualize. This new mode will then be used by P4VUtils.dll which is how we will run the virtualization process via p4v custom tools. - We need this second mode because: -- A workspace could have multiple projects in it, which means a user could try to virtualize a changelist with packages from more than one project. -- Projects can be associated with different engine installations so the user could try to virtualize a changelist with packages using different engine installations, which in turn means that they need to be virtualized by different versions of the stand-alone tool. -- This means we cannot rely on passing the CL number to the tool, install P4VUtils.dll will need to sort the package paths by engine installation, then pass those paths to the correct version of the tool. -- The paths need to be written to a text file to be passed to the tool as the total length of path data can easily exceed the commandline max length limit if someone were to try and virtualize an entire project in one go for example. - Note that the new mode does NOT submit the package files since there might be multiple calls to different versions of the stand-alone tool. In theory if all the packages are under one engine install (which is almost certainly going to be 99.999% of all submits) we could allow the tool to do the submit but that means we need to maintain two different code paths for the same operation. - The existing mode (virtualize and submit a changelist) remains unchanged. It is not likely that an end user will ever use this mode, but it will make for a useful debugging tool in the future. ### Changes - When the tool first starts we now print the log path to the user so that they can more easily find it. This might be something that we want to remove after a while, but during the initial release of the tool it should make getting good debug info back from users with problems somewhat easier. - Some logging and error messages have had small updates made, to make the final output to the user more readable (based on the many many times this tool was run while testing). In some places additional logging has been added where I felt it was not clear to the user as to what was going on. - Connecting to source control has been moved from the changelist parsing code to it's own method ::TryConnectToSourceControl since all modes will require it. - The cmdline help output has been redone to support the multiple command line options we now have. - Although we could infer the mode to run based on the commandline options, we require the users to explicitly state the mode via -Mode=XXX. In general the tool will be invoked from a gui so adding extra commands doesn't really affect the usability of the system and it makes things a bit clearer. [CL 19665307 by paul chipchase in ue5-main branch]
2022-04-07 02:08:03 -04:00
UE_LOG(LogVirtualizationTool, Error, TEXT("No valid source control connection found!"));
return false;
}
SCCProvider->Init(true);
if (!SCCProvider->UsesChangelists())
{
UE_LOG(LogVirtualizationTool, Error, TEXT("The source control provider does not support the use of changelists"));
return false;
}
TArray<FSourceControlChangelistRef> Changelists = SCCProvider->GetChangelists(EStateCacheUsage::ForceUpdate);
if (Changelists.IsEmpty())
{
UE_LOG(LogVirtualizationTool, Error, TEXT("Failed to find any changelists"));
return false;
}
TArray<FSourceControlChangelistStateRef> ChangelistsStates;
if (SCCProvider->GetState(Changelists, ChangelistsStates, EStateCacheUsage::Use) != ECommandResult::Succeeded)
{
UE_LOG(LogVirtualizationTool, Error, TEXT("Failed to find changelist data"));
return false;
}
for (FSourceControlChangelistStateRef& ChangelistState : ChangelistsStates)
{
const FText DisplayText = ChangelistState->GetDisplayText();
if (ChangelistNumber == DisplayText.ToString())
{
TSharedRef<FUpdatePendingChangelistsStatus, ESPMode::ThreadSafe> Operation = ISourceControlOperation::Create<FUpdatePendingChangelistsStatus>();
// TODO: Updating only the CL we want does not currently work and even if it did we still end up with a pointless
// p4 changes command before updating the files. Given we know the changelist number via FSourceControlChangelistRef
// we should be able to just request the file states be updated.
// This is also a lot of code to write for a simple "give me all files in a changelist" operation, if we don't add
// support directly in the API we should move this to a utility namespace in the source control module.
FSourceControlChangelistRef Changelist = ChangelistState->GetChangelist();
Operation->SetChangelistsToUpdate(MakeArrayView(&Changelist, 1));
Operation->SetUpdateFilesStates(true);
if (SCCProvider->Execute(Operation, EConcurrency::Synchronous) != ECommandResult::Succeeded)
{
UE_LOG(LogVirtualizationTool, Error, TEXT("Failed to find the files in changelist '%s'"), *ChangelistNumber);
return false;
}
const TArray<FSourceControlStateRef>& FilesinChangelist = ChangelistState->GetFilesStates();
for (const FSourceControlStateRef& FileState : FilesinChangelist)
{
if (::IsPackageFile(FileState->GetFilename()))
{
Extend the VA stand-alone tool to work on both changelists (the existing functionality) and lists of packages stored in a text file (the new functionality) #rb PJ.Kack #jira UE-143675 #rnx #preflight 624c6a1814634fba5c7e3d71 ### Rationale - The new mode being added allows us to write a list of package file paths to be virtualized to a file on disk and then the file to be passed to the tool so that it knows what to virtualize. This new mode will then be used by P4VUtils.dll which is how we will run the virtualization process via p4v custom tools. - We need this second mode because: -- A workspace could have multiple projects in it, which means a user could try to virtualize a changelist with packages from more than one project. -- Projects can be associated with different engine installations so the user could try to virtualize a changelist with packages using different engine installations, which in turn means that they need to be virtualized by different versions of the stand-alone tool. -- This means we cannot rely on passing the CL number to the tool, install P4VUtils.dll will need to sort the package paths by engine installation, then pass those paths to the correct version of the tool. -- The paths need to be written to a text file to be passed to the tool as the total length of path data can easily exceed the commandline max length limit if someone were to try and virtualize an entire project in one go for example. - Note that the new mode does NOT submit the package files since there might be multiple calls to different versions of the stand-alone tool. In theory if all the packages are under one engine install (which is almost certainly going to be 99.999% of all submits) we could allow the tool to do the submit but that means we need to maintain two different code paths for the same operation. - The existing mode (virtualize and submit a changelist) remains unchanged. It is not likely that an end user will ever use this mode, but it will make for a useful debugging tool in the future. ### Changes - When the tool first starts we now print the log path to the user so that they can more easily find it. This might be something that we want to remove after a while, but during the initial release of the tool it should make getting good debug info back from users with problems somewhat easier. - Some logging and error messages have had small updates made, to make the final output to the user more readable (based on the many many times this tool was run while testing). In some places additional logging has been added where I felt it was not clear to the user as to what was going on. - Connecting to source control has been moved from the changelist parsing code to it's own method ::TryConnectToSourceControl since all modes will require it. - The cmdline help output has been redone to support the multiple command line options we now have. - Although we could infer the mode to run based on the commandline options, we require the users to explicitly state the mode via -Mode=XXX. In general the tool will be invoked from a gui so adding extra commands doesn't really affect the usability of the system and it makes things a bit clearer. [CL 19665307 by paul chipchase in ue5-main branch]
2022-04-07 02:08:03 -04:00
OutPackages.Add(FileState->GetFilename());
}
else
{
UE_LOG(LogVirtualizationTool, Log, TEXT("Ignoring non-package file '%s'"), *FileState->GetFilename());
}
}
ChangelistToSubmit = Changelist;
return true;
}
}
UE_LOG(LogVirtualizationTool, Error, TEXT("Failed to find the changelist '%s'"), *ChangelistNumber);
return false;
}
Extend the VA stand-alone tool to work on both changelists (the existing functionality) and lists of packages stored in a text file (the new functionality) #rb PJ.Kack #jira UE-143675 #rnx #preflight 624c6a1814634fba5c7e3d71 ### Rationale - The new mode being added allows us to write a list of package file paths to be virtualized to a file on disk and then the file to be passed to the tool so that it knows what to virtualize. This new mode will then be used by P4VUtils.dll which is how we will run the virtualization process via p4v custom tools. - We need this second mode because: -- A workspace could have multiple projects in it, which means a user could try to virtualize a changelist with packages from more than one project. -- Projects can be associated with different engine installations so the user could try to virtualize a changelist with packages using different engine installations, which in turn means that they need to be virtualized by different versions of the stand-alone tool. -- This means we cannot rely on passing the CL number to the tool, install P4VUtils.dll will need to sort the package paths by engine installation, then pass those paths to the correct version of the tool. -- The paths need to be written to a text file to be passed to the tool as the total length of path data can easily exceed the commandline max length limit if someone were to try and virtualize an entire project in one go for example. - Note that the new mode does NOT submit the package files since there might be multiple calls to different versions of the stand-alone tool. In theory if all the packages are under one engine install (which is almost certainly going to be 99.999% of all submits) we could allow the tool to do the submit but that means we need to maintain two different code paths for the same operation. - The existing mode (virtualize and submit a changelist) remains unchanged. It is not likely that an end user will ever use this mode, but it will make for a useful debugging tool in the future. ### Changes - When the tool first starts we now print the log path to the user so that they can more easily find it. This might be something that we want to remove after a while, but during the initial release of the tool it should make getting good debug info back from users with problems somewhat easier. - Some logging and error messages have had small updates made, to make the final output to the user more readable (based on the many many times this tool was run while testing). In some places additional logging has been added where I felt it was not clear to the user as to what was going on. - Connecting to source control has been moved from the changelist parsing code to it's own method ::TryConnectToSourceControl since all modes will require it. - The cmdline help output has been redone to support the multiple command line options we now have. - Although we could infer the mode to run based on the commandline options, we require the users to explicitly state the mode via -Mode=XXX. In general the tool will be invoked from a gui so adding extra commands doesn't really affect the usability of the system and it makes things a bit clearer. [CL 19665307 by paul chipchase in ue5-main branch]
2022-04-07 02:08:03 -04:00
bool FUnrealVirtualizationToolApp::TryParsePackageList(TArray<FString>& OutPackages)
{
TRACE_CPUPROFILER_EVENT_SCOPE(TrySortFilesByProject);
UE_LOG(LogVirtualizationTool, Display, TEXT("Parsing the package list..."));
Extend the VA stand-alone tool to work on both changelists (the existing functionality) and lists of packages stored in a text file (the new functionality) #rb PJ.Kack #jira UE-143675 #rnx #preflight 624c6a1814634fba5c7e3d71 ### Rationale - The new mode being added allows us to write a list of package file paths to be virtualized to a file on disk and then the file to be passed to the tool so that it knows what to virtualize. This new mode will then be used by P4VUtils.dll which is how we will run the virtualization process via p4v custom tools. - We need this second mode because: -- A workspace could have multiple projects in it, which means a user could try to virtualize a changelist with packages from more than one project. -- Projects can be associated with different engine installations so the user could try to virtualize a changelist with packages using different engine installations, which in turn means that they need to be virtualized by different versions of the stand-alone tool. -- This means we cannot rely on passing the CL number to the tool, install P4VUtils.dll will need to sort the package paths by engine installation, then pass those paths to the correct version of the tool. -- The paths need to be written to a text file to be passed to the tool as the total length of path data can easily exceed the commandline max length limit if someone were to try and virtualize an entire project in one go for example. - Note that the new mode does NOT submit the package files since there might be multiple calls to different versions of the stand-alone tool. In theory if all the packages are under one engine install (which is almost certainly going to be 99.999% of all submits) we could allow the tool to do the submit but that means we need to maintain two different code paths for the same operation. - The existing mode (virtualize and submit a changelist) remains unchanged. It is not likely that an end user will ever use this mode, but it will make for a useful debugging tool in the future. ### Changes - When the tool first starts we now print the log path to the user so that they can more easily find it. This might be something that we want to remove after a while, but during the initial release of the tool it should make getting good debug info back from users with problems somewhat easier. - Some logging and error messages have had small updates made, to make the final output to the user more readable (based on the many many times this tool was run while testing). In some places additional logging has been added where I felt it was not clear to the user as to what was going on. - Connecting to source control has been moved from the changelist parsing code to it's own method ::TryConnectToSourceControl since all modes will require it. - The cmdline help output has been redone to support the multiple command line options we now have. - Although we could infer the mode to run based on the commandline options, we require the users to explicitly state the mode via -Mode=XXX. In general the tool will be invoked from a gui so adding extra commands doesn't really affect the usability of the system and it makes things a bit clearer. [CL 19665307 by paul chipchase in ue5-main branch]
2022-04-07 02:08:03 -04:00
if (!IFileManager::Get().FileExists(*PackageListPath))
{
UE_LOG(LogVirtualizationTool, Error, TEXT("\tThe package list '%s' does not exist"), *PackageListPath);
Extend the VA stand-alone tool to work on both changelists (the existing functionality) and lists of packages stored in a text file (the new functionality) #rb PJ.Kack #jira UE-143675 #rnx #preflight 624c6a1814634fba5c7e3d71 ### Rationale - The new mode being added allows us to write a list of package file paths to be virtualized to a file on disk and then the file to be passed to the tool so that it knows what to virtualize. This new mode will then be used by P4VUtils.dll which is how we will run the virtualization process via p4v custom tools. - We need this second mode because: -- A workspace could have multiple projects in it, which means a user could try to virtualize a changelist with packages from more than one project. -- Projects can be associated with different engine installations so the user could try to virtualize a changelist with packages using different engine installations, which in turn means that they need to be virtualized by different versions of the stand-alone tool. -- This means we cannot rely on passing the CL number to the tool, install P4VUtils.dll will need to sort the package paths by engine installation, then pass those paths to the correct version of the tool. -- The paths need to be written to a text file to be passed to the tool as the total length of path data can easily exceed the commandline max length limit if someone were to try and virtualize an entire project in one go for example. - Note that the new mode does NOT submit the package files since there might be multiple calls to different versions of the stand-alone tool. In theory if all the packages are under one engine install (which is almost certainly going to be 99.999% of all submits) we could allow the tool to do the submit but that means we need to maintain two different code paths for the same operation. - The existing mode (virtualize and submit a changelist) remains unchanged. It is not likely that an end user will ever use this mode, but it will make for a useful debugging tool in the future. ### Changes - When the tool first starts we now print the log path to the user so that they can more easily find it. This might be something that we want to remove after a while, but during the initial release of the tool it should make getting good debug info back from users with problems somewhat easier. - Some logging and error messages have had small updates made, to make the final output to the user more readable (based on the many many times this tool was run while testing). In some places additional logging has been added where I felt it was not clear to the user as to what was going on. - Connecting to source control has been moved from the changelist parsing code to it's own method ::TryConnectToSourceControl since all modes will require it. - The cmdline help output has been redone to support the multiple command line options we now have. - Although we could infer the mode to run based on the commandline options, we require the users to explicitly state the mode via -Mode=XXX. In general the tool will be invoked from a gui so adding extra commands doesn't really affect the usability of the system and it makes things a bit clearer. [CL 19665307 by paul chipchase in ue5-main branch]
2022-04-07 02:08:03 -04:00
return false;
}
if (FFileHelper::LoadFileToStringArray(OutPackages, *PackageListPath))
{
// We don't have control over how the package list was generated so make sure that the paths
// are in the format that we want.
for (FString& PackagePath : OutPackages)
{
FPaths::NormalizeFilename(PackagePath);
}
return true;
}
else
{
UE_LOG(LogVirtualizationTool, Error, TEXT("\tFailed to parse the package list '%s'"), *PackageListPath);
Extend the VA stand-alone tool to work on both changelists (the existing functionality) and lists of packages stored in a text file (the new functionality) #rb PJ.Kack #jira UE-143675 #rnx #preflight 624c6a1814634fba5c7e3d71 ### Rationale - The new mode being added allows us to write a list of package file paths to be virtualized to a file on disk and then the file to be passed to the tool so that it knows what to virtualize. This new mode will then be used by P4VUtils.dll which is how we will run the virtualization process via p4v custom tools. - We need this second mode because: -- A workspace could have multiple projects in it, which means a user could try to virtualize a changelist with packages from more than one project. -- Projects can be associated with different engine installations so the user could try to virtualize a changelist with packages using different engine installations, which in turn means that they need to be virtualized by different versions of the stand-alone tool. -- This means we cannot rely on passing the CL number to the tool, install P4VUtils.dll will need to sort the package paths by engine installation, then pass those paths to the correct version of the tool. -- The paths need to be written to a text file to be passed to the tool as the total length of path data can easily exceed the commandline max length limit if someone were to try and virtualize an entire project in one go for example. - Note that the new mode does NOT submit the package files since there might be multiple calls to different versions of the stand-alone tool. In theory if all the packages are under one engine install (which is almost certainly going to be 99.999% of all submits) we could allow the tool to do the submit but that means we need to maintain two different code paths for the same operation. - The existing mode (virtualize and submit a changelist) remains unchanged. It is not likely that an end user will ever use this mode, but it will make for a useful debugging tool in the future. ### Changes - When the tool first starts we now print the log path to the user so that they can more easily find it. This might be something that we want to remove after a while, but during the initial release of the tool it should make getting good debug info back from users with problems somewhat easier. - Some logging and error messages have had small updates made, to make the final output to the user more readable (based on the many many times this tool was run while testing). In some places additional logging has been added where I felt it was not clear to the user as to what was going on. - Connecting to source control has been moved from the changelist parsing code to it's own method ::TryConnectToSourceControl since all modes will require it. - The cmdline help output has been redone to support the multiple command line options we now have. - Although we could infer the mode to run based on the commandline options, we require the users to explicitly state the mode via -Mode=XXX. In general the tool will be invoked from a gui so adding extra commands doesn't really affect the usability of the system and it makes things a bit clearer. [CL 19665307 by paul chipchase in ue5-main branch]
2022-04-07 02:08:03 -04:00
return false;
}
}
bool FUnrealVirtualizationToolApp::TrySortFilesByProject(const TArray<FString>& Packages)
{
TRACE_CPUPROFILER_EVENT_SCOPE(TrySortFilesByProject);
UE_LOG(LogVirtualizationTool, Display, TEXT("Sorting files by project..."));
Extend the VA stand-alone tool to work on both changelists (the existing functionality) and lists of packages stored in a text file (the new functionality) #rb PJ.Kack #jira UE-143675 #rnx #preflight 624c6a1814634fba5c7e3d71 ### Rationale - The new mode being added allows us to write a list of package file paths to be virtualized to a file on disk and then the file to be passed to the tool so that it knows what to virtualize. This new mode will then be used by P4VUtils.dll which is how we will run the virtualization process via p4v custom tools. - We need this second mode because: -- A workspace could have multiple projects in it, which means a user could try to virtualize a changelist with packages from more than one project. -- Projects can be associated with different engine installations so the user could try to virtualize a changelist with packages using different engine installations, which in turn means that they need to be virtualized by different versions of the stand-alone tool. -- This means we cannot rely on passing the CL number to the tool, install P4VUtils.dll will need to sort the package paths by engine installation, then pass those paths to the correct version of the tool. -- The paths need to be written to a text file to be passed to the tool as the total length of path data can easily exceed the commandline max length limit if someone were to try and virtualize an entire project in one go for example. - Note that the new mode does NOT submit the package files since there might be multiple calls to different versions of the stand-alone tool. In theory if all the packages are under one engine install (which is almost certainly going to be 99.999% of all submits) we could allow the tool to do the submit but that means we need to maintain two different code paths for the same operation. - The existing mode (virtualize and submit a changelist) remains unchanged. It is not likely that an end user will ever use this mode, but it will make for a useful debugging tool in the future. ### Changes - When the tool first starts we now print the log path to the user so that they can more easily find it. This might be something that we want to remove after a while, but during the initial release of the tool it should make getting good debug info back from users with problems somewhat easier. - Some logging and error messages have had small updates made, to make the final output to the user more readable (based on the many many times this tool was run while testing). In some places additional logging has been added where I felt it was not clear to the user as to what was going on. - Connecting to source control has been moved from the changelist parsing code to it's own method ::TryConnectToSourceControl since all modes will require it. - The cmdline help output has been redone to support the multiple command line options we now have. - Although we could infer the mode to run based on the commandline options, we require the users to explicitly state the mode via -Mode=XXX. In general the tool will be invoked from a gui so adding extra commands doesn't really affect the usability of the system and it makes things a bit clearer. [CL 19665307 by paul chipchase in ue5-main branch]
2022-04-07 02:08:03 -04:00
for (const FString& PackagePath : Packages)
{
FString ProjectFilePath;
FString PluginFilePath;
if (TryFindProject(PackagePath, ProjectFilePath, PluginFilePath))
{
Add a rehydration command to the stand alone virtualization tool, making it easier to reverse the effects of asset virtualization. Unlike previous processes, this one does not require that we load the package and will just manipulate the data storaged in the package trailer. #rb Sebastian.Nordgren #rnx #jira UE-156436 #preflight 62c287f9a3568e30664eb94f ### VA Standalone Tool - We now plan to add much more functionality to the tool than just virtualizing and submitting changelists, so to make this easier I am moving the tool towards a design where it should be fairly easy to add new functionality. - Added FCommand, which is a base class for adding new functionality, simple derive from FCommand and hook it up at the appropriate locations. -- In the future it should be possible for new command types to automatically register themselves to be initiated from the command line. There should be no need to edit UnrealVirtualizationToolApp to add a new command but this will be done as an additional work item. -- At the moment FCommand comes with a number of utility methods to call that cover some common source control commands. -- The original functionality has not yet been moved to the command system and so the code is a little bit weird at the moment. Updating older code to the new system will be done as an additional work item. - FProject/FPlugin have been moved to their own code files. ### Rehydrate Command - The rehydrate command will take a number of packages, check them out of source control and then attempt to virtualize them. - At the moment the chekout logic is fairly basic, we just check out every package supplied, we don't check if the package is virtualized or not yet. This can be improved in additional work items. Ideally by the end of command the only packages that we have checked out should also be rehydrated. - At the moment the command can either take a path of a specific package, a path of a directory to find packages in, or a changelist containing packages that should be rehydrated. - A cleint spec (workspace) can optionally be provided, but if not supplied we will attempt to find a client spec for which to check out the packages. - Currently we will check out the packages to the default change list. ### Rehydrate process - Added the rehydration process in it's own code files in the virtualization module. Like the virtualization process this is exposed in a public header file and no via the Core interface which means it is very specific to our module/implementation. - The process expects that the caller will have checked out any required packages from source control. It will treat being unable to update a package file as an error. - Added PackageUtils.h/.cpp and moved some of the generic code from the virtualization process code there so that it can be shared by the rehydration process. ### Misc Moving away from the using things like FPackagePath as that requires that the correct mount points have been registered for a project and at the moment (with the flakiness of FConfig*) it seems that the best idea would be to prefer absolute file paths where possible. [CL 20982284 by paul chipchase in ue5-main branch]
2022-07-07 06:54:33 -04:00
FProject& Project = FindOrAddProject(MoveTemp(ProjectFilePath));
if (PluginFilePath.IsEmpty())
{
Project.AddFile(PackagePath);
}
else
{
Add a rehydration command to the stand alone virtualization tool, making it easier to reverse the effects of asset virtualization. Unlike previous processes, this one does not require that we load the package and will just manipulate the data storaged in the package trailer. #rb Sebastian.Nordgren #rnx #jira UE-156436 #preflight 62c287f9a3568e30664eb94f ### VA Standalone Tool - We now plan to add much more functionality to the tool than just virtualizing and submitting changelists, so to make this easier I am moving the tool towards a design where it should be fairly easy to add new functionality. - Added FCommand, which is a base class for adding new functionality, simple derive from FCommand and hook it up at the appropriate locations. -- In the future it should be possible for new command types to automatically register themselves to be initiated from the command line. There should be no need to edit UnrealVirtualizationToolApp to add a new command but this will be done as an additional work item. -- At the moment FCommand comes with a number of utility methods to call that cover some common source control commands. -- The original functionality has not yet been moved to the command system and so the code is a little bit weird at the moment. Updating older code to the new system will be done as an additional work item. - FProject/FPlugin have been moved to their own code files. ### Rehydrate Command - The rehydrate command will take a number of packages, check them out of source control and then attempt to virtualize them. - At the moment the chekout logic is fairly basic, we just check out every package supplied, we don't check if the package is virtualized or not yet. This can be improved in additional work items. Ideally by the end of command the only packages that we have checked out should also be rehydrated. - At the moment the command can either take a path of a specific package, a path of a directory to find packages in, or a changelist containing packages that should be rehydrated. - A cleint spec (workspace) can optionally be provided, but if not supplied we will attempt to find a client spec for which to check out the packages. - Currently we will check out the packages to the default change list. ### Rehydrate process - Added the rehydration process in it's own code files in the virtualization module. Like the virtualization process this is exposed in a public header file and no via the Core interface which means it is very specific to our module/implementation. - The process expects that the caller will have checked out any required packages from source control. It will treat being unable to update a package file as an error. - Added PackageUtils.h/.cpp and moved some of the generic code from the virtualization process code there so that it can be shared by the rehydration process. ### Misc Moving away from the using things like FPackagePath as that requires that the correct mount points have been registered for a project and at the moment (with the flakiness of FConfig*) it seems that the best idea would be to prefer absolute file paths where possible. [CL 20982284 by paul chipchase in ue5-main branch]
2022-07-07 06:54:33 -04:00
Project.AddPluginFile(PackagePath, MoveTemp(PluginFilePath));
}
}
}
UE_LOG(LogVirtualizationTool, Display, TEXT("\tThe package files are associated with '%d' projects(s)"), Projects.Num());
return true;
}
bool FUnrealVirtualizationToolApp::TryFindProject(const FString& PackagePath, FString& ProjectFilePath, FString& PluginFilePath) const
{
TRACE_CPUPROFILER_EVENT_SCOPE(TryFindProject);
// TODO: This could be heavily optimized by caching known project files
int32 ContentIndex = PackagePath.Find(TEXT("/content/"), ESearchCase::IgnoreCase, ESearchDir::FromEnd);
// Early out if there is not a single content directory in the path
if (ContentIndex == INDEX_NONE)
{
UE_LOG(LogVirtualizationTool, Warning, TEXT("'%s' is not under a content directory"), *PackagePath);
return false;
}
while (ContentIndex != INDEX_NONE)
{
// Assume that the project directory is the parent of the /content/ directory
FString ProjectDirectory = PackagePath.Left(ContentIndex);
FString PluginDirectory;
TArray<FString> ProjectFile;
TArray<FString> PluginFile;
IFileManager::Get().FindFiles(ProjectFile, *ProjectDirectory, TEXT(".uproject"));
if (ProjectFile.IsEmpty())
{
// If there was no project file, the package could be in a plugin, so lets check for that
PluginDirectory = ProjectDirectory;
IFileManager::Get().FindFiles(PluginFile, *PluginDirectory, TEXT(".uplugin"));
if (PluginFile.Num() == 1)
{
PluginFilePath = PluginDirectory / PluginFile[0];
// We have a valid plugin file, so we should be able to find a /plugins/ directory which will be just below the project directory
const int32 PluginIndex = PluginDirectory.Find(TEXT("/plugins/"), ESearchCase::IgnoreCase, ESearchDir::FromEnd);
if (PluginIndex != INDEX_NONE)
{
// We found the plugin root directory so the one above it should be the project directory
ProjectDirectory = PluginDirectory.Left(PluginIndex);
IFileManager::Get().FindFiles(ProjectFile, *ProjectDirectory, TEXT(".uproject"));
}
}
else if (PluginFile.Num() > 1)
{
UE_LOG(LogVirtualizationTool, Warning, TEXT("Found multiple .uplugin files for '%s' at '%s'"), *PackagePath, *PluginDirectory);
return false;
}
}
if (ProjectFile.Num() == 1)
{
ProjectFilePath = ProjectDirectory / ProjectFile[0];
return true;
}
else if (!ProjectFile.IsEmpty())
{
UE_LOG(LogVirtualizationTool, Warning, TEXT("Found multiple .uproject files for '%s' at '%s'"), *PackagePath, *ProjectDirectory);
return false;
}
// Could be more than one content directory in the path so lets keep looking
ContentIndex = PackagePath.Find(TEXT("/content/"), ESearchCase::IgnoreCase, ESearchDir::FromEnd, ContentIndex);
}
// We found one or more content directories but none of them contained a project file
UE_LOG(LogVirtualizationTool, Warning, TEXT("Failed to find project file for '%s'"), *PackagePath);
return false;
}
Add a rehydration command to the stand alone virtualization tool, making it easier to reverse the effects of asset virtualization. Unlike previous processes, this one does not require that we load the package and will just manipulate the data storaged in the package trailer. #rb Sebastian.Nordgren #rnx #jira UE-156436 #preflight 62c287f9a3568e30664eb94f ### VA Standalone Tool - We now plan to add much more functionality to the tool than just virtualizing and submitting changelists, so to make this easier I am moving the tool towards a design where it should be fairly easy to add new functionality. - Added FCommand, which is a base class for adding new functionality, simple derive from FCommand and hook it up at the appropriate locations. -- In the future it should be possible for new command types to automatically register themselves to be initiated from the command line. There should be no need to edit UnrealVirtualizationToolApp to add a new command but this will be done as an additional work item. -- At the moment FCommand comes with a number of utility methods to call that cover some common source control commands. -- The original functionality has not yet been moved to the command system and so the code is a little bit weird at the moment. Updating older code to the new system will be done as an additional work item. - FProject/FPlugin have been moved to their own code files. ### Rehydrate Command - The rehydrate command will take a number of packages, check them out of source control and then attempt to virtualize them. - At the moment the chekout logic is fairly basic, we just check out every package supplied, we don't check if the package is virtualized or not yet. This can be improved in additional work items. Ideally by the end of command the only packages that we have checked out should also be rehydrated. - At the moment the command can either take a path of a specific package, a path of a directory to find packages in, or a changelist containing packages that should be rehydrated. - A cleint spec (workspace) can optionally be provided, but if not supplied we will attempt to find a client spec for which to check out the packages. - Currently we will check out the packages to the default change list. ### Rehydrate process - Added the rehydration process in it's own code files in the virtualization module. Like the virtualization process this is exposed in a public header file and no via the Core interface which means it is very specific to our module/implementation. - The process expects that the caller will have checked out any required packages from source control. It will treat being unable to update a package file as an error. - Added PackageUtils.h/.cpp and moved some of the generic code from the virtualization process code there so that it can be shared by the rehydration process. ### Misc Moving away from the using things like FPackagePath as that requires that the correct mount points have been registered for a project and at the moment (with the flakiness of FConfig*) it seems that the best idea would be to prefer absolute file paths where possible. [CL 20982284 by paul chipchase in ue5-main branch]
2022-07-07 06:54:33 -04:00
FProject& FUnrealVirtualizationToolApp::FindOrAddProject(FString&& ProjectFilePath)
{
FProject* Project = Projects.FindByPredicate([&ProjectFilePath](const FProject& Project)->bool
{
Add a rehydration command to the stand alone virtualization tool, making it easier to reverse the effects of asset virtualization. Unlike previous processes, this one does not require that we load the package and will just manipulate the data storaged in the package trailer. #rb Sebastian.Nordgren #rnx #jira UE-156436 #preflight 62c287f9a3568e30664eb94f ### VA Standalone Tool - We now plan to add much more functionality to the tool than just virtualizing and submitting changelists, so to make this easier I am moving the tool towards a design where it should be fairly easy to add new functionality. - Added FCommand, which is a base class for adding new functionality, simple derive from FCommand and hook it up at the appropriate locations. -- In the future it should be possible for new command types to automatically register themselves to be initiated from the command line. There should be no need to edit UnrealVirtualizationToolApp to add a new command but this will be done as an additional work item. -- At the moment FCommand comes with a number of utility methods to call that cover some common source control commands. -- The original functionality has not yet been moved to the command system and so the code is a little bit weird at the moment. Updating older code to the new system will be done as an additional work item. - FProject/FPlugin have been moved to their own code files. ### Rehydrate Command - The rehydrate command will take a number of packages, check them out of source control and then attempt to virtualize them. - At the moment the chekout logic is fairly basic, we just check out every package supplied, we don't check if the package is virtualized or not yet. This can be improved in additional work items. Ideally by the end of command the only packages that we have checked out should also be rehydrated. - At the moment the command can either take a path of a specific package, a path of a directory to find packages in, or a changelist containing packages that should be rehydrated. - A cleint spec (workspace) can optionally be provided, but if not supplied we will attempt to find a client spec for which to check out the packages. - Currently we will check out the packages to the default change list. ### Rehydrate process - Added the rehydration process in it's own code files in the virtualization module. Like the virtualization process this is exposed in a public header file and no via the Core interface which means it is very specific to our module/implementation. - The process expects that the caller will have checked out any required packages from source control. It will treat being unable to update a package file as an error. - Added PackageUtils.h/.cpp and moved some of the generic code from the virtualization process code there so that it can be shared by the rehydration process. ### Misc Moving away from the using things like FPackagePath as that requires that the correct mount points have been registered for a project and at the moment (with the flakiness of FConfig*) it seems that the best idea would be to prefer absolute file paths where possible. [CL 20982284 by paul chipchase in ue5-main branch]
2022-07-07 06:54:33 -04:00
return Project.GetProjectFilePath() == ProjectFilePath;
});
if (Project != nullptr)
{
return *Project;
}
else
{
Add a rehydration command to the stand alone virtualization tool, making it easier to reverse the effects of asset virtualization. Unlike previous processes, this one does not require that we load the package and will just manipulate the data storaged in the package trailer. #rb Sebastian.Nordgren #rnx #jira UE-156436 #preflight 62c287f9a3568e30664eb94f ### VA Standalone Tool - We now plan to add much more functionality to the tool than just virtualizing and submitting changelists, so to make this easier I am moving the tool towards a design where it should be fairly easy to add new functionality. - Added FCommand, which is a base class for adding new functionality, simple derive from FCommand and hook it up at the appropriate locations. -- In the future it should be possible for new command types to automatically register themselves to be initiated from the command line. There should be no need to edit UnrealVirtualizationToolApp to add a new command but this will be done as an additional work item. -- At the moment FCommand comes with a number of utility methods to call that cover some common source control commands. -- The original functionality has not yet been moved to the command system and so the code is a little bit weird at the moment. Updating older code to the new system will be done as an additional work item. - FProject/FPlugin have been moved to their own code files. ### Rehydrate Command - The rehydrate command will take a number of packages, check them out of source control and then attempt to virtualize them. - At the moment the chekout logic is fairly basic, we just check out every package supplied, we don't check if the package is virtualized or not yet. This can be improved in additional work items. Ideally by the end of command the only packages that we have checked out should also be rehydrated. - At the moment the command can either take a path of a specific package, a path of a directory to find packages in, or a changelist containing packages that should be rehydrated. - A cleint spec (workspace) can optionally be provided, but if not supplied we will attempt to find a client spec for which to check out the packages. - Currently we will check out the packages to the default change list. ### Rehydrate process - Added the rehydration process in it's own code files in the virtualization module. Like the virtualization process this is exposed in a public header file and no via the Core interface which means it is very specific to our module/implementation. - The process expects that the caller will have checked out any required packages from source control. It will treat being unable to update a package file as an error. - Added PackageUtils.h/.cpp and moved some of the generic code from the virtualization process code there so that it can be shared by the rehydration process. ### Misc Moving away from the using things like FPackagePath as that requires that the correct mount points have been registered for a project and at the moment (with the flakiness of FConfig*) it seems that the best idea would be to prefer absolute file paths where possible. [CL 20982284 by paul chipchase in ue5-main branch]
2022-07-07 06:54:33 -04:00
int32 Index = Projects.Emplace(MoveTemp(ProjectFilePath));
return Projects[Index];
}
}
} // namespace UE::Virtualization