#rb Per.Larsson
#jira UE-169626
#rnx
#preflight 63bd670a71079a8d1c0e837b
- Since the API was forcing the caller to pass in a results structure to be filled in, we might as well make it the return value.
- Added a ::WasSuccessful method to the results structures that can be used instead of checking if the result had errors or not.
- Remove the reset method from FVirtualizationResult/FRehydrationResult as they no longer need it.
- The older deprecated methods still use the results enum, so we cannot easily deprecate those enums yet.
[CL 23626072 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-169626
#rnx
#preflight 63bd0ebfd862fdd347bce1fe
### VA System
- This allows us to provide the user with more ways to customize the rehydration and return more detailed info about it if the calling code wishes to log additional info. In both cases we can extend the options and the data returned without changing the API.
- At the moment the only flag we support is 'Checkout', which requests that the rehydration process checkout any package that it needs to modify rather than giving an error. This means that the user does not need to check packages out from revision control before running the rehydration process.
-- We still check if packages can be modified and warn the user if they can't, as package files could be locked in other ways.
- The rehydration process will now long the time taken if verbose logging is set for the category 'LogVirtualization'
### UnrealVirtualizationTool
- The virtualize command now reports how many packages were checked out if the flag was set.
- The rehydration command now supports a '-Checkout' commandline flag, which when enabled will use the new api to checkout the packages that need to be checked out when rehydrated.
[CL 23625132 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-169626
#rnx
#preflight 639c4112012902cb8db43e13
- This allows us to provide the user with more ways to customize the virtualization and return more detailed info about it if the calling code wishes to log additional info. In both cases we can extend the options and the data returned without changing the API.
- Previously if we virtualized a package that was not checked out in revision control we would warn the user and then skip updating the package file on disk. This means the payloads would be uploaded but the user would be left with no local changes. Since sometimes we know we don't need to check out any package (virtualizing the packages in a change list for example) we don't want to always incur the cost of polling reivision control to see which packages do need checking out. This is why we now allow the caller to request package files be checked out via the new options enum EVirtualizationOptions.
-- If the EVirtualizationOptions::Checkout flag is provided we will poll the revision control status of all package files and then check out those which need it.
-- We still check if packages can be modified and warn the user if they can't, as package files could be locked in other ways.
- Added a new utility function to SourceControlUtilties to make it easier to check out packages. There is similar functionality elsewhere in the code base but the virtualization module is too low level to make use of it.
- Updated existing code that calls ::TryVirtualizePackages and add cases of ''using namespace UE::Virtualization' where required to improve readability.
- The UnrealVirtualizationTool now supports a new cmdline option "-checkout" that can be used when virtualizing packages. This will checkout any package that was actually virtualized so the result can be saved back out to the workspace domain. This means we no longer require the caller to have checked out the packages before running the tool.
[CL 23536832 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-163093
#rnx
#preflight 63526abcae33b04ec1ee4a65
- The caller can now request more than one payload to be pulled froim the system at a time. In theory this will allow backends to pull data quicker.
-- The file system backend works with the new system but makes no attempt to take advantage as the backend is not intended for serious production use.
-- The DDC backend should be taking full advantage of the new batching API
-- Note that although the source control API does attempt to take advantage of batching, the internal source control API implementation is not doing so. This will be addressed in a future submit.
- This does make the pulling logic a bit more complicated in FVirtualizationManager as we need to deal with some payloads being found in the first backend, some in the second and some in the third etc.
-- To help deal with this a new class FPullRequestCollection has been added which abstracts a lot of complexity away.
[CL 22707955 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-165308
#rnx
#preflight 634ff17e8449f85a47390fbe
- It was pointed out that a command line option containing '-' characters could confuse parsers and it is not worth the risk to keep them.
- Now we accept versions of each command line without the '-' character which will be the version that we document for users in 5.1
[CL 22650940 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
#preflight 634805f7f93be0f634b8ed5b
- Added a new FArchive 'FFixedBufferWriterArchive' that will let me serialize to an already allocated, fixed sized memory buffer.
- Moved the code for opening and reading packages to shared utility functions so that it is easier to keep the error messages consistent.
- Moved common code to a new function ::TryRehydrateBuilder so that both rehydration methods can share it.
- Note that technically we are changing the API since now we accept a TConstArrayView for the package file paths for both the virtualization and rehydration paths, but existing code will be compatible with this change.
[CL 22511406 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-163103
#rnx
#preflight 6318989c2b7fe03eb664e9f0
### VirtualizationSystem/VirtualizationManager
- Added an overload of ::Push taking just one FPUshRequest so that people don't have to keep adding MakeArrayView boiler plate when pushing a single request
- Change the order of the last two parameters for the raw ::Push call as this will group all of the payload specific parameters together and leave the target storage type at the end. It is unlikely that anyone is calling the older version but it has been deprecated for safety.
### IVirtualizationBackend
- The none FPushRequest overload of ::PushData is no longer virtual, it just converts the parameters to FPushRequest and calls that overload instead. In this way we now only have one pushing code path in our backends. We could probably look into removing this overload at this point (since the higher level IVirtualizationSystem will now convert all push requests into a FPushRequest form) but it is not considered worth it at the moment when the simple overload covers our needs.
- Removed EPushResult in favour of just returning true/false for the overall operation.If the caller needs a more detailed breakdown then they will have to use an overload that takes an FPushRequest over raw parameters.
-- At the moment FPushRequest does not contain a full breakdown of what happened, so with this submit we are effectively losing the ability to find out if the payload was already in the backends or not, however the batch version of push was already not returning this info so it is not a big loss. Fixing FPushRequest to return a better break down of what happened will be done in UE-160942
- Removed the none batch Push paths from the source control and ddc backends as they already supported batch pushing.
- File backend needed to be converted to supporting batch pushing, which is pretty much the same code as before except we need to iterate over the container of FPushRequests.
-- The backend does not early out on error as it tends to be quite fast. We might want to consider an official policy for the VA system, if we should early out of errors or not.
[CL 21907558 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-161673
#rnx
#preflight 6304ad0da45b007ea26e3087
- We currently provide ways to change these options via code define, config file options and command line options but it has been requested that we also support this level of customization via cvars as well.
- The cvar 'VA.DisableSystem' when enabled will act as though [Core.ContentVirtualization]SystemName is set to 'None' and disable the VA system entirely by mounting the null reference implementation.
- The cvar 'VA.LazyInitSystem' when enabled will act as though [Core.ContentVirtualization]LazyInit is set to true and the VA system will not be initialized until first used.
- The logic for both of these was getting a little verbose, so I moved them to their own functions to make ::Initialize easier to read.
[CL 21512236 by robomerge in ue5-main branch]
#rb Per.Larsson
#jira UE-161416
#rnx
#preflight 62ff8fb5086f90bbc4ffd4ae
- When the VA is disabled we mount the FNullVirtualizationSystem instead.
-- Added logging when this system is used to push or pull payloads. The logging can be disabled by changing the define 'UE_LOG_ON_NULLSYSTEM_USE' to 0
-- The new logging gives errors, but if the define UE_USING_NULLSYSTEM_IS_ERROR is set to 0 then we will log warnings instead.
- EditorBulkData now checks before pulling payloads and if the current VA system is disabled then we print a specific error for this case.
- Overall this should make it easier to identify that the problem is that VA is disabled if we only have access to a small snippet of the log surrounding the crash
[CL 21500336 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-161296
#rnx
#preflight 62fe3ce73d3fb466b229bcc0
- There are some usecases that require the VA system to initialize the first time it is accessed (usually the first time we attempt to pull a virtualized payload) rather than be initialized in the program start up. This change provides three different methods to achieve this:
-- Setting the define 'UE_VIRTUALIZATION_SYSTEM_LAZY_INIT' to 1 in a programs .target.cs
-- Setting [Core.ContentVirtualization]LazyInit=true in the Engine ini file
-- Running with the commandline option -VA-LazyInit
- If we detect that the source control backend is being initialized on a background thread we do not try to run the FConnect operation. The backend will still work but this does reduce the potential error checking on initialization. This is done because the FConnect operation currently only works on the main thread and to change this would be a bigger work item than we can schedule at the moment.
- UE::Virtualization::Initialize functions now take a EInitializationFlags enum as a parameter. This enum allows the call to ignore all lazy init settings and force the initialization immediately. This is useful for programs like the Virtualization standalone tool which just needs to start the system when needed.
-- The call to ::Initialize in LaunchEngineLoop passes in None and does not ignore lazy initialization.
-- Calls to ::Initialize in the UnrealVirtualizationTool however all use EInitializationFlags::ForceInitialize and ignore lazy initialization settings.
- Fixed an odd bug in UE::Virtualization::Initialize where the error path (if the config file cannot be found) was using a different start up code path.
- Add asserts when assigning to GVirtualizationSystem to make sure that it is null. This is not 100% safe but should catch some potential threading issues, if any.
- Add an assert after lazy initialization (IVirtualizationSystem::Get) to make sure that GVirtualizationSystem was assigned a valid object.
- Improve how we check for legacy values in [Core.ContentVirtualization]. We now support multiple allowed values.
- Added a way to poll if a VA system has been initialize yet or not, this allows us to avoid initializing a VA system if one has not yet been created and we try to:
-- Dump VA profiling stats after cooking
-- Send VA stats to studio analytics
- Note that currently using lazy init loading will probably cause the VA statistics panel not to work, this will be fixed in future work where we will allow the panel to register for a callback when the system is initialized.
[CL 21467510 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-156750
#rnx
#preflight 62f212e13b773d04161ee7dd
### Problem
- The payloads stored in map files tend to change more than other assets and would cause a lot more churn in the VA system.
- Some other systems like the landscape component are not able to sensibly continue if their payloads cannot be accessed (heightmaps for exmaple) and would prefer not to allow virtualization.
- As a short term fix we need an option to disable the virtualization on all payloads in map files. Future improvements to the filtering system will allow systems to more easily opt their payloads out of virtualization. When this is functional we might want to change the default from true to false.
### Feature
- The config optionf for this filtering is "[Core.VirtualizationModule]FilterMapContent=True"
- Testing if the owning UObject for a payload is in a umap can get tricky, because we not only need to check the umap but we also need to check if it is in a "_builddata.uasset" file, which is an additional file we store next to a umap containing things like lightmaps etc.
- At the moment we check for this by finding the outermost object for the given owner and check to see if it is a ULevel, UWorld or UMapBuildDataRegistry. This is a bit of a kludge but the types we need to check against are not accessible by this module and making them accessible will pull in a lot of dependencies that we'd prefer not to add.
-- One improvement might be to tag the FLinkerSave with the info we need and passing that into the serialization process rather than trying to work it out ourselves but I am wary of making that change until we are 100% sure that we want to keep this feature.
### Refactor
- Removed IVirtualizationSystem::IsDisabledForObject and replaced it with ::FilterPayload which can return multiple reasons for preventing a payload from virtualizing. (the method was added during 5.1 development so it should be fine to just replace it without deprecation)
-- The original behaviour for FVirtualizationManager::IsDisabledForObject has been moved to FVirtualizationManager::ShouldVirtualizeAsset
- Added a new header to declare enums/types used by the various parts of the virtualization system and started by moving EPayloadFilterReason there from the package trailer header. This allows both the core API and PackageTrailer to use EPayloadFilterReason without creating overburdened header dependencies.
-- EPayloadFilterReason has moved from the UE namespace to UE::Virtualization so the package trailer code needed updating accordingly.
- EditorBulkData will ask the virtualization system for the base filter reason, then add it's own reasons if UE_ENABLE_VIRTUALIZATION_TOGGLE is enabled. This bit of code will be removed for 5.1
[CL 21283179 by paul chipchase in ue5-main branch]
#rb none
#jira UE-159596
#rnx
#preflight 62d94160110da0f944afdd32
### Problem
- When rolling our virtualization to projects we have run into a number of set up problems which has caused users to fail to submit packages from the editor as when the virtualization failed we prevented the overall submit from going through.
- Although we will eventually work through all of these problems and add better fault tolerance for now it is annoying for the content team so we need an easy way for a project to allow package submits to be completed even if the virtualization process part of the submit failed.
- Worth noting that the virtualization process failing should never affect the state of the package files, the only downside is that the packages may not be virtualized.
### Fix
- The virtualization/rehydration of packages now return an enum instead of a bool although at the moment we are only returning Success and Failed, but it will allow us to expand in the future
- Combined the code calling the virtualization process from both the source control window and the source control changelist window into a single utility function.
-- Now the two windows only diverge on how they report failure to the user.
- This new utility function now checks to see if the virtualization manager suggests if failing the virtualization process should block any package submission to source control or not.
- I am not happy with this solution, but I haven't come up with anything that feels right, this solution will at least work.
[CL 21199597 by paul chipchase in ue5-main branch]
#rb Sebastian.Nordgren
#rnx
#jira UE-156436
#preflight 62c287f9a3568e30664eb94f
### VA Standalone Tool
- We now plan to add much more functionality to the tool than just virtualizing and submitting changelists, so to make this easier I am moving the tool towards a design where it should be fairly easy to add new functionality.
- Added FCommand, which is a base class for adding new functionality, simple derive from FCommand and hook it up at the appropriate locations.
-- In the future it should be possible for new command types to automatically register themselves to be initiated from the command line. There should be no need to edit UnrealVirtualizationToolApp to add a new command but this will be done as an additional work item.
-- At the moment FCommand comes with a number of utility methods to call that cover some common source control commands.
-- The original functionality has not yet been moved to the command system and so the code is a little bit weird at the moment. Updating older code to the new system will be done as an additional work item.
- FProject/FPlugin have been moved to their own code files.
### Rehydrate Command
- The rehydrate command will take a number of packages, check them out of source control and then attempt to virtualize them.
- At the moment the chekout logic is fairly basic, we just check out every package supplied, we don't check if the package is virtualized or not yet. This can be improved in additional work items. Ideally by the end of command the only packages that we have checked out should also be rehydrated.
- At the moment the command can either take a path of a specific package, a path of a directory to find packages in, or a changelist containing packages that should be rehydrated.
- A cleint spec (workspace) can optionally be provided, but if not supplied we will attempt to find a client spec for which to check out the packages.
- Currently we will check out the packages to the default change list.
### Rehydrate process
- Added the rehydration process in it's own code files in the virtualization module. Like the virtualization process this is exposed in a public header file and no via the Core interface which means it is very specific to our module/implementation.
- The process expects that the caller will have checked out any required packages from source control. It will treat being unable to update a package file as an error.
- Added PackageUtils.h/.cpp and moved some of the generic code from the virtualization process code there so that it can be shared by the rehydration process.
### Misc
Moving away from the using things like FPackagePath as that requires that the correct mount points have been registered for a project and at the moment (with the flakiness of FConfig*) it seems that the best idea would be to prefer absolute file paths where possible.
[CL 20982284 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
#preflight 62ab3b4a634e82e5d145f5a3
- Dumping of stats to the log file is now something that can be invoked via the IVirtualizationSystem api, so anything can call it, if desired.
- Removed the OnExit callback from the virtualization manager, we no longer print the stats to the log file on exit, but instead print them on demand.
- Add a call at the end of the cooking commandlet to dump the virtualization stats so that they appear around the same part of the log file as the cooking stats.
- Changed the verbosity from 'log' to 'display' so that we can see the stats more easily in horde.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 20699898 via CL 20699907 via CL 20699911
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v968-20684695)
[CL 20700183 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
#preflight 62a9bff813004691f9830b8d
- I think when I was first adding it, the type was originally a struct but was changed to an enum before it was submitted, however I failed to update the name.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 20668490 via CL 20668521 via CL 20668532
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v955-20579017)
[CL 20669764 by paul chipchase in ue5-main branch]
#rb trivial
#rnx
#preflight
- A new define has been added 'UE_DISABLE_VIRTUALIZATION_SYSTEM' which when set to 1 will prevent the virtualization system from finding a valid system to mount, meaning it will default to the null implementation.
- The expected use it to add 'GlobalDefinitions.Add("UE_DISABLE_VIRTUALIZATION_SYSTEM=1");' to a target.cs to disable virtualization for a target
- This wouldn't be needed if the virtualization module was a plugin as it could be disabled by a .uproject file. We should consider moving the module.
[CL 20278057 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
#jira UE-151377
#preflight 628364050039ea57a52d6989
### Virtualization
- [Core.ContentVirtualization] in the engine ini file now supports an array called 'DisabledAsset' which can be used to name asset types that should not virtualize their payloads.
-- By default (in BaseEngine.ini) we have disabled the StaticMesh asset as we know it will crash if a payload is missing and the SoundWave asset as it still is pending testing.
- This new way to disable virtualization is data driven. The older hard coded method has not been removed but will likely be reworked in a future submit.
- Now when an editor bulkdata is adding it's payload to the package trailer builder during package save it will poll the virtualization system with a call to the new method ::IsDisabledForObject by passing in it's owner.
-- If the owner is valid and was present in the 'DisabledAsset' array then the method will return true and the EPayloadFlags::DisableVirtualization flag will be applied.
### Package Trailer
- The pre-existing functionality of enum EPayloadFilter has been moved to a new enum EPayloadStorageType as will only filter payloads based on their storage type.
- EPayloadFilter has been modified to filter payloads based on functionality although at the moment the only thing it can filter for is to return payloads that can be virtualized, it is left for future expansion.
- EPayloadFlags has been reduced to a uint16 with the remaining 2bytes being turned into a new member EPayloadFilterReason.
- This new member allows us to record the exact reason why a payload is excluded from virtualization. If it is zero then the payload can virtualize, otherwise it will contain one or more reasons as to why it is being excluded. For this reason the enum is a bitfield.
- Added overloads of ::GetPayloads and ::GetNumPayloads that take EPayloadFilter rather than a EPayloadStorageType
- Added wrappers around all AccessMode types for FLookupTableEntry.
- FPackageTrailerBuilder has been extended to take a EPayloadFilterReason so that the caller can already provide a reason why the payload cannot be virtualized.
-- As a future peace of work this will probably be changed and we will ask the caller to pass in the owner UObject pointer instead and then we will process the filtering when building the package trailer to a) keep all of the filtering code in one place b) keep the filtering consistent
### PackageSubmissionChecks
- The virtualization process in will now request the payloads that can be virtualized from the package trailer, which respects the new payload flag, rather than requesting all locally stored payloads.
### UObjects
- There is no need for the SoundWave or MeshDescription classes to opt out of virtualization on construction. This will be done when the package is saved and is now data driven rather than being hardcoded.
### DumpPackagePayloadInfo
- The command has been updated to also display the filter reasons applied to each payload
[CL 20240971 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#rnx
#preflight 623ae40e7b69b01ec15da75c
#robomerge FNNC
- We need to expose the code that checks a set of packages for non virtualized payloads and virtualizes them so that the stand alone submission tool can access it.
- Currently I am adding this to the IVirtualizationSystem interface to avoid adding additional interface, but we might want to consider moving this some day as it is not a great fit with the rest of the interface.
[CL 19479757 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#rnx
#preflight 62276138671c913c0515c3b4
- To make it easier to extend the number of parameters when initializing the virtualization system, the function has been changed to accept a single struct, FInitParams, which will contains all potential parameters.
- Calling the version of UE::Virtualization::Initialize without any parameters will fallback to using the default values.
- The virtualization manager now passes the provided project name onto each backend that it creates.
- The source control backend now stored the provided project name and uses that when creating the submit description for new payloads rather than polling FApp for the current project name.
-- This is required for the stand alone virtualization application which will not have a specific project set.
[CL 19303638 by paul chipchase in ue5-main branch]
#rb trivial
#rnx
#preflight 62270345e83598518f08fb11
- The warning is correct, if we failed to find a valid config file it would be a nullptr and would be incorrectly accessed when setting up the fallback FNullVirtualizationSystem.
- We now log an error if no config file is provided by the caller and we are unable to find one to use.
- When setting up the fallback FNullVirtualizationSystem we pass a reference to an empty FConfigFile on the stack instead since we know that it is not used. This avoids any potential invalid access.
[CL 19300174 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#rnx
#preflight 6225cf4cb57e715222ca15aa
- IVirtualizationSystem now has a new ::Initialize method which is called after it is first created. This allows the system to return a proper error when setting up which was not possible when the set up code ran in the constructor.
- The method also takes a FConfigFile reference which should be used when parsing settings from the config file system.
-- The config file being passed to this method, is either the optional config file provided when calling UE::Virtualization::Initialize, or (if that was not set) we find the GEngineIni config file from the global cache to use.
- This should ensure that the entire virtualization system is now loading the settings from the same set of config files where as before the code was a bit inconsistent (loading it in place, using the GConfig global, or trying to find the file from the global cache)
- If the system fails to create we now log if the problem was that we could not find a valid factory for the given system (meaning the module the system lives in was not yet loaded) or if the initialization phase failed.
- FVirtualizationManager now passes a single FConfigFile into all of it's methods that are parsing settings from it.
[CL 19283763 by paul chipchase in ue5-main branch]
#rb trivial
#jira UE-134434
#rnx
#preflight 620d1a10742ffef420223b15
- Changing a number of log items from 'Log' to 'Display' so that the info shows up more easily.
- Removed a number of log items that added no useful infomation
- Edited some of the existing log items to be clearer.
[CL 19031890 by paul chipchase in ue5-main branch]
