#rb Per.Larsson
#jira UE-156750
#rnx
#preflight 62f212e13b773d04161ee7dd
### Problem
- The payloads stored in map files tend to change more than other assets and would cause a lot more churn in the VA system.
- Some other systems like the landscape component are not able to sensibly continue if their payloads cannot be accessed (heightmaps for exmaple) and would prefer not to allow virtualization.
- As a short term fix we need an option to disable the virtualization on all payloads in map files. Future improvements to the filtering system will allow systems to more easily opt their payloads out of virtualization. When this is functional we might want to change the default from true to false.
### Feature
- The config optionf for this filtering is "[Core.VirtualizationModule]FilterMapContent=True"
- Testing if the owning UObject for a payload is in a umap can get tricky, because we not only need to check the umap but we also need to check if it is in a "_builddata.uasset" file, which is an additional file we store next to a umap containing things like lightmaps etc.
- At the moment we check for this by finding the outermost object for the given owner and check to see if it is a ULevel, UWorld or UMapBuildDataRegistry. This is a bit of a kludge but the types we need to check against are not accessible by this module and making them accessible will pull in a lot of dependencies that we'd prefer not to add.
-- One improvement might be to tag the FLinkerSave with the info we need and passing that into the serialization process rather than trying to work it out ourselves but I am wary of making that change until we are 100% sure that we want to keep this feature.
### Refactor
- Removed IVirtualizationSystem::IsDisabledForObject and replaced it with ::FilterPayload which can return multiple reasons for preventing a payload from virtualizing. (the method was added during 5.1 development so it should be fine to just replace it without deprecation)
-- The original behaviour for FVirtualizationManager::IsDisabledForObject has been moved to FVirtualizationManager::ShouldVirtualizeAsset
- Added a new header to declare enums/types used by the various parts of the virtualization system and started by moving EPayloadFilterReason there from the package trailer header. This allows both the core API and PackageTrailer to use EPayloadFilterReason without creating overburdened header dependencies.
-- EPayloadFilterReason has moved from the UE namespace to UE::Virtualization so the package trailer code needed updating accordingly.
- EditorBulkData will ask the virtualization system for the base filter reason, then add it's own reasons if UE_ENABLE_VIRTUALIZATION_TOGGLE is enabled. This bit of code will be removed for 5.1
[CL 21283179 by paul chipchase in ue5-main branch]
#rb none
#rnx
#jira UE-159597
#preflight
- The option is [Core.VirtualizationModule]VirtualizationProcessTag=
- If left blank then no tag will be added
- The internal virtualization process code no longer needs to return a tag since it will now be applied by the virtualization manager instead.
-- Note that we still follow the same logic, we only stamp if no errors were encountered.
[CL 21217286 by paul chipchase in ue5-main branch]
#rb trivial
#rnx
#preflight 62d96169110da0f944b69f4f
- The bucket entry was already marked as optional in the code documentation but was giving a fatal error if it was not set. The value is now optional and will default to 'BulkData' if not set.
- We no longer consider both the local and remote transfer policy being disabled as a fatal error, when the work UE-143676 is done this can be improved further.
[CL 21200968 by paul chipchase in ue5-main branch]
#rb none
#jira UE-159596
#rnx
#preflight 62d94160110da0f944afdd32
### Problem
- When rolling our virtualization to projects we have run into a number of set up problems which has caused users to fail to submit packages from the editor as when the virtualization failed we prevented the overall submit from going through.
- Although we will eventually work through all of these problems and add better fault tolerance for now it is annoying for the content team so we need an easy way for a project to allow package submits to be completed even if the virtualization process part of the submit failed.
- Worth noting that the virtualization process failing should never affect the state of the package files, the only downside is that the packages may not be virtualized.
### Fix
- The virtualization/rehydration of packages now return an enum instead of a bool although at the moment we are only returning Success and Failed, but it will allow us to expand in the future
- Combined the code calling the virtualization process from both the source control window and the source control changelist window into a single utility function.
-- Now the two windows only diverge on how they report failure to the user.
- This new utility function now checks to see if the virtualization manager suggests if failing the virtualization process should block any package submission to source control or not.
- I am not happy with this solution, but I haven't come up with anything that feels right, this solution will at least work.
[CL 21199597 by paul chipchase in ue5-main branch]
#rb trivial
#rnx
#preflight 62d92875185da2495f3a8d09
- The missing default values were taken from BaseEngine.ini.
- Made it a bit clearer in the header, which members are set from the config files and which are not.
[CL 21199008 by paul chipchase in ue5-main branch]
#rb trivial
#rnx
#jira UE-159992
#preflight 62d7c9eb110da0f944670ecb
- When the depot path is parsed from the config file we now add the trailing '/' if it is not already there, this allows us to make the input more forgiving without changing the existing code.
- Removed FString::Printf when creating the path for payload_metainfo in favour of using WriteToString, this doesn't add much but gets rid of the %s.
- Reformatted some messages that had odd tab alignment.
[CL 21176124 by paul chipchase in ue5-main branch]
#rb trivial
#rnx
#preflight 62d5509047779a730a1e4602
- The code in the files isn't really a check anymore but the actual process for virtualizing packages (taking the payloads stored in the package trailer, pushing to the backends then modifying the trailer) updating the name reflects this.
- It also helps match the PackageRehydrationProcess.
[CL 21147691 by paul chipchase in ue5-main branch]
#rb trivial
#rnx
#preflight 62d15f427e1c26dcd715699c
- Now that the IVirtualizationSystem interface supports both virtualization and rehydration of package files, it seems pointless to keep a delegate that occurs on pre-submit, rather than just calling the system we really want to call directly.
- Deprecating the public methods accessing the delegate from ISourceControlModule should prevent licensees from trying to use it. The delegate was added in 5.0 so even though it is unlikely to have been used by anyone we need to go through the deprecation steps to remove it.
-- This leaves the calling code a little more complicated than I'd like, but cannot be avoided.
[CL 21109464 by paul chipchase in ue5-main branch]
#rb Sebastian.Nordgren
#jira UE-159595
#rnx
#preflight 62d13965a66919b6700c8069
### SVirtualAssetsStatistics
- We do not currently have a virtualization specific editor module and have been piggybacking onto the DDC editor module, so although this new code has nothing to do with the VA statistics panel it seems easier to keep the code in a single file and then split it later when we move it to a virtualization editor module.
- At the moment if the files being hydrated are not checked out then the process will fail. Support for auto checkout will be added in a future submit.
### EditorExperimentalSettings
- By default no change will be made to the context menu. The user must edit the experimental editor settings and opt in to be able to see it.
- This is because the feature should not really be used in day to day workflows and is provided to either fix problems or to make it easier for people developing the virtualization system.
### PackageRehydrationProcess
- Fixed some typos in error messages
- Being unable to write to the package is considered an error rather than a warning. It was demoted to a warning when virtualizing to try and reduce the number of blocked submits but we do not have this problem with rehydration.
- Added code to reset the loader of a package if it happens to be already loaded in the editor now that we can invoke it from the editor.
[CL 21107763 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-158771
#rnx
#preflight 62c6c8f2f5590c326d77b0f3
- This is a temp fix to an internal problem, where we have a group of people unable to access the source control payload storage system due to permissions but do not actually need to access it at the moment.
- By supressing the connection failure pop up we can allow them to keep working undistrubed.
- Improving how we handle error notifications like this is already on the VA roadmap, at which point this change can be removed.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 20982887 via CL 20982891 via CL 20982896
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)
[CL 20984627 by paul chipchase in ue5-main branch]
#rb Sebastian.Nordgren
#rnx
#jira UE-156436
#preflight 62c287f9a3568e30664eb94f
### VA Standalone Tool
- We now plan to add much more functionality to the tool than just virtualizing and submitting changelists, so to make this easier I am moving the tool towards a design where it should be fairly easy to add new functionality.
- Added FCommand, which is a base class for adding new functionality, simple derive from FCommand and hook it up at the appropriate locations.
-- In the future it should be possible for new command types to automatically register themselves to be initiated from the command line. There should be no need to edit UnrealVirtualizationToolApp to add a new command but this will be done as an additional work item.
-- At the moment FCommand comes with a number of utility methods to call that cover some common source control commands.
-- The original functionality has not yet been moved to the command system and so the code is a little bit weird at the moment. Updating older code to the new system will be done as an additional work item.
- FProject/FPlugin have been moved to their own code files.
### Rehydrate Command
- The rehydrate command will take a number of packages, check them out of source control and then attempt to virtualize them.
- At the moment the chekout logic is fairly basic, we just check out every package supplied, we don't check if the package is virtualized or not yet. This can be improved in additional work items. Ideally by the end of command the only packages that we have checked out should also be rehydrated.
- At the moment the command can either take a path of a specific package, a path of a directory to find packages in, or a changelist containing packages that should be rehydrated.
- A cleint spec (workspace) can optionally be provided, but if not supplied we will attempt to find a client spec for which to check out the packages.
- Currently we will check out the packages to the default change list.
### Rehydrate process
- Added the rehydration process in it's own code files in the virtualization module. Like the virtualization process this is exposed in a public header file and no via the Core interface which means it is very specific to our module/implementation.
- The process expects that the caller will have checked out any required packages from source control. It will treat being unable to update a package file as an error.
- Added PackageUtils.h/.cpp and moved some of the generic code from the virtualization process code there so that it can be shared by the rehydration process.
### Misc
Moving away from the using things like FPackagePath as that requires that the correct mount points have been registered for a project and at the moment (with the flakiness of FConfig*) it seems that the best idea would be to prefer absolute file paths where possible.
[CL 20982284 by paul chipchase in ue5-main branch]
#rb trivial
#rnx
#preflight 62c59751756222ced479cdaf
- Switched the data sizes from integer to double type to avoid small data sizes not showing up
[CL 20969534 by paul chipchase in ue5-main branch]
#rb trivial
#rnx
#preflight 62b986ba5d29d0d10aa1969b
- We now log how many of the original file list turned out to be packages and how many of those packages had package trailers in addition to how many potential payloads for virtualization we found.
- Recently we ran into the problem where a team was unable to see why their virtualization command didn't virtualize a package without debugging, which uncovered that the packages had not been re-saved as expected and lacked trailers. If this logging had been present we would've seen that none of the packages being considered for virtualization had the packages and could've zeroed in on the problem without the aid of the debugger.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 20829071 via CL 20829084 via CL 20829089
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v971-20777995)
[CL 20829642 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
#preflight 62ab3b4a634e82e5d145f5a3
- Dumping of stats to the log file is now something that can be invoked via the IVirtualizationSystem api, so anything can call it, if desired.
- Removed the OnExit callback from the virtualization manager, we no longer print the stats to the log file on exit, but instead print them on demand.
- Add a call at the end of the cooking commandlet to dump the virtualization stats so that they appear around the same part of the log file as the cooking stats.
- Changed the verbosity from 'log' to 'display' so that we can see the stats more easily in horde.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 20699898 via CL 20699907 via CL 20699911
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v968-20684695)
[CL 20700183 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
#preflight 62ab39985c69124c4082aad6
- Ideally the VA tool would not need to deal with this like package mount points (the less project/plugin set up, the faster the tool can run) so we want to move back towards using file paths rather than FPackageNames to reference things where possible. Since virtualization does not need to do anything with the package data itself we can somewhat ignore a lot of Unreal conventions and treat the package files as just files that happen to have a package trailer on the end.
- The package builder was previously storing the name of it's package via an FName and converting it to a string when we needed to print a message. Now we store a generic FString of 'debug context' instead which can be anything.
- Although it is unlikely that anyone is using the FName overloads externally we need to deprecate them rather than just delete them.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 20699797 via CL 20699804 via CL 20699806
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v968-20684695)
[CL 20699936 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
#preflight 62a9bff813004691f9830b8d
- I think when I was first adding it, the type was originally a struct but was changed to an enum before it was submitted, however I failed to update the name.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 20668490 via CL 20668521 via CL 20668532
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v955-20579017)
[CL 20669764 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
#jira UE-156312
#preflight 62a99553943e7bb256fe174c
### Problem
- Firstly this fix is a hack and not intended for shipping 5.1, but the real fix is already planned work, which involves moving the filtering from the push process to being applied to each payload when the package is saved, just as the asset filtering is done at the moment.
-- Doing the proper fix will take a bit of time and testing so we need a quicker fix so that I can re-enable the virtualization process to unblock some workflows.
- All new code is wrapped in a single preprocessor define 'ENABLE_FILTERING_HACK' making the code easy to identify and remove.
- Submitting this hack will cause the priority of the real fix to be bumped in the backlog so that we can remove the hack asap.
- The problem code comes from a pass we do on the payloads to check if they are already in persistent storage or not. If the payload is then we just change it to be virtualized and don't attempt to push. In addition to fixing the filtering there is also a work item to investigate if we really want to keep this logic or if we want to remove it anyway. This might be looked into before I look into the filtering fix as an alternative way to remove the hack.
### Fix
- After we have queried the system to check which of the local payloads are already in persistent storage, BUT before we change them over to be virtualized we now run a new pass, checking which payloads should be filtered.
-- This is a bespoke code path expected only to be used here, so in this case it is not in the IVirtualizationSystem API but requires us to cast to FVirtualizationManager to gain access. This should stop anyone else from using the hack before it is removed.
- The filtering check mimics the filtering that would be run on a normal push operation.
- Once done we can check for any payload that would be filtered, and if so, set the status to not found. This prevents the payload from being changed to be virtualized and means it will be included in the normal push operation where it will be correctly filtered.
- None of this is the best way to do it, but it did seem like the easiest way to add the hack in isolation as no existing code was changed, so it should be simple to just 'delete' the hack when ready.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 20667712 via CL 20667730 via CL 20667733
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v955-20579017)
[CL 20668639 by paul chipchase in ue5-main branch]
#rb trivial
#rnx
#preflight 62a88a39cf54a658ee0de5f7
#jira UE-156319
- Ideally we'd break down the reporting to be per backend but at the moment the slow task system does not let us do that easily because it will only update the text ever 0.2seconds, so if we have a very fast backend followed by a slow one, we'd update the progress bar with the name of the fast backend, then when we update it with the name of the slow backend it will be skipped so the user will still see the name of the fast backend when waiting on the slow one.
- I would rather show less info than risk showing incorrect info.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 20647614 via CL 20647634 via CL 20647664
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v955-20579017)
[CL 20650901 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-156189
#rnx
#preflight 62a35285ec7332a25c95d7bb
- Now our internal projects are updated we can enable the warning about projects still using the old settings.
- Note that this remains backwards compatible and the legacy settings will be used.
[CL 20595697 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-156189
#rnx
#preflight 62a33c245a0ab464fb020897
- The main goal is to split the ini file option that selects the virtualization system to use (SystemName) from the options for our virtualization module.
- The old values should continue to be read and used, but if we detect that ini file values are still under [Core.ContentVirtualization] we will warn about it.
[CL 20594207 by paul chipchase in ue5-main branch]
