Saves several seconds when saving lots of actor files.
#rb francis hurteau
#preflight 62f29af13b773d0416317135
#rnx
#ROBOMERGE-AUTHOR: jeanfrancois.dube
#ROBOMERGE-SOURCE: CL 21316192 via CL 21316196 via CL 21316314
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)
[CL 21317425 by jeanfrancois dube in ue5-main branch]
The following types have moved to ITransactionObjectAnnotation.h:
- ITransactionObjectAnnotation
The following types have moved to TransactionObjectEvent.h:
- FTransactionObjectDeltaChange
- ETransactionObjectEventType
- FTransactionObjectEvent
- FTransactionDiff
#preflight 62f141b7086f90bbc473f631, 62f1351a904c243122f5eacd
#rb Francis.Hurteau, Rex.Hill
#ROBOMERGE-OWNER: jamie.dale
#ROBOMERGE-AUTHOR: jamie.dale
#ROBOMERGE-SOURCE: CL 21272390 via CL 21285937 via CL 21285973 via CL 21286088
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)
[CL 21290425 by jamie dale in ue5-main branch]
#rb Per.Larsson
#jira UE-156750
#rnx
#preflight 62f212e13b773d04161ee7dd
### Problem
- The payloads stored in map files tend to change more than other assets and would cause a lot more churn in the VA system.
- Some other systems like the landscape component are not able to sensibly continue if their payloads cannot be accessed (heightmaps for exmaple) and would prefer not to allow virtualization.
- As a short term fix we need an option to disable the virtualization on all payloads in map files. Future improvements to the filtering system will allow systems to more easily opt their payloads out of virtualization. When this is functional we might want to change the default from true to false.
### Feature
- The config optionf for this filtering is "[Core.VirtualizationModule]FilterMapContent=True"
- Testing if the owning UObject for a payload is in a umap can get tricky, because we not only need to check the umap but we also need to check if it is in a "_builddata.uasset" file, which is an additional file we store next to a umap containing things like lightmaps etc.
- At the moment we check for this by finding the outermost object for the given owner and check to see if it is a ULevel, UWorld or UMapBuildDataRegistry. This is a bit of a kludge but the types we need to check against are not accessible by this module and making them accessible will pull in a lot of dependencies that we'd prefer not to add.
-- One improvement might be to tag the FLinkerSave with the info we need and passing that into the serialization process rather than trying to work it out ourselves but I am wary of making that change until we are 100% sure that we want to keep this feature.
### Refactor
- Removed IVirtualizationSystem::IsDisabledForObject and replaced it with ::FilterPayload which can return multiple reasons for preventing a payload from virtualizing. (the method was added during 5.1 development so it should be fine to just replace it without deprecation)
-- The original behaviour for FVirtualizationManager::IsDisabledForObject has been moved to FVirtualizationManager::ShouldVirtualizeAsset
- Added a new header to declare enums/types used by the various parts of the virtualization system and started by moving EPayloadFilterReason there from the package trailer header. This allows both the core API and PackageTrailer to use EPayloadFilterReason without creating overburdened header dependencies.
-- EPayloadFilterReason has moved from the UE namespace to UE::Virtualization so the package trailer code needed updating accordingly.
- EditorBulkData will ask the virtualization system for the base filter reason, then add it's own reasons if UE_ENABLE_VIRTUALIZATION_TOGGLE is enabled. This bit of code will be removed for 5.1
[CL 21283179 by paul chipchase in ue5-main branch]
Add IMPCollector for generic method to pass messages from CookWorkers to Director.
#preflight 62efe345a035cdf05e82cc2e
#rb Zousar.Shaker
#rnx
[CL 21267593 by Matt Peters in ue5-main branch]
Headers are updated to contain any missing #includes needed to compile and #includes are sorted. Nothing is removed.
#ushell-cherrypick of 21065896 by bryan.sefcik
#preflight 62d4b1a5a6141b6adfb0c892
#jira
#ROBOMERGE-OWNER: Bryan.sefcik
#ROBOMERGE-AUTHOR: bryan.sefcik
#ROBOMERGE-SOURCE: CL 21150156 via CL 21151754 via CL 21154719
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)
#ROBOMERGE-CONFLICT from-shelf
[CL 21181076 by Bryan sefcik in ue5-main branch]
#rb Devin.Doucette
#jira UE-159985
#rnx
#preflight 62d6e8d2d76ea4b5032d5d56
### Problem
- When CL 21089705 was submitted, we changed the behaviour to assume that it is only safe to load payloads off disk if the AttachedAr pointer is valid rather than checking for the package path being valid. This allowed us to keep the package path intact after ::DetachFromDisk was called and give better error messages.
- However not all code paths were forcing the AttachedAr pointer to be set, TornOff bulkdata instances in the bulkdata registry for example causing these code paths to fail entirely.
- What we need is a way to not allow bulkdata instances that were attached but have been detached from loading (or at least force them to use the speculative loading path that was added in 21089705) and allow any other bulkdata instance that was never attached, to load it's payload as long as the path is set.
### Fix
- Instead of asking ::IsAttachedToPackageFile we instead ask ::HasAttachedArchive as in most places we are really checking if the AttachedAr pointer is null or not before we manipulate it
- There are two places where what we are really asking is if it is safe to load the payload from disk or not, these places now use a new method ::CanLoadDataFromDisk which checks that the bulkdata has access to a valid package path AND that the transient flag EFlags::WasDetached has not bee set.
-- This means that bulk data instances that are never attached to the packages FLinkerLoad can still load off disk, although this is less safe it reverts us back to the behaviour before CL 21089705was submitted.
### Misc
- Marked the transient flags as such in the code documentation.
- Ideally these flags would not have been added to the EFlags enum and a specific ETransientFlag enum added instead but the clean up for that would be irksome at this point.
[CL 21174617 by paul chipchase in ue5-main branch]
Add support for 32 and 64 bit variants of various int vector types.
IntVector2/3/4 - UPROPERTY support including auto-conversion of 32 and 64 bit variants of same signedness.
IntPoint - UPROPERTY support including auto-conversion of 32 and 64 bit variants of same signedness.
IntRect
#preflight 62d6c0a2d76ea4b503282a43
[CL 21162524 by Andrew Davidson in ue5-main branch]
#jira UE-159832
#rb Aaron.mcleran Phil.Popp Dan.Thompson Josh.Adams
#preflight 62d260033c3df323907eaa93
[CL 21153330 by jimmy smith in ue5-main branch]
This change completely removes Matinee from the codebase. The data types and runtime evaluation code are gone. The "InterpEdit" editor mode, asset factories, and other editor support for Matinee is gone. CameraAnims and their sequencer tracks are gone. FBX import/export for Matinee assets is gone.
#preflight 62d0591e562520a394bca1d1
#preflight 62d194f63c3df323904679c1
#jira UE-105313
#rb none
#ROBOMERGE-OWNER: max.chen
#ROBOMERGE-AUTHOR: ludovic.chabant
#ROBOMERGE-SOURCE: CL 21116312 via CL 21116339 via CL 21118051
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)
[CL 21142481 by max chen in ue5-main branch]
- LinkerLoad uses InstancingContext to do SoftObjectPath remaps
- This fixes SoftObjectPath remaps for OFPA Actors
- Fix WorldPartitionLevelHelper now that a first remapping is done in the LinkerLoad we need to change how we remap to cells for LevelInstance embedded actors
- Remove now useless remap in UWorldPartition::Initialize
#jira UE-158267
#rb jamie.dale, francis.hurteau, jeanfrancois.dube
#preflight 62d16f71a6141b6adf411c8f
#ROBOMERGE-AUTHOR: patrick.enfedaque
#ROBOMERGE-SOURCE: CL 21111456 via CL 21112041 via CL 21112295
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)
[CL 21142341 by patrick enfedaque in ue5-main branch]
#rb Per.Larsson, Matt.Peters
#jira UE-153904 , UE-156201
#preflight 62ce802c471a2c2886f592e8
### Problem
- Because of the package trailer we can easily find out which payloads are present in a package file and where they are. This means we don't need to rely on the cached offset into the file that bulkdata generally uses for payload loading. This means as long as the payload is still actually in a payload file then we can load it no matter what.
- We will still give errors in the following scenarios
-- The payload is not stored in a package trailer (either referencing older bulkdata or an older editor bulkdata format)
-- We cannot get a read handle to the package file.
-- The payload is no longer in the package file.
- If the attachment is still valid then we can guarantee that the payload can be loaded. Without an attachment we make our best efforts but in the case of the package file being reomoved/replaced this can fail. So ::ResetLoader will still be used to indicate that we want to remove the attachment but expect the package file to be changed and to load the payload into memory for safety. Work on improving this is another work item for the future.
### Fix
- We no longer remove the package path when ::DetachFromDisk is called nor do we remove flags relating to the payload. From this point onwards we use the attached archive being null to determine how we load. We continue to set OffsetInFile to an invalid value as we can no longer safely use it.
- Add a new method ::IsAttachedToPackageFile to explicitly show when we are checking to see if the bulkdata is still 'attached' to the package file.
- Explicitly set AttachedAr to null ptr when ::DetachBulkData is called on it. In theory the ::DetachBulkData call will call FEditorBulkData::DetachFromDisk which in turn will set AttachedAr but it can look confusing when reading the code. Explicitly setting the AttachedAr member to nullptr helps make the intent clearer.
- When loading the payload from disk there is now a new code branch that is followed if we detect that we are loading from a package file that we are no longer attached to.
-- This path gives bespoke errors in cases where it is not safe for us to try loading the payload, in addition there is a bespoke error message if loading from the package trailer fails.
-- Note we can only risk loading from the package trailer as it's format allows for us to find if the payload is still on disk or not. With the older formats we will not know if we are loading garbage or not.
-- Fixed a bug with CVarShouldLoadFromTrailer. When it is true we will only attempt to load payloads from the trailer if the payload is stored in the trailer. The comment for this development CVar has been updated.
- There has been a pass made to error reporting. If we are reporting an error relating to the package we will output the package name, if we are reporting on a error relating to the file itself then we will output the package path + extension
[CL 21089705 by paul chipchase in ue5-main branch]
This covers the following types and meta-data:
* Blueprint Generated Class (class, property, and function meta-data)
* User Defined Struct (struct and property meta-data)
* User Defined Enum (enum meta-data)
#preflight 62cc7d31173a4f3408716829
#rb Francis.Hurteau
#ROBOMERGE-AUTHOR: jamie.dale
#ROBOMERGE-SOURCE: CL 21057147 via CL 21057174 via CL 21057182 via CL 21057190
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)
[CL 21078081 by jamie dale in ue5-main branch]
Added a FCoreDelegates delegate called by the BP reinstancing path. The existing one is called per-batch of reinstanced objects, but this means that some objects will still be REINST_ at the point of the call. This was causing ordering issues when attempting to re-link linked anim graphs (which point to object internals). The new delegate is called once all references have been replaced so the reinstanced objects should be self-consistent.
Reworked reinstancing logic to now account for this self-consistency. Key to this is the caching of the 'source' object (i.e. the host anim instance) on nodes that inherit from FAnimNode_CustomProperty - this allows the reinstancing logic to correctly handle linking & property patchup at the point of the delegate call.
Fixed a memory stomp when compiling an anim BP that contains a control rig node. This was caused by the control rig's delegate still being pointed at the old instance due to the lambda binding. Switched this to a raw binding that is cleaned up on node destruction.
#jira UE-158849
#rb Phillip.Kavan,Jurre.deBaare
#preflight 62cec541d00f3c208e446dce
[CL 21070551 by Thomas Sarkanen in ue5-main branch]
Initial submit, compiled out by default.
#rb Peter.Engstrom,Mattias.Hornlund,Brian.Bekich,Ryan.Gerleve,John.Barrett
#jira UE-158381
#preflight /62cd6fc7ad490b52da0515a3
[CL 21068370 by Peter Engstrom in ue5-main branch]
#rb Sebastian.Nordgren
#rnx
#jira UE-156436
#preflight 62c287f9a3568e30664eb94f
### VA Standalone Tool
- We now plan to add much more functionality to the tool than just virtualizing and submitting changelists, so to make this easier I am moving the tool towards a design where it should be fairly easy to add new functionality.
- Added FCommand, which is a base class for adding new functionality, simple derive from FCommand and hook it up at the appropriate locations.
-- In the future it should be possible for new command types to automatically register themselves to be initiated from the command line. There should be no need to edit UnrealVirtualizationToolApp to add a new command but this will be done as an additional work item.
-- At the moment FCommand comes with a number of utility methods to call that cover some common source control commands.
-- The original functionality has not yet been moved to the command system and so the code is a little bit weird at the moment. Updating older code to the new system will be done as an additional work item.
- FProject/FPlugin have been moved to their own code files.
### Rehydrate Command
- The rehydrate command will take a number of packages, check them out of source control and then attempt to virtualize them.
- At the moment the chekout logic is fairly basic, we just check out every package supplied, we don't check if the package is virtualized or not yet. This can be improved in additional work items. Ideally by the end of command the only packages that we have checked out should also be rehydrated.
- At the moment the command can either take a path of a specific package, a path of a directory to find packages in, or a changelist containing packages that should be rehydrated.
- A cleint spec (workspace) can optionally be provided, but if not supplied we will attempt to find a client spec for which to check out the packages.
- Currently we will check out the packages to the default change list.
### Rehydrate process
- Added the rehydration process in it's own code files in the virtualization module. Like the virtualization process this is exposed in a public header file and no via the Core interface which means it is very specific to our module/implementation.
- The process expects that the caller will have checked out any required packages from source control. It will treat being unable to update a package file as an error.
- Added PackageUtils.h/.cpp and moved some of the generic code from the virtualization process code there so that it can be shared by the rehydration process.
### Misc
Moving away from the using things like FPackagePath as that requires that the correct mount points have been registered for a project and at the moment (with the flakiness of FConfig*) it seems that the best idea would be to prefer absolute file paths where possible.
[CL 20982284 by paul chipchase in ue5-main branch]
