This is because dependent BPs at least need a re-link to pick up layout changes in their property access 'bytecode'.
The also requires some re-arranging of where sparse class data gets created, linked and patched.
- Previously sparse class data was patched in the linker at the point of re-creation, but this meant that stale TRASHCLASS sparse class data was could still be in the struct hierarchy when we are BATCH compiling (when batch compiling, each compilation phase is performed in turn for all compiling/dependent BPs). This change removes the sparse class data when the class is sanitized prior to full compilation and stashes it for later patching. This ensures that sparse class data doesnt get linked into stale structs that can then get GCed later, invalidating property links.
- Sparse class data is now linked when its containing ABPGC is linked. It also ensures that the hierarchy of sparse class data mirrors that of the ABPGC. The 'mirror hierarchy' is currently only an invariant of anim blueprints.
Part of this change alters the behavior of UClass::SetSparseClassDataStruct when passed a NULL value. This will now ensure that subclasses are left in a valid state when this occurs. Previously if sparse class data was nulled-out, subclasses would be left pointing at a potentially stale struct in the case where the struct was being regenerated.
Added an override point for subsystems to participate in Link() and made sure property access offsets are patched there, to catch cases where dependent BPs get either a bytecode-only or link-only compile
#jira UE-136035 - Unexpected results from Property Access after changing Lyra animbp
#jira UE-130965 - Multiple crashes involving anim BP sparse class data, inheritance & GC
#rb Phillip.Kavan,Matt.Peters
#preflight 61af321f470c73a8c4933ff5
#ROBOMERGE-AUTHOR: thomas.sarkanen
#ROBOMERGE-SOURCE: CL 18393342 in //UE5/Release-5.0/... via CL 18393349
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v896-18170469)
[CL 18393353 by thomas sarkanen in ue5-release-engine-test branch]
This has the side-effect of releasing unused OS pages to the next GC (since they are released from the per-thread calls to FlushCurrentThreadCache, but doesn't has a noticeable impact on internal demo.
In practice, this makes FMemory::Trim detached from the GC reachability pass, and takes ~4ms instead of ~10-50ms depending on the other threads workload.
Original code by PJ.Kack.
#rb kenzo.terelst, richard.malo
#lockdown jeff.farris
#preflight 61697646823b7b0001acfb3a
#ROBOMERGE-AUTHOR: jeanfrancois.dube
#ROBOMERGE-SOURCE: CL 17829574 via CL 18002912 via CL 18368846 via CL 18368871
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)
[CL 18368879 by jeanfrancois dube in ue5-release-engine-test branch]
Reverse check during IsUnsaveable to emit class warning only on valid non transient object
#rb Phillip.Kavan, Matt.Peters
#lockdown Jeff.Farris
#preflight 614cc9a37657b800017ad050
#ROBOMERGE-OWNER: aurel.cordonnier
#ROBOMERGE-AUTHOR: francis.hurteau
#ROBOMERGE-SOURCE: CL 17614041 via CL 17963206 via CL 18365234 via CL 18365240
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)
[CL 18365241 by aurel cordonnier in ue5-release-engine-test branch]
#rb PJ.Kack
#jira UE-136374
#rnx
#preflight 61a8ad08ad6629a51eb1f8d8
- Having invalid payloads in the trailer's lookup table and writing empty payload data to disk was a waste of time and meant we had to add more error handling code elsewhere.
- Now when a VBD attempts to add one to the trailer it will be rejected, although the provided callback will still be invoked so that the VBD object will still mark itself as being serialized to disk and update it's flags etc.
- Asserts have been added when writing the FPackageTrailer to disk or creating a new FPackageTrailerBuilder from an existing FPackageTrailer if invalid FPayloadIds are somehow found. This is not an expected condition and the asserts are there to guard against future code changes that might break the contract.
- Now when virtualizing the package trailer during submission checks we can assume that all FPayloadIds reference actual data.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18351024 in //UE5/Release-5.0/... via CL 18351037
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)
[CL 18351039 by paul chipchase in ue5-release-engine-test branch]
SavePackage: turn off the check for missing toplevelflag when cooking. It is only a concern when saving editor packages, and it can have false positives when cooking.
#rb Francis.Hurteau
#rnx
#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 18346064 in //UE5/Release-5.0/... via CL 18346086
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)
[CL 18346098 by matt peters in ue5-release-engine-test branch]
Add 3 contexts when harvesting references during package saving: Game (cooking), Editor and the new Optional context
This will be able to be used for saving multiple package output using the new optional context but also fix problem when using `CanSkipEditorReferencedPackagesWhenCooking` and properly purge editor only internal references from the cooked context.
Add a new save flag to save the new optional context
#rb Matt.Peters
#jira UE-129801
#preflight 61a7a27ce01f3610f51e4a7a
#ROBOMERGE-AUTHOR: francis.hurteau
#ROBOMERGE-SOURCE: CL 18340567 in //UE5/Release-5.0/... via CL 18340584
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)
[CL 18340608 by francis hurteau in ue5-release-engine-test branch]
#rb PJ.Kack
#rnx
#preflight 61a795773c29b3cf13cd8250
### PackageSubmissionChecks
- Under the old model submitting a large number of packages could be very slow as each package would check each payload that it owns and is currently stored locally one at a time. For the source control backend this created quite a large overhead even when all of the payloads were already virtualized.
- Now we do a pass over the submitted files to find all valid packages that have package trailers with locally stored payloads, gather the payloads into a single list and then query that in one large batch.
- Once we find which payloads are not in permanent storage (note that in the case where a project is using multiple permanent storage solutions, if a payload is missing in one backend it counts as not being in permanent storage) we then attempt to virtualized them.
- Only after all of this is done will we create the truncated copy of each package and then append the updated trailer to each one. In theory doing it in this order this might slightly increase the change of submit failures that occur after virtualization that result in a package never being submitted and orphaned payloads being added to permanent storage, but this will always be a risk.
- Added an assert to fire if we detect a trailer with some virtualized and some local payloads. This should be a supported feature but needs proper testing first before we can allow it. With out current project settings no project should actually encounter this scenario.
- To make the code easier to follow we now early out of the entire check when errors are encountered.
- Added logging at various stages in the process to help show the user that something is happening and make problems easier to identify in the future.
- Notes
-- There is a lot of handling of invalid FPayloads. This is because it is currently possible to add empty payloads to the trailer which is inefficient and wastes space. The trailer will be modified to reject empty payloads in a future update at which point a lot of this handling can be removed.
-- This could've also been solved by not fully rehydrating a package on save by the end user, which will be added as a project setting in a future piece of work, but this approach will solve the edge case when the user does have a large amount of hydrated packages which contain payloads that are already virtualized so it was better to fix that now while we have good test cases for it.
-- We still have scaling problems with large number of package being submitted that do have payloads that need to be virtualized, this will be fixed by extending IVirtualizationSystem::Push to also accept batches of payloads in future work.
-- OnPrePackageSubmission could be broken up into smaller chunks to make the code easier to follow. This will be done after the batch payload submission work is done.
### VirtualizationSystem
- EStorageType has been promoted to enum class.
- Added a new enum FPayloadStatus to be used when querying if a payload exists in a backend storage system or not.
- Add a new method ::DoPayloadsExist which allows the caller to query if one or more payloads exists in the given backend storage system.
### VirtualizationManager
- Implemented ::DoPayloadsExist. First we get the results from each backend in the storage system (which return as true or false from each backend) then total how many backends found the payload in order to set the correct status.
### IVirtualizationBackend
- ::DoesPayloadExist which queries the existence of a single payload has been added to the interface. Most backends already implemented this for private use and if so have had their implementation renamed to match this.
- Also added ::DoPayloadsExist which takes a batch of FpayloadIdsto query. Some backends can deal with a batch of payload ids much more efficiently than one at a time, although the default implementation does call ::DoesPayloadExist for each requested payload.
-- The default implementation prevents every backend from needing to implement the same for loop but does allow backends that can gain from batching to override it.
### VirtualizationSourceControlBackend
- This backend does override ::DoPayloadsExist and implements it's own version as it tends to perform very poorly when not operating on larger batches.
- In this case ::DoesPayloadExist calls back to ::DoPayloadsExist to check each payload rather than implement as specific version.
### PackageTrailer
- The trailer can now be queries to request how many payloads of a given type it contains
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18339847 in //UE5/Release-5.0/... via CL 18339852
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)
[CL 18339859 by paul chipchase in ue5-release-engine-test branch]
#rb PJ.Kack
#rnx
#preflight 61a4b235405273b2c3daa7c7
### Virtualization
- The idea is to move from our current set up, where the virtualization happens when a package is saved and the payloads are saved to a local virtualization storage system with the final push to persistent storage occurring when the package is submitted to a system where we save payloads into the package file, but move them to persistent virtualization storage when the package file is submitted.
- The main advantage is that if someone submits a package file to source control without virtualizing it, we don't have to worry that others might not be able to load the package, the worst case scenario is that data sizes get bigger until the package is virtualized again.
- This is only the first pass, in the future we can do further optimizations, like not storing payloads locally if we know that they are already in the persistent storage system etc.
- In order to keep the virtualization process simple we want to be able to do it without needing to re-save the package, so to this end we will storage the payloads at the end of the package file as before, but instead of storing them inside of the package file format we will use a new container, the FPackageTrailer that is appended to the package file instead.
- In theory this will make future sidecar work easier as the trailer can just be moved to the sidecar file as as long as we know where to find the trailer things will just work (tm)
- For now VBD will continue to serialize it's offset/size to disk and use those values to read the payload directly when not virtualizing. Ideally we'd use the trailer but this will help reduce the risk.
### Current Issues
- The system does not work with the editor domain and has minor issues with text based assets
- The trailer system is disabled by default via the config system [Core]UsePackageTrailer=False and can then be enabled on select test projects until fully functional.
### PackageTrailer
- The trailer is split into two parts FPackageTrailerBuilder and FPackageTrailer
- FPackageTrailerBuilder is used when building the trailer during package save and FPackageTrailer is the structure we load and actually use.
- When saving the trailer we try to avoid using containers and such so that we can try to keep the specification for the file format very clear so that licensees can build their own tools for the virtualization process if they so desire.
- the header file contains a breakdown of the new format.
### VirtualizedBulkData
- FPayloadToc is now only used by the sidecar experimental feature. In a future update the sidecar will be changed to basically be the package trailer but stored in it's own file.
- FindPayloadsInPackageFile has been moved to PackageTrailer.h as it has a closer association with that code.
- Added a new method, ::LoadFromPackageTrailer, to load payloads from the package trailer without needing to know where the payload is on disk. Using this is opt in with the cvar "Serialization.LoadFromTrailer" and is only provided for debugging purposes. Although once this system has matured it will likely become the preferred way to access payloads.
- We no longer allow virtualization of bulkdata payloads on save, but I am not sure if we might want to allow this as an option in the future. So for now the code branch is disabled by the global constant bAllowVirtualizationOnSave.
- Added utility functions ::GetLinkerLoadFromOwner and ::GetTrailerFromOwner for easy access to the LinkerLoad/Trailer from the owning object.
- Added a utility ::UpdateArchiveData for seeking back to a known position in an archive, writing over it, then seeking back, which is fairly common in the package saving code paths. This might be worth moving to Archive.h for general use.
- ::LoadFromPackageFile has been refactored to prefer returning error values early over nested ifs.
- We no longer serialize the EFlags::IsVirtualized flag to disk. Instead it is applied when the package is loaded if we detect that a payload is virtualized and then removed before saved to disk. This allows the virtualization process to occur without resaving the package.
- We currently support the old serialize to disk path which is enabled when the FPackageTrailer system is disabled and the newer system when it is enabled.
- We do however continue to serialize the offset in the package file to disk when saved. Strictly speaking this is not needed as we can look this up from the package trailer when the package is loaded but continuing to serialize it out to disk helps keep the old and newer data format paths working together without adding too much special case code.
-- Once the FPackageTrailer feature is no longer optional we can consider changing this.
### SavePackage/SavePackage2
- Removed the functions that would create the FPayloadToc
- Added new functions to help create the trailer.
- Moved the end of package tag out of the if to it's own scope (still won't run if a package writer exists), the reasoning is to make it clearer where the package format ends and where we can start writing the payload trailer.
- We have checks to make sure that the trailer does not try to write to disk for both text based assets and if the editor domain is enabled. Support for these features will be added later, but since the package trailer system is disabled by default this shouldn't cause problems.
### LinkerLoad
- After we parse the FPackageFileSummary we now also parse the header of the package trailer. In the workspace domain this will incur additional seek costs (which may or not actually seek and invalidate the internal file cache depending on how large the package's exports are) as the trailer is found after the package
-- In the future, the package trailer header can be stored right after the FPackageFileSummary in the editor domain which will remove this overhead.
- By parsing the header of the trailer at this point, virtualized bulkdata objects in the package can use the look up info to determine if their payloads are virtualized or not, and if not then where they reside inside the package file.
### LinkerSave
- We no longer collect a list of all virtualized bulkdata in a package while saving as we no longer generate the FPayloadToc so BulkDataInPackage has been removed.
- We now gather data to be appended to the package trailer via TrailerBuilder.
### DumpPayloadToc
- This command now dumps info based on the package trailer if one can be found.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18307893 in //UE5/Release-5.0/... via CL 18307905
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)
[CL 18307913 by paul chipchase in ue5-release-engine-test branch]
EDLCookChecker: suppress missing export warnings for packages that were iteratively skipped and thus do have the exports but did not declare them.
#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 18288814 in //UE5/Release-5.0/... via CL 18288852
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)
[CL 18288863 by matt peters in ue5-release-engine-test branch]
- Gets rid of 572K reallocs from process start to end of skydiving for a replayrun
#rb Robert.Manuszewski
#ROBOMERGE-AUTHOR: danny.couture
#ROBOMERGE-SOURCE: CL 18286024 in //UE5/Release-5.0/... via CL 18286052
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)
[CL 18286062 by danny couture in ue5-release-engine-test branch]
ObjectSaveContext: Fix ObjectSaveContext never having IsUpdatingLoadPath set in the in OnPreSaveWorld, PreSavePackageWithContext, and OnPostSaveWorld events.
#rb Dave.Belanger
#rnx
#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 18279373 via CL 18279377 via CL 18279379 via CL 18279851 via CL 18279865
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)
[CL 18279867 by matt peters in ue5-release-engine-test branch]
#rb Francis.Hurteau
[RN] Core, Minor
#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 18279152 in //UE5/Release-5.0/... via CL 18279174
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)
[CL 18279186 by matt peters in ue5-release-engine-test branch]
#rb trivial
#ROBOMERGE-AUTHOR: robert.manuszewski
#ROBOMERGE-SOURCE: CL 18256279 via CL 18256282 via CL 18256283 via CL 18263293 via CL 18263359
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)
[CL 18263519 by robert manuszewski in ue5-release-engine-test branch]
Change IPackageResourceManager::OpenAsyncReadPackage to return the Format along with the IAsyncReadFileHandle, so that FAsyncArchive can abort with an error if it is a text archive. This is necessary for cooked editor, which tries to use FAsyncArchive when available, but has to support text-format packages.
#rb Francis.Hurteau
#rnx
#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 18259898 in //UE5/Release-5.0/... via CL 18259993
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)
[CL 18260204 by matt peters in ue5-release-engine-test branch]
AssetManager:
1) Warn and ignore if callers attempt to add a PrimaryAsset that is not a TopLevelAsset. Actors in external packages can be non-TopLevel Assets.
2) Cull non-TopLevelAssets from search results looking for PrimaryAssets.
3) Disallow SearchAssetRegistryPaths of the entire registry; require callers to provide at least one path. This prevents PrimaryAssetRules that are missing their paths from attempting to set every asset in the AssetRegistry as a primary asset.
#rb Francis.Hurteau
[FYI] Ben.Zeigler
#rnx
#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 18244088 in //UE5/Release-5.0/... via CL 18244407
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)
[CL 18244444 by matt peters in ue5-release-engine-test branch]
