Also fix two other minor issues: Make Move constructor available for FIntVirtualizedBulkData, and make Register public so UObjects can notify the registry that they own a BulkData.
#rb Paul.Chipchase
#rnx
#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 18431598 in //UE5/Release-5.0/... via CL 18435352
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v897-18405271)
[CL 18435594 by matt peters in ue5-release-engine-test branch]
#rb PJ.Kack
#jira UE-136126
#rnx
#preflight
### VirtualizationSystem
- Added a new overload for Push to VirtualizationSystem that takes an array of FPushRequest, which is a new structure representing a single payload request.
- Filtering by package name is currently disabled, this is because the API has been forced into changing and passing the package name in via a FString rather than FPackagePath which means we would need to be more careful. This will be done in a future submit.
- The backend interface has been extended to also have a batch version of PushData, by default this will attempt to submit each request one at a time so payloads don't have to try and implement a batched version if there is no need.
- The context being passed with a payload when being pushed has been changed from FPackagePath to FString due to include order issues, as the FPackagePath lives in CoreUObject and the API for virtualization lives in Core. Additionally in the future the payloads might not be owned by a package (there is nothing specifically enforcing this) so the context being a string makes more sense.
- NOTE: Due to the context change we currently no longer support the filtering feature, which allows for payloads belonging to packages under specific directories to be excluded from virtualization. This is something that will be solved in a future submit.
### SourceControlBackend
- Now that we can submit multiple payloads in the same submit, the CL description has been changed slightly. We will now print a list of payload identifiers -> the package trying to submit that payload. This will only tell the users which package originally caused the payload to submit. If a user submits a new package at a later date that contains the same payload we will not be updating the description.
### PackageSubmissionChecks
- Converted the submission process to use the new batch push operation in VirtualizationSystem.
-- This means that we do a single push and then have to update the package trailers to convert the now pushed payloads from local to virtualized.
- Added new define UE_PRECHECK_PAYLOAD_STATUS that makes it easy to toggle off the checks to see which payloads need to be submitted to the persistent backend. This is useful to test if it actually helps speed up the overall operations or if it is faster to just perform the batch push operations on all payloads and check the return values.
-- The hope is that over time the submission processes will become fast enough that we can remove the precheck.
- Fixed up logging to not always assume more than one package or payload.
### General Notes
- Errors and logging is now a bit more vague as we often not just report that X payloads failed etc rather than specific payload identifiers. This probably doesn't affect the user too much since those identifiers as fairly meaningless to them anyway.
- The source control submission could be further optimized by first checking the status of the files in thge depot and only then creating/switching workspace etc.
- As currently written, we need to load all of the payloads into memory, then the backends will do what they need (in the case of source control this results in the payloads being written to disk then submitted) which could create quite a large memory spike when submitting a large number of packages.
-- One solution would be to change the batch push API to take a "payload provider" interface and have the payloads requested as needed rather than passing in the FCompressedBuffer directly. This would let us immediately write the payload to disk for submission then discard it from memory, preventing larger spikes. Although it could cause overhead if there are multiple backends being submitted to. Internally we are unlikely to have more than one backend per storage solution so maybe we should just make it a config option?
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18403735 in //UE5/Release-5.0/... via CL 18403737
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v896-18170469)
[CL 18403738 by paul chipchase in ue5-release-engine-test branch]
Requiring the use of a separate reader type makes it more likely that readers will be reused, and makes it easier to audit reader usage going forward. Reusing readers is desirable to reduce the number of large temporary allocations made during partial decompression of a buffer.
- Added FCompressedBuffer::Save(FArchive&) and renamed FromCompressed(FArchive&) to Load(FArchive&).
- Added FCompressedBufferReaderSourceScope to set a buffer source within a scope.
- Added proper bounds checks to FNoneDecoder.
- Store the header checksum on the decoder context to allow raw blocks to be reused across sources.
- Decode the header on the fly to avoid a temporary header allocation when the header is in contiguous memory.
#rb Zousar.Shaker
#rnx
#preflight 61a98d53800738dbfbc84c73
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18382211 in //UE5/Release-5.0/... via CL 18382310
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v896-18170469)
[CL 18382377 by devin doucette in ue5-release-engine-test branch]
[FYI] Jack.Cai
#lockdown jeff.farris
#jira UE-129834
FCoreUObjectDelegates::OnEndLoadPackage:
Fix the implementation in AsyncLoading1 to handle recursive loads; FAsyncPackage->PackageObjLoaded includes UObjects loaded from packages that were import dependencies of the package, and we need to send OnEndLoadPackage for those as well.
Also filter out the transient package and script and memory packages from all OnEndLoadPackage calls; this was affecting all three of AsyncLoading1, AsyncLoading2, and UObjectGlobals.
Did some comment improvement and removal of dead code in AsyncLoading1 while I was working in the area.
#ushell-cherrypick of 17734361 by Matt.Peters
#rb None, cherrypick
#rnx
#preflight 615dd94eb04b57000197de53
#ROBOMERGE-OWNER: matt.peters
#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 17740393 via CL 17982902 via CL 18367272 via CL 18367316
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)
[CL 18367335 by matt peters in ue5-release-engine-test branch]
FCoreUObjectDelegates::OnEndLoadPackage:
Fix the implementation in AsyncLoading1 to handle recursive loads; FAsyncPackage->PackageObjLoaded includes UObjects loaded from packages that were import dependencies of the package, and we need to send OnEndLoadPackage for those as well.
Also filter out the transient package and script and memory packages from all OnEndLoadPackage calls; this was affecting all three of AsyncLoading1, AsyncLoading2, and UObjectGlobals.
Did some comment improvement and removal of dead code in AsyncLoading1 while I was working in the area.
#author Matt.Peters
#rb Francis.Hurteau Matt.Peters
[REVIEW] https://p4-swarm.epicgames.net/reviews/17725160#fileshttps://p4-swarm.epicgames.net/reviews/17730585/v1/
#preflight 615cf49f80e85c000155a276
[FYI] Halfdan.Ingvarsson
#lockdown jeff.farris
#ROBOMERGE-OWNER: jon.nabozny
#ROBOMERGE-AUTHOR: jack.cai
#ROBOMERGE-SOURCE: CL 17731463 via CL 17981926 via CL 18367097 via CL 18367104
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)
[CL 18367114 by jon nabozny in ue5-release-engine-test branch]
#rb PJ.Kack
#rnx
#preflight 61a8d6d8a7179bfa550218af
- This is an experimental feature being used during testing and development, it is not intended to be used by a shippable project.
-- The code is all wrapped by a single define, UE_ENABLE_VIRTUALIZATION_TOGGLE, making it easy to disable and remove
-- The actual method called on bulkdata, ::SetVirtualizationOptOut is being submitted already deprecated to prevent it's accidental use.
- The use case is to allow projects to opt out of virtualizing static meshes as they are more fragile (i.e. hard crashes) when their payloads cannot be accessed.
- Even when ::SetVirtualizationOptOut is called the feature will not work unless [Core.System.Experimental]AllowVirtualizationOptOut=True is set in the config file. This allows projects to opt into this.
- When a bulkdata is opting out, we simply send the payload to be stored at the end of the package like we used to, rather than send it to the package trailer. This way we do not need to store any additional state in the trailer to track if a payload is allowed to be virtualized or not. Since this feature is throw away I did not want to make any data format changes.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18352158 in //UE5/Release-5.0/... via CL 18352163
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)
[CL 18352180 by paul chipchase in ue5-release-engine-test branch]
#rb PJ.Kack
#jira UE-136374
#rnx
#preflight 61a8ad08ad6629a51eb1f8d8
- Having invalid payloads in the trailer's lookup table and writing empty payload data to disk was a waste of time and meant we had to add more error handling code elsewhere.
- Now when a VBD attempts to add one to the trailer it will be rejected, although the provided callback will still be invoked so that the VBD object will still mark itself as being serialized to disk and update it's flags etc.
- Asserts have been added when writing the FPackageTrailer to disk or creating a new FPackageTrailerBuilder from an existing FPackageTrailer if invalid FPayloadIds are somehow found. This is not an expected condition and the asserts are there to guard against future code changes that might break the contract.
- Now when virtualizing the package trailer during submission checks we can assume that all FPayloadIds reference actual data.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18351024 in //UE5/Release-5.0/... via CL 18351037
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)
[CL 18351039 by paul chipchase in ue5-release-engine-test branch]
FArchiveReplaceObjectRefBase no longer tracks replaced references by default. GetReferencedResources will ensure if archiver was not run with TrackReplacedReferences.
#rb Michael.Noland
#preflight 61a7c89c1a368fd6038f4b79
#ROBOMERGE-AUTHOR: marc.audy
#ROBOMERGE-SOURCE: CL 18343619 in //UE5/Release-5.0/... via CL 18343639
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)
[CL 18343679 by marc audy in ue5-release-engine-test branch]
#rb none
#jira UE-136287
#rnx
#preflight 61a63f4ef70a9e92dbebd70a
- The analysis warning complains that as bAllowVirtualizationOnSave is always false, was & insteaded instead of &&.
- However bAllowVirtualizationOnSave only exists as a development tool, to allow people to enable the old functionality more easily, so we expect bAllowVirtualizationOnSave to only be evaluated as false. It is currently a constant and not a define as we * might * expose this as a config option at some point.
- Flipping the order of evaluation is enough to satify the warning.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18336952 in //UE5/Release-5.0/... via CL 18336960
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)
[CL 18336963 by paul chipchase in ue5-release-engine-test branch]
#rb none
#rnx
- Removed an assert that isn't strictly needed and is firing now that GetLinkerLoadFromOwner is being called in other places.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18308949 in //UE5/Release-5.0/... via CL 18308957
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)
[CL 18308960 by paul chipchase in ue5-release-engine-test branch]
#rb PJ.Kack
#rnx
#preflight 61a4b235405273b2c3daa7c7
### Virtualization
- The idea is to move from our current set up, where the virtualization happens when a package is saved and the payloads are saved to a local virtualization storage system with the final push to persistent storage occurring when the package is submitted to a system where we save payloads into the package file, but move them to persistent virtualization storage when the package file is submitted.
- The main advantage is that if someone submits a package file to source control without virtualizing it, we don't have to worry that others might not be able to load the package, the worst case scenario is that data sizes get bigger until the package is virtualized again.
- This is only the first pass, in the future we can do further optimizations, like not storing payloads locally if we know that they are already in the persistent storage system etc.
- In order to keep the virtualization process simple we want to be able to do it without needing to re-save the package, so to this end we will storage the payloads at the end of the package file as before, but instead of storing them inside of the package file format we will use a new container, the FPackageTrailer that is appended to the package file instead.
- In theory this will make future sidecar work easier as the trailer can just be moved to the sidecar file as as long as we know where to find the trailer things will just work (tm)
- For now VBD will continue to serialize it's offset/size to disk and use those values to read the payload directly when not virtualizing. Ideally we'd use the trailer but this will help reduce the risk.
### Current Issues
- The system does not work with the editor domain and has minor issues with text based assets
- The trailer system is disabled by default via the config system [Core]UsePackageTrailer=False and can then be enabled on select test projects until fully functional.
### PackageTrailer
- The trailer is split into two parts FPackageTrailerBuilder and FPackageTrailer
- FPackageTrailerBuilder is used when building the trailer during package save and FPackageTrailer is the structure we load and actually use.
- When saving the trailer we try to avoid using containers and such so that we can try to keep the specification for the file format very clear so that licensees can build their own tools for the virtualization process if they so desire.
- the header file contains a breakdown of the new format.
### VirtualizedBulkData
- FPayloadToc is now only used by the sidecar experimental feature. In a future update the sidecar will be changed to basically be the package trailer but stored in it's own file.
- FindPayloadsInPackageFile has been moved to PackageTrailer.h as it has a closer association with that code.
- Added a new method, ::LoadFromPackageTrailer, to load payloads from the package trailer without needing to know where the payload is on disk. Using this is opt in with the cvar "Serialization.LoadFromTrailer" and is only provided for debugging purposes. Although once this system has matured it will likely become the preferred way to access payloads.
- We no longer allow virtualization of bulkdata payloads on save, but I am not sure if we might want to allow this as an option in the future. So for now the code branch is disabled by the global constant bAllowVirtualizationOnSave.
- Added utility functions ::GetLinkerLoadFromOwner and ::GetTrailerFromOwner for easy access to the LinkerLoad/Trailer from the owning object.
- Added a utility ::UpdateArchiveData for seeking back to a known position in an archive, writing over it, then seeking back, which is fairly common in the package saving code paths. This might be worth moving to Archive.h for general use.
- ::LoadFromPackageFile has been refactored to prefer returning error values early over nested ifs.
- We no longer serialize the EFlags::IsVirtualized flag to disk. Instead it is applied when the package is loaded if we detect that a payload is virtualized and then removed before saved to disk. This allows the virtualization process to occur without resaving the package.
- We currently support the old serialize to disk path which is enabled when the FPackageTrailer system is disabled and the newer system when it is enabled.
- We do however continue to serialize the offset in the package file to disk when saved. Strictly speaking this is not needed as we can look this up from the package trailer when the package is loaded but continuing to serialize it out to disk helps keep the old and newer data format paths working together without adding too much special case code.
-- Once the FPackageTrailer feature is no longer optional we can consider changing this.
### SavePackage/SavePackage2
- Removed the functions that would create the FPayloadToc
- Added new functions to help create the trailer.
- Moved the end of package tag out of the if to it's own scope (still won't run if a package writer exists), the reasoning is to make it clearer where the package format ends and where we can start writing the payload trailer.
- We have checks to make sure that the trailer does not try to write to disk for both text based assets and if the editor domain is enabled. Support for these features will be added later, but since the package trailer system is disabled by default this shouldn't cause problems.
### LinkerLoad
- After we parse the FPackageFileSummary we now also parse the header of the package trailer. In the workspace domain this will incur additional seek costs (which may or not actually seek and invalidate the internal file cache depending on how large the package's exports are) as the trailer is found after the package
-- In the future, the package trailer header can be stored right after the FPackageFileSummary in the editor domain which will remove this overhead.
- By parsing the header of the trailer at this point, virtualized bulkdata objects in the package can use the look up info to determine if their payloads are virtualized or not, and if not then where they reside inside the package file.
### LinkerSave
- We no longer collect a list of all virtualized bulkdata in a package while saving as we no longer generate the FPayloadToc so BulkDataInPackage has been removed.
- We now gather data to be appended to the package trailer via TrailerBuilder.
### DumpPayloadToc
- This command now dumps info based on the package trailer if one can be found.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18307893 in //UE5/Release-5.0/... via CL 18307905
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)
[CL 18307913 by paul chipchase in ue5-release-engine-test branch]
#rb Francis.Hurteau
[RN] Core, Minor
#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 18279152 in //UE5/Release-5.0/... via CL 18279174
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)
[CL 18279186 by matt peters in ue5-release-engine-test branch]
Change IPackageResourceManager::OpenAsyncReadPackage to return the Format along with the IAsyncReadFileHandle, so that FAsyncArchive can abort with an error if it is a text archive. This is necessary for cooked editor, which tries to use FAsyncArchive when available, but has to support text-format packages.
#rb Francis.Hurteau
#rnx
#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 18259898 in //UE5/Release-5.0/... via CL 18259993
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)
[CL 18260204 by matt peters in ue5-release-engine-test branch]
This represents UE4/Main @18073326, Release-5.0 @18081140 and Dev-PerfTest @18045971
[CL 18081471 by aurel cordonnier in ue5-release-engine-test branch]
#jira UE-132939
LargeMemoryArchive::ReleaseOwnership() sets TotalSize() to 0, so TotalSize must be called before calling ReleaseOwnership. Calling them both as arguments into a function was succeeding on some compilers and failing on others due to compiler optimization differences.
#rb Zousar.Shaker
#rnx
#[fyi] Francis.Hurteau
#ushell-cherrypick of 17947411 by Matt.Peters
#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 18008712 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v885-17909292)
[CL 18008728 by matt peters in ue5-release-engine-test branch]
This represents UE4/Main @17911760, Release-5.0 @17915875 and Dev-PerfTest @17914035
[CL 17918595 by aurel cordonnier in ue5-release-engine-test branch]
This represents UE4/Main @17774255, Release-5.0 @17791557 and Dev-PerfTest @17789485
[CL 17794212 by aurel cordonnier in ue5-release-engine-test branch]
FCoreUObjectDelegates::OnEndLoadPackage:
Fix the implementation in AsyncLoading1 to handle recursive loads; FAsyncPackage->PackageObjLoaded includes UObjects loaded from packages that were import dependencies of the package, and we need to send OnEndLoadPackage for those as well.
Also filter out the transient package and script and memory packages from all OnEndLoadPackage calls; this was affecting all three of AsyncLoading1, AsyncLoading2, and UObjectGlobals.
Did some comment improvement and removal of dead code in AsyncLoading1 while I was working in the area.
#ushell-cherrypick of 17686607 by jamie.dale
#ushell-cherrypick of 17734361 by Matt.Peters
#rb None, cherrypick
#rnx
#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 17735351 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v879-17706426)
[CL 17735376 by matt peters in ue5-release-engine-test branch]
#rb trivial
#rnx
#preflight 615af751e3b24d00014c878a
- Cleaned up some of the code comments documenting the utility macros and functions.
- Removed ::IsIoDispatcherEnabled as it was no longer used.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 17707280 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v879-17706426)
#ROBOMERGE[STARSHIP]: UE5-Release-Engine-Staging Release-5.0
[CL 17707296 by paul chipchase in ue5-release-engine-test branch]
* Decouple container concept from IoDispatcher
* Decoiuple PackageStore implementation from AsyncLoading2
* Restore ucas unmount fix that got kist when merrging from UE4
* Fix packages being left in the PackageStiore even after unmounting contaiiners
#rnx
#rb pj.kack, per.larsson
#preflight 61520cc52afc2d0001146ce7
#ROBOMERGE-AUTHOR: carlmagnus.nordin
#ROBOMERGE-SOURCE: CL 17641845 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v874-17637634)
[CL 17642353 by carlmagnus nordin in ue5-release-engine-test branch]
