#jira UE-132939
LargeMemoryArchive::ReleaseOwnership() sets TotalSize() to 0, so TotalSize must be called before calling ReleaseOwnership. Calling them both as arguments into a function was succeeding on some compilers and failing on others due to compiler optimization differences.
#rb Zousar.Shaker
#rnx
#[fyi] Francis.Hurteau
#ushell-cherrypick of 17947411 by Matt.Peters
#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 18008712 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v885-17909292)
[CL 18008728 by matt peters in ue5-release-engine-test branch]
This represents UE4/Main @17911760, Release-5.0 @17915875 and Dev-PerfTest @17914035
[CL 17918595 by aurel cordonnier in ue5-release-engine-test branch]
This represents UE4/Main @17774255, Release-5.0 @17791557 and Dev-PerfTest @17789485
[CL 17794212 by aurel cordonnier in ue5-release-engine-test branch]
FCoreUObjectDelegates::OnEndLoadPackage:
Fix the implementation in AsyncLoading1 to handle recursive loads; FAsyncPackage->PackageObjLoaded includes UObjects loaded from packages that were import dependencies of the package, and we need to send OnEndLoadPackage for those as well.
Also filter out the transient package and script and memory packages from all OnEndLoadPackage calls; this was affecting all three of AsyncLoading1, AsyncLoading2, and UObjectGlobals.
Did some comment improvement and removal of dead code in AsyncLoading1 while I was working in the area.
#ushell-cherrypick of 17686607 by jamie.dale
#ushell-cherrypick of 17734361 by Matt.Peters
#rb None, cherrypick
#rnx
#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 17735351 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v879-17706426)
[CL 17735376 by matt peters in ue5-release-engine-test branch]
#rb trivial
#rnx
#preflight 615af751e3b24d00014c878a
- Cleaned up some of the code comments documenting the utility macros and functions.
- Removed ::IsIoDispatcherEnabled as it was no longer used.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 17707280 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v879-17706426)
#ROBOMERGE[STARSHIP]: UE5-Release-Engine-Staging Release-5.0
[CL 17707296 by paul chipchase in ue5-release-engine-test branch]
#rb CarlMagnus.Nordin
#jira none
#rnx
#preflight 6156eabd7b2a62000169d451
### Current Changes
- By storing the location of the toc in the package file summary we can access the toc by reading the summary and then jumping to it. It should also be possible for external scripts to be able to do this as well.
- We currently store the identifier, the offset into the package file and the uncompressed size of payload, which will be stored in FCompressedBuffer format.
-- We don't need all this info right now but it means that the structure will be compatible with the experimental sidecar feature. It would also (hypothetically) allow us to move the offset/size info from the bulkdata objects themselves and simply rely on the toc, in which case the loading from within the package file or a sidecar file will be the same.
- When saving a package's exports we will gather a reference to each virtualized bulkdata serialized. The toc should appear before the payloads in the file but we do not know the correct info for the toc until after the payloads have been appended to the package file. So we need to write out a dummy toc that will reserve the space in the file and then after we have appended the payloads, seek back in the file and write out the correct info.
### Future Work
- Although this CL does have a number of code changes to allow this to work with text based assets there are some problems with how that format is currently saved which will prevent it from working (and currently prevents VBD from working with it as well). This will be fixed as a separate issue.
- The experimental sidecar feature uses FTocEntry but serializes a TArray to disk. This should be updated to use FPayloadToc in a later submit.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 17690578 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v875-17642767)
#ROBOMERGE[STARSHIP]: UE5-Release-Engine-Staging Release-5.0
[CL 17690585 by paul chipchase in ue5-release-engine-test branch]
They are empty as the value of bReferenceAdded shouldn't be copied from one instance to another as each instance needs to register itself.
#rb Steve.Robb
#preflight 6155c40c260f7d0001302c45
#ROBOMERGE-AUTHOR: jeanmichel.dignard
#ROBOMERGE-SOURCE: CL 17678459 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v875-17642767)
[CL 17678487 by jeanmichel dignard in ue5-release-engine-test branch]
* Strip names that are not used when cooking for UPS
* Strip names that are not used by the loader when staging for IoStore
#preflight 615424b7f4d2a4000107be68, 615450a875fae600010d4820
#rb pj.kack, francis.hurteau
#ROBOMERGE-AUTHOR: carlmagnus.nordin
#ROBOMERGE-SOURCE: CL 17674004 in //UE5/Release-5.0/... via CL 17674013
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v875-17642767)
#ROBOMERGE[STARSHIP]: UE5-Main
[CL 17674014 by carlmagnus nordin in ue5-release-engine-test branch]
#rb Devin.Doucette
[FYI] Nick.Darnell
#jira UE-129792
#rnx
#tests Loaded production map in editor (cold DDC), full EngineTest cook, retested the original use case for CL 17648118
### Problem
- If a virtualized bulkdata object, created from an old bulkdata object (so that the content id was derived from the old guid rather than the hash of the payload content) , was forced into memory by having it's loader reset, it incorrectly have it's LegacyKeyWasGuidDerived flag removed.
- The flag was removed because we'd assign the newly loaded payload via ::UpdatePayloadImpl which assumes that the provided FPayloadId is correct, but in this case it will not be (it's still derived from the old guid) and given that in most use cases the bulkdata will be destroyed after this call (making loading it from disk a bit pointless, but we currently need to do so for safety) we do not want to pay the cost of hashing the payload at this point.
- If the payload was then accessed then the corruption check would fail as we would assume that the content id should match the hash found in the FCompressedBuffer.
### The Fix
- Instead of calling ::UpdatePayloadImpl when detaching the virtualized bulkdata from disk we now just assign it directly and remove the flags/members relating to disk access.
- Added a cpu trace scope to ::DetachFromDisk as it would be great to find out how much time we are wasting here.
- I also noticed that when ::DetachFromDisk is called, we would not remove the links to the file on disk if bEnsurePayloadIsLoaded was true which will now be fixed as well.
- Since there is no easy way for a user to fix this mismatch and it would only be a problem for them if virtualization is enabled I have reduced the validation severity to a warning rather than error/fatal error until I can provide a way for a package to be 'fixed' so people don't waste work.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 17667806 via CL 17667845 via CL 17667866 via CL 17667872 via CL 17668025
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v875-17642767)
#ROBOMERGE[STARSHIP]: UE5-Main
[CL 17668041 by paul chipchase in ue5-release-engine-test branch]
#rb Danny Couture
#rnx
### Problem
- The crash would occur when the texture being since contains a reference to another texture (via composite) which is also in need of syncing on the users machine.
- After the files on disk have been updated the current packages are reloaded but the original version of the texture will still be in memory when the composite texture notifies that it has been reloaded causing a texture compilation job with the older now invalid texture.
-- VirtualizedBulkData relied on out of date packages not trying to access their payloads on disk once the file has been changed, but with the above scenario it was possible. There could also be other paths causing this (even if the access of out of date package payloads at this point is a waste of time)
### Fix
- The virtualized bulkdata objects now reference themselves with the FLinkerLoader if the payload is on disk, just as the old bulkdata system did. When file changing operations are invoked, the FLinkerLoader will inform it's referenced bulkdata to load the payloads off disk and hold them in memory.
-- Note that virtualized bulkdata was trying to avoid this memory bloat by design, but if we cannot be certain that the bulkdata will not try to access the payload after the file changes (as was thought) then we must take the payload into memory.
- Added a log message to VBD if we detect that the loaded payload does not match the VBD members expectations. This log message can either be an error or a fatal error depending on the define 'VBD_CORRUPTED_PAYLOAD_IS_FATAL' (defaulted to on)
- The registration to FLinkerLoader can be easily disabled by setting the define 'VBD_ALLOW_LINKERLOADER_ATTACHMENT' to 0
### Outstanding Issues
- Cloned bulkdata objects (via assignment) do not currently register themselves to the FLinkerLoader (the old bulkdata system did not) if we were to add this then we need to make sure that the FLinkerLoader registration is thread safe. This should be done as a separate work item so that we can unblock the current workflows that this changelist addresses.
-- Note that if we did not update cloned bulkdata to work with this system then we could drop the registration and just iterate over the package's bulkdata and force them to load into memory when required. However that would leave the cloned virtual bulkdata in an uncertain state and is not desirable.
- When saving a package, the virtualized bulkdata objects that it contains will change their members so that future payload access can come from the newly saved files, where as the old bulkdata system would force the payload to remain in memory for the rest of the editor session. If we want to keep this then we will need to try and make sure that the newly updated bulkdata can be registered so that it receives future notifications about the file being changed.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 17648118 via CL 17648147 via CL 17648161 via CL 17648163 via CL 17648175
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v875-17642767)
#ROBOMERGE[STARSHIP]: UE5-Main
[CL 17648184 by paul chipchase in ue5-release-engine-test branch]
* Decouple container concept from IoDispatcher
* Decoiuple PackageStore implementation from AsyncLoading2
* Restore ucas unmount fix that got kist when merrging from UE4
* Fix packages being left in the PackageStiore even after unmounting contaiiners
#rnx
#rb pj.kack, per.larsson
#preflight 61520cc52afc2d0001146ce7
#ROBOMERGE-AUTHOR: carlmagnus.nordin
#ROBOMERGE-SOURCE: CL 17641845 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v874-17637634)
[CL 17642353 by carlmagnus nordin in ue5-release-engine-test branch]
1) Batch up calls to AssetRegistry->GetAssetsByPath instead of calling it once per plugin, as the function has a high overhead.
2) Remove trace scope from UE:AssetRegistry FFiltering::ShouldSkipAsset as the function is called very frequently (large projects can have a million assets).
#rb Francis.Hurteau, Jamie.Dale
#rnx
#jira None, slack thread
#ushell-cherrypick of 17591138 by Matt.Peters
#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 17628939 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v871-17566257)
[CL 17628957 by matt peters in ue5-release-engine-test branch]
