- Add a new partial loading mode where flushing the requestID will exit after serialization is done (after exportdone but before postload)
- Automatically opt-in partial loading when an explicit load is called recursively from another load (treated as an import)
- During recursive loading, attach postload of the recursive load to the caller's postload
- Add loading unit tests to validate the new behavior for zenloader
- Add a way to detect which loader is currently active to run unit tests on zenloader only
#rnx
#jira UE-190649
#rb Francis.Hurteau
[CL 28227792 by danny couture in ue5-main branch]
[FYI] danny.couture
Original CL Desc
-----------------------------------------------------------------
[ZenLoader] Fix recursive loading behavior in zenloader by following more closely how legacy loading works
- Add a new partial loading mode where flushing the requestID will exit after serialization is done (after exportdone but before postload)
- Automatically opt-in partial loading when an explicit load is called recursively from another load (treated as an import)
- During recursive loading, attach postload of the recursive load to the caller's postload
- Add loading unit tests to validate the new behavior for zenloader
- Add a way to detect which loader is currently active to run unit tests on zenloader only
#rnx
#jira UE-190649
#rb Francis.Hurteau
[CL 28214800 by bob tellez in ue5-main branch]
- Add a new partial loading mode where flushing the requestID will exit after serialization is done (after exportdone but before postload)
- Automatically opt-in partial loading when an explicit load is called recursively from another load (treated as an import)
- During recursive loading, attach postload of the recursive load to the caller's postload
- Add loading unit tests to validate the new behavior for zenloader
- Add a way to detect which loader is currently active to run unit tests on zenloader only
#rnx
#jira UE-190649
#rb Francis.Hurteau
[CL 28184263 by danny couture in ue5-main branch]
#rb Paul.Chipchase
#rnx
## Background
The on demand I/O backend doesn't store complete/hole I/O chunks in the local file cache, only the parts that has been requested by the game. There are currently
two ways for querying the I/O backend(s) whether a specific chunk exists or not. The primary one used by the texture streaming system is DoesChunkExist(ChunkId). This
method is called via FBulkData::DoesExist(). In order to be able to fufill this query for partially cached I/O chunks, two additional methods has been
added to the I/O dispatcher API. Currently private, only accessible by FBulkData.
- DoesChunkExist(ChunkId, OffsetAndLength)
- GetSizeForChunk(ChunkId, OffsetAndLength)
The method GetSizeForChunk is only ever used by the IAsyncFileHandle wrapper for bulk data and is a tricky compromise. The file handle
represents the hole chunk/file where the bulk data instance resides, i.e. .ubulk, .uptnl etc. Calling get file size on this file handle should return
the size for the entire chunk/file. It is technically possible to read outside of the bulk data range and this would result in the read request
coming back as failed, since the resolve function in the I/O backend wouldn't be found. The file handle wrapper for bulk data is currently only used
by virtual textures and should be deprecated in the long run.
[CL 27337787 by per larsson in ue5-main branch]
- Allows callees of LoadPackageAsync to be notified when serialization is done so they can schedule the next load right from the async loading thread without having to pay for a round-trip to the game-thread
- Add UnattendedAsyncLoading test as a sample of how this feature can be used
- Add thread-safety tests for LoadPackageAsync
- Can be exercised using a cooked client build with the following command-line
-ExecCmds="Automation RunTest System.Engine.Loading.UnattendedAsyncLoadingTest+System.Engine.Loading.ThreadSafeAsyncLoadingTest;Quit"
#jira UE-188744
#rb Francis.Hurteau
[CL 26569151 by danny couture in ue5-main branch]
* Moved dllexport from type to methods/staticvar. This improves compile times, memory and performance in dll builds
#rb
[CL 26081848 by henrik karlsson in ue5-main branch]
The code for parsing a view of the ZenPackageSummary has been extracted from AsyncLoading2.cpp into a private h/cpp and is possible to use from the header diffing code when printing out detailed diffs, but is not used yet.
#rb matt.peters
#rb pj.kack
#jira UE-175237
[CL 25549055 by zousar shaker in ue5-main branch]
#rb Per.Larsson
#jira none
#preflight 64674758c571db717b037aaf
- BULKDATA_ForceStreamPayload and BULKDATA_SerializeCompressedBitWindow have not done anything for sometime.
- BULKDATA_BadDataVersion was a bug added some years ago which could've caused DDC pollution. Since then the DDC keys for the problematic assets have been changed several times and there should no longer be any danger.
- BULKDATA_Unused technically works but is unused and there does not seem to be any demand for it. This flag will probably remain deprecated for longer than the others to check for any valid external feedback.
[CL 25541994 by paul chipchase in ue5-main branch]
#rb Danny.Couture, Devin.Doucette, Per.Larsson
#jira UE-140944
#preflight 645e2c76b71b8186aaca29db
*** FLinkerLoad
- Attaching/detaching both FBulkData and FEditorBulkdata to a FLinkerLoad should now be thread safe.
- Note that this does NOT fix the long standing issue when a FPackage is GCed before some of the UObjects (assets) inside of that package, which causes any remaining bulkdata to be detached (and unable to load off disk) without forcing payloads into memory. This will be addressed as its own work item.
-- FEditorBulkData can still attempt to load its payload from a package after detachment as a fall back measure that is preferable to a crash.
*** FEditorBulkData
- Forced to use FRecursiveMutex at the moment some methods that need locking make calls into the bulkdata registry which can make calls back to the calling bulkdata object ending up with recursive locking, at least for now.
- This might look like a lot of locks being introduced, but FEditorBulkData methods are called relatively infrequently., in a test project with a semi large amount of data (~16GB) cooking from a cold DDC benchmarked at roughly the same wall time before/after these changes.
- Added a comment block to the header describing the thread safety expectations of the class.
- Made the class final as we really don't want to deal with inherited bulkdata types as we did with the old system.
- Since FLinkerLoad attachment/detachment is now thread safe we can detach from any pre-existing AttachedAr when running the assignment operator and attach to the AttachedAr of the provided bulkdata. This means copied/assigned bulkdata do not need to force their payloads into memory.
- Cleaned up some comments in the header.
[CL 25485916 by paul chipchase in ue5-main branch]
- FSoftObjectPath properties report as containing weak object references for serializers.
- Mark PIE ID fixup archive as ArIsObjectReferenceCollector.
#rb francis.hurteau
[CL 25415885 by robert millar in ue5-main branch]
Currently, Landscape will only apply the new storage provider to it's heightmaps at cook time, when it sees that the landscape has been flagged to use the new compressed storage path (enabled under Landscape -> Advanced), and only for Windows platforms (to be expanded as we test other platforms).
The compressed storage path option on the landscape is hidden by default, and can be made visible via CVAR ("landscape.ShowCompressHeightMapsOption")
Makes use of a new All Mip Provider interface to Texture2Ds, that expands on the existing Mip Data Provider (that only supported providing streaming mips).
The new interface assumes you want to completely override the Texture2D mip data.
When Texture2D sees that it has an All Mip Provider, it will skip serializing it's own heavyweight mip data during cook, as it assumes the provider will take care of providing all mip data.
The majority of the implementation is in LandscapeTextureStorageProvider, which implements the AllMipProvider that can store landscape heightmaps in a custom compressed format. Each mip's height data is stored using delta encoding, and normals are reconstructed.
#rb jonathan.bard, jian.ru
#jira UE-180654
#preflight 64541acf6c35ad81e61890cc
#preflight 64542713fd4b8f4e0d9a2ce9
#preflight 6459378bfd4b8f4e0dc206d2
[CL 25381191 by chris tchou in ue5-main branch]
#rb Per.Larsson
#jira none
#rnx
#preflight 6453ce50d863ba2621851360
- FCompressedBuffer will calculate the FIoHash of its data and FEditorBulkData can take the hash from there rather than calculating one itself. If you create one with no compression codec set you could use that to control when the hashing takes place before passing the data to FEditorBulkData.
- In the future we might change FEditorBulkData to hold its data in a FCompressedBuffer, allowing us to avoid decompressing the payload unless someone actually asks for it in uncompressed format.
- Added some unit tests
[CL 25352762 by paul chipchase in ue5-main branch]
