Only preload the current package instead of the whole hierarchy recursively
#rnx
#rb PJ.Kack, Francis.Hurteau
#preflight 64788f45947ff6973c0a1e7c
[CL 25788188 by danny couture in ue5-main branch]
I've reviewed all code using async requests and we had multiple places incorrectly handling the stat: some placed failed to decrease the counter, while others would decrease it when they shouldn't. The main cause was the fact that when we create an async request and pass nullptr as the target memory, the request allocates the memory itself and increases the STAT_AsyncFileMemory. However, when we call GetReadResults() on such requests, the responsibility is on the caller to balance the calls and decrease the stat, which is not very obvious and leads to the mentioned bugs. On top of that, when we call GetReadResults() we may not have the knowledge as to whether the returned memory was allocated by the request or not (which may affect whether we should touch the stat at all).
Resolved by adding IAsyncReadRequest::ReleaseMemoryOwnershipImpl member, which may be used by request implementations to do the proper clean up (balance the stat). This way requests take the full responsibility for their changes and users of the requests don't need to know their implementation details.
#preflight none
#rb Patrick.Laflamme
#jira UE-185064
[CL 25664733 by Wojciech Krywult in ue5-main branch]
The code for parsing a view of the ZenPackageSummary has been extracted from AsyncLoading2.cpp into a private h/cpp and is possible to use from the header diffing code when printing out detailed diffs, but is not used yet.
#rb matt.peters
#rb pj.kack
#jira UE-175237
[CL 25549055 by zousar shaker in ue5-main branch]
#rb Per.Larsson
#jira none
#preflight 64674758c571db717b037aaf
- BULKDATA_ForceStreamPayload and BULKDATA_SerializeCompressedBitWindow have not done anything for sometime.
- BULKDATA_BadDataVersion was a bug added some years ago which could've caused DDC pollution. Since then the DDC keys for the problematic assets have been changed several times and there should no longer be any danger.
- BULKDATA_Unused technically works but is unused and there does not seem to be any demand for it. This flag will probably remain deprecated for longer than the others to check for any valid external feedback.
[CL 25541994 by paul chipchase in ue5-main branch]
#rb Danny.Couture, Devin.Doucette, Per.Larsson
#jira UE-140944
#preflight 645e2c76b71b8186aaca29db
*** FLinkerLoad
- Attaching/detaching both FBulkData and FEditorBulkdata to a FLinkerLoad should now be thread safe.
- Note that this does NOT fix the long standing issue when a FPackage is GCed before some of the UObjects (assets) inside of that package, which causes any remaining bulkdata to be detached (and unable to load off disk) without forcing payloads into memory. This will be addressed as its own work item.
-- FEditorBulkData can still attempt to load its payload from a package after detachment as a fall back measure that is preferable to a crash.
*** FEditorBulkData
- Forced to use FRecursiveMutex at the moment some methods that need locking make calls into the bulkdata registry which can make calls back to the calling bulkdata object ending up with recursive locking, at least for now.
- This might look like a lot of locks being introduced, but FEditorBulkData methods are called relatively infrequently., in a test project with a semi large amount of data (~16GB) cooking from a cold DDC benchmarked at roughly the same wall time before/after these changes.
- Added a comment block to the header describing the thread safety expectations of the class.
- Made the class final as we really don't want to deal with inherited bulkdata types as we did with the old system.
- Since FLinkerLoad attachment/detachment is now thread safe we can detach from any pre-existing AttachedAr when running the assignment operator and attach to the AttachedAr of the provided bulkdata. This means copied/assigned bulkdata do not need to force their payloads into memory.
- Cleaned up some comments in the header.
[CL 25485916 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#jira none
#rnx
#preflight 645a13fd013f3751583f0380
- If the value being stored in ExportBundleEntriesCount exceeds MAX_int32 then we will end up failing the check below and printing an error anyway so we might as well just cast at this point.
[CL 25385553 by paul chipchase in ue5-main branch]
#rb Sebastien.Lussier
#jira none
#rnx
#preflight 6453d22b4d593c0b428dacfe
- This should only change the hash when FArchiveCrc32::Serialize is called on buffers larger than MAX_int32 in size, which given the current use cases of FArchiveCrc32 is probably not occuring anywhere anyway.
- Found this issue when fixing truncation warnings.
- Serialize accepts data lengths of type int64 but FCrc::MemCrc32 would silently truncate the data to int32.
-- This doesn't result in the data being hashed up to length MAX_int32 as the iteration inside of MemCrc32 (at least in some implementations) uses a signed index, passing in larger values will result in this index being negative and our loop checks for that index being reduced to 0.
- To fix this we should call FCrc::MemCrc32 with the data length clamped to MAX_int32 and then make repeated calls until the entire buffer hash been hashed.
[CL 25384518 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira none
#rnx
#preflight 6453ce50d863ba2621851360
- FCompressedBuffer will calculate the FIoHash of its data and FEditorBulkData can take the hash from there rather than calculating one itself. If you create one with no compression codec set you could use that to control when the hashing takes place before passing the data to FEditorBulkData.
- In the future we might change FEditorBulkData to hold its data in a FCompressedBuffer, allowing us to avoid decompressing the payload unless someone actually asks for it in uncompressed format.
- Added some unit tests
[CL 25352762 by paul chipchase in ue5-main branch]
#rb trivial
#jira none
#rnx
#preflight 644a6b7f877716c8784935db
- LastDifferenceCallstackOffsetIndex was only ever being used as a int32 so changed it to that type.
[CL 25298576 by paul chipchase in ue5-main branch]
Also use the PostLoadGroup to run the deferred post load using the group to prevent race with ConditionalPostLoad with async and non async PostLoad
#rb PJ.Kack, Danny.Couture
#preflight 644700b8641e2c3cb4dbcaf0, 644ad94eb208f61af83567ea
[CL 25238989 by francis hurteau in ue5-main branch]
#rb Per.Larsson
#jira UE-139530
#rnx
#preflight 644bc6e4f502608229ff7e68
- The functionality remains mostly the same and is still disabled by default.
- We use the older legacy serialization path if sidecar saving is enabled.
- Updating the code to use FPackageTrailer allows us to finally delete the older which was the precursor to the FPackageTrailer system. This will allow us to combine some code paths in the future and reduce complexity.
[CL 25232459 by paul chipchase in ue5-main branch]
