* Moved dllexport from type to methods/staticvar. This improves compile times, memory and performance in dll builds
#rb
[CL 26081848 by henrik karlsson in ue5-main branch]
The code for parsing a view of the ZenPackageSummary has been extracted from AsyncLoading2.cpp into a private h/cpp and is possible to use from the header diffing code when printing out detailed diffs, but is not used yet.
#rb matt.peters
#rb pj.kack
#jira UE-175237
[CL 25549055 by zousar shaker in ue5-main branch]
#rb Per.Larsson
#jira none
#preflight 64674758c571db717b037aaf
- BULKDATA_ForceStreamPayload and BULKDATA_SerializeCompressedBitWindow have not done anything for sometime.
- BULKDATA_BadDataVersion was a bug added some years ago which could've caused DDC pollution. Since then the DDC keys for the problematic assets have been changed several times and there should no longer be any danger.
- BULKDATA_Unused technically works but is unused and there does not seem to be any demand for it. This flag will probably remain deprecated for longer than the others to check for any valid external feedback.
[CL 25541994 by paul chipchase in ue5-main branch]
#rb Danny.Couture, Devin.Doucette, Per.Larsson
#jira UE-140944
#preflight 645e2c76b71b8186aaca29db
*** FLinkerLoad
- Attaching/detaching both FBulkData and FEditorBulkdata to a FLinkerLoad should now be thread safe.
- Note that this does NOT fix the long standing issue when a FPackage is GCed before some of the UObjects (assets) inside of that package, which causes any remaining bulkdata to be detached (and unable to load off disk) without forcing payloads into memory. This will be addressed as its own work item.
-- FEditorBulkData can still attempt to load its payload from a package after detachment as a fall back measure that is preferable to a crash.
*** FEditorBulkData
- Forced to use FRecursiveMutex at the moment some methods that need locking make calls into the bulkdata registry which can make calls back to the calling bulkdata object ending up with recursive locking, at least for now.
- This might look like a lot of locks being introduced, but FEditorBulkData methods are called relatively infrequently., in a test project with a semi large amount of data (~16GB) cooking from a cold DDC benchmarked at roughly the same wall time before/after these changes.
- Added a comment block to the header describing the thread safety expectations of the class.
- Made the class final as we really don't want to deal with inherited bulkdata types as we did with the old system.
- Since FLinkerLoad attachment/detachment is now thread safe we can detach from any pre-existing AttachedAr when running the assignment operator and attach to the AttachedAr of the provided bulkdata. This means copied/assigned bulkdata do not need to force their payloads into memory.
- Cleaned up some comments in the header.
[CL 25485916 by paul chipchase in ue5-main branch]
- FSoftObjectPath properties report as containing weak object references for serializers.
- Mark PIE ID fixup archive as ArIsObjectReferenceCollector.
#rb francis.hurteau
[CL 25415885 by robert millar in ue5-main branch]
Currently, Landscape will only apply the new storage provider to it's heightmaps at cook time, when it sees that the landscape has been flagged to use the new compressed storage path (enabled under Landscape -> Advanced), and only for Windows platforms (to be expanded as we test other platforms).
The compressed storage path option on the landscape is hidden by default, and can be made visible via CVAR ("landscape.ShowCompressHeightMapsOption")
Makes use of a new All Mip Provider interface to Texture2Ds, that expands on the existing Mip Data Provider (that only supported providing streaming mips).
The new interface assumes you want to completely override the Texture2D mip data.
When Texture2D sees that it has an All Mip Provider, it will skip serializing it's own heavyweight mip data during cook, as it assumes the provider will take care of providing all mip data.
The majority of the implementation is in LandscapeTextureStorageProvider, which implements the AllMipProvider that can store landscape heightmaps in a custom compressed format. Each mip's height data is stored using delta encoding, and normals are reconstructed.
#rb jonathan.bard, jian.ru
#jira UE-180654
#preflight 64541acf6c35ad81e61890cc
#preflight 64542713fd4b8f4e0d9a2ce9
#preflight 6459378bfd4b8f4e0dc206d2
[CL 25381191 by chris tchou in ue5-main branch]
#rb Per.Larsson
#jira none
#rnx
#preflight 6453ce50d863ba2621851360
- FCompressedBuffer will calculate the FIoHash of its data and FEditorBulkData can take the hash from there rather than calculating one itself. If you create one with no compression codec set you could use that to control when the hashing takes place before passing the data to FEditorBulkData.
- In the future we might change FEditorBulkData to hold its data in a FCompressedBuffer, allowing us to avoid decompressing the payload unless someone actually asks for it in uncompressed format.
- Added some unit tests
[CL 25352762 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-139530
#rnx
#preflight 644bc6e4f502608229ff7e68
- The functionality remains mostly the same and is still disabled by default.
- We use the older legacy serialization path if sidecar saving is enabled.
- Updating the code to use FPackageTrailer allows us to finally delete the older which was the precursor to the FPackageTrailer system. This will allow us to combine some code paths in the future and reduce complexity.
[CL 25232459 by paul chipchase in ue5-main branch]
- In CL 3462726, Max.Chen moved the FObjectWriter(TArray<uint8>& InBytes) constructor into public so that it can be used with UClass::SerializeTaggedProperties. This change proposes we do the same with FObjectReader.
#rb Matt.Peters
#jira none
#preflight 64348f8ab4329f6f1a04729e
[CL 24985648 by jordan hoffmann in ue5-main branch]
Always silence warning about missing package for script packages, just like GetPackageLinker() simply returns nullptr for PKG_InMemoryOnly.
submitted on behalf of [at]PJ.Kack
#rb Matt.Peters
#preflight 63fe8765ae54ee4ce9e59742
[CL 24469377 by francis hurteau in ue5-main branch]
#rb Per.Larsson
#rnx
#preflight 63f3b1fa52460c3a1f4684ce
- The problem is that our allocation functions take allocation sizes as type SIZE_T which on older 32bit platforms will be int32, but bulkdata stores sizes as int64. If the requested allocation is large enough the size will be truncated and the allocation could still return a valid memory allocation but one much smaller than requested. Later when we try to load into that allocation we will overwrite the end of the buffer. Given our supported platforms and how bulkdata is generally used this is an unlikely scenario but one we still should protect against.
- Added two functions ::SafeMalloc and ::SafeRealloc to replace direct calls to malloc/realloc. These utility functions will check to make sure that we can make allocations of the requested size by making sure that SIZE_T (the type that our allocators take for allocation sizes) is a large enough type.
- If the platform cannot service the request we will issue a fatal log error and halt the process. Technically we could error and return nullptr but in practice very little code checks to see if bulkdata allocations are valid or not and will end up crashing anyway. Given that this problem is only going to show up on 32bit machines and only for very large assets (most likely invalid or broken ones at that) we might as well halt execution as close to the problem as possible to make debugging easier.
- The changes are only applied for platforms where PLATFORM_32BITS is true, should be no cost to 64 bit platforms.
[CL 24335588 by paul chipchase in ue5-main branch]
