#rb Per.Larsson
#rnx
#preflight 63f3b1fa52460c3a1f4684ce
- The problem is that our allocation functions take allocation sizes as type SIZE_T which on older 32bit platforms will be int32, but bulkdata stores sizes as int64. If the requested allocation is large enough the size will be truncated and the allocation could still return a valid memory allocation but one much smaller than requested. Later when we try to load into that allocation we will overwrite the end of the buffer. Given our supported platforms and how bulkdata is generally used this is an unlikely scenario but one we still should protect against.
- Added two functions ::SafeMalloc and ::SafeRealloc to replace direct calls to malloc/realloc. These utility functions will check to make sure that we can make allocations of the requested size by making sure that SIZE_T (the type that our allocators take for allocation sizes) is a large enough type.
- If the platform cannot service the request we will issue a fatal log error and halt the process. Technically we could error and return nullptr but in practice very little code checks to see if bulkdata allocations are valid or not and will end up crashing anyway. Given that this problem is only going to show up on 32bit machines and only for very large assets (most likely invalid or broken ones at that) we might as well halt execution as close to the problem as possible to make debugging easier.
- The changes are only applied for platforms where PLATFORM_32BITS is true, should be no cost to 64 bit platforms.
[CL 24335588 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#jira UE-172853
#rnx
#preflight 63f3834a61378b7ea0065346
- The main issue we are trying to solve is that we serialize the number of elements when saving/loading the package. On load we multiple the number of elements (int64) by the size of each element (int64) and if either of those values is large enough we could overflow and end up with an incorrect size. We do check for this when saving the package so this is only likely to come up if the package has been corrupted in some way.
- Change FBulkMetaData::FromSerialized to do the actual serialization as well. This lets us set the error flag on the archive if we do detect problems and keep all the error handling in a single place.
-- Note that setting the error flag on an archive while a package is being loaded doesn't seem to do much at the moment, so this is more of an attempt to future proof.
- If we detect an error with serializing FBulkMetaResource we will log an error and immediately return so that FBulkMetaData will remain set to defaults.
- We now calculate the largest value of ElementCount that we can support, given the ElementCount size and if the count is exceeded we will log an error, set the error flag on the archive and return leaving FBulkMetaData as default.
- Note that both before and after this change the package is effectively broken and won't work properly, but this change will stop us from potentially stomping memory.
- Note that at runtime we generally serialize the bulkdata info from the BulkDataMap in the package header, so this is mostly an editor thing.
[CL 24322887 by paul chipchase in ue5-main branch]
#rb carlmagnus.nordin
#rnx
#preflight 63ca7b97e3138815655cf971
- Changed a ranged for loop in FEventLoadGraph::RemoveNode to be index based to avoid pointer arithmetic to index casting.
- FAsyncPackage::PostLoadDeferredObjects and FAsyncPackage::CreateClusters were not modifying the parameter InOutTimeLimit that was being passed in by reference, so it has been changed to be passed by value to avoid confusion.
- Changed a number of methods that were taking float values to doubles, in most cases double variables were being passed into these methods and they were being compared to doubles inside of the methods, so there didn't seem to be much point in the parameter type being float.
### Double->Float explicit cast
- Need to do this when using the IAsyncReadRequest api.
- Need to do this when using FLinkerLoad
- FAsyncLoadingThread::TickAsyncThread otherwise we'd need to change FAsyncLoadingThread::ProcessAsyncLoading to take doubles, which in turn would require a large number of double->float casting when calling out to the rest of the engine.
[CL 23788745 by paul chipchase in ue5-main branch]
- extracted out callstack offets and stack traces into FArchiveCallstacks
- created FArchiveStackTraceWriter that wraps an archive and writes into a FArchiveCallstacks
- refactored existing FArchiveStackTrace to use the new writer and callstack collection
- added IPackageWriter::CreateLinkerExportArchive in order to create a stack trace archive writer in DiffPackageWriter
#rb Matt.Peters, Francis.Hurteau
#jira UE-173560
#preflight 63c7bd232e714f64ad89fd2c
#rnx
[CL 23787282 by Per Larsson in ue5-main branch]
- Made a couple of simple getter member functions const
- Added missing virtual and override specifiers
#rb trivial
#preflight 63be92846729b05ec93fda5f
[CL 23648569 by nick edwards in ue5-main branch]
* Removed CoreMinimal.h include from a ton of headers all the way to Actor.h. This shaved off 15% of the compile time of small files that has uht markup.
#preflight 63b915baaf3ebedd9931496e
#rb none
[CL 23607296 by henrik karlsson in ue5-main branch]
Allow EditorDomain to save BulkDataType FBulkDataInfo::BulkSegment.
Declare a Region for each BulkData passed into the EditorDomain PackageWriter.
Mark BulkDatas saved by reference as PayloadInSeparateFile.
Allow RegionTypeNone in SerializePayload.
#rb Francis.Hurteau
#rnx
#preflight 639cec7c776b61ba3b905f8a
[CL 23542385 by Matt Peters in ue5-main branch]
This change aims to unify how BulkData and DerivedData references are stored and loaded from disk.
The structured part of Bulk Data i.e. Flags,Offset and Size has been moved out from the export data
and into a lookup table in the package header. This allows for bulk data to be rearranged w/o needing to recook but it will
also be used for derived data references to allow loading DDC data both from the runtime.
Currenlty this change ONLY effects cooked output.
Inline data can now be reloaded when loading from I/O store, all consistency checks
when discarding/reloading inline data has been removed.
When cooking, exports are serialized into it's own archive enabling the package header
to be modified after cooking the exports. This also makes it easer to rearrange the exports
in the export bundles for optimal loading performance, i.e. there is no longer a dependency
of the cooked serial offset from the begnning of the .uasset file.
#rb CarlMagnus.Nordi, PJ.Kack, Paul.Chipchase
#jira none
#preflight 637387db9e3bea807941a850
#rnx
[CL 23489396 by Per Larsson in ue5-main branch]
#rnx
#jira UE-171080
PR #9856: int64 instead of int32 type when copy-assigning in FBulkDataBuffer (Contributed by gtreshchev)
#preflight 6384a3bffa053c489ae4acba
- The problem was introduced in CL 16399515 which converted the ViewType from TArrayView to TArrayView64
[CL 23287171 by gtreshchev in ue5-main branch]
Change consist of only forward declaration and additional includes
#preflight 63789c1de30d438849c48188
#rb none
[CL 23218412 by henrik karlsson in ue5-main branch]
* Adding forward declarations
* Adding "inline" in front of const/constexpr variables in headers
* Adding includes
* A few moves of ctor/dtor to cpp file
#preflight 6372b3ac0c74adb48b49f313
#rb none
[CL 23130896 by henrik karlsson in ue5-main branch]
Takeaways. Try to use "friend" on functions/operators that are frequently overloaded EXCEPT if they are in a templated type that is frequently instantiated. So do not put friends in TMap, TSet, TObjectPtr etc, this will slow down compile times. There is a break-even somewhere and hard to tell where it is but taking a class that is templatized on character type probably don't matter either way and then it is nicer to use a friend since that simplies error messages when compiler can't resolve functions/operators.
If it is possible to use member functions instead of friend that is the best option in terms of compile time performance. With c++20 you only have to write operator==(Foo, Bar) and the compiler will automatically provide operator==(Bar, Foo), operator!=(Foo, Bar) and operator!=(Bar, Foo).
Changes in this changelist involes
* Making operator<< friends in non-template types and not friends in template types
* Making operator==/!= members where possible and if not possible moved out if type is a frequently instantiated templated type.
#preflight 636970f5376a9cd6a80da54a
#rb steve.robb
[CL 23038965 by henrik karlsson in ue5-main branch]
Moved GetTypeHash function to be hidden friend instead of put directly in global namespace.
Note that the function/operator needs to be fully inlined in the type or placed in the cpp. If the function is added as friend but then implemented outside the type then hidden friend optimization won't work.
This should improve compile time somewhat according to msvc devs.
#rb Steve.Robb
#preflight 6360b7052b5338aceb26471b
[CL 22889837 by henrik karlsson in ue5-main branch]
This is required for c++ modules/header units in order to decide if constexpr/const value should be available to outside module
#preflight 635c0b5c1803be35c767e0ea
#rb none
[CL 22835882 by henrik karlsson in ue5-main branch]
Handle loaded objects being moved to another (transient) package by cleaning up all public export state when removing a loaded package.
Handle reloading of a package with internally renamed objects by allowing StoreGlobalObject to replace the previous public export.
Note: Renaming of loaded packages is still only supported via the delegate GetOnLeakedPackageRenameDelegate.
Compile all debug verification in all builds and make it possible to toggle with s.VerifyUnreachableObject=0/1 and s.VerifyObjectLoadFlags=0/1, while enabling it by default it debug and development builds.
#jira UE-167186, FORT-524537
#rb carlmagnus.nordin
[FYI] robert.millar
#preflight 63526bdc9e14ee3c798aa429, 63527f560313c24974216206
#rnx
[CL 22710443 by pj kack in ue5-main branch]
- This change removes the explicit usage of the package resource manager when loading bulk data from loose files or the Editor Domain. A new
I/O dispatcher backend reads data from the package resource manager with a new I/O chunk type constructed from a package name/package segment. When
serializing bulk data the I/O chunk type is set to the correct chunk type depending on whether the package is cooked or not.
#rb CarlMagnus.Nordin, Paul.Chipchase
#jira none
#preflight 633bdbc66b10157eac8e1807
[CL 22523877 by Per Larsson in ue5-main branch]
Flags have moved into FCookedData to avoid padding, and there is now a static_assert on the size.
A new constructor allows creating references from the private FCookedData, and FLinkerSave now returns FDerivedData instead of FCookedData.
#preflight 633ef62f2a0a2c1ead2f4061
#rb Zousar.Shaker
#rnx
[CL 22377528 by Devin Doucette in ue5-main branch]
