- FDerivedDataIoBatch::Compress requests the entire compressed value.
- FDerivedDataIoResponse::GetCompressedData gets the compressed buffer from the response when available.
- FDerivedDataIoResponse::GetCacheValueId has been added to get the FValueId when available.
- FDerivedDataIoResponse::GetCacheKey has been fixed to return a non-null cache key when appropriate.
- Asserts about FDerivedData now log the derived data reference too.
#preflight 631a5f6fa60c539c98ab5fa9
#rb Zousar.Shaker
[CL 21920523 by Devin Doucette in ue5-main branch]
- This change fixes an issue when batching bulk data I/O requests where the bulk data is layed out contiguous in the chunk/file. Internally
the max read size was previously capped to the size of the bulk data but when merging read requests we read past the first bulk data
instance, i.e. from first to last. This is safe as long as the range is stored contiguously in the same chunk/file.
#rb Paul.Chipchase
#jira UE-162910
#preflight 6314d66ba20b67673bca3713
[CL 21788475 by Per Larsson in ue5-main branch]
#rb CarlMagnus.Nordin, Paul.Chipchase
#jira none
#preflight 62fdf44787319bacfb44bdcb
- deprecated and replaced the usage of StartAsyncLoading and CreateStreamingRequestForRange with a new batch API for requesting bulk data. The
new API fits better with the I/O dispatcher API and does not assume that the bulk data resides in the same I/O chunk. The new API also
removes state handling away from the bulk data instances.
[CL 21439218 by Per Larsson in ue5-main branch]
Replace FinalizeInitialLoad with CompleteRegistrationEvent and use the same code path with/without the async loading thread.
Strip the FindAllScriptObjects logic from test and shipping builds.
Optimize initial load NotifyRegistrationEvent handling by passing the finished object instead of looking it up by name.
#jira FORT-492925
#rb carlmagnus.nordin
#rnx
#preflight 62fb3e5b4b04bcb92d500a68
[CL 21402146 by PJ Kack in ue5-main branch]
- Array.h is not needed by CompactBinarySerialization.h because the use of TArray is in a template function.
- SharedBuffer.h is not needed by DerivedData.h because its types are only used as return types or function parameters.
- FString and FCompositeBuffer are both declared in Fwd.h headers and do not need duplicate forward declarations.
#rb none
#rnx
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 21391840 via CL 21394622 via CL 21395405
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v975-21357124)
[CL 21398955 by devin doucette in ue5-main branch]
Headers are updated to contain any missing #includes needed to compile and #includes are sorted. Nothing is removed.
#ushell-cherrypick of 21065896 by bryan.sefcik
#preflight 62d4b1a5a6141b6adfb0c892
#jira
#ROBOMERGE-OWNER: Bryan.sefcik
#ROBOMERGE-AUTHOR: bryan.sefcik
#ROBOMERGE-SOURCE: CL 21150156 via CL 21151754 via CL 21154719
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)
#ROBOMERGE-CONFLICT from-shelf
[CL 21181076 by Bryan sefcik in ue5-main branch]
#rb Devin.Doucette
#jira UE-159985
#rnx
#preflight 62d6e8d2d76ea4b5032d5d56
### Problem
- When CL 21089705 was submitted, we changed the behaviour to assume that it is only safe to load payloads off disk if the AttachedAr pointer is valid rather than checking for the package path being valid. This allowed us to keep the package path intact after ::DetachFromDisk was called and give better error messages.
- However not all code paths were forcing the AttachedAr pointer to be set, TornOff bulkdata instances in the bulkdata registry for example causing these code paths to fail entirely.
- What we need is a way to not allow bulkdata instances that were attached but have been detached from loading (or at least force them to use the speculative loading path that was added in 21089705) and allow any other bulkdata instance that was never attached, to load it's payload as long as the path is set.
### Fix
- Instead of asking ::IsAttachedToPackageFile we instead ask ::HasAttachedArchive as in most places we are really checking if the AttachedAr pointer is null or not before we manipulate it
- There are two places where what we are really asking is if it is safe to load the payload from disk or not, these places now use a new method ::CanLoadDataFromDisk which checks that the bulkdata has access to a valid package path AND that the transient flag EFlags::WasDetached has not bee set.
-- This means that bulk data instances that are never attached to the packages FLinkerLoad can still load off disk, although this is less safe it reverts us back to the behaviour before CL 21089705was submitted.
### Misc
- Marked the transient flags as such in the code documentation.
- Ideally these flags would not have been added to the EFlags enum and a specific ETransientFlag enum added instead but the clean up for that would be irksome at this point.
[CL 21174617 by paul chipchase in ue5-main branch]
#jira UE-159832
#rb Aaron.mcleran Phil.Popp Dan.Thompson Josh.Adams
#preflight 62d260033c3df323907eaa93
[CL 21153330 by jimmy smith in ue5-main branch]
#rb Per.Larsson, Matt.Peters
#jira UE-153904 , UE-156201
#preflight 62ce802c471a2c2886f592e8
### Problem
- Because of the package trailer we can easily find out which payloads are present in a package file and where they are. This means we don't need to rely on the cached offset into the file that bulkdata generally uses for payload loading. This means as long as the payload is still actually in a payload file then we can load it no matter what.
- We will still give errors in the following scenarios
-- The payload is not stored in a package trailer (either referencing older bulkdata or an older editor bulkdata format)
-- We cannot get a read handle to the package file.
-- The payload is no longer in the package file.
- If the attachment is still valid then we can guarantee that the payload can be loaded. Without an attachment we make our best efforts but in the case of the package file being reomoved/replaced this can fail. So ::ResetLoader will still be used to indicate that we want to remove the attachment but expect the package file to be changed and to load the payload into memory for safety. Work on improving this is another work item for the future.
### Fix
- We no longer remove the package path when ::DetachFromDisk is called nor do we remove flags relating to the payload. From this point onwards we use the attached archive being null to determine how we load. We continue to set OffsetInFile to an invalid value as we can no longer safely use it.
- Add a new method ::IsAttachedToPackageFile to explicitly show when we are checking to see if the bulkdata is still 'attached' to the package file.
- Explicitly set AttachedAr to null ptr when ::DetachBulkData is called on it. In theory the ::DetachBulkData call will call FEditorBulkData::DetachFromDisk which in turn will set AttachedAr but it can look confusing when reading the code. Explicitly setting the AttachedAr member to nullptr helps make the intent clearer.
- When loading the payload from disk there is now a new code branch that is followed if we detect that we are loading from a package file that we are no longer attached to.
-- This path gives bespoke errors in cases where it is not safe for us to try loading the payload, in addition there is a bespoke error message if loading from the package trailer fails.
-- Note we can only risk loading from the package trailer as it's format allows for us to find if the payload is still on disk or not. With the older formats we will not know if we are loading garbage or not.
-- Fixed a bug with CVarShouldLoadFromTrailer. When it is true we will only attempt to load payloads from the trailer if the payload is stored in the trailer. The comment for this development CVar has been updated.
- There has been a pass made to error reporting. If we are reporting an error relating to the package we will output the package name, if we are reporting on a error relating to the file itself then we will output the package path + extension
[CL 21089705 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
#preflight 62c6a9faf85287873280cef7
EFlags::TransientFlags has been removed from the enum and instead is stored as a constant. This will allow the debugger to more easily display the flag member.
Move the removal of transient bulkdata flags when saving to ::BuildFlagsForSerialization so that all flag manipulation is done in the same place.
We now update the flags at the end of serialization and remove the EFlags::IsVirtualized bit (if set) if the payload is being stored in a package trailer
-- I originally tried to do this by removing the bit in ::BuildFlagsForSerialization, but that was partially reverted in CL 19976045 as it was causing the editor bulkdata is rehydrate as we check for EFlags::IsVirtualized in several places when saving out the bulkdata
-- Moving the bit removal to the end of serialization is the easiest and safest way to fix this. However with some more work we could probably remove more branches from this logic but I'd rather move that to a separate work item.
Moved the paranoid checks to make sure that the package trailer builder was created correctly to a static method ::ValidatePackageTrailerBuilder in order to make the ::Serialize method easier to read.
[CL 20982025 by paul chipchase in ue5-main branch]
Actual payload support will be implemented separately
#rb Paul.Chipchase
#preflight 62c367cf5751c961938bbd3f
#ushell-cherrypick of 20965910 by Francis.Hurteau
#ROBOMERGE-OWNER: francis.hurteau
#ROBOMERGE-AUTHOR: francis.hurteau
#ROBOMERGE-SOURCE: CL 20967184 via CL 20968121 via CL 20968405 via CL 20970220
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)
[CL 20973707 by robomerge in ue5-main branch]
#rb none
#rnx
#robomerge EngineMerge
#preflight 62bc6744fd062511870fdbcc
- Bulkdata2 made a distinction between allocating for the first time and re-allocating were as the original bulkdata system only called realloc even if we knew that we have no pre-existing allocation.
- Using the Bulkdata2 system in the editor was causing some code paths to crash as they were relying on being able to serialize a bulkdata instance repeatedly without resetting it.
- For now, I have changed the code to default to the old behavior so we only ever call reallocate.
-- A better fix in the future would be to ::Reset the bulkdata before a loading serialization call, but I will add a new JIRA ticket for that.
[CL 20889881 by paul chipchase in ue5-main branch]
NotifyRegistrationEvent:
Remove all NRP_Started events since they add no value.
Move the NRP_Finished event for type NRT_Class from InitializePrivateStaticClass (Class.cpp) to UObjectLoadAllCompiledInDefaultProperties (UObjectBase.cpp) since this is where the outer registration function for the class is actually executed.
Add a new NRT_NoExportObject event in ConstructUPackage for top level noexport objects like UScriptStruct, UDelegateFunction and USparseDelegateFunction that are just part of the package and has no registration info.
AsyncLoader:
Move EDLBootNotificationManager from AsyncPackageLoader to AsyncLoading.
Add FEarlyRegistrationEventsRecorder to replay early events that occur before the loader is created.
AsyncLoading:
Ignore the new NRT_NoExportObject event.
Allow for objects to skip the NRP_Added and NRP_Started phases.
AsyncLoading2:
Remove LazyInitializeFromLoadPackage and the reading and parsing of the ScriptObjects binary blob.
Remove FindScriptImportObjectFromIndex/GFindExistingScriptImport that was dependent on the ScriptObject blob, instead all script objects are added to the global import store directly from the notification system callbacks.
Change GetUnresolvedCDOs logic to check for imported classes with missing or not-yet-initialized CDOs.
Add FAsyncPackageDesc2::ReferencerRequestId to let each package track its most recent load package request referencer.
Change ProcessPendingCDOs logic to avoid recursive reentrancy and to process the most relevant CDOs first, based on the new ReferencerRequestId.
Tests:
Lyra: Editor, Runtime EDL and Zen loader on Win64 and Console
ShooterGame: Editor, Runtime EDL and Zen loader on Win64 and Console
AncientGame: Editor, Runtime EDL and Zen loader on Win64
Other: Editor, CookedEditor, Runtime EDL and Zen loader on Win64 (client and server)
#jira none
#rb carlmagnus.nordin
#robomerge EngineMerge
#rnx
#preflight 62a84ba7cf54a658eeffc200
[CL 20643384 by PJ Kack in ue5-main branch]
#rb Per.Larsson
#rnx
#preflight 629f530958032863438a0b30
- Storing the segment type as a member made the code would imply to the reader that it could be changed, even though in all paths it was being set to EPackageSegment::Header. Removing the member and just hard coding the segment use should make the code easier to follow.
- Editor bulkdata always stores it's payload in the package trailer, which for now counts as the package header.
- The only deviation for this is the experimental feature that allows us to store the payloads in a sidecar file. This code path is not currently supported, but has been updated to use hardcoded EPackageSegment::PayloadSidecar.
- Given that PackageSegment was always set to EPackageSegment::Header we can easily remove any assert of if statement checking for that.
[CL 20553326 by paul chipchase in ue5-main branch]
