#rb Paul.Chipchase, PJ.Kack, CarlMagnus.Nordin
#jira none
#preflight 629085d14f63120d8e170977
###
This change attempts to merge the editor and the runtime version of bulk data, i.e. bulk data one and two.
The major difference between the two is the fact that the editor version supported element vice
serialization via inheritance. This has been complelety removed in the runtime version. In order
to mitigate this divergance a non-virtual base class has been extracted that forms the base
for the untyped and typed versions of bulk data.
The untyped version is only used in editor and will be deprecated. However this is a breaking change,
previously it was possible to inherit from FByteBulkData in editor builds.
Other differences between the two includes:
* IsBulkDataLoaded - returns true for bulk data 1 if an allocation has been made even if the size is zero
* StartStreaming - was removed in bulk data 2 and has been removed
* FlushAsyncLoading - bulk data 2 was busy waiting, now an event is used
All I/O related functionality has been extracted into BulkDataStreaming.cpp and is using
a FBulkDataChunkId that is either a package path or a package ID.This could potentially
be extended to contain a cache key in development builds to support loading from DDC, i.e derived
data references.
[CL 20391050 by Per Larsson in ue5-main branch]
Remove bValid from FMetaData and FData return structs; it is redundant with the RawHash/Buffer data already present in those structs.
#preflight 62680b42430b9997ebe0a46d
#rb Devin.Doucette
#rnx
[CL 19923039 by Matt Peters in ue5-main branch]
Automatically propagate explicit package load request association to a package dependencies which allow flush request to process only related packages while skipping others
This should also allow FlushAsyncLoading to be re-entrant down the line
Suppress the transaction system while ticking async loading in the main thread
Added an editor config to enabled the async loading thread. Currently non functional
#rb CarlMagnus.Nordin
#preflight 623e0081a67e4e1ab70598d7
[CL 19626072 by Francis Hurteau in ue5-main branch]
331s -> 0.28s to consolidate objects when cloning a small shooting range map
#rb Robert.Manuszewski,Chris.Gagnon,Francis.Hurteau,Jamie.Dale,JeanLuc.Corenthin,Phillip.Kavan
#ROBOMERGE-AUTHOR: johan.torp
#ROBOMERGE-SOURCE: CL 19507095 via CL 19507103 via CL 19507108 via CL 19507110
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v937-19513599)
[CL 19514683 by johan torp in ue5-main branch]
Adds a define UE_FNAME_OUTLINE_NUMBER.
Removes of FName, FMinimalName from memory image support. Adds of FMemoryImageName.
Removal of FMinimalName operator<<, all fields made private, size made variable.
All fields of FScriptName made private.
Added console commands for dumping numbered/unnumbered names and stats.
#rb johan.torp
#ROBOMERGE-OWNER: robert.millar
#ROBOMERGE-AUTHOR: robert.millar
#ROBOMERGE-SOURCE: CL 19058026 via CL 19058611 via CL 19058656 via CL 19061727 via CL 19061740 via CL 19064047
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v918-19018356)
[CL 19066701 by robert millar in ue5-main branch]
#rb Andrew.Scheidecker
#preflight 620f3a108863746a104d6c36
#ROBOMERGE-AUTHOR: robert.manuszewski
#ROBOMERGE-SOURCE: CL 19047989 via CL 19048186 via CL 19048197 via CL 19048203 via CL 19049414
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v918-19018356)
[CL 19065767 by robert manuszewski in ue5-main branch]
BulkDataRegistry: Wait until the top-level EndLoadPackage, and wait until after OnEnginePostInit, before deleting bulkdatasize records. Deleting them sooner can make them be deleted before they're used by the EditorDomain if the EditorDomain's TrySavePackage triggers package loading and a recursive OnEndLoadPackage.
#rb Zousar.Shaker
#rnx
#preflight 620f00e7e8554a6f64f74d5a
[CL 19046732 by Matt Peters in ue5-main branch]
- Text format now includes import and export tables instead of discovering import while serializating exports. This is required to have enough information about objects with an override package that have their Outer in another package.
- UObject Json formatters now export import/export indices instead of relying on the objects full paths, which index into the exported import/export tables.
- Merged code to serialize UObject* and FObjectPtr in Json formatters to avoid code duplication.
#rb francis.hurteau, steve.robb
#preflight 61fd34030a43b689e1750d28
#rnx
#ROBOMERGE-AUTHOR: jeanfrancois.dube
#ROBOMERGE-SOURCE: CL 18978952 via CL 18980785 via CL 18981531 via CL 18981661 via CL 18992713 via CL 18993052
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v917-18934589)
[CL 18997360 by jeanfrancois dube in ue5-main branch]
#rb Per.Larsson, Devin.Doucette
#jira UE-133497
#rnx
#preflight 61f835491c5ac5523462810a
- A lot of files have been touched but most of this is changing FPayload::IsValid to !FIoHash::IsZero, the main changes are in EditorBulkData/EditorBulkDataTests
- The only difference between FPayloadId and FIoHash is that the former considered the hash of an invalid or empty buffer to be 'invalid' too where as FIoHash would return a valid hash
-- To keep behaviour the same, we only hash payloads in EditorBulkData using a utility method ::HashBuffer which will not hash empty or invalid payloads and return a default FIoHash instead.
-- Unit tests have been extended to prove that the behaviour has not changed.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18806362 in //UE5/Release-5.0/... via CL 18808527 via CL 18821790
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v908-18788545)
[CL 18822154 by paul chipchase in ue5-main branch]
FIoDispatcher::IsInitialized() will now return true if it exists (which is almost always)
Renamed FArchive::IsUsingEventDrivenLoader() to IsLoadingFromCookedPackage and based it on the PKG_Cooked flag
#rnx
#rb pj.kack
#preflight 61f3ab6f74510448a6740570
#ROBOMERGE-AUTHOR: carlmagnus.nordin
#ROBOMERGE-SOURCE: CL 18769198 in //UE5/Release-5.0/... via CL 18769204 via CL 18769272
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v903-18687472)
[CL 18769277 by carlmagnus nordin in ue5-main branch]
#rb PJ.Kack
#rnx
#jira UE-139708
#preflight 61eaaedfc92021e535b6d1e0
- The goal is to show a clear separation between the bulkdata object and the virtualization system.
- NOTE that all of these classes were added during 5.0 development so we are just renaming and moving them, no deprecation paths.
- Renamed FVirtualizedUntypedBulkData to FEditorBulkData
-- Also removed the derived templated types, they were never actually used
- Renamed FVirtualizedBulkDataReader to FEditorBulkDataReader
- Renamed FVirtualizedBulkDataWriter to FEditorBulkDataWriter
- Moved the renamed classes to the UE::Serialization namespace from UE::Virtualization
- Renamed the files of the renamed classes where required.
- Replaced use of LogVirtualization with LogSerialization.
- Renamed defines prefixed with VBD_* to UE_ and make sure they are undefed by the end of the cpp
- Edited unit tests to show up under "System.CoreUObject.Serialization.EditorBulkData.*"
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18688778 in //UE5/Release-5.0/... via CL 18688787 via CL 18688810
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v903-18687472)
[CL 18688823 by paul chipchase in ue5-main branch]
Add support for importing uncooked source packages from cooked packages
#rnx
#jira UE-136265, FORT-426538
#rb pj.kack
#preflight 61e6ff6a3778a195debba826
#ROBOMERGE-OWNER: marc.audy
#ROBOMERGE-AUTHOR: carlmagnus.nordin
#ROBOMERGE-SOURCE: CL 18655480 via CL 18664834 via CL 18665656 via CL 18665916 via CL 18667722 via CL 18667764 via CL 18668174
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v901-18665521)
[CL 18668352 by marc audy in ue5-main branch]
#rb CarlMagnus.Nordin, pj.kack
#rnx
#jira UE-133429
#preflight 61e7c020b875abaf33179ccc
### Token Removal
- Now that we support the IoStore in the editor there is no reason to keep the bulkdata streaming token as both the editor and runtime systems can use the same code paths.
- Existing use of USE_BULKDATA_STREAMING_TOKEN will now give deprecation warnings however code should continue to work as long as STREAMINGTOKEN_PARAM was correctly used.
- ::CreateStreamingToken has been removed from the bulkdata api, but any calls to this should've been wrapped with #if USE_BULKDATA_STREAMING_TOKEN
### API Fixes
- We can remove all code that relied on USE_BULKDATA_STREAMING_TOKEN being == 1
- LANDSCAPE_LOD_STREAMING_USE_TOKEN required USE_BULKDATA_STREAMING_TOKEN to be enabled, so we can now assume that it is always 0 and remove the associated code paths.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18656055 in //UE5/Release-5.0/... via CL 18656061 via CL 18656063
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v900-18638592)
[CL 18656065 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#jira UE-139348
#rnx
#preflight 61e6742a3778a195dea02199
### PackageTrailer
- We no longer store the access mode for all payloads in the trailer. Instead each payload entry in the lookup table will store the access mode for that specific payload. This allow us to store many payloads of different access types in the same trailer.
- Note that this format change is currently versioned (going from 0 to 1) and is compatible with older version.
-- To allow this, the << operator has been removed from FLookupTableEntry and replaced with a ::Serialize method that takes the version of the package trailer header as an input so that it can run it's own versioning checks.
- When loading trailers with version 0 we apply the access mode to all non virtualized payload entries in the lookup table to reproduce the single access mode for the entire trailer.
- A FPackageTrailerBuilder can now be created directly from an existing FPackageTrailer that will reference all of the trailers local payloads via the new static method ::CreateReferenceToTrailer
- Removed FPackageTrailer::TrySave as it is no longer used when creating a trailer for the editor domain.
- Renamed FPackageTrailer::LoadPayload to ::LoadLocalPayload to make it clear the type of payload that can be loaded.
-- Someday the package trailer should be able to load payloads of any type.
### VirtualizedBulkData Serialization Changes
- We now record if the bulkdata's payload is stored in a package trailer or not via the 'StoredInPackageTrailer' flag, which makes it easier to identify critical errors in serialization (the trailer is missing for example)
-- Storing this info as a flag is now possible as the design for the trailer has changed and a fully virtualized package will still contain a trailer.
- We now only write out OffsetInFile if the payload is not stored in a FPackageTrailer, if the bulkdata object does use a package trailer via the new flag then we take the offset from the trailer instead.
- We also no longer write out a path when saving a payload as referenced to the editor domain, it is assumed that the trailer will know where the workspace domain data is (namely the owning package path)
-- Due to the 'StoredInPackageTrailer' only being added now, this means that packages already saved with a package trailer will work with this code logic as those packages will have serialized the offsets to disk, but won't have the new flag and so will serialize the offsets when loading until re-saved. (worth noting that nobody should encounter this as the package trailer system is disabled by default)
- The loading code is a lot easier to read now and a lot of the branches have been removed although we still have a few extra checks for older formats that might have been saved during development.
- When loading the only real consideration we have is if the payload is in the trailer or not.
- Future Work:
-- If we are not writing to the trailer then we use the legacy serialization system which appends the bulkdata to the end of the package. This is used in two cases, 1) When the package trailer system is disabled 2) If the payload is updated in postload and we are writing out the package to the editor domain as the package trailer does not support writing out local payloads outside of the workspace domain at the moment. The trailer should handle this itself and not rely on the legacy serialization system so that we can remove it.
-- Note that if a payload is stored by reference, we are not adding it explicitly to the payload trailer and we are relying on the trailer builder already containing the reference information when it is created in SavePackage. We do however run an assert to ensure that the reference does exist in the trailer but it might be worth revisiting this.
### VirtualizedBulkData Misc
- The member 'PackageSegment' is now initialized in the constructor to EPackageSegment::Header.
- Moved ::GetPackagePathFromOwner from being a method to a utility function as it accessed no members.
- Validating the payload for saving during serialization has been moved from ::Serialize to it's own method ::TryPayloadValidationForSaving.
### PackageSubmissionChecks
- Added an error message if we detect referenced payloads in a trailer we are trying to virtualize as this condition should never occur. If we encounter it, we give a user facing error message and fail the submit process. It might be worth demoting this to an assert instead as it is not an error the user is expected to see.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18637906 in //UE5/Release-5.0/... via CL 18637926 via CL 18637929
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v899-18417669)
[CL 18637932 by paul chipchase in ue5-main branch]
