#rb PJ.Kack
#jira UE-136131
#rnx
#preflight 61eacb3d445cebac10c3d046
- A cvar named "Serialization.RehydrateOnSave" now controls if we rehydrate a package on save, defaulting to false.
-- At the very least we will need to be able to turn this to true to run any "rehydrate the project" UAT command in the future.
- The package trailer no longer stores a valid compressed size for a payload. In the best case scenario we only know the compressed size of the payload when it was still stored on locally on disk. Each virtualization backend is free to take the payload and recompress it anyway it sees fit etc.
- When duplicate payloads entries are added to a FPackageTrailerBuilder we now keep the original payload entry and discard the duplicate rather than replacing the existing entry with the duplicate, which was the original intent.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18721599 in //UE5/Release-5.0/... via CL 18721602 via CL 18721661
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v903-18687472)
[CL 18721674 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#rnx
#jira UE-139708
#preflight 61eaaedfc92021e535b6d1e0
- The goal is to show a clear separation between the bulkdata object and the virtualization system.
- NOTE that all of these classes were added during 5.0 development so we are just renaming and moving them, no deprecation paths.
- Renamed FVirtualizedUntypedBulkData to FEditorBulkData
-- Also removed the derived templated types, they were never actually used
- Renamed FVirtualizedBulkDataReader to FEditorBulkDataReader
- Renamed FVirtualizedBulkDataWriter to FEditorBulkDataWriter
- Moved the renamed classes to the UE::Serialization namespace from UE::Virtualization
- Renamed the files of the renamed classes where required.
- Replaced use of LogVirtualization with LogSerialization.
- Renamed defines prefixed with VBD_* to UE_ and make sure they are undefed by the end of the cpp
- Edited unit tests to show up under "System.CoreUObject.Serialization.EditorBulkData.*"
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18688778 in //UE5/Release-5.0/... via CL 18688787 via CL 18688810
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v903-18687472)
[CL 18688823 by paul chipchase in ue5-main branch]
Deprecate IsEventDrivenLoaderEnabledInCookedBuilds() function and always return true.
Delete s.DisableEDLDeprecationWarnings option and NOEDL command line argument.
Keep IsEventDrivenLoaderEnabled() but let it return true for FPlatformProperties::RequiresCookedData().
Delete UseSeperateBulkDataFiles option and always write bulkdata to different files/segments when cooking.
#jira none
#rb francis.hurteau,paul.chipchase,matt.peters
#rnx
#preflight 61e90fc8c32d25dadc27fb0c
#ROBOMERGE-AUTHOR: pj.kack
#ROBOMERGE-SOURCE: CL 18672462 in //UE5/Release-5.0/... via CL 18672465 via CL 18672467
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v901-18665521)
[CL 18672468 by pj kack in ue5-main branch]
Restore propagation of the RF_Loaded flag when duplicating objects as it no longer has any negative effect on the loader
#rb pj.kack
#rnx
#preflight none
#ROBOMERGE-AUTHOR: carlmagnus.nordin
#ROBOMERGE-SOURCE: CL 18657189 via CL 18664846 via CL 18665679 via CL 18665921 via CL 18667765 via CL 18667809 via CL 18668182
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v901-18665521)
[CL 18668365 by carlmagnus nordin in ue5-main branch]
This is in preparation of the deprecation of the usage of public data members in UPackage
ResetLoaders now also empty a package custom versions array when detaching a valid linker from it.
#rb Frank.Fella, PJ.Kack
#jira UE-138957
#preflight 61e71c84b56c33b8ecf46535
#ROBOMERGE-AUTHOR: francis.hurteau
#ROBOMERGE-SOURCE: CL 18646219 in //UE5/Release-5.0/... via CL 18646233 via CL 18646258
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v900-18638592)
[CL 18648154 by francis hurteau in ue5-main branch]
Added similar support to the package store manifest
Zen Store support will be added separately
Added some extra data to linker object resources to be used by the iostore package optimizer to properly support optional package chunks.
Multiple cooking outputs is used to support "optional objects" which are editor data package exports that would be put in separate iostore containers than the normal cooked game content.
#jira UE-129801
#rb Matt.Peters, CarlMagnus.Nordin
#preflight 61e5d3bb873f2ea48f48166c
#ROBOMERGE-AUTHOR: francis.hurteau
#ROBOMERGE-SOURCE: CL 18639742 in //UE5/Release-5.0/... via CL 18639751 via CL 18639758
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v900-18638592)
[CL 18639762 by francis hurteau in ue5-main branch]
#rb PJ.Kack
#jira UE-139348
#rnx
#preflight 61e6742a3778a195dea02199
### PackageTrailer
- We no longer store the access mode for all payloads in the trailer. Instead each payload entry in the lookup table will store the access mode for that specific payload. This allow us to store many payloads of different access types in the same trailer.
- Note that this format change is currently versioned (going from 0 to 1) and is compatible with older version.
-- To allow this, the << operator has been removed from FLookupTableEntry and replaced with a ::Serialize method that takes the version of the package trailer header as an input so that it can run it's own versioning checks.
- When loading trailers with version 0 we apply the access mode to all non virtualized payload entries in the lookup table to reproduce the single access mode for the entire trailer.
- A FPackageTrailerBuilder can now be created directly from an existing FPackageTrailer that will reference all of the trailers local payloads via the new static method ::CreateReferenceToTrailer
- Removed FPackageTrailer::TrySave as it is no longer used when creating a trailer for the editor domain.
- Renamed FPackageTrailer::LoadPayload to ::LoadLocalPayload to make it clear the type of payload that can be loaded.
-- Someday the package trailer should be able to load payloads of any type.
### VirtualizedBulkData Serialization Changes
- We now record if the bulkdata's payload is stored in a package trailer or not via the 'StoredInPackageTrailer' flag, which makes it easier to identify critical errors in serialization (the trailer is missing for example)
-- Storing this info as a flag is now possible as the design for the trailer has changed and a fully virtualized package will still contain a trailer.
- We now only write out OffsetInFile if the payload is not stored in a FPackageTrailer, if the bulkdata object does use a package trailer via the new flag then we take the offset from the trailer instead.
- We also no longer write out a path when saving a payload as referenced to the editor domain, it is assumed that the trailer will know where the workspace domain data is (namely the owning package path)
-- Due to the 'StoredInPackageTrailer' only being added now, this means that packages already saved with a package trailer will work with this code logic as those packages will have serialized the offsets to disk, but won't have the new flag and so will serialize the offsets when loading until re-saved. (worth noting that nobody should encounter this as the package trailer system is disabled by default)
- The loading code is a lot easier to read now and a lot of the branches have been removed although we still have a few extra checks for older formats that might have been saved during development.
- When loading the only real consideration we have is if the payload is in the trailer or not.
- Future Work:
-- If we are not writing to the trailer then we use the legacy serialization system which appends the bulkdata to the end of the package. This is used in two cases, 1) When the package trailer system is disabled 2) If the payload is updated in postload and we are writing out the package to the editor domain as the package trailer does not support writing out local payloads outside of the workspace domain at the moment. The trailer should handle this itself and not rely on the legacy serialization system so that we can remove it.
-- Note that if a payload is stored by reference, we are not adding it explicitly to the payload trailer and we are relying on the trailer builder already containing the reference information when it is created in SavePackage. We do however run an assert to ensure that the reference does exist in the trailer but it might be worth revisiting this.
### VirtualizedBulkData Misc
- The member 'PackageSegment' is now initialized in the constructor to EPackageSegment::Header.
- Moved ::GetPackagePathFromOwner from being a method to a utility function as it accessed no members.
- Validating the payload for saving during serialization has been moved from ::Serialize to it's own method ::TryPayloadValidationForSaving.
### PackageSubmissionChecks
- Added an error message if we detect referenced payloads in a trailer we are trying to virtualize as this condition should never occur. If we encounter it, we give a user facing error message and fail the submit process. It might be worth demoting this to an assert instead as it is not an error the user is expected to see.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18637906 in //UE5/Release-5.0/... via CL 18637926 via CL 18637929
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v899-18417669)
[CL 18637932 by paul chipchase in ue5-main branch]
Split FPackageName's initialization into two events so that FPackageName::TryConvertFilenameToLongPackageName can be called with limited functionality before pluginmanager and coreobject have been initialized.
#preflight 61e58568006b17a6865ac944
#rb Graeme.Thornton
#rnx
#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 18634271 in //UE5/Release-5.0/... via CL 18634294 via CL 18634301
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v899-18417669)
[CL 18637425 by matt peters in ue5-main branch]
#rb Matt.Peters
#rnx
#jira UE-136891
#preflight 61dbf8eeaf44b43628343f00
- Now when saving a package to the editor domain we will load the existing package trailer (if one exists) from the workspace domain and then create a new trailer that references those payloads rather than store them locally.
- VirtualizedBulkData serialization is still overly complicated and this code adds more branches there. If we stopped the VBD objects from serializing their offsets at all and rely entirely on the package trailer to know where the payloads are stored then we could simplify this, but that requires the package trailer system to be turned on so will be done as future work.
### VirtualizedBulkData
- Split bAllowVirtualizationOnSave into a second variable. The original now controls if we push the payload on save (off by default and not something we are likely to keep) and the new variable (bForceRehydrationOnSave) controls if we must pull payloads when the package is saved (on by default) which will be moved to a config value once properly supported.
-- The split was done to make the logic in ::Serialize clearer
- We now correctly Register/Unregister the bulkdata object when ::DetachFromDisk is called. This prevents the bulkdata registry from trying to access the workspace domain file after it has been invalidated.
-- In addition we now attach to the FLinkerLoader's cachable archive (if there is one) when using the ::IsReferencingByPackagePath branch of serialization.
- When saving the bulkdata as part of an editor domain save we can keep the virtualization flag as we don't have to worry about the workspace domain file changing without the editor domain version being rebuilt anyway.
### SavePackage
- If the package is part of a procedural save and not a cook we will try to load the trailer from the workspace domain and then convert it to a referenced trailer (which contains the correct info to load the payload from the workspace domain package file but does not contain the payloads itself) which is then appended to the package.
### PackageTrailer
- Removed the fatal error if a project has both the editor domain and the package trailer system enabled.
- Added a method ::CreateReference that creates a trailer that references the payloads in an existing trailer.
- Added a method ::TrySave, allowing direct writing of a trailer to an archive.
-- Now that we have TrySave and TryLoad it probably makes sense to add the << operator as the public interface and move the methods to be private.
- FFooter and FHeader have their own serialization overloads now rather than relying on the higher level structures to do the work.
- When building a new trailer we now create a new FPackageTrailer structure and then write that out in one go, rather than writing each value as it is calculated to make the code easier to read.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18557778 in //UE5/Release-5.0/... via CL 18557792 via CL 18557812
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Test -> Main) (v899-18417669)
[CL 18557816 by paul chipchase in ue5-main branch]
#rb Matt.Peters
#rnx
#jira UE-136891
#preflight 61dbf8eeaf44b43628343f00
- Now when saving a package to the editor domain we will load the existing package trailer (if one exists) from the workspace domain and then create a new trailer that references those payloads rather than store them locally.
- VirtualizedBulkData serialization is still overly complicated and this code adds more branches there. If we stopped the VBD objects from serializing their offsets at all and rely entirely on the package trailer to know where the payloads are stored then we could simplify this, but that requires the package trailer system to be turned on so will be done as future work.
### VirtualizedBulkData
- Split bAllowVirtualizationOnSave into a second variable. The original now controls if we push the payload on save (off by default and not something we are likely to keep) and the new variable (bForceRehydrationOnSave) controls if we must pull payloads when the package is saved (on by default) which will be moved to a config value once properly supported.
-- The split was done to make the logic in ::Serialize clearer
- We now correctly Register/Unregister the bulkdata object when ::DetachFromDisk is called. This prevents the bulkdata registry from trying to access the workspace domain file after it has been invalidated.
-- In addition we now attach to the FLinkerLoader's cachable archive (if there is one) when using the ::IsReferencingByPackagePath branch of serialization.
- When saving the bulkdata as part of an editor domain save we can keep the virtualization flag as we don't have to worry about the workspace domain file changing without the editor domain version being rebuilt anyway.
### SavePackage
- If the package is part of a procedural save and not a cook we will try to load the trailer from the workspace domain and then convert it to a referenced trailer (which contains the correct info to load the payload from the workspace domain package file but does not contain the payloads itself) which is then appended to the package.
### PackageTrailer
- Removed the fatal error if a project has both the editor domain and the package trailer system enabled.
- Added a method ::CreateReference that creates a trailer that references the payloads in an existing trailer.
- Added a method ::TrySave, allowing direct writing of a trailer to an archive.
-- Now that we have TrySave and TryLoad it probably makes sense to add the << operator as the public interface and move the methods to be private.
- FFooter and FHeader have their own serialization overloads now rather than relying on the higher level structures to do the work.
- When building a new trailer we now create a new FPackageTrailer structure and then write that out in one go, rather than writing each value as it is calculated to make the code easier to read.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18557778 in //UE5/Release-5.0/... via CL 18557792
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)
[CL 18557812 by paul chipchase in ue5-release-engine-test branch]
Fixed issue where SavePackage2 would fail to gather custom versions for CDOs
* FPackageHarvester::ProcessExport() is using UClass::SerializeDefaultObject() which will not call CDO->Serialize()
[at]francis.hurteau
#localization none
#tests Resaved BP assets packages, child classes of AWaterBodyIsland - those now have a FWaterCustomVersion as expected
#preferred_whitelister bob.tellez
#ROBOMERGE-AUTHOR: bob.tellez
#ROBOMERGE-SOURCE: CL 18550470 via CL 18550475 via CL 18550481 via CL 18550486 via CL 18550487 via CL 18550665 via CL 18550671 via CL 18550677
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Test -> Main) (v899-18417669)
[CL 18550685 by bob tellez in ue5-main branch]
Fixed issue where SavePackage2 would fail to gather custom versions for CDOs
* FPackageHarvester::ProcessExport() is using UClass::SerializeDefaultObject() which will not call CDO->Serialize()
[at]francis.hurteau
#localization none
#tests Resaved BP assets packages, child classes of AWaterBodyIsland - those now have a FWaterCustomVersion as expected
#preferred_whitelister bob.tellez
#ROBOMERGE-AUTHOR: bob.tellez
#ROBOMERGE-SOURCE: CL 18550470 via CL 18550475 via CL 18550481 via CL 18550486 via CL 18550487 via CL 18550665 via CL 18550671
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)
[CL 18550677 by bob tellez in ue5-release-engine-test branch]
Some minor deprecation cleanup
#rb louisphilippe.seguin, jon.nabozny
#ROBOMERGE-AUTHOR: brian.bekich
#ROBOMERGE-SOURCE: CL 18546386 via CL 18546409 via CL 18546424 via CL 18547591 via CL 18547656 via CL 18547712
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Test -> Main) (v899-18417669)
[CL 18547788 by brian bekich in ue5-main branch]
Some minor deprecation cleanup
#rb louisphilippe.seguin, jon.nabozny
#ROBOMERGE-AUTHOR: brian.bekich
#ROBOMERGE-SOURCE: CL 18546386 via CL 18546409 via CL 18546424 via CL 18547591 via CL 18547656
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)
[CL 18547712 by brian bekich in ue5-release-engine-test branch]
