#rb PJ.Kack
#jira UE-136131
#rnx
#preflight 61eacb3d445cebac10c3d046
- A cvar named "Serialization.RehydrateOnSave" now controls if we rehydrate a package on save, defaulting to false.
-- At the very least we will need to be able to turn this to true to run any "rehydrate the project" UAT command in the future.
- The package trailer no longer stores a valid compressed size for a payload. In the best case scenario we only know the compressed size of the payload when it was still stored on locally on disk. Each virtualization backend is free to take the payload and recompress it anyway it sees fit etc.
- When duplicate payloads entries are added to a FPackageTrailerBuilder we now keep the original payload entry and discard the duplicate rather than replacing the existing entry with the duplicate, which was the original intent.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18721599 in //UE5/Release-5.0/... via CL 18721602 via CL 18721661
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v903-18687472)
[CL 18721674 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#jira UE-140005
#rnx
[FYI] Mark.Lintott
#preflight 61ee5ec4e6107b23a020d0f5
- EditorBulkData payloads that are updated in PostLoad were being added to the package trailer builder for local access when saving the package for the editor domain.
- The package trailer system is not able to support this (although this will change soon) so when this occurs we should force those payloads to the legacy serialization path.
- Cleaned up ::ShouldUseLegacySerialization by factoring out the code that is actually affected by UE_ENABLE_VIRTUALIZATION_TOGGLE is wrapped in it, making things easier to read.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18706628 in //UE5/Release-5.0/... via CL 18706631 via CL 18706657
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v903-18687472)
[CL 18706667 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#rnx
#jira UE-139708
#preflight 61eaaedfc92021e535b6d1e0
- The goal is to show a clear separation between the bulkdata object and the virtualization system.
- NOTE that all of these classes were added during 5.0 development so we are just renaming and moving them, no deprecation paths.
- Renamed FVirtualizedUntypedBulkData to FEditorBulkData
-- Also removed the derived templated types, they were never actually used
- Renamed FVirtualizedBulkDataReader to FEditorBulkDataReader
- Renamed FVirtualizedBulkDataWriter to FEditorBulkDataWriter
- Moved the renamed classes to the UE::Serialization namespace from UE::Virtualization
- Renamed the files of the renamed classes where required.
- Replaced use of LogVirtualization with LogSerialization.
- Renamed defines prefixed with VBD_* to UE_ and make sure they are undefed by the end of the cpp
- Edited unit tests to show up under "System.CoreUObject.Serialization.EditorBulkData.*"
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18688778 in //UE5/Release-5.0/... via CL 18688787 via CL 18688810
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v903-18687472)
[CL 18688823 by paul chipchase in ue5-main branch]
Deprecate IsEventDrivenLoaderEnabledInCookedBuilds() function and always return true.
Delete s.DisableEDLDeprecationWarnings option and NOEDL command line argument.
Keep IsEventDrivenLoaderEnabled() but let it return true for FPlatformProperties::RequiresCookedData().
Delete UseSeperateBulkDataFiles option and always write bulkdata to different files/segments when cooking.
#jira none
#rb francis.hurteau,paul.chipchase,matt.peters
#rnx
#preflight 61e90fc8c32d25dadc27fb0c
#ROBOMERGE-AUTHOR: pj.kack
#ROBOMERGE-SOURCE: CL 18672462 in //UE5/Release-5.0/... via CL 18672465 via CL 18672467
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v901-18665521)
[CL 18672468 by pj kack in ue5-main branch]
#preflight
#ROBOMERGE-AUTHOR: marc.audy
#ROBOMERGE-SOURCE: CL 18668118 in //UE5/Release-5.0/... via CL 18668651 via CL 18668756
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v901-18665521)
[CL 18668788 by marc audy in ue5-main branch]
Add support for importing uncooked source packages from cooked packages
#rnx
#jira UE-136265, FORT-426538
#rb pj.kack
#preflight 61e6ff6a3778a195debba826
#ROBOMERGE-OWNER: marc.audy
#ROBOMERGE-AUTHOR: carlmagnus.nordin
#ROBOMERGE-SOURCE: CL 18655480 via CL 18664834 via CL 18665656 via CL 18665916 via CL 18667722 via CL 18667764 via CL 18668174
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v901-18665521)
[CL 18668352 by marc audy in ue5-main branch]
#rb CarlMagnus.Nordin, pj.kack
#rnx
#jira UE-133429
#preflight 61e7c020b875abaf33179ccc
### Token Removal
- Now that we support the IoStore in the editor there is no reason to keep the bulkdata streaming token as both the editor and runtime systems can use the same code paths.
- Existing use of USE_BULKDATA_STREAMING_TOKEN will now give deprecation warnings however code should continue to work as long as STREAMINGTOKEN_PARAM was correctly used.
- ::CreateStreamingToken has been removed from the bulkdata api, but any calls to this should've been wrapped with #if USE_BULKDATA_STREAMING_TOKEN
### API Fixes
- We can remove all code that relied on USE_BULKDATA_STREAMING_TOKEN being == 1
- LANDSCAPE_LOD_STREAMING_USE_TOKEN required USE_BULKDATA_STREAMING_TOKEN to be enabled, so we can now assume that it is always 0 and remove the associated code paths.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18656055 in //UE5/Release-5.0/... via CL 18656061 via CL 18656063
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v900-18638592)
[CL 18656065 by paul chipchase in ue5-main branch]
#rb trivial
#rnx
#preflight 61e6ee517a4131bf358cbfc0
- Simplified the assert checking that the trailer exists by moving inside the code branch that has already checked that the bulkdata object should have a trailer.
- This makes the code easier to read and shows that it is okay to use the Trailer pointer as the assert will have validated it.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18641371 in //UE5/Release-5.0/... via CL 18641373 via CL 18641389
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v900-18638592)
[CL 18641396 by paul chipchase in ue5-main branch]
Added similar support to the package store manifest
Zen Store support will be added separately
Added some extra data to linker object resources to be used by the iostore package optimizer to properly support optional package chunks.
Multiple cooking outputs is used to support "optional objects" which are editor data package exports that would be put in separate iostore containers than the normal cooked game content.
#jira UE-129801
#rb Matt.Peters, CarlMagnus.Nordin
#preflight 61e5d3bb873f2ea48f48166c
#ROBOMERGE-AUTHOR: francis.hurteau
#ROBOMERGE-SOURCE: CL 18639742 in //UE5/Release-5.0/... via CL 18639751 via CL 18639758
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v900-18638592)
[CL 18639762 by francis hurteau in ue5-main branch]
#rb PJ.Kack
#jira UE-139348
#rnx
#preflight 61e6742a3778a195dea02199
### PackageTrailer
- We no longer store the access mode for all payloads in the trailer. Instead each payload entry in the lookup table will store the access mode for that specific payload. This allow us to store many payloads of different access types in the same trailer.
- Note that this format change is currently versioned (going from 0 to 1) and is compatible with older version.
-- To allow this, the << operator has been removed from FLookupTableEntry and replaced with a ::Serialize method that takes the version of the package trailer header as an input so that it can run it's own versioning checks.
- When loading trailers with version 0 we apply the access mode to all non virtualized payload entries in the lookup table to reproduce the single access mode for the entire trailer.
- A FPackageTrailerBuilder can now be created directly from an existing FPackageTrailer that will reference all of the trailers local payloads via the new static method ::CreateReferenceToTrailer
- Removed FPackageTrailer::TrySave as it is no longer used when creating a trailer for the editor domain.
- Renamed FPackageTrailer::LoadPayload to ::LoadLocalPayload to make it clear the type of payload that can be loaded.
-- Someday the package trailer should be able to load payloads of any type.
### VirtualizedBulkData Serialization Changes
- We now record if the bulkdata's payload is stored in a package trailer or not via the 'StoredInPackageTrailer' flag, which makes it easier to identify critical errors in serialization (the trailer is missing for example)
-- Storing this info as a flag is now possible as the design for the trailer has changed and a fully virtualized package will still contain a trailer.
- We now only write out OffsetInFile if the payload is not stored in a FPackageTrailer, if the bulkdata object does use a package trailer via the new flag then we take the offset from the trailer instead.
- We also no longer write out a path when saving a payload as referenced to the editor domain, it is assumed that the trailer will know where the workspace domain data is (namely the owning package path)
-- Due to the 'StoredInPackageTrailer' only being added now, this means that packages already saved with a package trailer will work with this code logic as those packages will have serialized the offsets to disk, but won't have the new flag and so will serialize the offsets when loading until re-saved. (worth noting that nobody should encounter this as the package trailer system is disabled by default)
- The loading code is a lot easier to read now and a lot of the branches have been removed although we still have a few extra checks for older formats that might have been saved during development.
- When loading the only real consideration we have is if the payload is in the trailer or not.
- Future Work:
-- If we are not writing to the trailer then we use the legacy serialization system which appends the bulkdata to the end of the package. This is used in two cases, 1) When the package trailer system is disabled 2) If the payload is updated in postload and we are writing out the package to the editor domain as the package trailer does not support writing out local payloads outside of the workspace domain at the moment. The trailer should handle this itself and not rely on the legacy serialization system so that we can remove it.
-- Note that if a payload is stored by reference, we are not adding it explicitly to the payload trailer and we are relying on the trailer builder already containing the reference information when it is created in SavePackage. We do however run an assert to ensure that the reference does exist in the trailer but it might be worth revisiting this.
### VirtualizedBulkData Misc
- The member 'PackageSegment' is now initialized in the constructor to EPackageSegment::Header.
- Moved ::GetPackagePathFromOwner from being a method to a utility function as it accessed no members.
- Validating the payload for saving during serialization has been moved from ::Serialize to it's own method ::TryPayloadValidationForSaving.
### PackageSubmissionChecks
- Added an error message if we detect referenced payloads in a trailer we are trying to virtualize as this condition should never occur. If we encounter it, we give a user facing error message and fail the submit process. It might be worth demoting this to an assert instead as it is not an error the user is expected to see.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18637906 in //UE5/Release-5.0/... via CL 18637926 via CL 18637929
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v899-18417669)
[CL 18637932 by paul chipchase in ue5-main branch]
The default configuration for ue5 is to run with both package data and bulk data in iostore.
The only remaining alternative is to disable iostore completely and run with both package data and bulk data in pak.
#jira none
#rb carlmagnus.nordin, paul.chipchase
#rnx
#preflight 61e53b77873f2ea48f1dcd94
#ROBOMERGE-AUTHOR: pj.kack
#ROBOMERGE-SOURCE: CL 18631455 in //UE5/Release-5.0/... via CL 18631662 via CL 18631683
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v899-18417669)
[CL 18636672 by pj kack in ue5-main branch]
#rb PJ.Kack
#jira UE-138715
#rnx
#preflight 61de7c826a076ddb53d85e76
- We now only validate the payloads contents when ::GetDataInternal is called. This prevents duplicate log messages being raised.
-- Duplicate calls to IsValid have been removed.
-- This also means we don't need ::GetCorruptedPayloadErrorMsgForLoad anymore as it is only used in a single place.
- Renamed ::IsValid to ::IsDataValid to make it clearer that we are checking the contents of the payload.
-- The utility no longer checks if the payload fails to load (size should be >0 but the payload is null) but only checks if the payload is corrupted or does not match the expected content hash.
--- This is because the payload failing to load is an error, but it is something that we expect to see happen on occasion, perhaps when the network is flaky etc. However the wrong or corrupted data being loaded is not expected and so needs to be reported as it's own specific error.
--- Splitting these types of failures up, should allow us to consider enabling VBD_CORRUPTED_PAYLOAD_IS_FATAL again. Although now we prevent the saving of packages with corrupted payloads we probably don't need to treat a corrupted payload as a reason to terminate the editor.
- Removed warning from FVirtualizedUntypedBulkData::PullData when the payload failed to pull as IVirtualizationSystem will have already logged an error and the warning was incorrect as it was displaying the payload identifier as the bulkdata guid.
- VBD_CORRUPTED_DATA_SEVERITY is now set as an Error if VBD_CORRUPTED_PAYLOAD_IS_FATAL is disabled.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18581869 in //UE5/Release-5.0/... via CL 18581873 via CL 18581874
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Test -> Main) (v899-18417669)
[CL 18581878 by paul chipchase in ue5-main branch]
Convert BulkData to use a separate file handle (not AttachedAr) when MakeSureBulkDataIsLoaded is called from non-gamethread. This prevents unguarded multithreaded access to AttachedAr.
#preflight 61dc386a5b6f9bf2e6cf6a96
#rb Paul.Chipchase
#rnx
#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 18561037 in //UE5/Release-5.0/... via CL 18561055
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)
[CL 18561076 by matt peters in ue5-release-engine-test branch]
#rb Matt.Peters
#rnx
#jira UE-136891
#preflight 61dbf8eeaf44b43628343f00
- Now when saving a package to the editor domain we will load the existing package trailer (if one exists) from the workspace domain and then create a new trailer that references those payloads rather than store them locally.
- VirtualizedBulkData serialization is still overly complicated and this code adds more branches there. If we stopped the VBD objects from serializing their offsets at all and rely entirely on the package trailer to know where the payloads are stored then we could simplify this, but that requires the package trailer system to be turned on so will be done as future work.
### VirtualizedBulkData
- Split bAllowVirtualizationOnSave into a second variable. The original now controls if we push the payload on save (off by default and not something we are likely to keep) and the new variable (bForceRehydrationOnSave) controls if we must pull payloads when the package is saved (on by default) which will be moved to a config value once properly supported.
-- The split was done to make the logic in ::Serialize clearer
- We now correctly Register/Unregister the bulkdata object when ::DetachFromDisk is called. This prevents the bulkdata registry from trying to access the workspace domain file after it has been invalidated.
-- In addition we now attach to the FLinkerLoader's cachable archive (if there is one) when using the ::IsReferencingByPackagePath branch of serialization.
- When saving the bulkdata as part of an editor domain save we can keep the virtualization flag as we don't have to worry about the workspace domain file changing without the editor domain version being rebuilt anyway.
### SavePackage
- If the package is part of a procedural save and not a cook we will try to load the trailer from the workspace domain and then convert it to a referenced trailer (which contains the correct info to load the payload from the workspace domain package file but does not contain the payloads itself) which is then appended to the package.
### PackageTrailer
- Removed the fatal error if a project has both the editor domain and the package trailer system enabled.
- Added a method ::CreateReference that creates a trailer that references the payloads in an existing trailer.
- Added a method ::TrySave, allowing direct writing of a trailer to an archive.
-- Now that we have TrySave and TryLoad it probably makes sense to add the << operator as the public interface and move the methods to be private.
- FFooter and FHeader have their own serialization overloads now rather than relying on the higher level structures to do the work.
- When building a new trailer we now create a new FPackageTrailer structure and then write that out in one go, rather than writing each value as it is calculated to make the code easier to read.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18557778 in //UE5/Release-5.0/... via CL 18557792
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)
[CL 18557812 by paul chipchase in ue5-release-engine-test branch]
#rb trivial
#jira UE-138459
#preflight 61d888e64c252480ca397e49
- If the asyc read being waited on takes a very long time (1000ms+) then we log a warning that there was a stall. However the warning prints the size of the bulkdata payload to the warning to give the user an idea of if the wait was due to the length of the data or some other reason.
- Calling ::GetBulkDataSize cannot be done in the destructor of the bulkdata object because the method is virtual, which was causing the crash.
- We can however call ::GetBulkDataSizeOnDisk as it is not virtual and gives a more accurate picture of the length of the data being loaded, as it can be smaller than the result of ::GetBulkDataSize if compression is used.
- Switched use of %lld to use INT64_FMT instead.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18548835 in //UE5/Release-5.0/... via CL 18548858
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)
[CL 18548885 by paul chipchase in ue5-release-engine-test branch]
A payload was conceptually a value with an ID. That has been formalized by removing the ID from the payload and having separate FValue and FValueId types. This separation cleans up the API in a few areas, and provides a more natural path to providing a basic key/value API.
#rb Zousar.Shaker
#rnx
#preflight 61d704c04c252480ca284d61
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18531844 in //UE5/Release-5.0/... via CL 18531856
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)
[CL 18531864 by devin doucette in ue5-release-engine-test branch]
#rb PJ.Kack
#rnx
#preflight 61b79447f807c623046e91a5
### FPackageTrailer
- FPackageTrailerBuilder::Create now renamed to ::CreateFromTrailer to make it a bit clearer.
- Added 'AccessMode' to the file format documentation.
- FPackageTrailerBuilder no longer tracks infomation about the trailer once it is written to disk. Instead when writing to disk we will create a valid FPackageTrailer as part of the serialization. This will be passed to callbacks and can be queried by virtualized bulkdata.
-- Since the builder no longer tracks data after creating the trailer, we can remove checks from methods like ::AddPayload.
-- Removed FPackageTrailerBuilder::FindPayloadOffset and added a version to FPackageTrailer instead.
- Changed a number of methods to take FPayloadId by reference rather than by value.
### VirtualizedBulkData
- Replaced areas of code responsible for updating previously written out values with the utility function ::UpdateArchiveData to help with code readability.
### Misc
- FLinkerSave no longer creates a FPackageTrailerBuilder by default, instead one should be added to the FLinkerSave if we want to build a trailer for the package being saved.
- FLinkerLoad now considers 0 to be an invalid value for PayloadTocOffset in addition to INDEX_NONE, so we now check if the value is > than 0. This is because when FPackageFileSummary objects are created, all members are set to zero and we know that a FPackageTrailer cannot be 0 and the rest of the package still exist so we can safely consider it invalid.
- SavePackage/SavePackage2 now create the FPackageTrailerBuilder when the linker is created if the package trailer feature is enabled.
-- We now always call BuildAndWriteTrailer and then only skip the creation of the trailer if the builder is missing or conditions are not met.
-- If we do not write out the trailer, we make sure to set PayloadTocOffset in the FPackageFileSummary to INDEX_NONE as it is clearer than leaving it at the default 0.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18453324 in //UE5/Release-5.0/... via CL 18453328
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v898-18417669)
[CL 18453333 by paul chipchase in ue5-release-engine-test branch]
The process of cooking would ensure that every BP derived from a C++ base using sparse class data would have its own sparse class data instance, as the serialization function would create an instance on save/load. This could potentially result in a lot of redundant instances that don't override any data from their archetype.
These changes allow the API for accessing sparse class data to optionally fallback to using the archetype data when possible (controlled via the EGetSparseClassDataMethod enum), though any existing code already using GetOrCreateSparseClassData (or the UHT wrapper) will continue to always create a new instance.
BPGC serialization has been updated to only save the sparse class data when it has overridden data from its archetype, and the loading code will now avoid creating the instance when no override data has been saved. This allows new code to take advantage of EGetSparseClassDataMethod by choosing to reference the archetype data instead, and the engine has also been updated to avoid calling GetOrCreateSparseClassData where possible.
Anything that requires a mutable sparse class data instance should still use GetOrCreateSparseClassData (or the UHT wrapper that doesn't take EGetSparseClassDataMethod), as otherwise you may edit data belonging to your archetype (which is why GetSparseClassData returns const data).
#preflight 61b28ecdc01c89f906c4d72f
#rb Fred.Kimberley, Thomas.Sarkanen
#ROBOMERGE-AUTHOR: jamie.dale
#ROBOMERGE-SOURCE: CL 18442650 via CL 18443219 via CL 18443223 via CL 18443224 via CL 18444418 via CL 18445355
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v897-18405271)
[CL 18445518 by jamie dale in ue5-release-engine-test branch]
#rb Jamie.Dale
#ROBOMERGE-AUTHOR: robert.manuszewski
#ROBOMERGE-SOURCE: CL 18441838 via CL 18441841 via CL 18441842 via CL 18441918 via CL 18444783
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v897-18405271)
[CL 18444957 by robert manuszewski in ue5-release-engine-test branch]
Also fix two other minor issues: Make Move constructor available for FIntVirtualizedBulkData, and make Register public so UObjects can notify the registry that they own a BulkData.
#rb Paul.Chipchase
#rnx
#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 18431598 in //UE5/Release-5.0/... via CL 18435352
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v897-18405271)
[CL 18435594 by matt peters in ue5-release-engine-test branch]
#rb PJ.Kack
#jira UE-136126
#rnx
#preflight
### VirtualizationSystem
- Added a new overload for Push to VirtualizationSystem that takes an array of FPushRequest, which is a new structure representing a single payload request.
- Filtering by package name is currently disabled, this is because the API has been forced into changing and passing the package name in via a FString rather than FPackagePath which means we would need to be more careful. This will be done in a future submit.
- The backend interface has been extended to also have a batch version of PushData, by default this will attempt to submit each request one at a time so payloads don't have to try and implement a batched version if there is no need.
- The context being passed with a payload when being pushed has been changed from FPackagePath to FString due to include order issues, as the FPackagePath lives in CoreUObject and the API for virtualization lives in Core. Additionally in the future the payloads might not be owned by a package (there is nothing specifically enforcing this) so the context being a string makes more sense.
- NOTE: Due to the context change we currently no longer support the filtering feature, which allows for payloads belonging to packages under specific directories to be excluded from virtualization. This is something that will be solved in a future submit.
### SourceControlBackend
- Now that we can submit multiple payloads in the same submit, the CL description has been changed slightly. We will now print a list of payload identifiers -> the package trying to submit that payload. This will only tell the users which package originally caused the payload to submit. If a user submits a new package at a later date that contains the same payload we will not be updating the description.
### PackageSubmissionChecks
- Converted the submission process to use the new batch push operation in VirtualizationSystem.
-- This means that we do a single push and then have to update the package trailers to convert the now pushed payloads from local to virtualized.
- Added new define UE_PRECHECK_PAYLOAD_STATUS that makes it easy to toggle off the checks to see which payloads need to be submitted to the persistent backend. This is useful to test if it actually helps speed up the overall operations or if it is faster to just perform the batch push operations on all payloads and check the return values.
-- The hope is that over time the submission processes will become fast enough that we can remove the precheck.
- Fixed up logging to not always assume more than one package or payload.
### General Notes
- Errors and logging is now a bit more vague as we often not just report that X payloads failed etc rather than specific payload identifiers. This probably doesn't affect the user too much since those identifiers as fairly meaningless to them anyway.
- The source control submission could be further optimized by first checking the status of the files in thge depot and only then creating/switching workspace etc.
- As currently written, we need to load all of the payloads into memory, then the backends will do what they need (in the case of source control this results in the payloads being written to disk then submitted) which could create quite a large memory spike when submitting a large number of packages.
-- One solution would be to change the batch push API to take a "payload provider" interface and have the payloads requested as needed rather than passing in the FCompressedBuffer directly. This would let us immediately write the payload to disk for submission then discard it from memory, preventing larger spikes. Although it could cause overhead if there are multiple backends being submitted to. Internally we are unlikely to have more than one backend per storage solution so maybe we should just make it a config option?
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18403735 in //UE5/Release-5.0/... via CL 18403737
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v896-18170469)
[CL 18403738 by paul chipchase in ue5-release-engine-test branch]
Requiring the use of a separate reader type makes it more likely that readers will be reused, and makes it easier to audit reader usage going forward. Reusing readers is desirable to reduce the number of large temporary allocations made during partial decompression of a buffer.
- Added FCompressedBuffer::Save(FArchive&) and renamed FromCompressed(FArchive&) to Load(FArchive&).
- Added FCompressedBufferReaderSourceScope to set a buffer source within a scope.
- Added proper bounds checks to FNoneDecoder.
- Store the header checksum on the decoder context to allow raw blocks to be reused across sources.
- Decode the header on the fly to avoid a temporary header allocation when the header is in contiguous memory.
#rb Zousar.Shaker
#rnx
#preflight 61a98d53800738dbfbc84c73
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18382211 in //UE5/Release-5.0/... via CL 18382310
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v896-18170469)
[CL 18382377 by devin doucette in ue5-release-engine-test branch]
