#rb trivial
#jira UE-136515
#rnx
#preflight 61b21e3b764790bee6bdb8b3
- This backend is still most likely going to be deleted before 5.0 ships but just in case it is not I am doing the renaming now to keep in step with the names we are using for Jupiter when communicating with externals.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18419810 in //UE5/Release-5.0/... via CL 18422457
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v897-18405271)
[CL 18422718 by paul chipchase in ue5-release-engine-test branch]
#rb PJ.Kack
#jira UE-136126
#rnx
#preflight
### VirtualizationSystem
- Added a new overload for Push to VirtualizationSystem that takes an array of FPushRequest, which is a new structure representing a single payload request.
- Filtering by package name is currently disabled, this is because the API has been forced into changing and passing the package name in via a FString rather than FPackagePath which means we would need to be more careful. This will be done in a future submit.
- The backend interface has been extended to also have a batch version of PushData, by default this will attempt to submit each request one at a time so payloads don't have to try and implement a batched version if there is no need.
- The context being passed with a payload when being pushed has been changed from FPackagePath to FString due to include order issues, as the FPackagePath lives in CoreUObject and the API for virtualization lives in Core. Additionally in the future the payloads might not be owned by a package (there is nothing specifically enforcing this) so the context being a string makes more sense.
- NOTE: Due to the context change we currently no longer support the filtering feature, which allows for payloads belonging to packages under specific directories to be excluded from virtualization. This is something that will be solved in a future submit.
### SourceControlBackend
- Now that we can submit multiple payloads in the same submit, the CL description has been changed slightly. We will now print a list of payload identifiers -> the package trying to submit that payload. This will only tell the users which package originally caused the payload to submit. If a user submits a new package at a later date that contains the same payload we will not be updating the description.
### PackageSubmissionChecks
- Converted the submission process to use the new batch push operation in VirtualizationSystem.
-- This means that we do a single push and then have to update the package trailers to convert the now pushed payloads from local to virtualized.
- Added new define UE_PRECHECK_PAYLOAD_STATUS that makes it easy to toggle off the checks to see which payloads need to be submitted to the persistent backend. This is useful to test if it actually helps speed up the overall operations or if it is faster to just perform the batch push operations on all payloads and check the return values.
-- The hope is that over time the submission processes will become fast enough that we can remove the precheck.
- Fixed up logging to not always assume more than one package or payload.
### General Notes
- Errors and logging is now a bit more vague as we often not just report that X payloads failed etc rather than specific payload identifiers. This probably doesn't affect the user too much since those identifiers as fairly meaningless to them anyway.
- The source control submission could be further optimized by first checking the status of the files in thge depot and only then creating/switching workspace etc.
- As currently written, we need to load all of the payloads into memory, then the backends will do what they need (in the case of source control this results in the payloads being written to disk then submitted) which could create quite a large memory spike when submitting a large number of packages.
-- One solution would be to change the batch push API to take a "payload provider" interface and have the payloads requested as needed rather than passing in the FCompressedBuffer directly. This would let us immediately write the payload to disk for submission then discard it from memory, preventing larger spikes. Although it could cause overhead if there are multiple backends being submitted to. Internally we are unlikely to have more than one backend per storage solution so maybe we should just make it a config option?
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18403735 in //UE5/Release-5.0/... via CL 18403737
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v896-18170469)
[CL 18403738 by paul chipchase in ue5-release-engine-test branch]
#rb PJ.Kack
#rnx
#preflight 61af1090c6650f98a97782a4
- The source control backend will now attempt to create the temporary workspace as a partitioned workspace by default.
- This can be overriden via the config file when defining the virtualization graph.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18392795 in //UE5/Release-5.0/... via CL 18392806
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v896-18170469)
[CL 18392807 by paul chipchase in ue5-release-engine-test branch]
Requiring the use of a separate reader type makes it more likely that readers will be reused, and makes it easier to audit reader usage going forward. Reusing readers is desirable to reduce the number of large temporary allocations made during partial decompression of a buffer.
- Added FCompressedBuffer::Save(FArchive&) and renamed FromCompressed(FArchive&) to Load(FArchive&).
- Added FCompressedBufferReaderSourceScope to set a buffer source within a scope.
- Added proper bounds checks to FNoneDecoder.
- Store the header checksum on the decoder context to allow raw blocks to be reused across sources.
- Decode the header on the fly to avoid a temporary header allocation when the header is in contiguous memory.
#rb Zousar.Shaker
#rnx
#preflight 61a98d53800738dbfbc84c73
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18382211 in //UE5/Release-5.0/... via CL 18382310
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v896-18170469)
[CL 18382377 by devin doucette in ue5-release-engine-test branch]
[FYI] paul.chipchase
Original CL Desc
-----------------------------------------------------------------
CIS - Fix mac non unity compile
#rb trivial
#jira UE-136544
#rnx
#preflight 61a8f87ae01f3610f5316fff
- We need to include "Virtualization/PayloadId.h" due to the use of PayloadId in DoPayloadsExist.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18380572 in //UE5/Release-5.0/... via CL 18380574
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)
[CL 18380579 by paul chipchase in ue5-release-engine-test branch]
#rb trivial
#jira UE-136544
#rnx
#preflight 61a8f87ae01f3610f5316fff
- We need to include "Virtualization/PayloadId.h" due to the use of PayloadId in DoPayloadsExist.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18380510 in //UE5/Release-5.0/... via CL 18380511
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)
[CL 18380515 by paul chipchase in ue5-release-engine-test branch]
#rb PJ.Kack
#jira UE-136374
#rnx
#preflight 61a8ad08ad6629a51eb1f8d8
- Having invalid payloads in the trailer's lookup table and writing empty payload data to disk was a waste of time and meant we had to add more error handling code elsewhere.
- Now when a VBD attempts to add one to the trailer it will be rejected, although the provided callback will still be invoked so that the VBD object will still mark itself as being serialized to disk and update it's flags etc.
- Asserts have been added when writing the FPackageTrailer to disk or creating a new FPackageTrailerBuilder from an existing FPackageTrailer if invalid FPayloadIds are somehow found. This is not an expected condition and the asserts are there to guard against future code changes that might break the contract.
- Now when virtualizing the package trailer during submission checks we can assume that all FPayloadIds reference actual data.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18351024 in //UE5/Release-5.0/... via CL 18351037
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)
[CL 18351039 by paul chipchase in ue5-release-engine-test branch]
#rb Per.Larsson
#rnx
#preflight 61a8940b9c77d610079ce7da
- Made all backends final, they should not be derived from.
- Added header files for the remaining backends that were cpp only. The headers are in the modules private directory so will not be used elsewhere and it makes it easier to see the interface.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18350848 in //UE5/Release-5.0/... via CL 18350854
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)
[CL 18350856 by paul chipchase in ue5-release-engine-test branch]
#rb trivial
#rnx
#preflight 61a88963800738dbfbb4983b
- Removed the code documentation for the methods overriden from IVirtualizationSystem as the interface is already documented.
- Set FVirtualizationManager as final since it will not be derived from.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18350562 in //UE5/Release-5.0/... via CL 18350564
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)
[CL 18350565 by paul chipchase in ue5-release-engine-test branch]
#rb PJ.Kack
#rnx
#preflight 61a795773c29b3cf13cd8250
### PackageSubmissionChecks
- Under the old model submitting a large number of packages could be very slow as each package would check each payload that it owns and is currently stored locally one at a time. For the source control backend this created quite a large overhead even when all of the payloads were already virtualized.
- Now we do a pass over the submitted files to find all valid packages that have package trailers with locally stored payloads, gather the payloads into a single list and then query that in one large batch.
- Once we find which payloads are not in permanent storage (note that in the case where a project is using multiple permanent storage solutions, if a payload is missing in one backend it counts as not being in permanent storage) we then attempt to virtualized them.
- Only after all of this is done will we create the truncated copy of each package and then append the updated trailer to each one. In theory doing it in this order this might slightly increase the change of submit failures that occur after virtualization that result in a package never being submitted and orphaned payloads being added to permanent storage, but this will always be a risk.
- Added an assert to fire if we detect a trailer with some virtualized and some local payloads. This should be a supported feature but needs proper testing first before we can allow it. With out current project settings no project should actually encounter this scenario.
- To make the code easier to follow we now early out of the entire check when errors are encountered.
- Added logging at various stages in the process to help show the user that something is happening and make problems easier to identify in the future.
- Notes
-- There is a lot of handling of invalid FPayloads. This is because it is currently possible to add empty payloads to the trailer which is inefficient and wastes space. The trailer will be modified to reject empty payloads in a future update at which point a lot of this handling can be removed.
-- This could've also been solved by not fully rehydrating a package on save by the end user, which will be added as a project setting in a future piece of work, but this approach will solve the edge case when the user does have a large amount of hydrated packages which contain payloads that are already virtualized so it was better to fix that now while we have good test cases for it.
-- We still have scaling problems with large number of package being submitted that do have payloads that need to be virtualized, this will be fixed by extending IVirtualizationSystem::Push to also accept batches of payloads in future work.
-- OnPrePackageSubmission could be broken up into smaller chunks to make the code easier to follow. This will be done after the batch payload submission work is done.
### VirtualizationSystem
- EStorageType has been promoted to enum class.
- Added a new enum FPayloadStatus to be used when querying if a payload exists in a backend storage system or not.
- Add a new method ::DoPayloadsExist which allows the caller to query if one or more payloads exists in the given backend storage system.
### VirtualizationManager
- Implemented ::DoPayloadsExist. First we get the results from each backend in the storage system (which return as true or false from each backend) then total how many backends found the payload in order to set the correct status.
### IVirtualizationBackend
- ::DoesPayloadExist which queries the existence of a single payload has been added to the interface. Most backends already implemented this for private use and if so have had their implementation renamed to match this.
- Also added ::DoPayloadsExist which takes a batch of FpayloadIdsto query. Some backends can deal with a batch of payload ids much more efficiently than one at a time, although the default implementation does call ::DoesPayloadExist for each requested payload.
-- The default implementation prevents every backend from needing to implement the same for loop but does allow backends that can gain from batching to override it.
### VirtualizationSourceControlBackend
- This backend does override ::DoPayloadsExist and implements it's own version as it tends to perform very poorly when not operating on larger batches.
- In this case ::DoesPayloadExist calls back to ::DoPayloadsExist to check each payload rather than implement as specific version.
### PackageTrailer
- The trailer can now be queries to request how many payloads of a given type it contains
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18339847 in //UE5/Release-5.0/... via CL 18339852
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)
[CL 18339859 by paul chipchase in ue5-release-engine-test branch]
#rb PJ.Kack
#rnx
#preflight 61a5cb34801b3619784062d0
- If packages were edited but not submitted and the editor shutdown then restarted there is a possibility that the edited packages will be loaded by the editor, meaning that a FLinkerLoader will have a file lock on the package file. So now we call ::ResetLoadersForSave on each package file that needs to be replaced to remove the file lock.
- When verbose logging is enabled we will now print how long the prepackagesubmission check took as a development aid.
- When packages have passed the prepackagesubmission process we now just append #virtualized to the CL description to mark the assets as having been through the process. The previous marker (a guid) was some what meaningless at the moment.
- Removed the legacy submit code, it is no longer needed for reference
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18322298 in //UE5/Release-5.0/... via CL 18322317
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)
[CL 18322321 by paul chipchase in ue5-release-engine-test branch]
#rb PJ.Kack
#rnx
#preflight 61a4b235405273b2c3daa7c7
### Virtualization
- The idea is to move from our current set up, where the virtualization happens when a package is saved and the payloads are saved to a local virtualization storage system with the final push to persistent storage occurring when the package is submitted to a system where we save payloads into the package file, but move them to persistent virtualization storage when the package file is submitted.
- The main advantage is that if someone submits a package file to source control without virtualizing it, we don't have to worry that others might not be able to load the package, the worst case scenario is that data sizes get bigger until the package is virtualized again.
- This is only the first pass, in the future we can do further optimizations, like not storing payloads locally if we know that they are already in the persistent storage system etc.
- In order to keep the virtualization process simple we want to be able to do it without needing to re-save the package, so to this end we will storage the payloads at the end of the package file as before, but instead of storing them inside of the package file format we will use a new container, the FPackageTrailer that is appended to the package file instead.
- In theory this will make future sidecar work easier as the trailer can just be moved to the sidecar file as as long as we know where to find the trailer things will just work (tm)
- For now VBD will continue to serialize it's offset/size to disk and use those values to read the payload directly when not virtualizing. Ideally we'd use the trailer but this will help reduce the risk.
### Current Issues
- The system does not work with the editor domain and has minor issues with text based assets
- The trailer system is disabled by default via the config system [Core]UsePackageTrailer=False and can then be enabled on select test projects until fully functional.
### PackageTrailer
- The trailer is split into two parts FPackageTrailerBuilder and FPackageTrailer
- FPackageTrailerBuilder is used when building the trailer during package save and FPackageTrailer is the structure we load and actually use.
- When saving the trailer we try to avoid using containers and such so that we can try to keep the specification for the file format very clear so that licensees can build their own tools for the virtualization process if they so desire.
- the header file contains a breakdown of the new format.
### VirtualizedBulkData
- FPayloadToc is now only used by the sidecar experimental feature. In a future update the sidecar will be changed to basically be the package trailer but stored in it's own file.
- FindPayloadsInPackageFile has been moved to PackageTrailer.h as it has a closer association with that code.
- Added a new method, ::LoadFromPackageTrailer, to load payloads from the package trailer without needing to know where the payload is on disk. Using this is opt in with the cvar "Serialization.LoadFromTrailer" and is only provided for debugging purposes. Although once this system has matured it will likely become the preferred way to access payloads.
- We no longer allow virtualization of bulkdata payloads on save, but I am not sure if we might want to allow this as an option in the future. So for now the code branch is disabled by the global constant bAllowVirtualizationOnSave.
- Added utility functions ::GetLinkerLoadFromOwner and ::GetTrailerFromOwner for easy access to the LinkerLoad/Trailer from the owning object.
- Added a utility ::UpdateArchiveData for seeking back to a known position in an archive, writing over it, then seeking back, which is fairly common in the package saving code paths. This might be worth moving to Archive.h for general use.
- ::LoadFromPackageFile has been refactored to prefer returning error values early over nested ifs.
- We no longer serialize the EFlags::IsVirtualized flag to disk. Instead it is applied when the package is loaded if we detect that a payload is virtualized and then removed before saved to disk. This allows the virtualization process to occur without resaving the package.
- We currently support the old serialize to disk path which is enabled when the FPackageTrailer system is disabled and the newer system when it is enabled.
- We do however continue to serialize the offset in the package file to disk when saved. Strictly speaking this is not needed as we can look this up from the package trailer when the package is loaded but continuing to serialize it out to disk helps keep the old and newer data format paths working together without adding too much special case code.
-- Once the FPackageTrailer feature is no longer optional we can consider changing this.
### SavePackage/SavePackage2
- Removed the functions that would create the FPayloadToc
- Added new functions to help create the trailer.
- Moved the end of package tag out of the if to it's own scope (still won't run if a package writer exists), the reasoning is to make it clearer where the package format ends and where we can start writing the payload trailer.
- We have checks to make sure that the trailer does not try to write to disk for both text based assets and if the editor domain is enabled. Support for these features will be added later, but since the package trailer system is disabled by default this shouldn't cause problems.
### LinkerLoad
- After we parse the FPackageFileSummary we now also parse the header of the package trailer. In the workspace domain this will incur additional seek costs (which may or not actually seek and invalidate the internal file cache depending on how large the package's exports are) as the trailer is found after the package
-- In the future, the package trailer header can be stored right after the FPackageFileSummary in the editor domain which will remove this overhead.
- By parsing the header of the trailer at this point, virtualized bulkdata objects in the package can use the look up info to determine if their payloads are virtualized or not, and if not then where they reside inside the package file.
### LinkerSave
- We no longer collect a list of all virtualized bulkdata in a package while saving as we no longer generate the FPayloadToc so BulkDataInPackage has been removed.
- We now gather data to be appended to the package trailer via TrailerBuilder.
### DumpPayloadToc
- This command now dumps info based on the package trailer if one can be found.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18307893 in //UE5/Release-5.0/... via CL 18307905
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)
[CL 18307913 by paul chipchase in ue5-release-engine-test branch]
This represents UE4/Main @18073326, Release-5.0 @18081140 and Dev-PerfTest @18045971
[CL 18081471 by aurel cordonnier in ue5-release-engine-test branch]
This represents UE4/Main @17911760, Release-5.0 @17915875 and Dev-PerfTest @17914035
[CL 17918595 by aurel cordonnier in ue5-release-engine-test branch]
This represents UE4/Main @17774255, Release-5.0 @17791557 and Dev-PerfTest @17789485
[CL 17794212 by aurel cordonnier in ue5-release-engine-test branch]
#rb CarlMagnus.Nordin
#jira none
#rnx
#preflight 6156fd08475173000182663d
- This command is intended for debug use as an easy way to dump out the payload info in a package file that can then be inspected.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 17691116 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v875-17642767)
#ROBOMERGE[STARSHIP]: UE5-Release-Engine-Staging Release-5.0
[CL 17691124 by paul chipchase in ue5-release-engine-test branch]
The new flags are meant to prevent a cache store from evicting the associated cache records for the duration of the session.
#rb Zousar.Shaker
#rnx
#preflight 614954b611925a00017c525f
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 17579887 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v871-17566257)
[CL 17579899 by devin doucette in ue5-release-engine-test branch]
#rb Per.Larsson
#rnx
#preflight 61408a979dc6c8000148d0cf
### Misc
- Updated the documentation for setting up a file system backend and added descriptions to the backend's members.
- Renamed the member 'Name' to 'DebugName' to bring it in line with the other backends (still a candidate to be moved to the base class)
### Reading payloads from disk
- Added the ability to retry opening file handles for read if they fail. This can commonly occur when multiple threads or processes are both pulling and pushing to the same root directory.
-- By default we will retry to open the file handle up to 10 times, waiting for 100ms between each attempt. The user can configure the number of retries and the length of the pause between each attempt via the config files.
-- Each failed attempt at opening the file handle will result in a warning being logged, so if the user is stalled for a long length of time while we rety over and over they are at least made aware of the cause.
### Writing payloads to disk
- When attempting to write a payload to disk, we will now write to a temp file in the project save directory using a randomly generated name. Once the payload has been fully saved to disk it will then be moved to the correct location. Although not perfect this will reduce the potential for writing corrupted data to a location where it might be read in the future.
-- We do attempt to clean up the temp file if the write fails but we do not print error messages if the delete fails as it is considered an optional bonus rather than an essential feature.
- Previously we would only log an error if we failed to open a file handle for writing the payload to disk. Now we also log an error if the handle is opened but the actual writes fail.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 17550126 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v870-17433530)
[CL 17550133 by paul chipchase in ue5-release-engine-test branch]
#rb Devin.Doucette
#rnx
#preflight 61307f101a52e20001978a10
- This backend replaces the plans for a backend that interfaces with Zen directly.
- The backend has no real context for each payload being pushed at the moment, for now we pass in a dummy string.
- Currently we make blocking calls to the cache. When making the initial push to local storage this is desired as it will take place during package save. However both pulling the payload and making the submission to persistent storage should be able to be made async in the future.
- The storage policy is currently exposed via the ini file for now. We might want to enforce policies for this at a later time.
#ROBOMERGE-SOURCE: CL 17416948 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v865-17346139)
[CL 17416956 by paul chipchase in ue5-release-engine-test branch]
