#rb PJ.Kack
#rnx
#preflight 61af1090c6650f98a97782a4
- The source control backend will now attempt to create the temporary workspace as a partitioned workspace by default.
- This can be overriden via the config file when defining the virtualization graph.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18392795 in //UE5/Release-5.0/... via CL 18392806
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v896-18170469)
[CL 18392807 by paul chipchase in ue5-release-engine-test branch]
Requiring the use of a separate reader type makes it more likely that readers will be reused, and makes it easier to audit reader usage going forward. Reusing readers is desirable to reduce the number of large temporary allocations made during partial decompression of a buffer.
- Added FCompressedBuffer::Save(FArchive&) and renamed FromCompressed(FArchive&) to Load(FArchive&).
- Added FCompressedBufferReaderSourceScope to set a buffer source within a scope.
- Added proper bounds checks to FNoneDecoder.
- Store the header checksum on the decoder context to allow raw blocks to be reused across sources.
- Decode the header on the fly to avoid a temporary header allocation when the header is in contiguous memory.
#rb Zousar.Shaker
#rnx
#preflight 61a98d53800738dbfbc84c73
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18382211 in //UE5/Release-5.0/... via CL 18382310
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v896-18170469)
[CL 18382377 by devin doucette in ue5-release-engine-test branch]
#rb Per.Larsson
#rnx
#preflight 61a8940b9c77d610079ce7da
- Made all backends final, they should not be derived from.
- Added header files for the remaining backends that were cpp only. The headers are in the modules private directory so will not be used elsewhere and it makes it easier to see the interface.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18350848 in //UE5/Release-5.0/... via CL 18350854
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)
[CL 18350856 by paul chipchase in ue5-release-engine-test branch]
#rb PJ.Kack
#rnx
#preflight 61a795773c29b3cf13cd8250
### PackageSubmissionChecks
- Under the old model submitting a large number of packages could be very slow as each package would check each payload that it owns and is currently stored locally one at a time. For the source control backend this created quite a large overhead even when all of the payloads were already virtualized.
- Now we do a pass over the submitted files to find all valid packages that have package trailers with locally stored payloads, gather the payloads into a single list and then query that in one large batch.
- Once we find which payloads are not in permanent storage (note that in the case where a project is using multiple permanent storage solutions, if a payload is missing in one backend it counts as not being in permanent storage) we then attempt to virtualized them.
- Only after all of this is done will we create the truncated copy of each package and then append the updated trailer to each one. In theory doing it in this order this might slightly increase the change of submit failures that occur after virtualization that result in a package never being submitted and orphaned payloads being added to permanent storage, but this will always be a risk.
- Added an assert to fire if we detect a trailer with some virtualized and some local payloads. This should be a supported feature but needs proper testing first before we can allow it. With out current project settings no project should actually encounter this scenario.
- To make the code easier to follow we now early out of the entire check when errors are encountered.
- Added logging at various stages in the process to help show the user that something is happening and make problems easier to identify in the future.
- Notes
-- There is a lot of handling of invalid FPayloads. This is because it is currently possible to add empty payloads to the trailer which is inefficient and wastes space. The trailer will be modified to reject empty payloads in a future update at which point a lot of this handling can be removed.
-- This could've also been solved by not fully rehydrating a package on save by the end user, which will be added as a project setting in a future piece of work, but this approach will solve the edge case when the user does have a large amount of hydrated packages which contain payloads that are already virtualized so it was better to fix that now while we have good test cases for it.
-- We still have scaling problems with large number of package being submitted that do have payloads that need to be virtualized, this will be fixed by extending IVirtualizationSystem::Push to also accept batches of payloads in future work.
-- OnPrePackageSubmission could be broken up into smaller chunks to make the code easier to follow. This will be done after the batch payload submission work is done.
### VirtualizationSystem
- EStorageType has been promoted to enum class.
- Added a new enum FPayloadStatus to be used when querying if a payload exists in a backend storage system or not.
- Add a new method ::DoPayloadsExist which allows the caller to query if one or more payloads exists in the given backend storage system.
### VirtualizationManager
- Implemented ::DoPayloadsExist. First we get the results from each backend in the storage system (which return as true or false from each backend) then total how many backends found the payload in order to set the correct status.
### IVirtualizationBackend
- ::DoesPayloadExist which queries the existence of a single payload has been added to the interface. Most backends already implemented this for private use and if so have had their implementation renamed to match this.
- Also added ::DoPayloadsExist which takes a batch of FpayloadIdsto query. Some backends can deal with a batch of payload ids much more efficiently than one at a time, although the default implementation does call ::DoesPayloadExist for each requested payload.
-- The default implementation prevents every backend from needing to implement the same for loop but does allow backends that can gain from batching to override it.
### VirtualizationSourceControlBackend
- This backend does override ::DoPayloadsExist and implements it's own version as it tends to perform very poorly when not operating on larger batches.
- In this case ::DoesPayloadExist calls back to ::DoPayloadsExist to check each payload rather than implement as specific version.
### PackageTrailer
- The trailer can now be queries to request how many payloads of a given type it contains
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18339847 in //UE5/Release-5.0/... via CL 18339852
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)
[CL 18339859 by paul chipchase in ue5-release-engine-test branch]
This represents UE4/Main @17911760, Release-5.0 @17915875 and Dev-PerfTest @17914035
[CL 17918595 by aurel cordonnier in ue5-release-engine-test branch]
#rb PJ.Kack
#rnx
#preflight 60fff7d57f21c90001f9399d
- When we check the source control provider name, we will now attempt to set it to "Perforce" if it is current the default "None" provider.
- If the user has specified a different source control provider than "Perforce" we will give a fatal error that the content virtualization graph containing the source control backend will not work with that setup.
- This part of the workflow might need to change in the future or at least the Mirage documentation must make very clear that when using source control as the persistent storage backend that a connection to perforce will be required, at least until we support our source control solutions but even then the connect to that provider will be required.
#ROBOMERGE-SOURCE: CL 16968160 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v838-16927207)
[CL 16968164 by paul chipchase in ue5-release-engine-test branch]
#rb Per.Larsson
#rnx
#preflight 60f9212d1f926d0001d41223
* VirtualizationJupiterBackend
- When pulling a payload we should assume that a 400 error response when trying to GET the payload header means that the payload is not in Jupiter.
-- Not being able to find the payload should not log an error, instead we can make a note of it in the verbose log (similar to the file system backend)
* VirtualizationFileBackend
- Moved the formatting of system errors to it's own function.
- Log the system error when failing to write a payload during a push as well as a pull.
- We now check that the FileArchive wrote correctly to disk and delete the output file and fail the push if it did not.
-- A future piece of work will change the logic to write to a tmp file at the root of the file store and them move the file to the final location to cut down on the potential of leaving corrupted files around (similar to the process when we save packages)
* Perforce
- The FDownloadFile command now takes an optional parameter EVerbosity that can allow the caller to choose the level of logging output that the command will generate.
- The source control backend for Mirage now opts to supress the logging of the full perforce command when we are pulling payloads as we can generate many hundreds or thousands of requests and the info is not useful to users.
-- We continue to log the command when validating the depot as this is the most likely command to fail so having the info in the log may prove useful.
#ROBOMERGE-SOURCE: CL 16921815 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v836-16769935)
[CL 16921818 by paul chipchase in ue5-release-engine-test branch]
