#rb Per.Larsson
#rnx
#preflight 61408a979dc6c8000148d0cf
### Misc
- Updated the documentation for setting up a file system backend and added descriptions to the backend's members.
- Renamed the member 'Name' to 'DebugName' to bring it in line with the other backends (still a candidate to be moved to the base class)
### Reading payloads from disk
- Added the ability to retry opening file handles for read if they fail. This can commonly occur when multiple threads or processes are both pulling and pushing to the same root directory.
-- By default we will retry to open the file handle up to 10 times, waiting for 100ms between each attempt. The user can configure the number of retries and the length of the pause between each attempt via the config files.
-- Each failed attempt at opening the file handle will result in a warning being logged, so if the user is stalled for a long length of time while we rety over and over they are at least made aware of the cause.
### Writing payloads to disk
- When attempting to write a payload to disk, we will now write to a temp file in the project save directory using a randomly generated name. Once the payload has been fully saved to disk it will then be moved to the correct location. Although not perfect this will reduce the potential for writing corrupted data to a location where it might be read in the future.
-- We do attempt to clean up the temp file if the write fails but we do not print error messages if the delete fails as it is considered an optional bonus rather than an essential feature.
- Previously we would only log an error if we failed to open a file handle for writing the payload to disk. Now we also log an error if the handle is opened but the actual writes fail.
[CL 17550126 by paul chipchase in ue5-main branch]
#rb Devin.Doucette
#rnx
#preflight 61307f101a52e20001978a10
- This backend replaces the plans for a backend that interfaces with Zen directly.
- The backend has no real context for each payload being pushed at the moment, for now we pass in a dummy string.
- Currently we make blocking calls to the cache. When making the initial push to local storage this is desired as it will take place during package save. However both pulling the payload and making the submission to persistent storage should be able to be made async in the future.
- The storage policy is currently exposed via the ini file for now. We might want to enforce policies for this at a later time.
[CL 17416948 by paul chipchase in ue5-main branch]
#rb Mikko.Mononen
#rnx
#preflight 612cbc029db30900012a0dca
- Added a header file and moved the class declaration thereto make it easier for people to find the ini file documentation.
- Split some logging into multiple lines to reduce the overall width of the file.
[CL 17365675 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#rnx
#preflight 60fff7d57f21c90001f9399d
- When we check the source control provider name, we will now attempt to set it to "Perforce" if it is current the default "None" provider.
- If the user has specified a different source control provider than "Perforce" we will give a fatal error that the content virtualization graph containing the source control backend will not work with that setup.
- This part of the workflow might need to change in the future or at least the Mirage documentation must make very clear that when using source control as the persistent storage backend that a connection to perforce will be required, at least until we support our source control solutions but even then the connect to that provider will be required.
[CL 16968160 by paul chipchase in ue5-main branch]
#rb trivial
#rnx
#preflight 60faab53915a92000124d080
Unshelved the wrong version after preflight for submission
[CL 16936466 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
#preflight 60fa86d7915a92000121c919
- Increase the number of potential requests from 32 to 64 (64 to 96 would reduce the time spent by another 5% but that doesn't seem worth the cost for now)
- Change the index we use to iterate over the requests in the pool from a 8bit value to 32bit value to prevent easy overflows.
[CL 16936132 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
#preflight 60f9212d1f926d0001d41223
* VirtualizationJupiterBackend
- When pulling a payload we should assume that a 400 error response when trying to GET the payload header means that the payload is not in Jupiter.
-- Not being able to find the payload should not log an error, instead we can make a note of it in the verbose log (similar to the file system backend)
* VirtualizationFileBackend
- Moved the formatting of system errors to it's own function.
- Log the system error when failing to write a payload during a push as well as a pull.
- We now check that the FileArchive wrote correctly to disk and delete the output file and fail the push if it did not.
-- A future piece of work will change the logic to write to a tmp file at the root of the file store and them move the file to the final location to cut down on the potential of leaving corrupted files around (similar to the process when we save packages)
* Perforce
- The FDownloadFile command now takes an optional parameter EVerbosity that can allow the caller to choose the level of logging output that the command will generate.
- The source control backend for Mirage now opts to supress the logging of the full perforce command when we are pulling payloads as we can generate many hundreds or thousands of requests and the info is not useful to users.
-- We continue to log the command when validating the depot as this is the most likely command to fail so having the info in the log may prove useful.
[CL 16921815 by paul chipchase in ue5-main branch]
- Cleaned up some allocations caused by string manipulation.
*Add a place for shared utility code that could have use in multiple places in the virtualization system.
- First utility is a function to generate a file path based off a FPayloadId.
- The path produced is similar to how we organize files in the DDC but instead of 0->9 we use the range 0->ff in hex.
- To generate the path we take the string form of FPayloadId and then use the first 6 characters to create the directories (each directory using 2 characters) and then the remaining 34 characters becomes the filename.
#rb Stefan.Boberg
#rnx
#ushell-cherrypick of 16167644 by paul.chipchase
#preflight 608f9330f5b27a0001b2d5ab
[CL 16182581 by paul chipchase in ue5-main branch]
- The mirage specific code is disabled behind the define UE_USE_VIRTUALBULKDATA, this means that some code paths in Texture/Mesh are much more complex than they need to be as we support both old and new paths. Once the system has been turned on and confirmed to cause no issues then this will be stripped out.
- SavePackageUtilities.cpp, SavePackage.cpp and SavePackage2.cpp are editgrates rather than integrations as those files have changes in DevCooker that we don't want to bring over immediately.
- Also includes a prototype system for storing bulkdata in a sidecar file in the workspace domain rather than in the .uasset/.umap file which although has been discontinued as part of mirage, will have applications for future work for non-virtualized projects and/or text based assets.
#rb Patrick.Finegan (all changes have been reviewed when submitted to Dev-Cooker)
#tests Cooking and running ShooterGame/Frosty and other sample programs using megascan assets
#rnx
#preflight 608be50d870cf400013ff99d
[CL 16167285 by paul chipchase in ue5-main branch]
