#rb Per.Larsson
#rnx
#preflight 61408a979dc6c8000148d0cf
### Misc
- Updated the documentation for setting up a file system backend and added descriptions to the backend's members.
- Renamed the member 'Name' to 'DebugName' to bring it in line with the other backends (still a candidate to be moved to the base class)
### Reading payloads from disk
- Added the ability to retry opening file handles for read if they fail. This can commonly occur when multiple threads or processes are both pulling and pushing to the same root directory.
-- By default we will retry to open the file handle up to 10 times, waiting for 100ms between each attempt. The user can configure the number of retries and the length of the pause between each attempt via the config files.
-- Each failed attempt at opening the file handle will result in a warning being logged, so if the user is stalled for a long length of time while we rety over and over they are at least made aware of the cause.
### Writing payloads to disk
- When attempting to write a payload to disk, we will now write to a temp file in the project save directory using a randomly generated name. Once the payload has been fully saved to disk it will then be moved to the correct location. Although not perfect this will reduce the potential for writing corrupted data to a location where it might be read in the future.
-- We do attempt to clean up the temp file if the write fails but we do not print error messages if the delete fails as it is considered an optional bonus rather than an essential feature.
- Previously we would only log an error if we failed to open a file handle for writing the payload to disk. Now we also log an error if the handle is opened but the actual writes fail.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 17550126 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v870-17433530)
[CL 17550133 by paul chipchase in ue5-release-engine-test branch]
#rb Mikko.Mononen
#rnx
#preflight 612cbc029db30900012a0dca
- Added a header file and moved the class declaration thereto make it easier for people to find the ini file documentation.
- Split some logging into multiple lines to reduce the overall width of the file.
#ROBOMERGE-SOURCE: CL 17365675 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v865-17346139)
[CL 17365683 by paul chipchase in ue5-release-engine-test branch]
#rb Per.Larsson
#rnx
#preflight 60f9212d1f926d0001d41223
* VirtualizationJupiterBackend
- When pulling a payload we should assume that a 400 error response when trying to GET the payload header means that the payload is not in Jupiter.
-- Not being able to find the payload should not log an error, instead we can make a note of it in the verbose log (similar to the file system backend)
* VirtualizationFileBackend
- Moved the formatting of system errors to it's own function.
- Log the system error when failing to write a payload during a push as well as a pull.
- We now check that the FileArchive wrote correctly to disk and delete the output file and fail the push if it did not.
-- A future piece of work will change the logic to write to a tmp file at the root of the file store and them move the file to the final location to cut down on the potential of leaving corrupted files around (similar to the process when we save packages)
* Perforce
- The FDownloadFile command now takes an optional parameter EVerbosity that can allow the caller to choose the level of logging output that the command will generate.
- The source control backend for Mirage now opts to supress the logging of the full perforce command when we are pulling payloads as we can generate many hundreds or thousands of requests and the info is not useful to users.
-- We continue to log the command when validating the depot as this is the most likely command to fail so having the info in the log may prove useful.
#ROBOMERGE-SOURCE: CL 16921815 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v836-16769935)
[CL 16921818 by paul chipchase in ue5-release-engine-test branch]
- Cleaned up some allocations caused by string manipulation.
*Add a place for shared utility code that could have use in multiple places in the virtualization system.
- First utility is a function to generate a file path based off a FPayloadId.
- The path produced is similar to how we organize files in the DDC but instead of 0->9 we use the range 0->ff in hex.
- To generate the path we take the string form of FPayloadId and then use the first 6 characters to create the directories (each directory using 2 characters) and then the remaining 34 characters becomes the filename.
#rb Stefan.Boberg
#rnx
#ushell-cherrypick of 16167644 by paul.chipchase
#preflight 608f9330f5b27a0001b2d5ab
[CL 16182581 by paul chipchase in ue5-main branch]
- The mirage specific code is disabled behind the define UE_USE_VIRTUALBULKDATA, this means that some code paths in Texture/Mesh are much more complex than they need to be as we support both old and new paths. Once the system has been turned on and confirmed to cause no issues then this will be stripped out.
- SavePackageUtilities.cpp, SavePackage.cpp and SavePackage2.cpp are editgrates rather than integrations as those files have changes in DevCooker that we don't want to bring over immediately.
- Also includes a prototype system for storing bulkdata in a sidecar file in the workspace domain rather than in the .uasset/.umap file which although has been discontinued as part of mirage, will have applications for future work for non-virtualized projects and/or text based assets.
#rb Patrick.Finegan (all changes have been reviewed when submitted to Dev-Cooker)
#tests Cooking and running ShooterGame/Frosty and other sample programs using megascan assets
#rnx
#preflight 608be50d870cf400013ff99d
[CL 16167285 by paul chipchase in ue5-main branch]
