Commit Graph

6 Commits

Author SHA1 Message Date
paul chipchase
7d754545ee Reorganised the virtualization filesystem backend code.
#rb Mikko.Mononen
#rnx
#preflight 612cbc029db30900012a0dca

- Added a header file and moved the class declaration thereto make it easier for people to find the ini file documentation.
- Split some logging into multiple lines to reduce the overall width of the file.

#ROBOMERGE-SOURCE: CL 17365675 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v865-17346139)

[CL 17365683 by paul chipchase in ue5-release-engine-test branch]
2021-08-31 06:30:26 -04:00
paul chipchase
5c51f3f814 Improve error handling and logging in the file and jupiter Mirage backends.
#rb Per.Larsson
#rnx
#preflight 60f9212d1f926d0001d41223

* VirtualizationJupiterBackend
- When pulling a payload we should assume that a 400 error response when trying to GET the payload header means that the payload is not in Jupiter.
-- Not being able to find the payload should not log an error, instead we can make a note of it in the verbose log (similar to the file system backend)

* VirtualizationFileBackend
- Moved the formatting of system errors to it's own function.
- Log the system error when failing to write a payload during a push as well as a pull.
- We now check that the FileArchive wrote correctly to disk and delete the output file and fail the push if it did not.
-- A future piece of work will change the logic to write to a tmp file at the root of the file store and them move the file to the final location to cut down on the potential of leaving corrupted files around (similar to the process when we save packages)

* Perforce
- The FDownloadFile command now takes an optional parameter EVerbosity that can allow the caller to choose the level of logging output that the command will generate.
- The source control backend for Mirage now opts to supress the logging of the full perforce command when we are pulling payloads as we can generate many hundreds or thousands of requests and the info is not useful to users.
-- We continue to log the command when validating the depot as this is the most likely command to fail so having the info in the log may prove useful.

#ROBOMERGE-SOURCE: CL 16921815 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v836-16769935)

[CL 16921818 by paul chipchase in ue5-release-engine-test branch]
2021-07-22 05:45:28 -04:00
paul chipchase
04229b992f IVirtualizationBackend::PushData now returns an enum describing what happened to the operation rather than a straight true/false.
#rb Per.Larsson
#rnx
#preflight 60a4f14665d94d00015be8d8

[CL 16383162 by paul chipchase in ue5-main branch]
2021-05-19 07:46:07 -04:00
paul chipchase
92b08f8d89 CIS: Fix typo in include
#rb trivial
#rnx
#preflight 608fb0de44491c0001005038

[CL 16182833 by paul chipchase in ue5-main branch]
2021-05-03 05:01:37 -04:00
paul chipchase
d1060bb972 Change the file path to be three directories deep from the root, with each directory being named 00->ff, we then attempt to place each file in a directory based on a hash of the PayloadId, ideally evenly distributing between all available leaf directories.
- Cleaned up some allocations caused by string manipulation.

*Add a place for shared utility code that could have use in multiple places in the virtualization system.
- First utility is a function to generate a file path based off a FPayloadId.
- The path produced is similar to how we organize files in the DDC but instead of 0->9 we use the range 0->ff in hex.
- To generate the path we take the string form of FPayloadId and then use the first 6 characters to create the directories (each directory using 2 characters) and then the remaining 34 characters becomes the filename.

#rb Stefan.Boberg
#rnx

#ushell-cherrypick of 16167644 by paul.chipchase
#preflight 608f9330f5b27a0001b2d5ab

[CL 16182581 by paul chipchase in ue5-main branch]
2021-05-03 02:47:29 -04:00
paul chipchase
71b969d9c4 Cherrypicking Mirage code base (disabled) from Dev-Cooker
- The mirage specific code is disabled behind the define UE_USE_VIRTUALBULKDATA, this means that some code paths in Texture/Mesh are much more complex than they need to be as we support both old and new paths. Once the system has been turned on and confirmed to cause no issues then this will be stripped out.
- SavePackageUtilities.cpp, SavePackage.cpp and SavePackage2.cpp are editgrates rather than integrations as those files have changes in DevCooker that we don't want to bring over immediately.
- Also includes a prototype system for storing bulkdata in a sidecar file in the workspace domain rather than in the .uasset/.umap file which although has been discontinued as part of mirage, will have applications for future work for non-virtualized projects and/or text based assets.

#rb Patrick.Finegan (all changes have been reviewed when submitted to Dev-Cooker)
#tests Cooking and running ShooterGame/Frosty and other sample programs using megascan assets
#rnx
#preflight  608be50d870cf400013ff99d

[CL 16167285 by paul chipchase in ue5-main branch]
2021-04-30 08:14:54 -04:00