Contains changes to other structured cache file access to stream data from archives instead of loading the whole file into memory. The record package will now be split into separate allocations on load, rather than the previous behavior where the whole package was kept in memory as long as any part of it was referenced.
#rb Zousar.Shaker
#rnx
#preflight 61b11fd75238845473be0c30
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18413427 in //UE5/Release-5.0/... via CL 18413472
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v897-18405271)
[CL 18413485 by devin doucette in ue5-release-engine-test branch]
Requiring the use of a separate reader type makes it more likely that readers will be reused, and makes it easier to audit reader usage going forward. Reusing readers is desirable to reduce the number of large temporary allocations made during partial decompression of a buffer.
- Added FCompressedBuffer::Save(FArchive&) and renamed FromCompressed(FArchive&) to Load(FArchive&).
- Added FCompressedBufferReaderSourceScope to set a buffer source within a scope.
- Added proper bounds checks to FNoneDecoder.
- Store the header checksum on the decoder context to allow raw blocks to be reused across sources.
- Decode the header on the fly to avoid a temporary header allocation when the header is in contiguous memory.
#rb Zousar.Shaker
#rnx
#preflight 61a98d53800738dbfbc84c73
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18382211 in //UE5/Release-5.0/... via CL 18382310
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v896-18170469)
[CL 18382377 by devin doucette in ue5-release-engine-test branch]
Messages are deterministic output recorded explicitly by the build function. Logs will be automatically captured from the output log and are not necessarily deterministic. The presence of logs in a build output automatically disables caching of the build output.
#rb Zousar.Shaker
#rnx
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18286155 in //UE5/Release-5.0/... via CL 18286173
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)
[CL 18286201 by devin doucette in ue5-release-engine-test branch]
No functional changes, only bringing the code style in line with the current coding standards since nearly the entire file has been modified for 5.0 anyway.
#rb none
#rnx
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18135592 in //UE5/Release-5.0/... via CL 18135607
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v889-18060218)
#ROBOMERGE[STARSHIP]: UE5-Main
[CL 18135617 by devin doucette in ue5-release-engine-test branch]
- Implemented only for the structured cache initially.
- Records are stored as compact binary packages with payloads stored within the record if small enough.
- Records are stored in Records/{Bucket}/{01}/{23}/{456789abcdef0123456789abcdef01234567} based on the hash in the cache key.
- Content that exceeds the size allowed for storage within the record is stored in Content/{01}/{23}/{456789abcdef0123456789abcdef01234567} based on the hash of the content.
- Both records and content are written to disk with their hash as a suffix, and are always checked on load.
- Maximum size of compressed data stored in the package can be configured by MaxRecordSizeKB and MaxValueSizeKB.
- Maintenance of the cache, to delete unused files, has been rewritten and can now scan ~175k files/second on modern hardware, though is limited to 10k files/second in the config to avoid adding too much overhead.
#rb Zousar.Shaker
#preflight 618b75c14a0c609a29204032
#preflight 618bfb5ec717ba4e929b7ac0
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18133085 in //UE5/Release-5.0/... via CL 18133356
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v889-18060218)
#ROBOMERGE[STARSHIP]: UE5-Main
[CL 18133430 by devin doucette in ue5-release-engine-test branch]
This represents UE4/Main @18073326, Release-5.0 @18081140 and Dev-PerfTest @18045971
[CL 18081471 by aurel cordonnier in ue5-release-engine-test branch]
When ServerID is used, configuration is first loaded from a key matching the ServerID field, in the HordeStorageServers section of the same INI. This makes it easier to put Jupiter configuration in a consistent place and have it stripped by the cooker to avoid leaking any of the configuration.
#rb Zousar.Shaker
#rnx
#preflight 617a28570cec4300011619d4
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 17961730 in //UE5/Release-5.0/... via CL 17961740
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v885-17909292)
#ROBOMERGE[STARSHIP]: UE5-Main
[CL 17961744 by devin doucette in ue5-release-engine-test branch]
This represents UE4/Main @17911760, Release-5.0 @17915875 and Dev-PerfTest @17914035
[CL 17918595 by aurel cordonnier in ue5-release-engine-test branch]
- Build now takes optional inputs for consistency with BuildAction.
- BuildAction has been renamed to Build and works the same as Build except it starts from an action.
- Build now takes a FBuildPolicy, which is implicitly constructible from EBuildPolicy, and allows setting the policy by payload.
- FBuildContext and FBuildConfigContext now have policy masks that allow clearing flags across every payload in the build.
- EBuildPolicy has CacheStore split into CacheStoreOnQuery and CacheStoreOnBuild for more control of cache behavior.
- Builds now use consistent completion callback parameters, FBuildCompleteParams.
- Build types now have debugger visualization.
#rb Zousar.Shaker
#rnx
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 17874670 in //UE5/Release-5.0/... via CL 17874724
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v883-17842818)
#ROBOMERGE[STARSHIP]: UE5-Main
[CL 17874740 by devin doucette in ue5-release-engine-test branch]
This represents UE4/Main @17774255, Release-5.0 @17791557 and Dev-PerfTest @17789485
[CL 17794212 by aurel cordonnier in ue5-release-engine-test branch]
