Change Zen to ValueWithLegacyFallback mode now that it handles Value PUTs to Jupiter correctly. Add Jupiter auto test that ensures a PUT via UE->Zen->UpstreamZen->Jupiter produces something that can be GET from UE<-Jupiter. Confirmed test passes.
#rb devin.doucette
#jira UE-134381
#lockdown Aurel.Cordonnier
#preflight 620b4e22bb6a072d17cdd6dd
[CL 18998519 by Zousar Shaker in ue5-main branch]
- FileSystem, Http, Pak, S3 use the ValueWithLegacyFallback mode by default, which cause them to fall back to searching for uncompressed data if compressed data is not found.
- FileSystem has been fixed to store up to 1 MiB of compressed data inline with the value package rather than separately in content-addressable storage.
- Pak has been optimized to have GetChunks only load the required region of the requested value, rather than the whole value.
- Pak has been changed to stop storing data inline in the record package, since it will end up in the same file anyway when stored separately.
- Pak will upgrade the compressor and compression level when a compressed pak file is requested. Default cache compression uses Oodle Mermaid VeryFast and will upgrade to Oodle Kraken Optimal2.
- Zen does not have compression enabled by default, pending deployment of a new version that stores compressed values to Horde Storage.
- Added a missing request barrier when saving uncompressed data as compressed.
Example reduction in file system cache size when cooking for Windows:
- CitySample dropped from 66.5 GiB to 21.8 GiB.
- Lyra dropped from 2.54 GiB to 672 MiB.
- ShooterGame dropped from 1.21 GiB to 380 MiB.
Example reduction in compressed pak file cache size when cooking for Windows:
- CitySample dropped from 22.3 GiB to 18.5 GiB.
- Lyra dropped from 691 MiB to 543 MiB.
- ShooterGame dropped from 387 MiB to 313 MiB.
#jira UE-134381
#preflight 620a703f583261b0a658e043, 620a6fb2803d9066e6805310, 620a733117632e948459b6af
#lockdown Aurel.Cordonnier
#rb Zousar.Shaker
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18983671 in //UE5/Release-5.0/... via CL 18983890 via CL 18984096
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v917-18934589)
[CL 18984126 by devin doucette in ue5-main branch]
Add batched Put and PutValue.
Update Batched Get to send the RecordPolicy with each key.
Update GetChunks to send the CachePolicy as string and to handle SkipData in the response.
Add more unit tests for upstream behavior and GetChunks.
Update the CacheRecordPolicy Save/Load to match the new variable names on FCacheRecordPolicy.
Deploy new zenserver binaries with the required matching server-side changes.
#rb Per.Larsson,Devin.Doucette
#rnx
#preflight 61f94e5980608c7029b8fc5c
#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 18808743 in //UE5/Release-5.0/... via CL 18809399 via CL 18822531
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v908-18788545)
[CL 18823289 by matt peters in ue5-main branch]
Fix erroneous logging in Jupiter DDC backend when a cache record was missed.
Fix breakage in Jupiter DDC backend automated test caused by the header re-org.
Add GetChunks tests to Jupiter DDC backend automated test and increase the size and byte variety for values in the test content.
#rb devin.doucette
#preflight 61e19ae5076be0fc4e128941
#ROBOMERGE-AUTHOR: zousar.shaker
#ROBOMERGE-SOURCE: CL 18618080 in //UE5/Release-5.0/... via CL 18618090 via CL 18618106
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v899-18417669)
[CL 18618116 by zousar shaker in ue5-main branch]
This hides implementation details and reduces the number of files that need to change when refactoring backends.
Backends moved in this change: AsyncPut, Http, Memory, S3, Zen.
#rb Zousar.Shaker
#rnx
#preflight 61dd0d758d72a407aab89074
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18573340 in //UE5/Release-5.0/... via CL 18573347
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)
[CL 18573359 by devin doucette in ue5-release-engine-test branch]
Added a test that could reliably reproduce the problem to validate it is properly fixed
#rb Danny.Couture, Zousar.Shaker
#preflight 610010b247c93a0001335319
#ROBOMERGE-OWNER: francis.hurteau
#ROBOMERGE-AUTHOR: francis.hurteau
#ROBOMERGE-SOURCE: CL 16968978 via CL 16972104
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v838-16927207)
[CL 16972120 by francis hurteau in ue5-release-engine-test branch]
Replace the sleep by a wait on an event triggered by the thread freeing the connection
Improve the test to output stats seen below and use the proper amount of threads on low core systems
Automation RunTests System.DerivedDataCache.HttpDerivedDataBackend.ConcurrentCachedDataProbablyExistsBatch
Before
RPS: 238, AvgLatency: 90.09 ms, MaxLatency: 7.77 s
RPS: 231, AvgLatency: 87.66 ms, MaxLatency: 7.77 s
RPS: 245, AvgLatency: 139.07 ms, MaxLatency: 10.08 s
After
RPS: 634, AvgLatency: 102.97 ms, MaxLatency: 0.26 s
RPS: 653, AvgLatency: 102.51 ms, MaxLatency: 0.26 s
RPS: 630, AvgLatency: 102.29 ms, MaxLatency: 0.26 s
#rnx
#rb Zousar.Shaker, Francis.Hurteau
#preflight 60ffe8317f21c90001f6116a
#ROBOMERGE-SOURCE: CL 16968010 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v838-16927207)
[CL 16968015 by danny couture in ue5-release-engine-test branch]
Added a test that could reliably reproduce the problem to validate it is properly fixed
#preflight 60faeafc1f926d000113425b
#ROBOMERGE-SOURCE: CL 16938823 via CL 16938832
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v838-16927207)
[CL 16938835 by francis hurteau in ue5-release-engine-test branch]
Adding basic unit test for HttpDerivedDataBackend.
#rb devin.doucette
#ROBOMERGE-SOURCE: CL 16885295 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v836-16769935)
[CL 16885298 by zousar shaker in ue5-release-engine-test branch]
