Commit Graph

16 Commits

Author SHA1 Message Date
zousar shaker
886b2fe9cd Change HttpCacheStoreTest to pass access token rather than OAuth details to Zen intermediary instances. Also ensure that we don't overlap keys for Meta and NoMeta test data.
#rb joakim.lindqvist

[CL 24764335 by zousar shaker in ue5-main branch]
2023-03-23 13:05:19 -04:00
zousar shaker
d78e86696f Exclude Zen from HttpCacheStore test on non-windows platforms as it is not ready for use on non-windows platforms.
#rb trivial
[FYI] devin.doucette
[FYI] matt.peters

[CL 23658577 by zousar shaker in ue5-main branch]
2023-01-11 20:57:41 -05:00
Zousar Shaker
b9548ddad4 Move Zen module public headers into an "Experimental" subfolder to have them reflect the feature status.
Fix locations that include the headers to use the "Experimental" subfolder and re-sort includes.  No functional change, purely a move, search+replace, and sort of includes.

#rb matt.peters
#preflight 63939eebc415e8dc78d35d63

[CL 23468934 by Zousar Shaker in ue5-main branch]
2022-12-09 16:11:41 -05:00
devin doucette
1239a75353 DDC: Removed obsolete unstructured namespace parameter from the clients for Horde Storage and Zen
#rb Zousar.Shaker

[CL 22018759 by devin doucette in ue5-main branch]
2022-09-14 18:25:46 -04:00
Devin Doucette
9aee9a4bd8 DDC: Fixed unsafe type cast warnings
#preflight 6317fc04a20b67673b1de7c3
#rb Zousar.Shaker
#rnx

[CL 21850770 by Devin Doucette in ue5-main branch]
2022-09-07 10:37:16 -04:00
Zousar Shaker
860fdb7053 Fix bugs in the System.DerivedDataCache.HttpDerivedDataBackend.CacheStore automated test caused by:
1. Using the wrong structured namespace when instantiating Zen.
2. Assuming that the order that PUT requests will complete is equal to the order that they were in the original request array.

#rb none
#fyi dan.engelbrecht
#preflight 6317a9e72b7fe03eb64e3d7c

[CL 21829182 by Zousar Shaker in ue5-main branch]
2022-09-06 17:05:48 -04:00
Joakim Lindqvist
bd3223cb4c Support for using OidcToken in editor / cooker to acquire a user token, currently opt in.
#preflight https://horde.devtools.epicgames.com/job/62c4108c8d5e678759f258f0

[CL 20964270 by Joakim Lindqvist in ue5-main branch]
2022-07-06 05:59:22 -04:00
steve robb
fd2a0b1e20 Removal of TEXT literal concatenations, which doesn't work in UTF-8 mode.
#rb devin.doucette
#jira UE-132142
#preflight 62a8748fa76c84377748036b

#ROBOMERGE-OWNER: steve.robb
#ROBOMERGE-AUTHOR: steve.robb
#ROBOMERGE-SOURCE: CL 20647443 via CL 20648283 via CL 20648315
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v955-20579017)

[CL 20651786 by steve robb in ue5-main branch]
2022-06-14 12:46:34 -04:00
Zousar Shaker
ea6ad5fc8e Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately.
This change requires a fix (in progress) to EditorDomain to avoid a deadlock.

#rb devin.doucette
#fyi stefan.boberg
#preflight 62a3619670680d6143dc5425

[CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
Devin Doucette
1bbb384300 DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen
#preflight 62967fd60c151996e13038ec
#rb Zousar.Shaker

[CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
dan engelbrecht
e4e1ba5527 Allow configuration for separate namespace for structured data
#preflight 628e494382597362d1f8050d
#preflight 628e70d72c7cb2d10d71c033

[CL 20369511 by dan engelbrecht in ue5-main branch]
2022-05-25 14:55:57 -04:00
Devin Doucette
92a888c1f6 Replaced uses of _ASV with ANSITEXTVIEW
#preflight 6266acf50634d0904cdfe91c
#rb Steve.Robb
#rnx

[CL 19903251 by Devin Doucette in ue5-main branch]
2022-04-25 11:31:36 -04:00
Zousar Shaker
03207d24ea Allow OAuth access scope to be configurable for a Horde Storage DDC store. Supplied by partners.
#rb devin.doucette
#rnx
#preflight 623b9723c73745c600c264e6

[CL 19487258 by Zousar Shaker in ue5-main branch]
2022-03-23 18:05:53 -04:00
Matt Peters
9741234c21 HttpCacheStoreTest: Add test to confirm that downloading from Jupiter -> Upstream Zen -> Zen -> Client works.
#rb Zousar.Shaker
#rnx
#preflight 621674780f71e491cc07ee7e

[CL 19095244 by Matt Peters in ue5-main branch]
2022-02-23 13:13:43 -05:00
Zousar Shaker
a0c688d5fd Unshelved from pending changelist '18992789' in UE5/Release-5.0:
Change Zen to  ValueWithLegacyFallback mode now that it handles Value PUTs to Jupiter correctly.  Add Jupiter auto test that ensures a PUT via UE->Zen->UpstreamZen->Jupiter produces something that can be GET from UE<-Jupiter.  Confirmed test passes.

#rb devin.doucette
#jira UE-134381
#lockdown Aurel.Cordonnier
#preflight 620b4e22bb6a072d17cdd6dd

[CL 18998519 by Zousar Shaker in ue5-main branch]
2022-02-15 12:34:29 -05:00
devin doucette
2800210897 DDC: Enabled compression of legacy cache data
- FileSystem, Http, Pak, S3 use the ValueWithLegacyFallback mode by default, which cause them to fall back to searching for uncompressed data if compressed data is not found.
- FileSystem has been fixed to store up to 1 MiB of compressed data inline with the value package rather than separately in content-addressable storage.
- Pak has been optimized to have GetChunks only load the required region of the requested value, rather than the whole value.
- Pak has been changed to stop storing data inline in the record package, since it will end up in the same file anyway when stored separately.
- Pak will upgrade the compressor and compression level when a compressed pak file is requested. Default cache compression uses Oodle Mermaid VeryFast and will upgrade to Oodle Kraken Optimal2.
- Zen does not have compression enabled by default, pending deployment of a new version that stores compressed values to Horde Storage.
- Added a missing request barrier when saving uncompressed data as compressed.

Example reduction in file system cache size when cooking for Windows:
- CitySample dropped from 66.5 GiB to 21.8 GiB.
- Lyra dropped from 2.54 GiB to 672 MiB.
- ShooterGame dropped from 1.21 GiB to 380 MiB.

Example reduction in compressed pak file cache size when cooking for Windows:
- CitySample dropped from 22.3 GiB to 18.5 GiB.
- Lyra dropped from 691 MiB to 543 MiB.
- ShooterGame dropped from 387 MiB to 313 MiB.

#jira UE-134381
#preflight 620a703f583261b0a658e043, 620a6fb2803d9066e6805310, 620a733117632e948459b6af
#lockdown Aurel.Cordonnier
#rb Zousar.Shaker

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18983671 in //UE5/Release-5.0/... via CL 18983890 via CL 18984096
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v917-18934589)

[CL 18984126 by devin doucette in ue5-main branch]
2022-02-14 14:43:39 -05:00