Commit Graph

66 Commits

Author SHA1 Message Date
zousar shaker
df06aa0d18 Allow for region pinning when connecting to Jupiter. Fix value printing error. Increase concurrent PUT operations from 2 to 4 when batching is enabled.
#preflight 6201d9c7384d6a94e8fc20d7
#jira UE-141733
#rb devin.doucette

#ROBOMERGE-AUTHOR: zousar.shaker
#ROBOMERGE-SOURCE: CL 18899614 in //UE5/Release-5.0/... via CL 18899644 via CL 18899680
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v910-18824042)

[CL 18899684 by zousar shaker in ue5-main branch]
2022-02-08 01:56:17 -05:00
zousar shaker
b9fdfa4036 GetChunks value implementation for Jupiter.
#rb devin.doucette
#preflight 61f9db609e4d23cd93b7b28d

#ROBOMERGE-AUTHOR: zousar.shaker
#ROBOMERGE-SOURCE: CL 18819720 in //UE5/Release-5.0/... via CL 18819729 via CL 18822895
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v910-18824042)

[CL 18825048 by zousar shaker in ue5-main branch]
2022-02-02 08:18:44 -05:00
zousar shaker
c6907b8736 Implement DDC2 Value API for Jupiter backend.
#rb devin.doucette
#preflight 61f98cc6db42673a6008471b

#ROBOMERGE-AUTHOR: zousar.shaker
#ROBOMERGE-SOURCE: CL 18813356 in //UE5/Release-5.0/... via CL 18813377 via CL 18822759
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v910-18824042)

[CL 18824291 by zousar shaker in ue5-main branch]
2022-02-02 07:35:19 -05:00
devin doucette
169318850a DDC: Masked record-only cache policy flags out of the value policy
Fixing backends to respect the masked flags includes a few fixes to handling of ~Query by Put and PutValue.

#rb Matt.Peters
#rnx
#preflight 61f7f7ede55232619f887977

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18790481 in //UE5/Release-5.0/... via CL 18790769 via CL 18791304
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v908-18788545)

[CL 18791515 by devin doucette in ue5-main branch]
2022-01-31 11:52:13 -05:00
devin doucette
517947112b DDC: Reorganized cache policy code into its own files
- FCacheRecordPolicyBuilder now disallows adding value policies with flags that only make sense for records.
- FCacheRecordPolicyBuilder now disallows adding multiple value policies for the same value.
- FCacheRecordPolicyBuilder now skips addition of value policies that match the default policy.

#rb Matt.Peters
#rnx
#preflight 61f471cc6b5aea38e5c64cb6

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18782116 in //UE5/Release-5.0/... via CL 18782123 via CL 18782168
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v903-18687472)

[CL 18782178 by devin doucette in ue5-main branch]
2022-01-28 20:34:31 -05:00
devin doucette
8bc347b0be DDC: Fixed TryToPrefetch when there is not one backend containing every key
#rb Zousar.Shaker
#rnx
#preflight 61f327371f12674afbc5be32

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18768073 in //UE5/Release-5.0/... via CL 18768076 via CL 18768140
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v903-18687472)

[CL 18768147 by devin doucette in ue5-main branch]
2022-01-27 23:16:00 -05:00
zousar shaker
b6a8902058 Change the DDC Http backend to ensure that when downloading from the /refs and /compressed-blob routes, that the Http return code 404 is not logged at display level, but instead treated as an expected error code because unlike the /c route, they use 404 to indicate a cache miss.
#rb devin.doucette
#preflight 61f0dce8ff453b751b5130a0

#ROBOMERGE-AUTHOR: zousar.shaker
#ROBOMERGE-SOURCE: CL 18732663 in //UE5/Release-5.0/... via CL 18732713 via CL 18732743
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v903-18687472)

[CL 18732763 by zousar shaker in ue5-main branch]
2022-01-26 00:46:18 -05:00
devin doucette
6ab080890c DDC: Renamed cache request and response types from Chunk to GetChunk
#rb Zousar.Shaker
#rnx
#preflight 61e5ec410705ece855b8da74

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18635634 in //UE5/Release-5.0/... via CL 18635654 via CL 18635655
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v899-18417669)

[CL 18637582 by devin doucette in ue5-main branch]
2022-01-18 04:47:59 -05:00
zousar shaker
a455cb734d Ensure Jupiter DDC backend implements GetChunks in a functionally correct way.
Fix erroneous logging in Jupiter DDC backend when a cache record was missed.
Fix breakage in Jupiter DDC backend automated test caused by the header re-org.
Add GetChunks tests to Jupiter DDC backend automated test and increase the size and  byte variety for values in the test content.

#rb devin.doucette
#preflight 61e19ae5076be0fc4e128941

#ROBOMERGE-AUTHOR: zousar.shaker
#ROBOMERGE-SOURCE: CL 18618080 in //UE5/Release-5.0/... via CL 18618090 via CL 18618106
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v899-18417669)

[CL 18618116 by zousar shaker in ue5-main branch]
2022-01-14 10:53:17 -05:00
michael noland
a862007587 Fixed a crash in logging code when CreateHttpDerivedDataBackend couldn't connect to a backend (extra dereference on a TCHAR* format arg)
[FYI] Devin.Doucette
#preflight none

#ROBOMERGE-AUTHOR: michael.noland
#ROBOMERGE-SOURCE: CL 18594389 in //UE5/Release-5.0/... via CL 18594410 via CL 18594417
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Test -> Main) (v899-18417669)

[CL 18596594 by michael noland in ue5-main branch]
2022-01-13 10:04:43 -05:00
devin doucette
23ddaa8bab DDC: Moved cache backends out of headers
This hides implementation details and reduces the number of files that need to change when refactoring backends.

Backends moved in this change: AsyncPut, Http, Memory, S3, Zen.

#rb Zousar.Shaker
#rnx
#preflight 61dd0d758d72a407aab89074

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18573340 in //UE5/Release-5.0/... via CL 18573347
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)

[CL 18573359 by devin doucette in ue5-release-engine-test branch]
2022-01-11 11:57:38 -05:00
zousar shaker
87fc31409c Switch to using HEAD when doing existance checks against the structured cache in Jupiter. Was not possible before due to a bug in Jupiter, but that hs now been fixed.
#rb devin.doucette
#preflight 61dda4790134bb26f3bbd33b

#ROBOMERGE-AUTHOR: zousar.shaker
#ROBOMERGE-SOURCE: CL 18572483 in //UE5/Release-5.0/... via CL 18572521
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)

[CL 18572545 by zousar shaker in ue5-release-engine-test branch]
2022-01-11 10:53:46 -05:00
zousar shaker
831400efc2 Add support for structured cache operations through the Jupiter DDC backend.
#rb devin.doucette
#preflight 61dc76dc2d5fd64ff7d4d4e8

#ROBOMERGE-AUTHOR: zousar.shaker
#ROBOMERGE-SOURCE: CL 18562740 in //UE5/Release-5.0/... via CL 18562761
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)

[CL 18562791 by zousar shaker in ue5-release-engine-test branch]
2022-01-10 13:43:40 -05:00
devin doucette
35393bbb2b DDC: Split FPayload into separate FValue and FValueId types
A payload was conceptually a value with an ID. That has been formalized by removing the ID from the payload and having separate FValue and FValueId types. This separation cleans up the API in a few areas, and provides a more natural path to providing a basic key/value API.

#rb Zousar.Shaker
#rnx
#preflight 61d704c04c252480ca284d61

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18531844 in //UE5/Release-5.0/... via CL 18531856
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)

[CL 18531864 by devin doucette in ue5-release-engine-test branch]
2022-01-06 11:05:57 -05:00
devin doucette
8e3ba57e91 DDC: Moved the ICacheStore Context params to be a Name member of the request types and return them in the completion params
#rb Zousar.Shaker
#rnx
#preflight 61d3729eaf44b43628f4bdc9

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18507435 in //UE5/Release-5.0/... via CL 18507440
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)

[CL 18507451 by devin doucette in ue5-release-engine-test branch]
2022-01-04 09:26:33 -05:00
devin doucette
54235555db DDC: Moved FDerivedDataBackendInterface from Public to Private
The interface was in the public folder before 5.0, but was not usable from there in a meaningful way.

#rb Zousar.Shaker
#rnx
#preflight 61bbac388a62de8385c132c7

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18479761 in //UE5/Release-5.0/... via CL 18481541
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)

[CL 18481817 by devin doucette in ue5-release-engine-test branch]
2021-12-16 19:56:56 -05:00
devin doucette
4d164bdfd5 DDC: Added a uint64 UserData to ICacheStore requests to make completion callbacks more efficient
Requests to Put and Get now specify the policy independently for each record or key, but the implementations have not yet been updated to take this into account.

The existing Put and Get API has been maintained temporarily to allow the conversion to happen separately, and will be removed after callers have been converted to use the new API.

#rb Zousar.Shaker
#rnx
#preflight 61b2e550ee0de9822e2e84dc

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18443052 in //UE5/Release-5.0/... via CL 18445037
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v897-18405271)

[CL 18445201 by devin doucette in ue5-release-engine-test branch]
2021-12-13 13:32:28 -05:00
aurel cordonnier
fc542f6cfd Merge from Release-Engine-Staging @ 18081189 to Release-Engine-Test
This represents UE4/Main @18073326, Release-5.0 @18081140 and Dev-PerfTest @18045971

[CL 18081471 by aurel cordonnier in ue5-release-engine-test branch]
2021-11-07 23:43:01 -05:00
joakim lindqvist
eec8f48f1a Support reading HttpBackend secret from a file
#ROBOMERGE-AUTHOR: joakim.lindqvist
#ROBOMERGE-SOURCE: CL 17921575 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v885-17909292)

[CL 17922647 by joakim lindqvist in ue5-release-engine-test branch]
2021-10-26 09:14:51 -04:00
aurel cordonnier
a6e741e007 Merge from Release-Engine-Staging @ 17915896 to Release-Engine-Test
This represents UE4/Main @17911760, Release-5.0 @17915875 and Dev-PerfTest @17914035

[CL 17918595 by aurel cordonnier in ue5-release-engine-test branch]
2021-10-25 20:05:28 -04:00
aurel cordonnier
a12d56ff31 Merge from Release-Engine-Staging @ 17791557 to Release-Engine-Test
This represents UE4/Main @17774255, Release-5.0 @17791557 and Dev-PerfTest @17789485

[CL 17794212 by aurel cordonnier in ue5-release-engine-test branch]
2021-10-12 21:21:22 -04:00
Devin Doucette
a077feffbd DDC: Reworked ICacheStore to allow partial records, filtering of payloads, and loading parts of payloads
- ICacheStore::Put() has updated documentation to reflect the requirements of partial records.
- ICacheStore::Get() now takes a FCacheRecordPolicy, which is implicitly constructible from ECachePolicy, and allows setting the policy by payload.
- ICacheStore::GetPayload() is replaced by ICacheStore::GetChunks(), which allows loading parts of payloads.
- ICacheStore::CancelAll() is moved to ICache::CancelAll() because the cache can track requests at the top level and cancel them without exposing cancellation on individual cache stores.
- ECachePolicy::SkipLocalCopy has been removed because it is difficult to reason about.
- ECachePolicy::SkipData flags now have a documented meaning for put requests, to hint that record existence implies payload existence.
- The filesystem and memory cache stores have been updated to support partial records, filtering of payloads, and loading parts of payloads.
- Requesting part of a payload will decompress the entire payload for now, until compressed buffers expose a way to decompress only part.
- Fixed a bug in FTexturePlatformData::AreDerivedMipsAvailable() that caused it to return false for structured cache keys.

#rb Zousar.Shaker
#rnx
#preflight 615e03241ed62f0001b95454

#ROBOMERGE-OWNER: Devin.Doucette
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 17748550 in //UE5/Release-5.0/... via CL 17748555
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v879-17706426)
#ROBOMERGE-CONFLICT from-shelf
#ROBOMERGE[STARSHIP]: UE5-Main

[CL 17748602 by Devin Doucette in ue5-release-engine-test branch]
2021-10-07 09:11:32 -04:00
devin doucette
c39e30cfd7 Added missing includes of DerivedDataPayload.h found by the non-unity build
#rb none
#rnx

#ROBOMERGE-SOURCE: CL 17426590 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v865-17346139)

[CL 17426595 by devin doucette in ue5-release-engine-test branch]
2021-09-03 17:11:57 -04:00
andriy tylychko
6bf3101dcd deprecated FTicker and family and replaced by thread-safe FTSTicker
#jira UE-120090
#rb francis.hurteau


#ROBOMERGE-SOURCE: CL 17176325 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v855-17104924)

[CL 17176374 by andriy tylychko in ue5-release-engine-test branch]
2021-08-16 11:09:22 -04:00
devin doucette
bd60f3ef1b DDC: Removed ICacheFactory in favor of exported functions
FCacheBucket is now always exposed as FAnsiStringView.

#rb Zousar.Shaker
#rnx

#ROBOMERGE-SOURCE: CL 17084521 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)

[CL 17084549 by devin doucette in ue5-release-engine-test branch]
2021-08-06 12:53:08 -04:00