Commit Graph

518 Commits

Author SHA1 Message Date
devin doucette
8a6e7cabc1 DDC: Changed cache policy string conversion to use optimal encoding
Parsing uses UTF-8 to avoid conversion of UTF-8 and ANSI strings, and conversion to a string uses ANSI to avoid expensive conversion from UTF-16.

Includes a change to mask out record-only flags from values in FCacheRecordPolicy::Transform.

#rb Matt.Peters
#rnx
#preflight 61f9934d162e627eab43ebfd

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18814354 in //UE5/Release-5.0/... via CL 18814383 via CL 18822771
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v910-18824042)

[CL 18824304 by devin doucette in ue5-main branch]
2022-02-02 07:36:25 -05:00
matt peters
f2eed1a838 CacheStoreTest: Add a test case that catches ValuePolicies being parsed correctly.
#rb Zousar.Shaker
#rnx
#preflight 61f98f024404d5fadefb1c0b

#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 18813550 in //UE5/Release-5.0/... via CL 18813562 via CL 18822761
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v910-18824042)

[CL 18824293 by matt peters in ue5-main branch]
2022-02-02 07:35:30 -05:00
zousar shaker
c6907b8736 Implement DDC2 Value API for Jupiter backend.
#rb devin.doucette
#preflight 61f98cc6db42673a6008471b

#ROBOMERGE-AUTHOR: zousar.shaker
#ROBOMERGE-SOURCE: CL 18813356 in //UE5/Release-5.0/... via CL 18813377 via CL 18822759
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v910-18824042)

[CL 18824291 by zousar shaker in ue5-main branch]
2022-02-02 07:35:19 -05:00
devin doucette
c3443800ec DDC: Converted modular feature names to constant lazy names
#rb Zousar.Shaker
#rnx
#preflight 61f97050fa6554dff3d6fa76

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18812871 in //UE5/Release-5.0/... via CL 18812927 via CL 18822743
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v909-18823584)

[CL 18823987 by devin doucette in ue5-main branch]
2022-02-02 06:26:22 -05:00
matt peters
c98e98aa5f ZenServer CachePolicy support.
Add batched Put and PutValue.
Update Batched Get to send the RecordPolicy with each key.
Update GetChunks to send the CachePolicy as string and to handle SkipData in the response.
Add more unit tests  for upstream behavior and GetChunks.
Update the CacheRecordPolicy Save/Load to match the new variable names on FCacheRecordPolicy.
Deploy new zenserver binaries with the required matching server-side changes.

#rb Per.Larsson,Devin.Doucette
#rnx
#preflight 61f94e5980608c7029b8fc5c

#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 18808743 in //UE5/Release-5.0/... via CL 18809399 via CL 18822531
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v908-18788545)

[CL 18823289 by matt peters in ue5-main branch]
2022-02-02 05:33:16 -05:00
devin doucette
d0a5f83709 DDC: Moved the delete batch in the new hierarchy next to the delete function
#rb trivial
#rnx
#preflight 61f8cf94a6632a34f3689784

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18805848 in //UE5/Release-5.0/... via CL 18807992 via CL 18821772
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v908-18788545)

[CL 18822130 by devin doucette in ue5-main branch]
2022-02-02 02:20:15 -05:00
devin doucette
da469aebf0 DDC: Added Put and Get support to the new cache hierarchy
#rb Zousar.Shaker
#rnx
#preflight 61f864a868795b2f458b0c91

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18805714 in //UE5/Release-5.0/... via CL 18807981 via CL 18821770
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v908-18788545)

[CL 18822126 by devin doucette in ue5-main branch]
2022-02-02 02:20:02 -05:00
devin doucette
3cb1585ced DDC: Changed "Type&& Get() &&" accessors to "Type Get() &&"
#rb Zousar.Shaker
#rnx
#preflight 61f80d4868795b2f45779b4b

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18792049 in //UE5/Release-5.0/... via CL 18792224 via CL 18792941
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v908-18788545)

[CL 18793084 by devin doucette in ue5-main branch]
2022-01-31 13:23:12 -05:00
devin doucette
52ca2f071b DDC: Added PutValue and GetValue support to the new cache hierarchy
#rb Zousar.Shaker
#rnx
#preflight 61f78b4e6a7c7b134f2f289e

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18791013 in //UE5/Release-5.0/... via CL 18791372 via CL 18791928
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v908-18788545)

[CL 18792105 by devin doucette in ue5-main branch]
2022-01-31 12:25:15 -05:00
devin doucette
169318850a DDC: Masked record-only cache policy flags out of the value policy
Fixing backends to respect the masked flags includes a few fixes to handling of ~Query by Put and PutValue.

#rb Matt.Peters
#rnx
#preflight 61f7f7ede55232619f887977

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18790481 in //UE5/Release-5.0/... via CL 18790769 via CL 18791304
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v908-18788545)

[CL 18791515 by devin doucette in ue5-main branch]
2022-01-31 11:52:13 -05:00
devin doucette
517947112b DDC: Reorganized cache policy code into its own files
- FCacheRecordPolicyBuilder now disallows adding value policies with flags that only make sense for records.
- FCacheRecordPolicyBuilder now disallows adding multiple value policies for the same value.
- FCacheRecordPolicyBuilder now skips addition of value policies that match the default policy.

#rb Matt.Peters
#rnx
#preflight 61f471cc6b5aea38e5c64cb6

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18782116 in //UE5/Release-5.0/... via CL 18782123 via CL 18782168
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v903-18687472)

[CL 18782178 by devin doucette in ue5-main branch]
2022-01-28 20:34:31 -05:00
zousar shaker
887b43ee9a Avoid synchronization bug involving DDC memory backend and value-locking that can occur during disable of the memory backend.
#rb danny.couture
#preflight 61f461496b5aea38e5c43f49

#ROBOMERGE-AUTHOR: zousar.shaker
#ROBOMERGE-SOURCE: CL 18779214 in //UE5/Release-5.0/... via CL 18780677 via CL 18781162
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v903-18687472)

[CL 18781198 by zousar shaker in ue5-main branch]
2022-01-28 19:12:55 -05:00
zousar shaker
9c291342c6 Fix accidental copy of data being fetched by Zen DDC due to the behavior of TArray64->TArray move operations.
#rb devin.doucette
#preflight 61f4618c801201ab38966132

#ROBOMERGE-AUTHOR: zousar.shaker
#ROBOMERGE-SOURCE: CL 18779176 in //UE5/Release-5.0/... via CL 18780674 via CL 18781161
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v903-18687472)

[CL 18781193 by zousar shaker in ue5-main branch]
2022-01-28 19:12:47 -05:00
devin doucette
1838ebc31e DDC: Fixed legacy cache fill behavior in the new hierarchy
When a node with StopStore is present and contains the data, but comes later in the hierarchy than the node that first found the data, puts to subsequent nodes were not being disabled.

#rb Zousar.Shaker
#rnx
#preflight 61f433b16b5aea38e5bf86ca

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18778611 in //UE5/Release-5.0/... via CL 18780649 via CL 18780924
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v903-18687472)

[CL 18781004 by devin doucette in ue5-main branch]
2022-01-28 19:01:07 -05:00
devin doucette
625f0006ab DDC: Removed default completion callbacks for private cache put functions
#rb Zousar.Shaker
#rnx
#preflight 61f394571f12674afbd55579

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18771213 in //UE5/Release-5.0/... via CL 18771244 via CL 18771361
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v903-18687472)

[CL 18771392 by devin doucette in ue5-main branch]
2022-01-28 10:22:45 -05:00
devin doucette
e73e078093 DDC: Updated the async cache wrapper to allow async execution in the wrapped cache store
#rb Zousar.Shaker
#rnx
#preflight 61f392ce74510448a6711ab5

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18771211 in //UE5/Release-5.0/... via CL 18771240 via CL 18771357
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v903-18687472)

[CL 18771382 by devin doucette in ue5-main branch]
2022-01-28 10:22:36 -05:00
devin doucette
df74509d41 DDC: Added a new async cache hierarchy type
This new cache hierarchy does not require the cache stores within it to complete every operation synchronously. When the child cache stores start to do async I/O, this hierarchy will support that instead of blocking like the existing hierarchy does.

This is a snapshot of the work in progress that is only implemented for the functions in ILegacyCacheStore.

#rb Zousar.Shaker
#rnx
#preflight 61f37e487a7f0d39ddd2d38b

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18768547 in //UE5/Release-5.0/... via CL 18768556 via CL 18768581
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v903-18687472)

[CL 18768594 by devin doucette in ue5-main branch]
2022-01-28 00:58:19 -05:00
devin doucette
8bc347b0be DDC: Fixed TryToPrefetch when there is not one backend containing every key
#rb Zousar.Shaker
#rnx
#preflight 61f327371f12674afbc5be32

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18768073 in //UE5/Release-5.0/... via CL 18768076 via CL 18768140
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v903-18687472)

[CL 18768147 by devin doucette in ue5-main branch]
2022-01-27 23:16:00 -05:00
devin doucette
b6bc462f04 DDC: Made ICacheStore private and removed ICache::CancelAll
#rb Zousar.Shaker
#rnx
#preflight 61f2da78801201ab3862d5ac

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18756063 in //UE5/Release-5.0/... via CL 18756070 via CL 18758707
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v903-18687472)

[CL 18760714 by devin doucette in ue5-main branch]
2022-01-27 15:43:52 -05:00
devin doucette
5b730ad9d0 DDC: Added LegacyDebugOptions and LegacyStats to ILegacyCacheStore
#rb Zousar.Shaker
#rnx
#preflight 61f1bcafbe0f0e0a623e8a15

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18745532 in //UE5/Release-5.0/... via CL 18745716 via CL 18746304
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v903-18687472)

[CL 18746526 by devin doucette in ue5-main branch]
2022-01-26 17:32:11 -05:00
devin doucette
b412a532ed DDC: Added MakeResponse(EStatus) to the cache request types
#rb Zousar.Shaker
#rnx
#preflight 61f1b88fc0033b39f89c8d84

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18744557 in //UE5/Release-5.0/... via CL 18744970 via CL 18745683
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v903-18687472)

[CL 18745958 by devin doucette in ue5-main branch]
2022-01-26 17:07:56 -05:00
devin doucette
f8b46f197a DDC: Moved FBackendDebugOptions, ESpeedClass, LogDerivedDataCache to prepare to extend ILegacyCacheStore
#rb Zousar.Shaker
#rnx
#preflight 61f1b7c1fd5285142b4b3b01

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18744365 in //UE5/Release-5.0/... via CL 18744960 via CL 18745676
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v903-18687472)

[CL 18745954 by devin doucette in ue5-main branch]
2022-01-26 17:07:47 -05:00
zousar shaker
109bb67a95 Fix bug where serialization of delimeter character for CachePolicy serialization (comma ',') was being serialized as an int32 ("44") due to mixing of UTF8 builder and TCHAR character.
#rb matt.peters
#preflight 61f1997573238441cb66c9ea

#ROBOMERGE-AUTHOR: zousar.shaker
#ROBOMERGE-SOURCE: CL 18740488 in //UE5/Release-5.0/... via CL 18740829 via CL 18742806
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v903-18687472)

[CL 18743180 by zousar shaker in ue5-main branch]
2022-01-26 15:35:13 -05:00
matt peters
c19bac1163 FCacheKey: Add operator<< to write to an FCbWriter.
FCacheRecord: Add Save function that allows appending to an existing package.
#preflight 61f174dcf8088a3d298c6e48

#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 18736675 in //UE5/Release-5.0/... via CL 18736693 via CL 18737351
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v903-18687472)

[CL 18737679 by matt peters in ue5-main branch]
2022-01-26 12:31:58 -05:00
zousar shaker
b6a8902058 Change the DDC Http backend to ensure that when downloading from the /refs and /compressed-blob routes, that the Http return code 404 is not logged at display level, but instead treated as an expected error code because unlike the /c route, they use 404 to indicate a cache miss.
#rb devin.doucette
#preflight 61f0dce8ff453b751b5130a0

#ROBOMERGE-AUTHOR: zousar.shaker
#ROBOMERGE-SOURCE: CL 18732663 in //UE5/Release-5.0/... via CL 18732713 via CL 18732743
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v903-18687472)

[CL 18732763 by zousar shaker in ue5-main branch]
2022-01-26 00:46:18 -05:00