Commit Graph

531 Commits

Author SHA1 Message Date
devin doucette
c2a46c2242 DDC: Fixed a test failure for small values in the file system cache
#jira UE-141632
#preflight 6202b6c9bf28c072cbf308ce
#lockdown Mark.Lintott
#rb Matt.Peters
#rnx

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18909346 in //UE5/Release-5.0/... via CL 18909604 via CL 18909733
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v915-18905420)

[CL 18909753 by devin doucette in ue5-main branch]
2022-02-08 16:07:34 -05:00
devin doucette
2ae070347f DDC: Fixed a memory leak in the cache
#jira UE-141335
#preflight 620294f5bd4f846436e5f4c6
#lockdown Mark.Lintott
#rb Matt.Peters
#rnx

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18903655 in //UE5/Release-5.0/... via CL 18903892 via CL 18904098
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v912-18901109)

[CL 18904146 by devin doucette in ue5-main branch]
2022-02-08 12:17:15 -05:00
zousar shaker
df06aa0d18 Allow for region pinning when connecting to Jupiter. Fix value printing error. Increase concurrent PUT operations from 2 to 4 when batching is enabled.
#preflight 6201d9c7384d6a94e8fc20d7
#jira UE-141733
#rb devin.doucette

#ROBOMERGE-AUTHOR: zousar.shaker
#ROBOMERGE-SOURCE: CL 18899614 in //UE5/Release-5.0/... via CL 18899644 via CL 18899680
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v910-18824042)

[CL 18899684 by zousar shaker in ue5-main branch]
2022-02-08 01:56:17 -05:00
devin doucette
c03d680063 DDC: Fixed propagation of cache values when a value policy contains SkipData
#jira UE-141576
#preflight 61fe001fdb3d0b17a933391c
#lockdown Mark.Lintott
#rb Zousar.Shaker
#rnx

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18896581 in //UE5/Release-5.0/... via CL 18896633 via CL 18896789
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v910-18824042)

[CL 18897023 by devin doucette in ue5-main branch]
2022-02-07 20:06:56 -05:00
danny couture
dcff1b48cb Fix infinite wait on async compilation when in pause mode by using an unpausable priority when forced compilation is required to wait
#rb Francis.Hurteau
#jira ue-141319
#preflight 62010b324346561cadf30657
#lockdown mark.lintott

#ROBOMERGE-AUTHOR: danny.couture
#ROBOMERGE-SOURCE: CL 18886756 in //UE5/Release-5.0/... via CL 18886809 via CL 18887172
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v910-18824042)

[CL 18887216 by danny couture in ue5-main branch]
2022-02-07 12:22:00 -05:00
devin doucette
0818b14470 DDC: Fixed incorrect callbacks in put batches with failures
Also adds the intended inline allocators for arrays that were missing them.

#jira UE-134381
#lockdown Mark.Lintott
#preflight 61fd998ff370b0d3111fad17
#rb Matt.Peters
#rnx

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18874174 in //UE5/Release-5.0/... via CL 18874208 via CL 18874924
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v910-18824042)

[CL 18874951 by devin doucette in ue5-main branch]
2022-02-04 17:24:08 -05:00
devin doucette
e963d14d2a DDC: Fixed the Zen cache being accidentally made read-only
#jira UE-134381
#preflight 61fcb3a2b434724823d9f733
#lockdown Mark.Lintott
#rb Zousar.Shaker
#rnx

#ROBOMERGE-OWNER: Devin.Doucette
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-COMMAND: _robomerge UE5-Main
#ROBOMERGE-SOURCE: CL 18860637 in //UE5/Release-5.0/... via CL 18860644 via CL 18860646
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v910-18824042)

[CL 18860649 by devin doucette in ue5-main branch]
2022-02-04 00:27:54 -05:00
zousar shaker
2903c5331f Fix static analysis error in CacheStoreTest.cpp.
#rb matt.peters
#jira UE-141032
#lockdown aurel.cordonnier
#preflight 61fc4d11e058822bdaaaefcb

#ROBOMERGE-AUTHOR: zousar.shaker
#ROBOMERGE-SOURCE: CL 18853180 in //UE5/Release-5.0/... via CL 18853206 via CL 18853946
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v910-18824042)

[CL 18853959 by zousar shaker in ue5-main branch]
2022-02-03 17:43:34 -05:00
devin doucette
c7954aebce DDC: Switched to the async cache hierarchy and fully implemented the verify wrapper
The async cache hierarchy:
- Is required to add compression to the legacy cache by forwarding LegacyPut/LegacyGet to PutValue/GetValue.
- Is always present in the graph, unlike the previous cache hierarchy, which will allow significant simplification of leaf cache stores.
- Allows for the leaf cache store nodes to operate asynchronously without blocking a worker thread like the previous cache hierarchy.
- Shifts from controlling cache behavior by speed class to controlling cache behavior by local/remote classification, which is a critical distinction for a cache like Zen that can identify as both local and remote.
- Respects the local/remote and query/store flags in the cache policy.
- Does not fully implement the partial record cache policy at this time.
- Does not propagate records or values in GetChunks, which will be added in a future release.

The verify wrapper was previously missing an implementation for the new cache interface. This version is more efficient than the previous because requests through the new cache interface can compare data without loading it from storage, and load it only when a mismatch has been detected, to dump it to disk.

#jira UE-134381
#lockdown Mark.Lintott
#preflight 61fc32ac0a50c2606f266388
#rb Zousar.Shaker
#rnx

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18851861 in //UE5/Release-5.0/... via CL 18851943 via CL 18852169
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v910-18824042)

[CL 18852184 by devin doucette in ue5-main branch]
2022-02-03 16:49:32 -05:00
devin doucette
07e3b20f45 DDC: Fixed a missing decrement of the async completion counter
Cancellation of requets in the async put wrapper was not decrementing the async completion counter, which causes a hang on exit if any cancellations have occurred. Any cook that builds a lot of meshes changes the timing of other operations enough that they are often canceled.

#jira UE-140939
#preflight 61fa2af34404d5fade10bd72
#lockdown Mark.Lintott
#rb Zousar.Shaker
#rnx

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18824202 in //UE5/Release-5.0/... via CL 18825870 via CL 18826297
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v910-18824042)

[CL 18826347 by devin doucette in ue5-main branch]
2022-02-02 09:54:07 -05:00
zousar shaker
b9fdfa4036 GetChunks value implementation for Jupiter.
#rb devin.doucette
#preflight 61f9db609e4d23cd93b7b28d

#ROBOMERGE-AUTHOR: zousar.shaker
#ROBOMERGE-SOURCE: CL 18819720 in //UE5/Release-5.0/... via CL 18819729 via CL 18822895
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v910-18824042)

[CL 18825048 by zousar shaker in ue5-main branch]
2022-02-02 08:18:44 -05:00
devin doucette
20ea07085b DDC: Added GetChunks support to the new cache hierarchy
Includes policy fixes for other functions.

#rb Zousar.Shaker
#rnx
#preflight 61f9b31c9e4d23cd93b1dad3

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18817404 in //UE5/Release-5.0/... via CL 18817411 via CL 18822862
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v910-18824042)

[CL 18824693 by devin doucette in ue5-main branch]
2022-02-02 07:58:00 -05:00
devin doucette
7802a87e20 DDC: Fixed GetChunks in the pak file cache to fail gracefully when it receives a value request
#rb Zousar.Shaker
#rnx
#preflight 61f9acf04404d5fadefe2509

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18816538 in //UE5/Release-5.0/... via CL 18816555 via CL 18822829
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v910-18824042)

[CL 18824626 by devin doucette in ue5-main branch]
2022-02-02 07:55:06 -05:00
devin doucette
8a6e7cabc1 DDC: Changed cache policy string conversion to use optimal encoding
Parsing uses UTF-8 to avoid conversion of UTF-8 and ANSI strings, and conversion to a string uses ANSI to avoid expensive conversion from UTF-16.

Includes a change to mask out record-only flags from values in FCacheRecordPolicy::Transform.

#rb Matt.Peters
#rnx
#preflight 61f9934d162e627eab43ebfd

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18814354 in //UE5/Release-5.0/... via CL 18814383 via CL 18822771
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v910-18824042)

[CL 18824304 by devin doucette in ue5-main branch]
2022-02-02 07:36:25 -05:00
matt peters
f2eed1a838 CacheStoreTest: Add a test case that catches ValuePolicies being parsed correctly.
#rb Zousar.Shaker
#rnx
#preflight 61f98f024404d5fadefb1c0b

#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 18813550 in //UE5/Release-5.0/... via CL 18813562 via CL 18822761
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v910-18824042)

[CL 18824293 by matt peters in ue5-main branch]
2022-02-02 07:35:30 -05:00
zousar shaker
c6907b8736 Implement DDC2 Value API for Jupiter backend.
#rb devin.doucette
#preflight 61f98cc6db42673a6008471b

#ROBOMERGE-AUTHOR: zousar.shaker
#ROBOMERGE-SOURCE: CL 18813356 in //UE5/Release-5.0/... via CL 18813377 via CL 18822759
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v910-18824042)

[CL 18824291 by zousar shaker in ue5-main branch]
2022-02-02 07:35:19 -05:00
devin doucette
c3443800ec DDC: Converted modular feature names to constant lazy names
#rb Zousar.Shaker
#rnx
#preflight 61f97050fa6554dff3d6fa76

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18812871 in //UE5/Release-5.0/... via CL 18812927 via CL 18822743
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v909-18823584)

[CL 18823987 by devin doucette in ue5-main branch]
2022-02-02 06:26:22 -05:00
matt peters
c98e98aa5f ZenServer CachePolicy support.
Add batched Put and PutValue.
Update Batched Get to send the RecordPolicy with each key.
Update GetChunks to send the CachePolicy as string and to handle SkipData in the response.
Add more unit tests  for upstream behavior and GetChunks.
Update the CacheRecordPolicy Save/Load to match the new variable names on FCacheRecordPolicy.
Deploy new zenserver binaries with the required matching server-side changes.

#rb Per.Larsson,Devin.Doucette
#rnx
#preflight 61f94e5980608c7029b8fc5c

#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 18808743 in //UE5/Release-5.0/... via CL 18809399 via CL 18822531
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v908-18788545)

[CL 18823289 by matt peters in ue5-main branch]
2022-02-02 05:33:16 -05:00
devin doucette
d0a5f83709 DDC: Moved the delete batch in the new hierarchy next to the delete function
#rb trivial
#rnx
#preflight 61f8cf94a6632a34f3689784

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18805848 in //UE5/Release-5.0/... via CL 18807992 via CL 18821772
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v908-18788545)

[CL 18822130 by devin doucette in ue5-main branch]
2022-02-02 02:20:15 -05:00
devin doucette
da469aebf0 DDC: Added Put and Get support to the new cache hierarchy
#rb Zousar.Shaker
#rnx
#preflight 61f864a868795b2f458b0c91

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18805714 in //UE5/Release-5.0/... via CL 18807981 via CL 18821770
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v908-18788545)

[CL 18822126 by devin doucette in ue5-main branch]
2022-02-02 02:20:02 -05:00
devin doucette
3cb1585ced DDC: Changed "Type&& Get() &&" accessors to "Type Get() &&"
#rb Zousar.Shaker
#rnx
#preflight 61f80d4868795b2f45779b4b

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18792049 in //UE5/Release-5.0/... via CL 18792224 via CL 18792941
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v908-18788545)

[CL 18793084 by devin doucette in ue5-main branch]
2022-01-31 13:23:12 -05:00
devin doucette
52ca2f071b DDC: Added PutValue and GetValue support to the new cache hierarchy
#rb Zousar.Shaker
#rnx
#preflight 61f78b4e6a7c7b134f2f289e

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18791013 in //UE5/Release-5.0/... via CL 18791372 via CL 18791928
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v908-18788545)

[CL 18792105 by devin doucette in ue5-main branch]
2022-01-31 12:25:15 -05:00
devin doucette
169318850a DDC: Masked record-only cache policy flags out of the value policy
Fixing backends to respect the masked flags includes a few fixes to handling of ~Query by Put and PutValue.

#rb Matt.Peters
#rnx
#preflight 61f7f7ede55232619f887977

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18790481 in //UE5/Release-5.0/... via CL 18790769 via CL 18791304
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v908-18788545)

[CL 18791515 by devin doucette in ue5-main branch]
2022-01-31 11:52:13 -05:00
devin doucette
517947112b DDC: Reorganized cache policy code into its own files
- FCacheRecordPolicyBuilder now disallows adding value policies with flags that only make sense for records.
- FCacheRecordPolicyBuilder now disallows adding multiple value policies for the same value.
- FCacheRecordPolicyBuilder now skips addition of value policies that match the default policy.

#rb Matt.Peters
#rnx
#preflight 61f471cc6b5aea38e5c64cb6

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18782116 in //UE5/Release-5.0/... via CL 18782123 via CL 18782168
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v903-18687472)

[CL 18782178 by devin doucette in ue5-main branch]
2022-01-28 20:34:31 -05:00
zousar shaker
887b43ee9a Avoid synchronization bug involving DDC memory backend and value-locking that can occur during disable of the memory backend.
#rb danny.couture
#preflight 61f461496b5aea38e5c43f49

#ROBOMERGE-AUTHOR: zousar.shaker
#ROBOMERGE-SOURCE: CL 18779214 in //UE5/Release-5.0/... via CL 18780677 via CL 18781162
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v903-18687472)

[CL 18781198 by zousar shaker in ue5-main branch]
2022-01-28 19:12:55 -05:00