Commit Graph

583 Commits

Author SHA1 Message Date
Devin Doucette
fa8f3dbe82 DDC: Fixed duplicate puts caused by the in-flight memory cache
Without using StopGetStore, get hits on the in-flight cache would propagate as puts to subsequent nodes in the hierarchy. Since the in-flight cache is only used as temporary storage while a put is in flight, this caused duplicates of the in-flight puts, which manifests as move collision log spam from the file system cache store.

#preflight 625d772d691f49969ea93249
#rb Zousar.Shaker
#rnx

[CL 19785647 by Devin Doucette in ue5-main branch]
2022-04-18 11:06:05 -04:00
Devin Doucette
b74b8b9227 DDC: Fixed incorrect cache store flags on dynamically mounted pak files
#preflight 625d771248670f31a61f0def
#rb Zousar.Shaker
#rnx

[CL 19785545 by Devin Doucette in ue5-main branch]
2022-04-18 10:57:14 -04:00
Devin Doucette
14583c8432 DDC: Back out 19757142 because it broke cache puts
#preflight skip
#rb none
#rnx

[CL 19763306 by Devin Doucette in ue5-main branch]
2022-04-14 17:05:38 -04:00
Devin Doucette
e35dc9fd40 DDC: Fixed duplicate puts caused by the in-flight memory cache
Without using StopStore, get hits on the in-flight cache would propagate as puts to subsequent nodes in the hierarchy. Since the in-flight cache is only used as temporary storage while a put is in flight, this caused duplicates of the in-flight puts, which manifests as move collision log spam from the file system cache store.

#preflight 625835d97f628a9018d8acf6
#rb Zousar.Shaker
#rnx

[CL 19757142 by Devin Doucette in ue5-main branch]
2022-04-14 11:06:11 -04:00
Devin Doucette
83f49f7dce DDC: Changed FileSystem PutValue to match Put
Except when replacing an existing value, load the existing value and check for determinism, and replace the existing value in the presence of an invalid value or non-deterministic value with missing content. This resolves move collisions being reported when a value existed without its content.

#preflight 62582c69946114248db5561f
#rb Zousar.Shaker
#rnx

[CL 19756384 by Devin Doucette in ue5-main branch]
2022-04-14 10:32:14 -04:00
Devin Doucette
37820d6bbc DDC: Fixed incorrect skipping of get propagation past nodes with the flag StopStore
The wrong response status was being checked, and caused subsequent nodes to be skipped even if the StopStore node did not contain the data, if any previous node contained the data.

#preflight 62582d67946114248db62d58
#rb Zousar.Shaker
#rnx

[CL 19756375 by Devin Doucette in ue5-main branch]
2022-04-14 10:31:55 -04:00
Devin Doucette
f1c81d6d61 DDC: Reset the build definition later in the build job to ease debugging
#preflight 62582c26044bab1525fca794
#rb Zousar.Shaker
#rnx

[CL 19756364 by Devin Doucette in ue5-main branch]
2022-04-14 10:31:33 -04:00
charles bloom
7b58069f70 DDC Verify cache miss is just a message not a warning
#rb devin.doucette
#preflight none

[CL 19648508 by charles bloom in ue5-main branch]
2022-04-06 11:43:57 -04:00
Zousar Shaker
28c6283a57 2nd submit with fix for lifetime issue:
Allow async operations for Jupiter DDC backend.

#rb devin.doucette
#jira UE-144412
#preflight 624c5fb68d1db441a90f23fe

[CL 19631067 by Zousar Shaker in ue5-main branch]
2022-04-05 16:43:41 -04:00
Devin Doucette
8c27714a84 DDC: Added ConvertLegacyCacheKey to construct new cache keys from legacy cache keys
The function is shared with the legacy cache store to allow callers of the legacy API to migrate to the value API using this conversion function without invalidating existing cache keys.

#preflight 624c9ab68d1db441a913137e
#rb Zousar.Shaker

[CL 19630681 by Devin Doucette in ue5-main branch]
2022-04-05 16:27:17 -04:00
Zousar Shaker
f34273a349 [Backout] - CL19628588
#fyi Zousar.Shaker
Original CL Desc
-----------------------------------------------------------------
Allow async operations for Jupiter DDC backend.

#rb devin.doucette
#jira UE-144412
#preflight 624c5fb68d1db441a90f23fe

[CL 19629334 by Zousar Shaker in ue5-main branch]
2022-04-05 15:17:17 -04:00
Zousar Shaker
56e179a5f4 Allow async operations for Jupiter DDC backend.
#rb devin.doucette
#jira UE-144412
#preflight 624c5fb68d1db441a90f23fe

[CL 19628588 by Zousar Shaker in ue5-main branch]
2022-04-05 14:18:39 -04:00
charles bloom
9754c6cb2c Make DDC-Verify successes Verbose
#rb devin.doucette
#preflight none

[CL 19592154 by charles bloom in ue5-main branch]
2022-04-01 14:01:04 -04:00
Devin Doucette
455b4ece9e DDC: Extended -DDC-MissTypes to match -DDC-Verify
In the examples below, <Node> is the name of a node in the cache store graph, or All to apply to every node.
-DDC-<Node>-MissTypes=StaticMesh+Texture will simulate a miss on every access to to static mesh and texture keys.
-DDC-<Node>-MissTypes=StaticMesh@12.5+Texture will simulate misses on 12.5% of accesses to static mesh keys and every access to texture keys.
-DDC-<Node>-MissRate=5 -DDC-<Node>-MissTypes=Texture will simulate misses on every access to texture keys and 5% of other keys.
-DDC-<Node>-MissRate=5 will simulate misses on 5% of keys.
-DDC-<Node>-MissSalt=PositiveInt32 will set the salt used to match keys to simulate misses on. A key always has the same simulated miss behavior with the same salt.

#preflight 6244d102b33098a72dc136df
#rb Zousar.Shaker

[CL 19572200 by Devin Doucette in ue5-main branch]
2022-03-31 10:06:47 -04:00
Devin Doucette
cebf4161f9 DDC: Added -DDC-Verify arguments to control determinism verification
-DDC-Verify will verify determinism on every access to the cache.
-DDC-Verify=StaticMesh+Texture will verify determinism on every access static mesh and texture keys.
-DDC-Verify=StaticMesh@12.5+Texture will verify determinism on 12.5% of accesses to static mesh keys and every access to texture keys.
-DDC-VerifyRate=5 -DDC-Verify=Texture will verify detereminism on every access to texture keys and 5% of other keys.
-DDC-VerifyRate=5 -DDC-Verify will verify determinism on 5% of keys.
-DDC-VerifySalt=PositiveInt32 will set the salt used to match keys to verify. A key always has the same verification behavior with the same salt.

#preflight 6244ad99637925b5d3bc7fab
#rb Zousar.Shaker

[CL 19563192 by Devin Doucette in ue5-main branch]
2022-03-30 15:51:28 -04:00
Zousar Shaker
4b6d3f3a0a Fix bug where the Content-Type header had multiple conflicting entries when posting compact binary during Horde Storage communications:
+		[1]	L"Content-Type: application/x-ue-cb"	FString
+		[2]	L"Content-Type: application/x-www-form-urlencoded"	FString

#rb devin.doucette
#rnx
#preflight 623bfe051302f69e9a9b5e1e

[CL 19492306 by Zousar Shaker in ue5-main branch]
2022-03-24 01:28:28 -04:00
Zousar Shaker
03207d24ea Allow OAuth access scope to be configurable for a Horde Storage DDC store. Supplied by partners.
#rb devin.doucette
#rnx
#preflight 623b9723c73745c600c264e6

[CL 19487258 by Zousar Shaker in ue5-main branch]
2022-03-23 18:05:53 -04:00
Devin Doucette
436b6a6461 DDC: Added IRequestOwner::LaunchTask
- Added IRequestOwner::LaunchTask to launch a task as a request in the request owner.
- Renamed ExecuteInCacheThreadPool to LaunchTaskInCacheThreadPool.
- Made GCacheThreadPool private to one source file.
- Changed task scheduling throughout DDC to use these functions where appropriate.

#preflight 623b846d8900c14eecd4daa4
#rb Zousar.Shaker

[CL 19485686 by Devin Doucette in ue5-main branch]
2022-03-23 16:56:39 -04:00
Johan Torp
175fbdc61b Optimized BuildCacheKey using FAsciiSet for ~4x speedup
Estimated to save 45s in a 1h50m cook of a larger internal project

#rb devin.doucette
#preflight 6238648a10251d53d55d3d12

[CL 19450563 by Johan Torp in ue5-main branch]
2022-03-21 07:56:39 -04:00
steve robb
87fb605867 TCHAR array init fixes for UTF-8 mode.
#rb devin.doucette
#preflight 6231e536e2541b4ff3af6cd2

#ROBOMERGE-AUTHOR: steve.robb
#ROBOMERGE-SOURCE: CL 19405353 via CL 19408955 via CL 19419621 via CL 19419706
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v930-19419903)

[CL 19420584 by steve robb in ue5-main branch]
2022-03-17 10:29:53 -04:00
Devin Doucette
1318804bb2 DDC: Fixed timing scopes for FileSystem, PakFile, S3
Edited the timing scopes to exclude invocation of the completion callback.

#preflight 62325890262232d86f812368
#rb Zousar.Shaker
#rnx

[CL 19419775 by Devin Doucette in ue5-main branch]
2022-03-17 09:07:20 -04:00
devin doucette
e3f0c1564c DDC: Fixed handling of metadata in the memory cache
#jira UE-146033
#preflight 62321041ac0173aa4131355a
#rb Matt.Peters, Zousar.Shaker
#rnx

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 19408655 in //UE5/Release-5.0/... via CL 19409276
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v926-19321884)

[CL 19410469 by devin doucette in ue5-main branch]
2022-03-16 15:31:59 -04:00
Devin Doucette
d8c983bbe6 DDC: Optimized legacy cache key validation and shortening overhead
- Validate and shorten legacy keys lazily.
- Change to ValueOnly mode for PakFile and Zen.

#preflight 623235998a6037989395baf2
#rb Zousar.Shaker
#rnx

[CL 19410386 by Devin Doucette in ue5-main branch]
2022-03-16 15:28:29 -04:00
zousar shaker
a33be10d6b Fix issue (using server-side feature in Horde Storage) with inconsistency to DDC content from Horde Storage.
This would manifest as a cache entry being deemed to "exist", but later when you attempt to fetch it, it would not exist.

#rb devin.doucette
[FYI] joakim.lindqvist
#jira UE-145701
#lockdown simon.tourangeau
#preflight 6231f8866e25767a2189e99a

#ROBOMERGE-AUTHOR: zousar.shaker
#ROBOMERGE-SOURCE: CL 19407779 in //UE5/Release-5.0/... via CL 19408966
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v926-19321884)

[CL 19409941 by zousar shaker in ue5-main branch]
2022-03-16 15:13:23 -04:00
Devin Doucette
6c0885e988 DDC: Fixed the async cache hierarchy to advance to the next node when nodes are async
#preflight 622f9ce380715dc6dadebf59
#rb Zousar.Shaker
#rnx

[CL 19377851 by Devin Doucette in ue5-main branch]
2022-03-14 16:15:03 -04:00