Commit Graph

463 Commits

Author SHA1 Message Date
devin doucette
35393bbb2b DDC: Split FPayload into separate FValue and FValueId types
A payload was conceptually a value with an ID. That has been formalized by removing the ID from the payload and having separate FValue and FValueId types. This separation cleans up the API in a few areas, and provides a more natural path to providing a basic key/value API.

#rb Zousar.Shaker
#rnx
#preflight 61d704c04c252480ca284d61

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18531844 in //UE5/Release-5.0/... via CL 18531856
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)

[CL 18531864 by devin doucette in ue5-release-engine-test branch]
2022-01-06 11:05:57 -05:00
devin doucette
48f68493ef DDC: Removed the Put/Get compatibility functions from ICache
#rb Zousar.Shaker
#rnx
#preflight 61d49a971a3fd09dcbc87978

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18511490 in //UE5/Release-5.0/... via CL 18511542
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)

[CL 18511622 by devin doucette in ue5-release-engine-test branch]
2022-01-04 14:38:48 -05:00
devin doucette
f0dd93d7cf DDC: Changed FCacheRecord::Load to take the record as an object instead of a view
#rb Zousar.Shaker
#rnx
#preflight 61d489feec35d1b940d21bbf

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18510417 in //UE5/Release-5.0/... via CL 18510464
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)

[CL 18510496 by devin doucette in ue5-release-engine-test branch]
2022-01-04 13:28:36 -05:00
devin doucette
f8fbec038e DDC: Added FSharedString to minimize copies of cache request names
#rb Zousar.Shaker
#rnx
#preflight 61d4831f932a02483cbba5c1

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18510411 in //UE5/Release-5.0/... via CL 18510430
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)

[CL 18510447 by devin doucette in ue5-release-engine-test branch]
2022-01-04 13:25:54 -05:00
devin doucette
8e3ba57e91 DDC: Moved the ICacheStore Context params to be a Name member of the request types and return them in the completion params
#rb Zousar.Shaker
#rnx
#preflight 61d3729eaf44b43628f4bdc9

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18507435 in //UE5/Release-5.0/... via CL 18507440
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)

[CL 18507451 by devin doucette in ue5-release-engine-test branch]
2022-01-04 09:26:33 -05:00
devin doucette
0f14d13c88 DDC: Fixed the logged cache summary stats
Stats from the root node must be a superset of stats from the rest of the nodes. The key length limiter is always the root node at present, and has been updated to include stats for the structured cache.

#rb Zousar.Shaker
#rnx
#preflight 61d318cc932a02483cb29797

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18501391 in //UE5/Release-5.0/... via CL 18501397
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)

[CL 18501400 by devin doucette in ue5-release-engine-test branch]
2022-01-03 11:24:10 -05:00
devin doucette
ffe8432dc4 DDC: Fixed static analysis warnings about null checks
#rb trivial
#rnx
#preflight 61d30e4d1f62d3ad4d46cb23

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18500919 in //UE5/Release-5.0/... via CL 18500934
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)

[CL 18500948 by devin doucette in ue5-release-engine-test branch]
2022-01-03 10:04:48 -05:00
devin doucette
c6113ec521 DDC: Fixed file system cache store put stats
#rb Zousar.Shaker
#rnx
#preflight 61bbb694c32d8309558303b9

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18480250 in //UE5/Release-5.0/... via CL 18481558
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)

[CL 18481828 by devin doucette in ue5-release-engine-test branch]
2021-12-16 19:57:53 -05:00
devin doucette
54235555db DDC: Moved FDerivedDataBackendInterface from Public to Private
The interface was in the public folder before 5.0, but was not usable from there in a meaningful way.

#rb Zousar.Shaker
#rnx
#preflight 61bbac388a62de8385c132c7

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18479761 in //UE5/Release-5.0/... via CL 18481541
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)

[CL 18481817 by devin doucette in ue5-release-engine-test branch]
2021-12-16 19:56:56 -05:00
devin doucette
14bcbbe4c6 DDC: Updated the stat dialog layout and split Zen to include its local stats and upstream stats
#rb Zousar.Shaker
#rnx
#preflight 61bba91ac32d830955820a47

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18479360 in //UE5/Release-5.0/... via CL 18481538
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)

[CL 18481812 by devin doucette in ue5-release-engine-test branch]
2021-12-16 19:56:35 -05:00
devin doucette
68dcdf74a1 DDC: Folded the corruption wrapper into the S3 cache
#rb Zousar.Shaker
#rnx
#preflight 61badf093961edbe76d77bf1

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18475378 in //UE5/Release-5.0/... via CL 18481136
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)

[CL 18481156 by devin doucette in ue5-release-engine-test branch]
2021-12-16 19:04:45 -05:00
devin doucette
73b904baed DDC: Extended the S3 cache with the ability to do local generation and decompression of bundles
#rb Zousar.Shaker
#preflight 61ba653d055f301345a1457a

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18471703 in //UE5/Release-5.0/... via CL 18471718
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)

[CL 18471721 by devin doucette in ue5-release-engine-test branch]
2021-12-15 17:23:24 -05:00
devin doucette
f7ff6e2de1 DDC: Fixed the file access log to write paths relative to the cache directory
#rb Bryan.Johnson
#rnx
#preflight 61ba63d7055f301345a0f56b

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18471469 in //UE5/Release-5.0/... via CL 18471470
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)

[CL 18471478 by devin doucette in ue5-release-engine-test branch]
2021-12-15 17:03:40 -05:00
devin doucette
844d30fa29 DDC: Fold the corruption wrapper into the file system cache store
#rb Zousar.Shaker
#rnx
#preflight 61b98c448f57c5c0c234fb17

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18466778 in //UE5/Release-5.0/... via CL 18466797
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)

[CL 18466801 by devin doucette in ue5-release-engine-test branch]
2021-12-15 09:30:12 -05:00
charles bloom
d6c8d22b81 Fix DDC should not sanitize $ characters because it makes $ characters
SanitizeCacheKey and ValidateCacheKey should check the same condition

#rb devin.doucette, danny.couture

#ROBOMERGE-AUTHOR: charles.bloom
#ROBOMERGE-SOURCE: CL 18455963 in //UE5/Release-5.0/... via CL 18455975
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v898-18417669)

[CL 18455987 by charles bloom in ue5-release-engine-test branch]
2021-12-14 12:35:14 -05:00
devin doucette
1ff24789c0 DDC: Added FPayload::RemoveData() to simplify converting payloads to references
#rb Zousar.Shaker
#rnx
#preflight 61b7b81ff42e2f02a8fda898

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18449038 in //UE5/Release-5.0/... via CL 18449046
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v897-18405271)

[CL 18449051 by devin doucette in ue5-release-engine-test branch]
2021-12-13 16:38:01 -05:00
devin doucette
559754c569 DDC: Removed accessors for raw data from FCacheRecord
Use GetValue().GetData().Decompress() in place of GetValue(), or the equivalent for GetAttachment(Id). The removed accessors kept a cache of the decompressed payload and required synchronization to be safe to call from multiple threads.

#rb Zousar.Shaker
#rnx
#preflight 61b7a41efc570891338e17aa

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18447623 in //UE5/Release-5.0/... via CL 18448002
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v897-18405271)

[CL 18448136 by devin doucette in ue5-release-engine-test branch]
2021-12-13 15:24:25 -05:00
devin doucette
8161902f38 DDC: Exposed payload compression block size through the cache record and build function interfaces
#rb Zousar.Shaker
#rnx
#preflight 61b7721bdd0e83c861a420d8

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18444327 in //UE5/Release-5.0/... via CL 18445345
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v897-18405271)

[CL 18445509 by devin doucette in ue5-release-engine-test branch]
2021-12-13 13:44:06 -05:00
devin doucette
4d164bdfd5 DDC: Added a uint64 UserData to ICacheStore requests to make completion callbacks more efficient
Requests to Put and Get now specify the policy independently for each record or key, but the implementations have not yet been updated to take this into account.

The existing Put and Get API has been maintained temporarily to allow the conversion to happen separately, and will be removed after callers have been converted to use the new API.

#rb Zousar.Shaker
#rnx
#preflight 61b2e550ee0de9822e2e84dc

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18443052 in //UE5/Release-5.0/... via CL 18445037
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v897-18405271)

[CL 18445201 by devin doucette in ue5-release-engine-test branch]
2021-12-13 13:32:28 -05:00
devin doucette
86dfb790ef DDC: Optimized GetChunks in the file system cache store to read only the requested range from disk
Contains changes to other structured cache file access to stream data from archives instead of loading the whole file into memory. The record package will now be split into separate allocations on load, rather than the previous behavior where the whole package was kept in memory as long as any part of it was referenced.

#rb Zousar.Shaker
#rnx
#preflight 61b11fd75238845473be0c30

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18413427 in //UE5/Release-5.0/... via CL 18413472
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v897-18405271)

[CL 18413485 by devin doucette in ue5-release-engine-test branch]
2021-12-08 17:15:36 -05:00
devin doucette
27c1393427 CompressedBuffer: Removed partial decompression from FCompressedBuffer now that FCompressedBufferReader is available
Requiring the use of a separate reader type makes it more likely that readers will be reused, and makes it easier to audit reader usage going forward. Reusing readers is desirable to reduce the number of large temporary allocations made during partial decompression of a buffer.

- Added FCompressedBuffer::Save(FArchive&) and renamed FromCompressed(FArchive&) to Load(FArchive&).
- Added FCompressedBufferReaderSourceScope to set a buffer source within a scope.
- Added proper bounds checks to FNoneDecoder.
- Store the header checksum on the decoder context to allow raw blocks to be reused across sources.
- Decode the header on the fly to avoid a temporary header allocation when the header is in contiguous memory.

#rb Zousar.Shaker
#rnx
#preflight 61a98d53800738dbfbc84c73

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18382211 in //UE5/Release-5.0/... via CL 18382310
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v896-18170469)

[CL 18382377 by devin doucette in ue5-release-engine-test branch]
2021-12-06 10:16:05 -05:00
devin doucette
cea98e6ab4 DDC: Split build diagnostics into messages and logs
Messages are deterministic output recorded explicitly by the build function. Logs will be automatically captured from the output log and are not necessarily deterministic. The presence of logs in a build output automatically disables caching of the build output.

#rb Zousar.Shaker
#rnx

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18286155 in //UE5/Release-5.0/... via CL 18286173
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)

[CL 18286201 by devin doucette in ue5-release-engine-test branch]
2021-11-24 13:25:25 -05:00
mark lintott
a90eaa1e87 Optimise FZenDerivedDataBackend::IsRemote() by checking the once on backend creation and caching the result
#jira UE-135791
#rb trivial
[FYI] zousar.shaker
#preflight 619e27c3aa4521f9e7b6b567

#ROBOMERGE-AUTHOR: mark.lintott
#ROBOMERGE-SOURCE: CL 18282332 in //UE5/Release-5.0/... via CL 18282340
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)

[CL 18282346 by mark lintott in ue5-release-engine-test branch]
2021-11-24 07:17:42 -05:00
Marc Audy
0c3be2b6ad Merge Release-Engine-Staging to Test @ CL# 18240298
[CL 18241953 by Marc Audy in ue5-release-engine-test branch]
2021-11-18 14:37:34 -05:00
devin doucette
a6c9c59388 DDC: Fixed the async put wrapper with -NoThreading
#rb Zousar.Shaker
#rnx
#preflight 6194654c34377daf864d4f80

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18223152 in //UE5/Release-5.0/... via CL 18223191
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)
#ROBOMERGE[STARSHIP]: UE5-Main

[CL 18223207 by devin doucette in ue5-release-engine-test branch]
2021-11-17 09:49:24 -05:00