Commit Graph

37 Commits

Author SHA1 Message Date
Marc Audy
0c3be2b6ad Merge Release-Engine-Staging to Test @ CL# 18240298
[CL 18241953 by Marc Audy in ue5-release-engine-test branch]
2021-11-18 14:37:34 -05:00
devin doucette
53f316587b DDC: New filesystem cache store directory layout and file format
- Implemented only for the structured cache initially.
- Records are stored as compact binary packages with payloads stored within the record if small enough.
- Records are stored in Records/{Bucket}/{01}/{23}/{456789abcdef0123456789abcdef01234567} based on the hash in the cache key.
- Content that exceeds the size allowed for storage within the record is stored in Content/{01}/{23}/{456789abcdef0123456789abcdef01234567} based on the hash of the content.
- Both records and content are written to disk with their hash as a suffix, and are always checked on load.
- Maximum size of compressed data stored in the package can be configured by MaxRecordSizeKB and MaxValueSizeKB.
- Maintenance of the cache, to delete unused files, has been rewritten and can now scan ~175k files/second on modern hardware, though is limited to 10k files/second in the config to avoid adding too much overhead.

#rb Zousar.Shaker
#preflight 618b75c14a0c609a29204032
#preflight 618bfb5ec717ba4e929b7ac0

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18133085 in //UE5/Release-5.0/... via CL 18133356
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v889-18060218)
#ROBOMERGE[STARSHIP]: UE5-Main

[CL 18133430 by devin doucette in ue5-release-engine-test branch]
2021-11-10 13:08:00 -05:00
devin doucette
10df2328dd DDC: Added structured cache support to the throttle wrapper
#rb Zousar.Shaker
#rnx
[FYI] Danny.Couture

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 17872207 in //UE5/Release-5.0/... via CL 17872235
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v883-17842818)
#ROBOMERGE[STARSHIP]: UE5-Main

[CL 17874113 by devin doucette in ue5-release-engine-test branch]
2021-10-20 11:50:36 -04:00
devin doucette
cf19925e08 DDC: Made CacheRecord serialized field order consistent with other types
#rb none
#rnx

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 17583018 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v871-17566257)

[CL 17583024 by devin doucette in ue5-release-engine-test branch]
2021-09-21 11:31:48 -04:00
per larsson
c6fafd4b7a DDC2: Use compressed buffer as attachment
#rb Devin.Doucette
#rnx

#ROBOMERGE-AUTHOR: per.larsson
#ROBOMERGE-SOURCE: CL 17580737 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v871-17566257)

[CL 17580740 by per larsson in ue5-release-engine-test branch]
2021-09-21 04:51:59 -04:00
devin doucette
ec3fa9680f CompositeBuffer: Renamed Flatten() to ToShared()
#rb none
#rnx
[FYI] Zousar.Shaker
#preflight 613f872b3bbb48000114081d

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 17495384 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v870-17433530)

[CL 17495398 by devin doucette in ue5-release-engine-test branch]
2021-09-13 15:16:06 -04:00
devin doucette
9c01e2fc04 DDC: Moved FPayloadId into a dedicated header
#rb Zousar.Shaker
#rnx

#ROBOMERGE-SOURCE: CL 17426140 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v865-17346139)

[CL 17426194 by devin doucette in ue5-release-engine-test branch]
2021-09-03 16:30:53 -04:00
devin doucette
bd60f3ef1b DDC: Removed ICacheFactory in favor of exported functions
FCacheBucket is now always exposed as FAnsiStringView.

#rb Zousar.Shaker
#rnx

#ROBOMERGE-SOURCE: CL 17084521 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)

[CL 17084549 by devin doucette in ue5-release-engine-test branch]
2021-08-06 12:53:08 -04:00
devin doucette
733d9c1a97 DDC: Added conversions between FPayloadId and FCbObjectId
#rb Zousar.Shaker
#rnx

#ROBOMERGE-SOURCE: CL 17071877 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)

[CL 17071890 by devin doucette in ue5-release-engine-test branch]
2021-08-05 13:37:13 -04:00
devin doucette
bfda02d384 DDC: Replaced FRequest with FRequestGroup to handle nested and chained requests more robustly
Any function that may create a request now has a IRequestOwner& parameter, and uses the Begin and End functions on the owner to manage the lifetime of any requests that it creates, as well as using End to invoke the completion callback for any request which has one.

The new FRequestBarrier may be used to block a group from being considered complete in a scope where more requests may be added to it.

#rb Matt.Peters
#rnx
#preflight 6109b5c403d303000144cce5
#preflight 610acf7103d30300016fda94

#ROBOMERGE-SOURCE: CL 17060470 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v850-17047176)

[CL 17060649 by devin doucette in ue5-release-engine-test branch]
2021-08-04 18:08:50 -04:00
devin doucette
c29fdcf532 DDC: Compress payloads using Oodle Mermaid VeryFast
#rb Zousar.Shaker
#rnx

#ROBOMERGE-SOURCE: CL 16956293 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v838-16927207)

[CL 16956298 by devin doucette in ue5-release-engine-test branch]
2021-07-26 10:29:10 -04:00
Devin Doucette
eb2cda6fba Added conversions and constructors to/from 12-byte arrays for FCbObjectId and DerivedData::FPayloadId
#rb Zousar.Shaker
#rnx

[CL 16536271 by Devin Doucette in ue5-main branch]
2021-06-02 13:06:45 -04:00
Devin Doucette
bad1b2bfde DDC: Extended the cache record builder to take composite buffers
#rb Zousar.Shaker
#rnx

[CL 16534280 by Devin Doucette in ue5-main branch]
2021-06-02 10:33:08 -04:00
Devin Doucette
490a4a0db2 CompactBinary: Restricted compact binary attachments to objects to simplify package management
#rb Zousar.Shaker
#rnx
#preflight 60b71447ae46a1000144f1d9

[CL 16533710 by Devin Doucette in ue5-main branch]
2021-06-02 09:33:55 -04:00
Devin Doucette
d0e04a4a76 DDC: Factored out cache bucket name validation and aligned it more closely with build function name validation
[CL 16509861 by Devin Doucette in ue5-main branch]
2021-05-31 12:43:42 -04:00
Devin Doucette
600ee5377a DDC: Added serialization of FCacheRecord to/from FCbPackage
#rb Devin.Doucette
#rnx

#ushell-cherrypick of 16476127 by Matt.Peters

[CL 16483925 by Devin Doucette in ue5-main branch]
2021-05-27 10:16:47 -04:00
Devin Doucette
c20b3b16cf DDC: Made name and function asserts consistent and reordered members for debuggability
#rb Zousar.Shaker
#rnx
#preflight 60a6d5afb7ccb00001d42b6e

[CL 16414174 by Devin Doucette in ue5-main branch]
2021-05-20 21:54:47 -04:00
Devin Doucette
09748c85e3 DDC: Added a missing #include
#rb trivial
#rnx

[CL 16326010 by Devin Doucette in ue5-main branch]
2021-05-13 23:22:40 -04:00
Devin Doucette
8310ac341e DDC: Changed the builder types to delete themselves on Build()
#rb Zousar.Shaker
#rnx

[CL 16317751 by Devin Doucette in ue5-main branch]
2021-05-13 16:09:45 -04:00
Devin Doucette
dd9e1ec58b DDC: Added support for cache records with null data
#rb Zousar.Shaker
#rnx
#preflight 609b6698b16d45000155cb04

[CL 16298475 by Devin Doucette in ue5-main branch]
2021-05-12 09:28:05 -04:00
Devin Doucette
536b5fa4dc DDC: Updated documentation and removed unnecessary explicit hash construction
#rb trivial
#rnx

[CL 16207037 by Devin Doucette in ue5-main branch]
2021-05-05 10:15:34 -04:00
Devin Doucette
e39324b347 DDC: Changed FCacheRecord to be reference-counted instead of uniquely owned
#rb Zousar.Shaker
#rnx

[CL 16184511 by Devin Doucette in ue5-main branch]
2021-05-03 11:41:14 -04:00
Devin Doucette
84641e44fb DDC: Added an assert to FCacheRecordBuilder::BuildAsync to require a callback
#rb trivial
#rnx

[CL 16148043 by Devin Doucette in ue5-main branch]
2021-04-28 16:05:08 -04:00
Devin Doucette
e7b98e5582 DDC: Added FCacheRecord KeyFuncs, FCacheRecord::GetPayload, and FPayload(FPayloadId)
#rb Zousar.Shaker
#rnx
#preflight 608056e0d4e67000017ac573

[CL 16078733 by Devin Doucette in ue5-main branch]
2021-04-21 14:56:45 -04:00
Devin Doucette
cf7d6505f3 DDC: Tweak names and namespace formatting for consistency
#rb trivial
#rnx
#preflight 607d0e236e83200001593a45

[CL 16050392 by Devin Doucette in ue5-main branch]
2021-04-19 09:56:34 -04:00