Commit Graph

5 Commits

Author SHA1 Message Date
devin doucette
b958bf4af6 DDC: Fixed file system cache maintenance with -NoThreading
#rb Zousar.Shaker
#rnx

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18194101 in //UE5/Release-5.0/... via CL 18194138
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)
#ROBOMERGE[STARSHIP]: UE5-Main

[CL 18194155 by devin doucette in ue5-release-engine-test branch]
2021-11-15 13:56:02 -05:00
devin doucette
53f316587b DDC: New filesystem cache store directory layout and file format
- Implemented only for the structured cache initially.
- Records are stored as compact binary packages with payloads stored within the record if small enough.
- Records are stored in Records/{Bucket}/{01}/{23}/{456789abcdef0123456789abcdef01234567} based on the hash in the cache key.
- Content that exceeds the size allowed for storage within the record is stored in Content/{01}/{23}/{456789abcdef0123456789abcdef01234567} based on the hash of the content.
- Both records and content are written to disk with their hash as a suffix, and are always checked on load.
- Maximum size of compressed data stored in the package can be configured by MaxRecordSizeKB and MaxValueSizeKB.
- Maintenance of the cache, to delete unused files, has been rewritten and can now scan ~175k files/second on modern hardware, though is limited to 10k files/second in the config to avoid adding too much overhead.

#rb Zousar.Shaker
#preflight 618b75c14a0c609a29204032
#preflight 618bfb5ec717ba4e929b7ac0

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18133085 in //UE5/Release-5.0/... via CL 18133356
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v889-18060218)
#ROBOMERGE[STARSHIP]: UE5-Main

[CL 18133430 by devin doucette in ue5-release-engine-test branch]
2021-11-10 13:08:00 -05:00
Devin Doucette
7d5d9f3798 DDC: Clean up a few details in the new Cache API and remove the last exported type from DDC
#rb Zousar.Shaker
#rnx

[CL 15670275 by Devin Doucette in ue5-main branch]
2021-03-10 17:33:38 -04:00
ryan durand
627baf970a Updating copyright for Engine Editor.
#rnx
#rb none


#ROBOMERGE-SOURCE: CL 10869241 via CL 10869527 via CL 10869904
#ROBOMERGE-BOT: (v613-10869866)

[CL 10870586 by ryan durand in Main branch]
2019-12-26 15:33:43 -05:00
Robert Manuszewski
b3da7113cc Added DDCCleanup commandlet that iterates over shared DDC directories and removes old (unused) cache files
#rb none

[CL 5490981 by Robert Manuszewski in Dev-Core branch]
2019-03-21 01:47:37 -04:00