Commit Graph

9 Commits

Author SHA1 Message Date
Devin Doucette
8dcdcbe09f DDC: Improved file system cache store maintenance
- Added deletion of empty directories.
- Applied the rate limit to directory scans.
- Reduced the default scan rate by a factor of 10.

#preflight 626c3a8b0f5f22f922511514
#rb Zousar.Shaker

[CL 19984889 by Devin Doucette in ue5-main branch]
2022-04-29 16:18:09 -04:00
Devin Doucette
7b11944d76 DDC: Added a timing scope to the flush (deletion of everything) of the file system cache
#preflight 626ae314a5009ff191c3f211
#rb Zousar.Shaker
#rnx

[CL 19967035 by Devin Doucette in ue5-main branch]
2022-04-28 15:28:49 -04:00
Devin Doucette
83f49f7dce DDC: Changed FileSystem PutValue to match Put
Except when replacing an existing value, load the existing value and check for determinism, and replace the existing value in the presence of an invalid value or non-deterministic value with missing content. This resolves move collisions being reported when a value existed without its content.

#preflight 62582c69946114248db5561f
#rb Zousar.Shaker
#rnx

[CL 19756384 by Devin Doucette in ue5-main branch]
2022-04-14 10:32:14 -04:00
steve robb
87fb605867 TCHAR array init fixes for UTF-8 mode.
#rb devin.doucette
#preflight 6231e536e2541b4ff3af6cd2

#ROBOMERGE-AUTHOR: steve.robb
#ROBOMERGE-SOURCE: CL 19405353 via CL 19408955 via CL 19419621 via CL 19419706
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v930-19419903)

[CL 19420584 by steve robb in ue5-main branch]
2022-03-17 10:29:53 -04:00
Devin Doucette
1318804bb2 DDC: Fixed timing scopes for FileSystem, PakFile, S3
Edited the timing scopes to exclude invocation of the completion callback.

#preflight 62325890262232d86f812368
#rb Zousar.Shaker
#rnx

[CL 19419775 by Devin Doucette in ue5-main branch]
2022-03-17 09:07:20 -04:00
steve robb
33c800a605 TEXT(""_SV) changed to TEXTVIEW("") for UTF-8 support.
#rb trivial

#ROBOMERGE-OWNER: steve.robb
#ROBOMERGE-AUTHOR: steve.robb
#ROBOMERGE-SOURCE: CL 19135062 via CL 19136099 via CL 19136620 via CL 19138526
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v921-19075845)

[CL 19155676 by steve robb in ue5-main branch]
2022-02-25 15:48:16 -05:00
devin doucette
6dd39d0d04 DDC: Fixed -DDC-All-MissRate to work with callers that expect consistent hits on the same key
Fixes the accumulation of stats in the async wrapper as well.

#jira UE-143606
#preflight 6216a933a45a91013f3f4047
#lockdown Aurel.Cordonnier
#rb Zousar.Shaker
#rnx

#ROBOMERGE-OWNER: Devin.Doucette
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-COMMAND: _robomerge UE5-Main
#ROBOMERGE-SOURCE: CL 19102208 in //UE5/Release-5.0/... via CL 19102301
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v921-19075845)

[CL 19102315 by devin doucette in ue5-main branch]
2022-02-23 17:04:41 -05:00
danny couture
1f388a417c Fix audio crash when using -DDC-All-MissTypes on streamed audio by only returning DDC miss until a put is done.
- Infinite misses break the editor in too many ways as we often rely on the local cache to be working correctly as off-memory storage.

#rnx
#rb Devin.Doucette
#preflight 620e36ea7e2cef4bf1d255a2

[CL 19033237 by danny couture in ue5-main branch]
2022-02-17 06:59:36 -05:00
devin doucette
2800210897 DDC: Enabled compression of legacy cache data
- FileSystem, Http, Pak, S3 use the ValueWithLegacyFallback mode by default, which cause them to fall back to searching for uncompressed data if compressed data is not found.
- FileSystem has been fixed to store up to 1 MiB of compressed data inline with the value package rather than separately in content-addressable storage.
- Pak has been optimized to have GetChunks only load the required region of the requested value, rather than the whole value.
- Pak has been changed to stop storing data inline in the record package, since it will end up in the same file anyway when stored separately.
- Pak will upgrade the compressor and compression level when a compressed pak file is requested. Default cache compression uses Oodle Mermaid VeryFast and will upgrade to Oodle Kraken Optimal2.
- Zen does not have compression enabled by default, pending deployment of a new version that stores compressed values to Horde Storage.
- Added a missing request barrier when saving uncompressed data as compressed.

Example reduction in file system cache size when cooking for Windows:
- CitySample dropped from 66.5 GiB to 21.8 GiB.
- Lyra dropped from 2.54 GiB to 672 MiB.
- ShooterGame dropped from 1.21 GiB to 380 MiB.

Example reduction in compressed pak file cache size when cooking for Windows:
- CitySample dropped from 22.3 GiB to 18.5 GiB.
- Lyra dropped from 691 MiB to 543 MiB.
- ShooterGame dropped from 387 MiB to 313 MiB.

#jira UE-134381
#preflight 620a703f583261b0a658e043, 620a6fb2803d9066e6805310, 620a733117632e948459b6af
#lockdown Aurel.Cordonnier
#rb Zousar.Shaker

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18983671 in //UE5/Release-5.0/... via CL 18983890 via CL 18984096
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v917-18934589)

[CL 18984126 by devin doucette in ue5-main branch]
2022-02-14 14:43:39 -05:00