Commit Graph

16 Commits

Author SHA1 Message Date
Devin Doucette
62f66e3e49 DDC: Fixed verbose logging in the file system cache
#preflight 62da257cbc175ec68cb7b9db
#rb Zousar.Shaker
#rnx
#robomerge FNMain

[CL 21220401 by Devin Doucette in ue5-main branch]
2022-07-22 10:20:41 -04:00
zousar shaker
75d0e7277b Add status icon to indicate DDC cache store status with tooltip text for details. Currently used to show when a cache store has been deactivated for performance.
#rb devin.doucette

#ROBOMERGE-AUTHOR: zousar.shaker
#ROBOMERGE-SOURCE: CL 21152573 via CL 21155094 via CL 21156160
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)

[CL 21181761 by zousar shaker in ue5-main branch]
2022-07-20 11:55:16 -04:00
zousar shaker
01a55c47a3 Add an optional "DeactivateAt" attribute for DDC filesystem cache stores to allow the store to be deactivated for the session if a speedtest is run and the speedtest indicates that the latency is higher than the configured threshold. Default state is that there is no deactivation threshold.
#rb devin.doucette

#ROBOMERGE-AUTHOR: zousar.shaker
#ROBOMERGE-SOURCE: CL 20994964 via CL 20994988 via CL 20995000
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)

[CL 21002571 by zousar shaker in ue5-main branch]
2022-07-08 02:53:40 -04:00
steve robb
fd2a0b1e20 Removal of TEXT literal concatenations, which doesn't work in UTF-8 mode.
#rb devin.doucette
#jira UE-132142
#preflight 62a8748fa76c84377748036b

#ROBOMERGE-OWNER: steve.robb
#ROBOMERGE-AUTHOR: steve.robb
#ROBOMERGE-SOURCE: CL 20647443 via CL 20648283 via CL 20648315
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v955-20579017)

[CL 20651786 by steve robb in ue5-main branch]
2022-06-14 12:46:34 -04:00
devin doucette
3e2538a313 DDC: Removed the legacy cache store implementations and limit cache bucket names to 63 characters
The cache bucket name was already limited to 63 characters in practice by Horde Storage. This makes it consistent across every cache store.

#preflight 629e6f0a1c7e8f9a180e192a
#rb Zousar.Shaker

#ROBOMERGE-OWNER: Devin.Doucette
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-COMMAND: _robomerge[bot4] UE5-MAIN
#ROBOMERGE-SOURCE: CL 20536454 via CL 20536805 via CL 20536993
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v954-20466795)

[CL 20538633 by devin doucette in ue5-main branch]
2022-06-07 11:17:10 -04:00
mark lintott
da46ade47c Added Cache Speed Stats to Analytics
Added Zen Server Stats to Analytics
#rb zousar.shaker, devin.doucette
#jira none
#preflight 6295d84bd57da28cc8ee9eab

[CL 20434901 by mark lintott in ue5-main branch]
2022-05-31 05:11:05 -04:00
Devin Doucette
47e28f2fe0 DDC: Optimized the file system cache for SMB
- Reading files from the Game Thread now launches a task to check and update the modification time, which approximately halves the latency of these reads.
- Writing files now only asks to delete the temporary file upon failure, as it added unnecessary latency before by trying to delete it after it was renamed.

#preflight 6272d5b21f37fd32b0f07652
#rb Zousar.Shaker

[CL 20048278 by Devin Doucette in ue5-main branch]
2022-05-04 16:23:17 -04:00
Devin Doucette
8dcdcbe09f DDC: Improved file system cache store maintenance
- Added deletion of empty directories.
- Applied the rate limit to directory scans.
- Reduced the default scan rate by a factor of 10.

#preflight 626c3a8b0f5f22f922511514
#rb Zousar.Shaker

[CL 19984889 by Devin Doucette in ue5-main branch]
2022-04-29 16:18:09 -04:00
Devin Doucette
7b11944d76 DDC: Added a timing scope to the flush (deletion of everything) of the file system cache
#preflight 626ae314a5009ff191c3f211
#rb Zousar.Shaker
#rnx

[CL 19967035 by Devin Doucette in ue5-main branch]
2022-04-28 15:28:49 -04:00
Devin Doucette
83f49f7dce DDC: Changed FileSystem PutValue to match Put
Except when replacing an existing value, load the existing value and check for determinism, and replace the existing value in the presence of an invalid value or non-deterministic value with missing content. This resolves move collisions being reported when a value existed without its content.

#preflight 62582c69946114248db5561f
#rb Zousar.Shaker
#rnx

[CL 19756384 by Devin Doucette in ue5-main branch]
2022-04-14 10:32:14 -04:00
steve robb
87fb605867 TCHAR array init fixes for UTF-8 mode.
#rb devin.doucette
#preflight 6231e536e2541b4ff3af6cd2

#ROBOMERGE-AUTHOR: steve.robb
#ROBOMERGE-SOURCE: CL 19405353 via CL 19408955 via CL 19419621 via CL 19419706
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v930-19419903)

[CL 19420584 by steve robb in ue5-main branch]
2022-03-17 10:29:53 -04:00
Devin Doucette
1318804bb2 DDC: Fixed timing scopes for FileSystem, PakFile, S3
Edited the timing scopes to exclude invocation of the completion callback.

#preflight 62325890262232d86f812368
#rb Zousar.Shaker
#rnx

[CL 19419775 by Devin Doucette in ue5-main branch]
2022-03-17 09:07:20 -04:00
steve robb
33c800a605 TEXT(""_SV) changed to TEXTVIEW("") for UTF-8 support.
#rb trivial

#ROBOMERGE-OWNER: steve.robb
#ROBOMERGE-AUTHOR: steve.robb
#ROBOMERGE-SOURCE: CL 19135062 via CL 19136099 via CL 19136620 via CL 19138526
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v921-19075845)

[CL 19155676 by steve robb in ue5-main branch]
2022-02-25 15:48:16 -05:00
devin doucette
6dd39d0d04 DDC: Fixed -DDC-All-MissRate to work with callers that expect consistent hits on the same key
Fixes the accumulation of stats in the async wrapper as well.

#jira UE-143606
#preflight 6216a933a45a91013f3f4047
#lockdown Aurel.Cordonnier
#rb Zousar.Shaker
#rnx

#ROBOMERGE-OWNER: Devin.Doucette
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-COMMAND: _robomerge UE5-Main
#ROBOMERGE-SOURCE: CL 19102208 in //UE5/Release-5.0/... via CL 19102301
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v921-19075845)

[CL 19102315 by devin doucette in ue5-main branch]
2022-02-23 17:04:41 -05:00
danny couture
1f388a417c Fix audio crash when using -DDC-All-MissTypes on streamed audio by only returning DDC miss until a put is done.
- Infinite misses break the editor in too many ways as we often rely on the local cache to be working correctly as off-memory storage.

#rnx
#rb Devin.Doucette
#preflight 620e36ea7e2cef4bf1d255a2

[CL 19033237 by danny couture in ue5-main branch]
2022-02-17 06:59:36 -05:00
devin doucette
2800210897 DDC: Enabled compression of legacy cache data
- FileSystem, Http, Pak, S3 use the ValueWithLegacyFallback mode by default, which cause them to fall back to searching for uncompressed data if compressed data is not found.
- FileSystem has been fixed to store up to 1 MiB of compressed data inline with the value package rather than separately in content-addressable storage.
- Pak has been optimized to have GetChunks only load the required region of the requested value, rather than the whole value.
- Pak has been changed to stop storing data inline in the record package, since it will end up in the same file anyway when stored separately.
- Pak will upgrade the compressor and compression level when a compressed pak file is requested. Default cache compression uses Oodle Mermaid VeryFast and will upgrade to Oodle Kraken Optimal2.
- Zen does not have compression enabled by default, pending deployment of a new version that stores compressed values to Horde Storage.
- Added a missing request barrier when saving uncompressed data as compressed.

Example reduction in file system cache size when cooking for Windows:
- CitySample dropped from 66.5 GiB to 21.8 GiB.
- Lyra dropped from 2.54 GiB to 672 MiB.
- ShooterGame dropped from 1.21 GiB to 380 MiB.

Example reduction in compressed pak file cache size when cooking for Windows:
- CitySample dropped from 22.3 GiB to 18.5 GiB.
- Lyra dropped from 691 MiB to 543 MiB.
- ShooterGame dropped from 387 MiB to 313 MiB.

#jira UE-134381
#preflight 620a703f583261b0a658e043, 620a6fb2803d9066e6805310, 620a733117632e948459b6af
#lockdown Aurel.Cordonnier
#rb Zousar.Shaker

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18983671 in //UE5/Release-5.0/... via CL 18983890 via CL 18984096
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v917-18934589)

[CL 18984126 by devin doucette in ue5-main branch]
2022-02-14 14:43:39 -05:00