Commit Graph

600 Commits

Author SHA1 Message Date
danny couture
623896568d Improve frame rate stability in editor by reducing the priority of the DDC IO threads which are now doing CPU intensive compression to avoid them preempting more important threads (GT/RT/RHI)
#rnx
#rb Devin.Doucette
#preflight 62825177cf7e4667a991a6cc

[CL 20223281 by danny couture in ue5-main branch]
2022-05-16 09:52:19 -04:00
Zousar Shaker
d08f81b885 Ensure that http headers can be found in a case-insensitive way as expected by the callers of GetHeader.
#rb devin.doucette
#rnx
#preflight 627b53295b58374330b5484c
#preflight 627bf3ce0a5817c9d952fc57

[CL 20144831 by Zousar Shaker in ue5-main branch]
2022-05-11 13:52:26 -04:00
Devin Doucette
5021354ab8 DDC: Fixed HttpCacheStore to shut down its thread
FHttpSharedData is no longer constructed as a function-local static because that makes shutdown more difficult to handle correctly. The cache store now owns the shared data and destroys it when it shuts down.

Includes greater encapsulation of the shared data and migration from FRunnable/FRunnableThread to FThread.

#preflight 627aad2bc42338be65272366
#rb Zousar.Shaker
#rnx

[CL 20138109 by Devin Doucette in ue5-main branch]
2022-05-11 09:10:24 -04:00
Devin Doucette
da04ab9644 DDC: Fixed cache put propagation for nodes that store async
#preflight 62798a72242446ce6c69df25
#rb Zousar.Shaker
#rnx

[CL 20121708 by Devin Doucette in ue5-main branch]
2022-05-10 08:46:00 -04:00
Devin Doucette
7a50c98265 DDC: Added propagation of cache records that contain metadata with no values
#preflight 62797bdd4561731dbe10ecdf
#rb Matt.Peters

[CL 20111217 by Devin Doucette in ue5-main branch]
2022-05-09 17:18:23 -04:00
Zousar Shaker
4ee0c804e0 [Backout] - CL20029407
#fyi Zousar.Shaker
Original CL Desc
-----------------------------------------------------------------
Limit concurrent connections when using the Curl multi API for Jupiter DDC operations.

#rb devin.doucette
#preflight 627040b4645c64f3a25a14ca
#preflight 62716d2c9d6c2f8f5b2558f2

[CL 20057204 by Zousar Shaker in ue5-main branch]
2022-05-05 09:58:02 -04:00
Devin Doucette
47e28f2fe0 DDC: Optimized the file system cache for SMB
- Reading files from the Game Thread now launches a task to check and update the modification time, which approximately halves the latency of these reads.
- Writing files now only asks to delete the temporary file upon failure, as it added unnecessary latency before by trying to delete it after it was renamed.

#preflight 6272d5b21f37fd32b0f07652
#rb Zousar.Shaker

[CL 20048278 by Devin Doucette in ue5-main branch]
2022-05-04 16:23:17 -04:00
Zousar Shaker
5127ed3cb9 Limit concurrent connections when using the Curl multi API for Jupiter DDC operations.
#rb devin.doucette
#preflight 627040b4645c64f3a25a14ca
#preflight 62716d2c9d6c2f8f5b2558f2

[CL 20029407 by Zousar Shaker in ue5-main branch]
2022-05-03 14:23:07 -04:00
Zousar Shaker
05664ce5b2 Avoid querying Horde Storage during GetChunks calls for "Null hash" compressed blob value when you've experienced a miss for the record it belongs to.
#rb devin.doucette
#preflight 626ff4878063424f88a90ceb

[CL 20009546 by Zousar Shaker in ue5-main branch]
2022-05-02 11:52:58 -04:00
Zousar Shaker
f7a1ba41e6 Fixed bug in handling of -DDC-All-* commandline arguments.
#rb none
#rnx
#fyi devin.doucette
#preflight 626ec64edd1fe026f111ac4a

[CL 20002339 by Zousar Shaker in ue5-main branch]
2022-05-01 13:56:24 -04:00
Devin Doucette
8dcdcbe09f DDC: Improved file system cache store maintenance
- Added deletion of empty directories.
- Applied the rate limit to directory scans.
- Reduced the default scan rate by a factor of 10.

#preflight 626c3a8b0f5f22f922511514
#rb Zousar.Shaker

[CL 19984889 by Devin Doucette in ue5-main branch]
2022-04-29 16:18:09 -04:00
Devin Doucette
f75b88cce1 DDC: Fixed simulated cache miss salt differing for every node when using -DDC-All-Miss options
#preflight 626c0ecfee358ddf385638bc
#rb Zousar.Shaker
#rnx

[CL 19982661 by Devin Doucette in ue5-main branch]
2022-04-29 14:18:40 -04:00
Devin Doucette
a99439300d DDC: Added and clarified wait timing scopes in the Horde Storage cache client
#preflight 626ae32e836be6e73ac9a190
#rb Zousar.Shaker
#rnx

[CL 19967049 by Devin Doucette in ue5-main branch]
2022-04-28 15:29:34 -04:00
Devin Doucette
7b11944d76 DDC: Added a timing scope to the flush (deletion of everything) of the file system cache
#preflight 626ae314a5009ff191c3f211
#rb Zousar.Shaker
#rnx

[CL 19967035 by Devin Doucette in ue5-main branch]
2022-04-28 15:28:49 -04:00
Devin Doucette
92a888c1f6 Replaced uses of _ASV with ANSITEXTVIEW
#preflight 6266acf50634d0904cdfe91c
#rb Steve.Robb
#rnx

[CL 19903251 by Devin Doucette in ue5-main branch]
2022-04-25 11:31:36 -04:00
Devin Doucette
0b7aa65637 Replaced uses of _U8SV with UTF8TEXTVIEW
#preflight 626310ac006fa20b683d17b9
#rb Zousar.Shaker
#rnx

[CL 19901490 by Devin Doucette in ue5-main branch]
2022-04-25 09:36:34 -04:00
Devin Doucette
ceabbe51e8 DDC: Added more documentation to the cache policy
#preflight 626183407d162829ae811783
#rb Zousar.Shaker
#rnx

[CL 19851224 by Devin Doucette in ue5-main branch]
2022-04-21 12:40:49 -04:00
Devin Doucette
fa8f3dbe82 DDC: Fixed duplicate puts caused by the in-flight memory cache
Without using StopGetStore, get hits on the in-flight cache would propagate as puts to subsequent nodes in the hierarchy. Since the in-flight cache is only used as temporary storage while a put is in flight, this caused duplicates of the in-flight puts, which manifests as move collision log spam from the file system cache store.

#preflight 625d772d691f49969ea93249
#rb Zousar.Shaker
#rnx

[CL 19785647 by Devin Doucette in ue5-main branch]
2022-04-18 11:06:05 -04:00
Devin Doucette
b74b8b9227 DDC: Fixed incorrect cache store flags on dynamically mounted pak files
#preflight 625d771248670f31a61f0def
#rb Zousar.Shaker
#rnx

[CL 19785545 by Devin Doucette in ue5-main branch]
2022-04-18 10:57:14 -04:00
Devin Doucette
14583c8432 DDC: Back out 19757142 because it broke cache puts
#preflight skip
#rb none
#rnx

[CL 19763306 by Devin Doucette in ue5-main branch]
2022-04-14 17:05:38 -04:00
Devin Doucette
e35dc9fd40 DDC: Fixed duplicate puts caused by the in-flight memory cache
Without using StopStore, get hits on the in-flight cache would propagate as puts to subsequent nodes in the hierarchy. Since the in-flight cache is only used as temporary storage while a put is in flight, this caused duplicates of the in-flight puts, which manifests as move collision log spam from the file system cache store.

#preflight 625835d97f628a9018d8acf6
#rb Zousar.Shaker
#rnx

[CL 19757142 by Devin Doucette in ue5-main branch]
2022-04-14 11:06:11 -04:00
Devin Doucette
83f49f7dce DDC: Changed FileSystem PutValue to match Put
Except when replacing an existing value, load the existing value and check for determinism, and replace the existing value in the presence of an invalid value or non-deterministic value with missing content. This resolves move collisions being reported when a value existed without its content.

#preflight 62582c69946114248db5561f
#rb Zousar.Shaker
#rnx

[CL 19756384 by Devin Doucette in ue5-main branch]
2022-04-14 10:32:14 -04:00
Devin Doucette
37820d6bbc DDC: Fixed incorrect skipping of get propagation past nodes with the flag StopStore
The wrong response status was being checked, and caused subsequent nodes to be skipped even if the StopStore node did not contain the data, if any previous node contained the data.

#preflight 62582d67946114248db62d58
#rb Zousar.Shaker
#rnx

[CL 19756375 by Devin Doucette in ue5-main branch]
2022-04-14 10:31:55 -04:00
Devin Doucette
f1c81d6d61 DDC: Reset the build definition later in the build job to ease debugging
#preflight 62582c26044bab1525fca794
#rb Zousar.Shaker
#rnx

[CL 19756364 by Devin Doucette in ue5-main branch]
2022-04-14 10:31:33 -04:00
charles bloom
7b58069f70 DDC Verify cache miss is just a message not a warning
#rb devin.doucette
#preflight none

[CL 19648508 by charles bloom in ue5-main branch]
2022-04-06 11:43:57 -04:00