Commit Graph

602 Commits

Author SHA1 Message Date
mihnea balta
399e160546 Fix bad scoped locks in various places.
Added nodiscard on the scope lock helpers to prevent such errors in the future.

#jira none
#rnx
#preflight https://horde.devtools.epicgames.com/job/62b49fe89117e2324e187b8d
#rb Francis.Hurteau, Andriy.Tylychko, Rob.Millar

[CL 20797349 by mihnea balta in ue5-main branch]
2022-06-23 13:34:31 -04:00
Zousar Shaker
6e67702448 Add missing include.
#rb none
#rnx
#robomerge EngineTest
#preflight 62b3f918a02640e7ab84ea24

[CL 20790219 by Zousar Shaker in ue5-main branch]
2022-06-23 01:43:23 -04:00
devin doucette
e31239c9b3 DDC: Fixed a non-unity compile error
#rb none
#rnx

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 20770467 via CL 20770468 via CL 20770472
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v970-20704180)

[CL 20770741 by devin doucette in ue5-main branch]
2022-06-22 01:40:20 -04:00
devin doucette
fbfcde6644 DDC: Extended cache replay to read replay files async and with a more configurable console command
#jira UE-144410
#preflight 62b243c5de72d4f6b34d0252
#rb Zousar.Shaker
#rnx

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 20770298 via CL 20770311 via CL 20770320
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v970-20704180)

[CL 20770736 by devin doucette in ue5-main branch]
2022-06-22 01:40:12 -04:00
devin doucette
adb41a4dbd DDC: Added cache replay functionality
A cache replay can be created from any program that uses the cache, and can be replayed through any cache graph in any program that uses the cache.

Save a cache replay by adding -DDC-ReplaySave=PathToReplay.ddr to the command line, and compress when saving by adding -DDC-ReplayCompress to reduce replay size by approximately 85%.

Load a cache replay by adding -DDC-ReplayLoad=PathToReplay.ddr to the command line one or more times.
- Modify the cache priority on loaded replays with -DDC-ReplayLoadPriority=Priority.
- For example, -DDC-ReplayLoadPriority=Lowest will queue replayed requests to execute asynchronously.
- Modify the cache policy on loaded replays with -DDC-ReplayLoadAddPolicy=CachePolicy and -DDC-ReplayLoadRemovePolicy=CachePolicy.
- For example, -DDC-ReplayLoadAddPolicy=SkipData will make the replay prefetch into the cache stores earlier in the cache hierarchy.
- For example, -DDC-ReplayLoadRemovePolicy=QueryLocal will make the replay query only remote cache stores.

Filter save or load operations by combining these arguments:
-DDC-ReplayMethods=Get+GetValue+GetChunks will only process requests of the corresponding methods.
-DDC-ReplayTypes=Type1[[at]Rate1][+Type2[[at]Rate2]...] will only process requests of the corresponding types.
-DDC-ReplayRate=Rate will only process Rate% of the requests that do not match one of the type filters.
-DDC-ReplaySalt=Salt configures the salt used to filter by type and rate, which makes them match consistently between sessions instead of being random.

Load a cache replay with the console command: DDC.LoadReplay ReplayPath [Priority=PriorityName]

#jira UE-144410
#rb Zousar.Shaker

#ROBOMERGE-OWNER: devin.doucette
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 20741874 via CL 20745226 via CL 20745737
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v970-20704180)

[CL 20751395 by devin doucette in ue5-main branch]
2022-06-20 23:40:37 -04:00
devin doucette
56ca40dda2 DDC: Fixed deprecation warnings in the Zen cache store
#jira UE-157180
#rb none
#rnx

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 20745126 via CL 20745729
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v970-20704180)

[CL 20751392 by devin doucette in ue5-main branch]
2022-06-20 23:40:25 -04:00
devin doucette
6366a19bda DDC: Added functions to serialize cache types to and from compact binary, and strings where appropriate
#rb Zousar.Shaker

#ROBOMERGE-OWNER: devin.doucette
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 20741868 via CL 20744192 via CL 20745275
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v970-20704180)

[CL 20751385 by devin doucette in ue5-main branch]
2022-06-20 23:40:01 -04:00
Zousar Shaker
def117e022 Ensure the connection pool size matches the total number of connections for the DDC HTTP implementation.
#rb none
#fyi devin.doucette
#rnx
#preflight 62b0c838640ffd8ee9ce8be5

[CL 20742724 by Zousar Shaker in ue5-main branch]
2022-06-20 15:32:20 -04:00
Zousar Shaker
0b361c7c46 Select maximum HTTP connections for ZenCacheStore based on logical core count (clamped between 8 and 64). Speculative solution for farm cooks using Zen experiencing "couldn't connect errors".
#rb trivial
#fyi devin.doucette
#rnx
#preflight 62b00bd92405dc6102315616

[CL 20734103 by Zousar Shaker in ue5-main branch]
2022-06-20 02:08:49 -04:00
devin doucette
3933b8f407 DDC: Added TryLoadFromCompactBinary for FCacheKey
#rb Zousar.Shaker
#rnx

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 20705430 via CL 20705467 via CL 20705498
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v968-20684695)

[CL 20707645 by devin doucette in ue5-main branch]
2022-06-17 14:45:17 -04:00
steve robb
fd2a0b1e20 Removal of TEXT literal concatenations, which doesn't work in UTF-8 mode.
#rb devin.doucette
#jira UE-132142
#preflight 62a8748fa76c84377748036b

#ROBOMERGE-OWNER: steve.robb
#ROBOMERGE-AUTHOR: steve.robb
#ROBOMERGE-SOURCE: CL 20647443 via CL 20648283 via CL 20648315
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v955-20579017)

[CL 20651786 by steve robb in ue5-main branch]
2022-06-14 12:46:34 -04:00
devin doucette
a21cf2431c DDC: Removed the legacy cache backend interface
#preflight 62a7a3dc0a2f58855e0af685
#rb Zousar.Shaker
#rnx

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 20646770 via CL 20646804 via CL 20646821
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v955-20579017)

[CL 20649216 by devin doucette in ue5-main branch]
2022-06-14 11:05:24 -04:00
devin doucette
f6f4dcd722 DDC: Added detail to the error when cache graph creation fails
#preflight skip
#rb Zousar.Shaker
#rnx

#ROBOMERGE-OWNER: devin.doucette
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 20628288 via CL 20628340 via CL 20628354
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v955-20579017)

[CL 20636252 by devin doucette in ue5-main branch]
2022-06-13 18:36:59 -04:00
devin doucette
1d698f7814 DDC: Removed the remaining legacy cache implementations and made the default implementation non-virtual
#preflight 62a38afedf2d057cb2814ea4
#rb Steve.Robb
#rnx

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 20601122 via CL 20601125 via CL 20601130
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v955-20579017)

[CL 20602835 by devin doucette in ue5-main branch]
2022-06-10 17:21:00 -04:00
Zousar Shaker
ea6ad5fc8e Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately.
This change requires a fix (in progress) to EditorDomain to avoid a deadlock.

#rb devin.doucette
#fyi stefan.boberg
#preflight 62a3619670680d6143dc5425

[CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
Zousar Shaker
742cdfac66 Reduce HTTP connection limit from 32 to 8 for the DDC HTTP client.
#rb trivial
#rnx
#preflight 62a114563f1e313c6ada20c2

[CL 20565273 by Zousar Shaker in ue5-main branch]
2022-06-08 17:47:30 -04:00
devin doucette
539d11284b DDC: Fixed the pak file cache IsWritable function to take into account whether the file is closed
#jira UE-156101
#preflight skip
#rb Zousar.Shaker
#rnx

#ROBOMERGE-OWNER: Devin.Doucette
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-COMMAND: _robomerge[bot4] UE5-MAIN
#ROBOMERGE-SOURCE: CL 20558509 via CL 20558581 via CL 20558624
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v954-20466795)

[CL 20558669 by devin doucette in ue5-main branch]
2022-06-08 12:40:43 -04:00
David Harvey
410a83c149 only define CURL_NO_OLDIES if it isn't defined already.
#rb trivial
#rnx
#preflight 62a0a45e521254896faff875

[CL 20555685 by David Harvey in ue5-main branch]
2022-06-08 09:44:27 -04:00
devin doucette
3e2538a313 DDC: Removed the legacy cache store implementations and limit cache bucket names to 63 characters
The cache bucket name was already limited to 63 characters in practice by Horde Storage. This makes it consistent across every cache store.

#preflight 629e6f0a1c7e8f9a180e192a
#rb Zousar.Shaker

#ROBOMERGE-OWNER: Devin.Doucette
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-COMMAND: _robomerge[bot4] UE5-MAIN
#ROBOMERGE-SOURCE: CL 20536454 via CL 20536805 via CL 20536993
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v954-20466795)

[CL 20538633 by devin doucette in ue5-main branch]
2022-06-07 11:17:10 -04:00
Zousar Shaker
1505481e23 Limit the maximum open HTTP connections used by Jupiter and Zen DDC backends to 32 HTTP2 connections (w/ multiplexing). I want to limit it further. I've seen equally fast cold local editor load times in CitySample with only 8 maximum connections, but until I've had a chance to test that with others in differing network environments, I'm going to stick with 32 and consider reducing later.
#rb devin.doucette
#fyi joakim.lindqvist
#preflight 629e70658a67e261691e6751

[CL 20528598 by Zousar Shaker in ue5-main branch]
2022-06-06 17:44:09 -04:00
Devin Doucette
3490c0ee69 DDC: Restored the correct default of true for bResolveHostCanonicalName in HttpCacheStore
#preflight 629e348b5d78bd7cf1966b45
#rb trivial
#rnx
#fyi Zousar.Shaker

[CL 20523583 by Devin Doucette in ue5-main branch]
2022-06-06 13:33:27 -04:00
Devin Doucette
7d082d7531 DDC: Iteration on HttpClient
- Cleaned up some string allocations and conversions.
- Changed code formatting for greater consistency with DDC.

#preflight 62992f72f03d578360b2bfec
#rb Zousar.Shaker
#rnx

[CL 20485244 by Devin Doucette in ue5-main branch]
2022-06-03 09:12:05 -04:00
Ben Marsh
8da76cf563 Fix non-unity error.
#preflight none

[CL 20469945 by Ben Marsh in ue5-main branch]
2022-06-02 11:06:47 -04:00
Devin Doucette
5c2b7f7381 DDC: Temporary compile fix for platforms that should not include DerivedDataCache
#preflight skip
#rb none
#rnx

[CL 20445864 by Devin Doucette in ue5-main branch]
2022-05-31 23:25:37 -04:00
Devin Doucette
1bbb384300 DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen
#preflight 62967fd60c151996e13038ec
#rb Zousar.Shaker

[CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00