Commit Graph

32 Commits

Author SHA1 Message Date
zousar shaker
1e1378f013 Avoid indexing into a requests output datas array before we have established if we have output, as this can contribute to an out-of-bounds array lookup.
#rb devin.doucette
[FYI] francis.hurteau

#ROBOMERGE-SOURCE: CL 16867872 via CL 16867881
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v836-16769935)

[CL 16867888 by zousar shaker in ue5-release-engine-test branch]
2021-07-15 17:31:20 -04:00
Devin Doucette
8e3e3cc389 DDC: Fixed TryToPrefetch to work when there is only one backend
#rb Zousar.Shaker
#rnx

[CL 16549112 by Devin Doucette in ue5-main branch]
2021-06-03 12:57:44 -04:00
Zousar Shaker
e56633ec8e Avoid crashing in DDC if the Jupiter server sends a malformed response with a payload size that sends us off the end of the response buffer. Emit an error message instead.
#rb devin.doucette
#preflight 60ad3fdd04188d0001f4987b

[CL 16454717 by Zousar Shaker in ue5-main branch]
2021-05-25 15:28:50 -04:00
Devin Doucette
3f40f504cf DDC: Changed Put to take a const array of cache records now that records are copyable
#rb Zousar.Shaker
#rnx

[CL 16184736 by Devin Doucette in ue5-main branch]
2021-05-03 12:05:18 -04:00
Andriy Tylychko
9f1cf3eabf re-deprecated FPlatformProcess::CreateSynchEvent(), now on all platforms. Replaced FEvent* by FEventRef where it made sense. Deprecated FEvent::Create() as another method to bypass the event pool
#rb steve.robb

[CL 16163267 by Andriy Tylychko in ue5-main branch]
2021-04-29 18:52:48 -04:00
Devin Doucette
e42bbbedab DDC: Implement ICacheStore on FDerivedDataBackendInterface
This allows the new cache to be implemented more efficiently using the old cache backends because functionality like the corruption wrapper and key length limiter can be bypassed and the individual backends can store cache records in whatever way is most efficient for them.

The hierarchical backend may request payloads when they are not required due to incomplete tracking of backend state, and GetPayload will never fill other backends due to the inefficiency of the existing backend framework.

The filesystem backend does not cache any state in memory, which makes requests for individual payloads less efficient than if it cached a mapping of cache payload key to raw hash after the first request for a cache key.

The HTTP, S3, and pak file backends are not implemented for the new interface.

The backends do not implement ICacheStore::CancelAll() because the existing backend framework provides WaitForQuiescence to wait for completion of async requests, and the implementation of ICacheStore by those backends works with that mechanism.

The non-leaf backends (hierarchical, async put, etc.) do not update stats from the ICacheStore functions.

#rb Zousar.Shaker
#rnx
#preflight 60899f35d324590001b47517

[CL 16148296 by Devin Doucette in ue5-main branch]
2021-04-28 16:22:18 -04:00
Zousar Shaker
94bcce4d3e Put a minimum "transfer speed over duration" limit on curl operations done by the HTTP DDC backend.
#rb devin.doucette
#preflight 60808a0ca698b3000172743c

[CL 16081589 by Zousar Shaker in ue5-main branch]
2021-04-21 17:14:57 -04:00
Zousar Shaker
8b5a796e2a Add support for ReadOnly={true|false} and EnvHostOverride={string} as configurable parameters on an HTTP DDC backend.
#rb devin.doucette

[CL 16076564 by Zousar Shaker in ue5-main branch]
2021-04-21 13:19:11 -04:00
Zousar Shaker
3b9ff68e09 Fix bug in the HttpDerivedDataBackend where an FRequest may not be returned to the FRequestPool if we're aborting queued PUTs due to a shutdown request.
#jira UE-113612
#rb devin.doucette
#preflight 607e38a842dbff0001a4dc20

[CL 16058184 by Zousar Shaker in ue5-main branch]
2021-04-19 22:53:53 -04:00
Johan Berg
ce27da1f98 Remove UE4 strings
#jira UE-111642
#rb none

[CL 16002173 by Johan Berg in ue5-main branch]
2021-04-14 04:25:18 -04:00
Zousar Shaker
d014867e9f Ensure that the non-DataRequest-Helper codepath for putting DDC data into the cache also tears-down promptly during process shutdown if not running on a build machine.
#jira UE-112179
#rb devin.doucette

[CL 15930076 by Zousar Shaker in ue5-main branch]
2021-04-06 10:20:06 -04:00
Zousar Shaker
dc68fab37b Ensure HTTP DDC backend doesn't delay process shutdown due to a queue of pending PUTs for normal users.
Still allows build machine runs to be thorough about completing PUTs.
Doesn't interrupt any in-flight PUTs, just prevents processing of any remaining queued PUTs.

#jira UE-112179
#rb devin.doucette

[CL 15920539 by Zousar Shaker in ue5-main branch]
2021-04-05 14:36:58 -04:00
Devin Doucette
d87eb29d07 DDC: Switched to WriteToString in the Jupiter backend
#rb Zousar.Shaker
#rnx

[CL 15829627 by Devin Doucette in ue5-main branch]
2021-03-25 15:34:31 -04:00
Zousar Shaker
0ee734bf61 Fix HTTP/Jupiter DDC backend to account for the fact that Jupiter will return batched results in arbitrary order, not the same order that the requests/keys were listed.
#rb devin.doucette

[CL 15771001 by Zousar Shaker in ue5-main branch]
2021-03-22 18:23:46 -04:00
Marc Audy
0cbbc781ca Merge UE5/Release-Engine-Staging @ 15740152 to UE5/Main
This represents UE4/Main @ 15709114

[CL 15740605 by Marc Audy in ue5-main branch]
2021-03-18 15:20:03 -04:00
Joakim Lindqvist
5957a9c659 Added option to override Jupiter speed class
#rb devin.doucette

[CL 15736124 by Joakim Lindqvist in ue5-main branch]
2021-03-18 05:51:42 -04:00
Johan Torp
7b4d8d2785 Enable DDC1 Jupiter backend GET/PUT pooling and added dedicated main thread pools
#rb devin.doucette, johan.berg
#fyi jeff.farris
#jira none
#rnx

[CL 15718578 by Johan Torp in ue5-main branch]
2021-03-17 04:27:28 -04:00
Joakim Lindqvist
911fb5c9ab Send DDC payloads with IoHash instead of SHA1
#rb devin.doucette

[CL 15710022 by Joakim Lindqvist in ue5-main branch]
2021-03-16 06:11:58 -04:00
Johan Berg
cfc09f48f2 Batched Jupiter queries
Adds a helper class to make CachedDataProbablyExists, GetCachedData and PutCachedData operations. The helper checks the availble connections in the pool and decides if the thread should make a regular request or batch together with other requests. This will reduce the number of concurrent connections on each client.

Disabled by default.

#rb devin.doucette

[CL 15604924 by Johan Berg in ue5-main branch]
2021-03-04 09:35:03 -04:00
Joakim Lindqvist
22cf9dc2f7 Reduced Error output to Display in HttpBackend as this is a recoverable error.
#rb devin.doucette
#jira UE-109707

[CL 15580171 by Joakim Lindqvist in ue5-main branch]
2021-03-03 05:05:34 -04:00
Joakim Lindqvist
385e81f304 HttpDerivedBackend - Allow IoHash header as well
* Removed pointless double hashing for sha1 hashes.
* Fixed spelling recieved -> received

#rb devin.doucette

[CL 15489593 by Joakim Lindqvist in ue5-main branch]
2021-02-22 11:11:09 -04:00
Devin Doucette
6394ccc1cd DDC: Added TryToPrefetch to check multiple keys for existence and prefetch the data into a fast cache backend
#rb Zousar.Shaker

[CL 15422719 by Devin Doucette in ue5-main branch]
2021-02-16 17:24:48 -04:00
Joakim Lindqvist
9bfbbf2c75 Clarified comment on why transient data can be disabled.
[CL 15154162 by Joakim Lindqvist in ue5-main branch]
2021-01-21 10:41:53 -04:00
Joakim Lindqvist
87ef345981 Do not delete transient data from Jupiter, it will clean itself up as needed.
Also made Jupiter a slow backend, as this generally is closer to the typical speed people have to it. We should make this automatic based on the latency we find from the health check call.

#rb devin.doucette

[CL 15154130 by Joakim Lindqvist in ue5-main branch]
2021-01-21 10:36:11 -04:00
Devin Doucette
2fe890d75a DDC: Implemented batched CachedDataProbablyExists requests in the Jupiter backend
#rb Johan.Berg, Zousar.Shaker

[CL 15037688 by Devin Doucette in ue5-main branch]
2021-01-11 11:34:27 -04:00