Commit Graph

19 Commits

Author SHA1 Message Date
Zousar Shaker
0ee734bf61 Fix HTTP/Jupiter DDC backend to account for the fact that Jupiter will return batched results in arbitrary order, not the same order that the requests/keys were listed.
#rb devin.doucette

[CL 15771001 by Zousar Shaker in ue5-main branch]
2021-03-22 18:23:46 -04:00
Marc Audy
0cbbc781ca Merge UE5/Release-Engine-Staging @ 15740152 to UE5/Main
This represents UE4/Main @ 15709114

[CL 15740605 by Marc Audy in ue5-main branch]
2021-03-18 15:20:03 -04:00
Joakim Lindqvist
5957a9c659 Added option to override Jupiter speed class
#rb devin.doucette

[CL 15736124 by Joakim Lindqvist in ue5-main branch]
2021-03-18 05:51:42 -04:00
Johan Torp
7b4d8d2785 Enable DDC1 Jupiter backend GET/PUT pooling and added dedicated main thread pools
#rb devin.doucette, johan.berg
#fyi jeff.farris
#jira none
#rnx

[CL 15718578 by Johan Torp in ue5-main branch]
2021-03-17 04:27:28 -04:00
Joakim Lindqvist
911fb5c9ab Send DDC payloads with IoHash instead of SHA1
#rb devin.doucette

[CL 15710022 by Joakim Lindqvist in ue5-main branch]
2021-03-16 06:11:58 -04:00
Johan Berg
cfc09f48f2 Batched Jupiter queries
Adds a helper class to make CachedDataProbablyExists, GetCachedData and PutCachedData operations. The helper checks the availble connections in the pool and decides if the thread should make a regular request or batch together with other requests. This will reduce the number of concurrent connections on each client.

Disabled by default.

#rb devin.doucette

[CL 15604924 by Johan Berg in ue5-main branch]
2021-03-04 09:35:03 -04:00
Joakim Lindqvist
22cf9dc2f7 Reduced Error output to Display in HttpBackend as this is a recoverable error.
#rb devin.doucette
#jira UE-109707

[CL 15580171 by Joakim Lindqvist in ue5-main branch]
2021-03-03 05:05:34 -04:00
Joakim Lindqvist
385e81f304 HttpDerivedBackend - Allow IoHash header as well
* Removed pointless double hashing for sha1 hashes.
* Fixed spelling recieved -> received

#rb devin.doucette

[CL 15489593 by Joakim Lindqvist in ue5-main branch]
2021-02-22 11:11:09 -04:00
Devin Doucette
6394ccc1cd DDC: Added TryToPrefetch to check multiple keys for existence and prefetch the data into a fast cache backend
#rb Zousar.Shaker

[CL 15422719 by Devin Doucette in ue5-main branch]
2021-02-16 17:24:48 -04:00
Joakim Lindqvist
9bfbbf2c75 Clarified comment on why transient data can be disabled.
[CL 15154162 by Joakim Lindqvist in ue5-main branch]
2021-01-21 10:41:53 -04:00
Joakim Lindqvist
87ef345981 Do not delete transient data from Jupiter, it will clean itself up as needed.
Also made Jupiter a slow backend, as this generally is closer to the typical speed people have to it. We should make this automatic based on the latency we find from the health check call.

#rb devin.doucette

[CL 15154130 by Joakim Lindqvist in ue5-main branch]
2021-01-21 10:36:11 -04:00
Devin Doucette
2fe890d75a DDC: Implemented batched CachedDataProbablyExists requests in the Jupiter backend
#rb Johan.Berg, Zousar.Shaker

[CL 15037688 by Devin Doucette in ue5-main branch]
2021-01-11 11:34:27 -04:00
Devin Doucette
9010b3e1a5 DDC: Return a status from PutCachedData
The async put backend now uses this new status to avoid calling CachedDataProbablyExists after every put, which has a high cost for certain backends.

#rb Zousar.Shaker
#fyi Joakim.Lindqvist, Stefan.Boberg

[CL 15000972 by Devin Doucette in ue5-main branch]
2021-01-06 14:18:01 -04:00
Joakim Lindqvist
0f54b08b77 Considering Jupiter backend as a fast backend, in the same way the SMB share is configured. Works around problematic assumptions in the hirarchal backend were it will not accept puts that primarilly live in slow backends (e.g. if no fast backends are available)
#rb devin.doucette

[CL 14946668 by Joakim Lindqvist in ue5-main branch]
2020-12-17 10:10:46 -04:00
paul chipchase
45bef5bc27 Jupiter DDC head requests will no longer give errors if the response code is 204 (which is valid)
- The Io service currently returns response 204 when a HEAD request finds an entry, however the Europa service has been returning 200. The goal here is to update the DDC HEAD requests so that they will work with a response of 200 OR 204 so that we can unify the Jupiter apis.
- Added a utility method to FRequest that can be used to test if the response code is in the range that we consider a success.
- Updated FHttpDerivedDataBackend::CachedDataProbablyExists to use this.
- Updated the FRequest logging code to use this as well to avoid reporting successful operations as errors.

#rb Joakim.Lindqvist
#fyi Johan.Berg
#rnx

[CL 14940619 by paul chipchase in ue5-main branch]
2020-12-16 14:06:26 -04:00
Joakim Lindqvist
254b31ad63 Bumped http request pool size to attempt to reduce amount of time we spend waiting on the pool when fetching from Jupiter
#fyi johan.berg

[CL 14931109 by Joakim Lindqvist in ue5-main branch]
2020-12-15 11:28:39 -04:00
Johan Berg
9a55c52bbf Fix for Jupiter token expiring
For very long cooking sessions the Jupiter token could expire multiple times. That could cause the failed login counter to disable further login attempts. Fix this by resetting the counter on every successful login. Also reduced verbosity of 401 errors so that CI doesn't pick up expired tokens as errors.

#rb joakim.lindqvist

[CL 14887529 by Johan Berg in ue5-main branch]
2020-12-09 08:30:46 -04:00
Marcus Wassmer
3b81cf8201 Merging using //UE5/Main_to_//UE5/Release-Engine-Staging @14384769
autoresolved files
#rb none

[CL 14384911 by Marcus Wassmer in ue5-main branch]
2020-09-24 00:43:27 -04:00
Marc Audy
11f5b21210 Merging //UE5/Release-Engine-Staging @ 13752110 to Main (//UE5/Main)
#rnx

[CL 13753156 by Marc Audy in ue5-main branch]
2020-06-23 18:40:00 -04:00