Commit Graph

186 Commits

Author SHA1 Message Date
Devin Doucette
a05f939c6b CompactBinary: Renamed and added overloads to FCbWriter to clarify behavior and allow specifying name and value in the same function
#rb Zousar.Shaker
#rnx

[CL 15303766 by Devin Doucette in ue5-main branch]
2021-02-03 18:37:36 -04:00
Devin Doucette
d273616103 CompactBinary: Renamed Reference to Attachment to clarify the expected relationship between reference fields and package attachments
#rb trivial
#rnx

[CL 15299118 by Devin Doucette in ue5-main branch]
2021-02-03 14:33:20 -04:00
Devin Doucette
594e45bada DDC: Fixed an error in FCacheRecord::SetBinary
#rb none
#rnx

[CL 15278526 by Devin Doucette in ue5-main branch]
2021-02-02 02:24:52 -04:00
Devin Doucette
90298de5c2 DDC: Added the value hash to FCacheRecord and made it easier to access object values
#rb Matt.Peters
#rnx

[CL 15237973 by Devin Doucette in ue5-main branch]
2021-01-28 09:50:58 -04:00
Devin Doucette
e9b798ca9c CompactBinary: Converted from FBlake3Hash to FIoHash
#rb Zousar.Shaker
#rnx

[CL 15212418 by Devin Doucette in ue5-main branch]
2021-01-26 18:17:02 -04:00
danny couture
0790ec6eb0 Create a separate thread pool for the I/O portion of DDC requests to avoid clogging the workers with long I/O waits
Add queued work priority support to auto delete async tasks and schedule DDC Puts as Low priority
ThreadPool threads are now honoring the names they've been given on creation
Move the I/O only async DDC Get into the DDC ThreadPool with higher priority (Normal) than DCC Puts

#rb Devin.Doucette

[CL 15177897 by danny couture in ue5-main branch]
2021-01-25 10:55:26 -04:00
Devin Doucette
3fdb9bc4e5 CompactBinary: Rename Reference to CompactBinaryReference for clarity and consistency
#rb Zousar.Shaker
#rnx

[CL 15169906 by Devin Doucette in ue5-main branch]
2021-01-23 02:45:14 -04:00
mark lintott
6c3abefe2f #rb Devin.Doucette
Added stats for Cloud based DCC ( ie Jupiter ) to DDC.Summary
Added BackEnd name to DDC.Summary
Added HasLocalDDC, HasSharedDDC, HasCloudDDC to DDC.Summary

[CL 15164347 by mark lintott in ue5-main branch]
2021-01-22 12:20:09 -04:00
Marc Audy
bc88b73a29 Merge Release-Engine-Staging to Main @ CL# 15151250
Represents UE4/Main @ 15133763

[CL 15158774 by Marc Audy in ue5-main branch]
2021-01-21 16:22:06 -04:00
Devin Doucette
0c7d307086 DDC: Removed ECachePolicy::Overwrite because there is not a valid use case for it
#rb Zousar.Shaker
#rnx

[CL 15154520 by Devin Doucette in ue5-main branch]
2021-01-21 11:37:24 -04:00
Joakim Lindqvist
9bfbbf2c75 Clarified comment on why transient data can be disabled.
[CL 15154162 by Joakim Lindqvist in ue5-main branch]
2021-01-21 10:41:53 -04:00
Joakim Lindqvist
87ef345981 Do not delete transient data from Jupiter, it will clean itself up as needed.
Also made Jupiter a slow backend, as this generally is closer to the typical speed people have to it. We should make this automatic based on the latency we find from the health check call.

#rb devin.doucette

[CL 15154130 by Joakim Lindqvist in ue5-main branch]
2021-01-21 10:36:11 -04:00
Devin Doucette
7bd5b197e8 DDC: Added the new cache interface UE::DerivedData::ICache
ICache will eventually replace the existing cache access functions on FDerivedDataCacheInterface. The notable differences relative to the existing interface are:

- It is asynchronous with completion callbacks by default, and allows waiting when necessary.
- It allows batch requests containing multiple cache records by default.
- It uses a structured cache key made up of a bucket name and a hash.
- It supports prioritization and cancellation of requests.
- It supports structured cache records with metadata.
- It supports attachments to allow multi-part cache records.
- It uses a typed request object instead of an opaque integer handle.
- It provides cache policy flags to control how to query and store cache records, and which parts of a cache record to fetch.

This initial implementation of ICache is totally synchronous and uses the existing cache backends for storage.

#rb Zousar.Shaker

[CL 15152247 by Devin Doucette in ue5-main branch]
2021-01-21 01:57:01 -04:00
Devin Doucette
2fe890d75a DDC: Implemented batched CachedDataProbablyExists requests in the Jupiter backend
#rb Johan.Berg, Zousar.Shaker

[CL 15037688 by Devin Doucette in ue5-main branch]
2021-01-11 11:34:27 -04:00
Marc Audy
bf80889353 UE5/Release-Engine-Staging to UE5/Main
This represents UE4/Main up to CL# 14958402

[CL 15028197 by Marc Audy in ue5-main branch]
2021-01-08 19:56:07 -04:00
Devin Doucette
6b3ce5c706 DDC: Changed CachedDataProbablyExistsBatch to call CachedDataProbablyExists for a batch size of 1
#rb Zousar.Shaker
#rnx

[CL 15019812 by Devin Doucette in ue5-main branch]
2021-01-08 11:44:45 -04:00
Devin Doucette
9010b3e1a5 DDC: Return a status from PutCachedData
The async put backend now uses this new status to avoid calling CachedDataProbablyExists after every put, which has a high cost for certain backends.

#rb Zousar.Shaker
#fyi Joakim.Lindqvist, Stefan.Boberg

[CL 15000972 by Devin Doucette in ue5-main branch]
2021-01-06 14:18:01 -04:00
Devin Doucette
e6401b0897 DDC: Added variants of CachedDataProbablyExists that check multiple keys
#rb Matt.Peters
#fyi Joakim.Lindqvist, Stefan.Boberg

[CL 14998596 by Devin Doucette in ue5-main branch]
2021-01-06 09:27:11 -04:00
Joakim Lindqvist
0f54b08b77 Considering Jupiter backend as a fast backend, in the same way the SMB share is configured. Works around problematic assumptions in the hirarchal backend were it will not accept puts that primarilly live in slow backends (e.g. if no fast backends are available)
#rb devin.doucette

[CL 14946668 by Joakim Lindqvist in ue5-main branch]
2020-12-17 10:10:46 -04:00
paul chipchase
45bef5bc27 Jupiter DDC head requests will no longer give errors if the response code is 204 (which is valid)
- The Io service currently returns response 204 when a HEAD request finds an entry, however the Europa service has been returning 200. The goal here is to update the DDC HEAD requests so that they will work with a response of 200 OR 204 so that we can unify the Jupiter apis.
- Added a utility method to FRequest that can be used to test if the response code is in the range that we consider a success.
- Updated FHttpDerivedDataBackend::CachedDataProbablyExists to use this.
- Updated the FRequest logging code to use this as well to avoid reporting successful operations as errors.

#rb Joakim.Lindqvist
#fyi Johan.Berg
#rnx

[CL 14940619 by paul chipchase in ue5-main branch]
2020-12-16 14:06:26 -04:00
Joakim Lindqvist
254b31ad63 Bumped http request pool size to attempt to reduce amount of time we spend waiting on the pool when fetching from Jupiter
#fyi johan.berg

[CL 14931109 by Joakim Lindqvist in ue5-main branch]
2020-12-15 11:28:39 -04:00
Marc Audy
ada7c144fa Merge //UE5/Release-Engine-Staging @14903491 to //UE5/Main
[CL 14906022 by Marc Audy in ue5-main branch]
2020-12-11 14:21:20 -04:00
Johan Berg
9a55c52bbf Fix for Jupiter token expiring
For very long cooking sessions the Jupiter token could expire multiple times. That could cause the failed login counter to disable further login attempts. Fix this by resetting the counter on every successful login. Also reduced verbosity of 401 errors so that CI doesn't pick up expired tokens as errors.

#rb joakim.lindqvist

[CL 14887529 by Johan Berg in ue5-main branch]
2020-12-09 08:30:46 -04:00
Francis Hurteau
5f773af42b Revert the reentrancy guard from cl 14528803 which was uneeded and actually causing local ddc from being properly filled
Added profiling scope for Async DDC put

#rb Danny.Couture

[CL 14532323 by Francis Hurteau in ue5-main branch]
2020-10-21 06:45:37 -04:00
Francis Hurteau
036678deb5 Fix typo in comment
#rb trivial

[CL 14528903 by Francis Hurteau in ue5-main branch]
2020-10-20 16:45:50 -04:00