andrew grant bccf8a1464 DDC Optimization for slow caches in the hierarchy - ...

* Reduce the calls to CachedDataProbablyExists in DDC hierarchy when forward/backfilling other caches and skip entirely for slower caches.
* Added some comments for future visitors.

#tests Benchmark PIE test with warm & cold caches. On fiber shows a ~20 sec gain with a warm cache and 3mins with cold to AthenaTestBed (likely a 3-5x multiple on slower connections).

#rb swarm
[at]devin.doucette

#ROBOMERGE-SOURCE: CL 12669758 via CL 12669760 via CL 12669761 via CL 12669762
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v675-12543919)

[CL 12669764 by andrew grant in Main branch]

2020-04-07 21:48:22 -04:00

..

Private

DDC Optimization for slow caches in the hierarchy -

2020-04-07 21:48:22 -04:00

Public

Moved some DDC debug options into a common class that can be provided to a backend to simulate certain behavior.

2020-03-26 20:55:39 -04:00

DerivedDataCache.Build.cs

Add experimental DDC backend which fetches objects from an S3 bucket. Bucket is populated by a job running periodically on build machines, and generates a JSON manifest used to determine files to download.

2020-03-12 13:29:21 -04:00