andrew grant 3cc86b3d2f Optimizations for DDC access on remote / slow drives. ...

If a filesystem node is not available not prompt the user and optionally retry incase they need to mount a drive or start VPN

Fiilesystem nodes now perform a speed test using a selection of 'DDC sized' files to determine a classification (local, fast, ok, slow).

Add a new 'ConsiderSlowAt' property to the 'Filesystem' DDC node type. If latency to the node is >= this value then the node will be marked as slow which disables touch'ing and reduces file stats

Interface Changes

- Add the concept of a speed class to nodes
- Add GetName to nodes for better debugging / logging
- WouldCache query that allows caches to opt of of consideration early and avoid async tasks being created.
- Create a new 'FileBackedDerivedDataBackend' class that's the for the memory/boot backend and future classes
- TryToPrefetch interface functions for future use

Behavior Changes

- Moved parameter parsing into FileSysteDerivedDataBackend as things were getting out of hand
- FileSystemDerivedDataBackend now performs a speed test using 'DDC sized' files in separate directories and applies a classification
- Slow locations turn off touching of data on read
- Slow locations always return true for CachedDataProbablyExists. It's faster just to try to read and fail
- If the shared DDC is not available the user is prompted incase they need to mount it.

[at]ben.marsh [at]josh.engebretson
#rb swarm
#tests lots of PIE runs with / without this option


#ROBOMERGE-SOURCE: CL 12387516 via CL 12387517 via CL 12396622
#ROBOMERGE-BOT: (v671-12333473)

[CL 12396757 by andrew grant in Release-Engine-Staging branch]

2020-03-24 19:12:36 -04:00

13 KiB

Raw Blame History

View Raw