Files
UnrealEngineUWP/Engine/Source/Developer/DerivedDataCache/Public/DerivedDataCacheUsageStats.h
andrew grant 3cc86b3d2f Optimizations for DDC access on remote / slow drives.
If a filesystem node is not available not prompt the user and optionally retry incase they need to mount a drive or start VPN

Fiilesystem nodes now perform a speed test using a selection of 'DDC sized' files to determine a classification (local, fast, ok, slow).

Add a new 'ConsiderSlowAt' property to the 'Filesystem' DDC node type. If latency to the node is >= this value then the node will be marked as slow which disables touch'ing and reduces file stats

Interface Changes

- Add the concept of a speed class to nodes
- Add GetName to nodes for better debugging / logging
- WouldCache query that allows caches to opt of of consideration early and avoid async tasks being created.
- Create a new 'FileBackedDerivedDataBackend' class that's the for the memory/boot backend and future classes
- TryToPrefetch interface functions for future use

Behavior Changes

- Moved parameter parsing into FileSysteDerivedDataBackend as things were getting out of hand
- FileSystemDerivedDataBackend now performs a speed test using 'DDC sized' files in separate directories and applies a classification
- Slow locations turn off touching of data on read
- Slow locations always return true for CachedDataProbablyExists. It's faster just to try to read and fail
- If the shared DDC is not available the user is prompted incase they need to mount it.

[at]ben.marsh [at]josh.engebretson
#rb swarm
#tests lots of PIE runs with / without this option


#ROBOMERGE-SOURCE: CL 12387516 via CL 12387517 via CL 12396622
#ROBOMERGE-BOT: (v671-12333473)

[CL 12396757 by andrew grant in Release-Engine-Staging branch]
2020-03-24 19:12:36 -04:00

96 lines
4.0 KiB
C++

// Copyright Epic Games, Inc. All Rights Reserved.
#pragma once
#include "CoreMinimal.h"
#include "ProfilingDebugging/CookStats.h"
/**
* Usage stats for the derived data cache nodes. At the end of the app or commandlet, the DDC
* can be asked to gather usage stats for each of the nodes in the DDC graph, which are accumulated
* into a TMap of Name->Stats. The Stats portion is this class.
*
* The class exposes various high-level routines to time important aspects of the DDC, mostly
* focusing on performance of GetCachedData, PutCachedData, and CachedDataProbablyExists. This class
* will track time taken, calls made, hits, misses, bytes processed, and do it for two buckets:
* 1) the main thread and 2) all other threads. The reason is because any time spent in the DDC on the
* main thread is considered meaningful, as DDC access is generally expected to be async from helper threads.
*
* The class goes through a bit of trouble to use thread-safe access calls for task-thread usage, and
* simple, fast accumulators for game-thread usage, since it's guaranteed to not be written to concurrently.
* The class also limits itself to checking the thread once at construction.
*
* Usage would be something like this in a concrete FDerivedDataBackendInterface implementation:
* class MyBackend : public FDerivedDataBackendInterface
* {
* FDerivedDataCacheUsageStats UsageStats;
* public:
* <override CachedDataProbablyExists>
* {
* auto Timer = UsageStats.TimeExists();
* ...
* }
* <override GetCachedData>
* {
* auto Timer = UsageStats.TimeGet();
* ...
* <if it's a cache hit> Timer.AddHit(DataSize);
* // Misses are automatically tracked
* }
* <override PutCachedData>
* {
* auto Timer = UsageStats.TimePut();
* ...
* <if the data will really be Put> Timer.AddHit(DataSize);
* // Misses are automatically tracked
* }
* <override GatherUsageStats>
* {
* // Add this node's UsageStats to the usage map. Your Key name should be UNIQUE to the entire graph (so use the file name, or pointer to this if you have to).
* UsageStatsMap.Add(FString::Printf(TEXT("%s: <Some unique name for this node instance>"), *GraphPath), UsageStats);
* }
* }
*/
class FDerivedDataCacheUsageStats
{
#if ENABLE_COOK_STATS
public:
/** Call this at the top of the CachedDataProbablyExists override. auto Timer = TimeProbablyExists(); */
FCookStats::FScopedStatsCounter TimeProbablyExists()
{
return FCookStats::FScopedStatsCounter(ExistsStats);
}
/** Call this at the top of the GetCachedData override. auto Timer = TimeGet(); Use AddHit on the returned type to track a cache hit. */
FCookStats::FScopedStatsCounter TimeGet()
{
return FCookStats::FScopedStatsCounter(GetStats);
}
/** Call this at the top of the PutCachedData override. auto Timer = TimePut(); Use AddHit on the returned type to track a cache hit. */
FCookStats::FScopedStatsCounter TimePut()
{
return FCookStats::FScopedStatsCounter(PutStats);
}
/** Call this at the top of the PutCachedData override. auto Timer = TimePut(); Use AddHit on the returned type to track a cache hit. */
FCookStats::FScopedStatsCounter TimePrefetch()
{
return FCookStats::FScopedStatsCounter(PrefetchStats);
}
void LogStats(FCookStatsManager::AddStatFuncRef AddStat, const FString& StatName, const FString& NodeName) const
{
GetStats.LogStats(AddStat, StatName, NodeName, TEXT("Get"));
PutStats.LogStats(AddStat, StatName, NodeName, TEXT("Put"));
ExistsStats.LogStats(AddStat, StatName, NodeName, TEXT("Exists"));
PrefetchStats.LogStats(AddStat, StatName, NodeName, TEXT("Prefetch"));
}
// expose these publicly for low level access. These should really never be accessed directly except when finished accumulating them.
FCookStats::CallStats GetStats;
FCookStats::CallStats PutStats;
FCookStats::CallStats ExistsStats;
FCookStats::CallStats PrefetchStats;
#endif
};