izzy/UnrealEngineUWP
0
0
Fork 0
mirror of https://github.com/izzy2lost/UnrealEngineUWP.git synced 2026-03-26 18:15:20 -07:00
Code Issues Packages Projects Releases Wiki Activity
Files
f9272bdebab035df1dfd235448d40e577fedd8fb
UnrealEngineUWP/Engine/Source/Developer/DerivedDataCache/Private/ZenCacheStore.cpp

1426 lines
48 KiB
C++
Raw Normal View History

Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
// Copyright Epic Games, Inc. All Rights Reserved.
DDC: Temporary compile fix for platforms that should not include DerivedDataCache #preflight skip #rb none #rnx [CL 20445864 by Devin Doucette in ue5-main branch]
2022-05-31 23:25:37 -04:00
#include "DerivedDataLegacyCacheStore.h"
Move Zen module public headers into an "Experimental" subfolder to have them reflect the feature status. Fix locations that include the headers to use the "Experimental" subfolder and re-sort includes. No functional change, purely a move, search+replace, and sort of includes. #rb matt.peters #preflight 63939eebc415e8dc78d35d63 [CL 23468934 by Zousar Shaker in ue5-main branch]
2022-12-09 16:11:41 -05:00
#include "Experimental/ZenServerInterface.h"
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
#if UE_WITH_ZEN
#include "BatchView.h"
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
#include "DerivedDataBackendInterface.h"
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
#include "DerivedDataCachePrivate.h"
#include "DerivedDataCacheRecord.h"
#include "DerivedDataCacheUsageStats.h"
#include "DerivedDataChunk.h"
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
#include "DerivedDataRequest.h"
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
#include "DerivedDataRequestOwner.h"
Move Zen module public headers into an "Experimental" subfolder to have them reflect the feature status. Fix locations that include the headers to use the "Experimental" subfolder and re-sort includes. No functional change, purely a move, search+replace, and sort of includes. #rb matt.peters #preflight 63939eebc415e8dc78d35d63 [CL 23468934 by Zousar Shaker in ue5-main branch]
2022-12-09 16:11:41 -05:00
#include "Experimental/ZenStatistics.h"
Add support for Zen DDC sandboxes which are separate instances of zenserver with separate data paths and can be optionally flushed from run-to-run. It is used to implement Warm and Cold DDC graphs using Zen sandboxes for projects/platforms that use Zen for local DDC by default, and to provide new ZenColdDDC and ZenWarmDDC as new graphs that have Cold/Warm behavior for projects that don't use Zen for local DDC by default. #rb devin.doucette #preflight 63b493d5202bee5e27a645ed [CL 23569648 by Zousar Shaker in ue5-main branch]
2023-01-03 15:58:30 -05:00
#include "HAL/FileManager.h"
2nd submit: Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 #preflight 629642a1be0ae0b33a46d012 [CL 20438549 by Zousar Shaker in ue5-main branch]
2022-05-31 12:42:17 -04:00
#include "Http/HttpClient.h"
Select maximum HTTP connections for ZenCacheStore based on logical core count (clamped between 8 and 64). Speculative solution for farm cooks using Zen experiencing "couldn't connect errors". #rb trivial #fyi devin.doucette #rnx #preflight 62b00bd92405dc6102315616 [CL 20734103 by Zousar Shaker in ue5-main branch]
2022-06-20 02:08:49 -04:00
#include "Math/UnrealMathUtility.h"
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
#include "Misc/App.h"
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
#include "Misc/ConfigCacheIni.h"
#include "Misc/Optional.h"
#include "ProfilingDebugging/CountersTrace.h"
#include "ProfilingDebugging/CpuProfilerTrace.h"
#include "Serialization/CompactBinary.h"
#include "Serialization/CompactBinaryPackage.h"
#include "Serialization/CompactBinaryWriter.h"
2nd submit: Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 #preflight 629642a1be0ae0b33a46d012 [CL 20438549 by Zousar Shaker in ue5-main branch]
2022-05-31 12:42:17 -04:00
#include "Serialization/LargeMemoryWriter.h"
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
#include "Serialization/MemoryReader.h"
2nd submit: Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 #preflight 629642a1be0ae0b33a46d012 [CL 20438549 by Zousar Shaker in ue5-main branch]
2022-05-31 12:42:17 -04:00
#include "Templates/Function.h"
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
#include "ZenBackendUtils.h"
#include "ZenSerialization.h"
Only report local stats for Zen DDC. ZenCacheStore now only tracks what the client sees, regardless if the zenserver instance gets a hit from a configured upstream. As we don't currently have any upstreams configured in the zen server instance this should give a more accurate number for hit/miss stats that matches the file cache. #preflight 63eabb1bb96a6b2718326d15 #rb zousar.shaker devin.doucette [CL 24206594 by dan engelbrecht in ue5-main branch]
2023-02-14 03:09:44 -05:00
TRACE_DECLARE_INT_COUNTER(ZenDDC_Get, TEXT("ZenDDC Get"));
TRACE_DECLARE_INT_COUNTER(ZenDDC_GetHit, TEXT("ZenDDC Get Hit"));
TRACE_DECLARE_INT_COUNTER(ZenDDC_Put, TEXT("ZenDDC Put"));
TRACE_DECLARE_INT_COUNTER(ZenDDC_PutHit, TEXT("ZenDDC Put Hit"));
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
TRACE_DECLARE_INT_COUNTER(ZenDDC_BytesReceived, TEXT("ZenDDC Bytes Received"));
Only report local stats for Zen DDC. ZenCacheStore now only tracks what the client sees, regardless if the zenserver instance gets a hit from a configured upstream. As we don't currently have any upstreams configured in the zen server instance this should give a more accurate number for hit/miss stats that matches the file cache. #preflight 63eabb1bb96a6b2718326d15 #rb zousar.shaker devin.doucette [CL 24206594 by dan engelbrecht in ue5-main branch]
2023-02-14 03:09:44 -05:00
TRACE_DECLARE_INT_COUNTER(ZenDDC_BytesSent, TEXT("ZenDDC Bytes Sent"));
TRACE_DECLARE_INT_COUNTER(ZenDDC_CacheRecordRequestCountInFlight, TEXT("ZenDDC CacheRecord Request Count"));
TRACE_DECLARE_INT_COUNTER(ZenDDC_ChunkRequestCountInFlight, TEXT("ZenDDC Chunk Request Count"));
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
namespace UE::DerivedData
{
template<typename T>
void ForEachBatch(const int32 BatchSize, const int32 TotalCount, T&& Fn)
{
check(BatchSize > 0);
if (TotalCount > 0)
{
const int32 BatchCount = FMath::DivideAndRoundUp(TotalCount, BatchSize);
const int32 Last = TotalCount - 1;
for (int32 BatchIndex = 0; BatchIndex < BatchCount; BatchIndex++)
{
const int32 BatchFirstIndex = BatchIndex * BatchSize;
const int32 BatchLastIndex = FMath::Min(BatchFirstIndex + BatchSize - 1, Last);
Fn(BatchFirstIndex, BatchLastIndex);
}
}
}
/**
* Backend for a HTTP based caching service (Zen)
*/
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
class FZenCacheStore final : public ILegacyCacheStore
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
{
public:
/**
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
* Creates the cache store client, checks health status and attempts to acquire an access token.
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
*
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
* @param ServiceUrl Base url to the service including scheme.
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
* @param Namespace Namespace to use.
*/
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
FZenCacheStore(const TCHAR* ServiceUrl, const TCHAR* Namespace);
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
Add support for Zen DDC sandboxes which are separate instances of zenserver with separate data paths and can be optionally flushed from run-to-run. It is used to implement Warm and Cold DDC graphs using Zen sandboxes for projects/platforms that use Zen for local DDC by default, and to provide new ZenColdDDC and ZenWarmDDC as new graphs that have Cold/Warm behavior for projects that don't use Zen for local DDC by default. #rb devin.doucette #preflight 63b493d5202bee5e27a645ed [CL 23569648 by Zousar Shaker in ue5-main branch]
2023-01-03 15:58:30 -05:00
FZenCacheStore(UE::Zen::FServiceSettings&& InSettings, const TCHAR* Namespace);
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
inline FString GetName() const { return ZenService.GetInstance().GetURL(); }
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
/**
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
* Checks if cache service is usable (reachable and accessible).
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
* @return true if usable
*/
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
inline bool IsUsable() const { return bIsUsable; }
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
// ICacheStore
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
void Put(
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
TConstArrayView<FCachePutRequest> Requests,
IRequestOwner& Owner,
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
FOnCachePutComplete&& OnComplete = FOnCachePutComplete()) final;
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
void Get(
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
TConstArrayView<FCacheGetRequest> Requests,
IRequestOwner& Owner,
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
FOnCacheGetComplete&& OnComplete) final;
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
void PutValue(
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
TConstArrayView<FCachePutValueRequest> Requests,
IRequestOwner& Owner,
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
FOnCachePutValueComplete&& OnComplete = FOnCachePutValueComplete()) final;
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
void GetValue(
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
TConstArrayView<FCacheGetValueRequest> Requests,
IRequestOwner& Owner,
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
FOnCacheGetValueComplete&& OnComplete = FOnCacheGetValueComplete()) final;
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
void GetChunks(
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
TConstArrayView<FCacheGetChunkRequest> Requests,
IRequestOwner& Owner,
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
FOnCacheGetChunkComplete&& OnComplete) final;
// ILegacyCacheStore
void LegacyStats(FDerivedDataCacheStatsNode& OutNode) final;
bool LegacyDebugOptions(FBackendDebugOptions& Options) final;
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
private:
Add support for Zen DDC sandboxes which are separate instances of zenserver with separate data paths and can be optionally flushed from run-to-run. It is used to implement Warm and Cold DDC graphs using Zen sandboxes for projects/platforms that use Zen for local DDC by default, and to provide new ZenColdDDC and ZenWarmDDC as new graphs that have Cold/Warm behavior for projects that don't use Zen for local DDC by default. #rb devin.doucette #preflight 63b493d5202bee5e27a645ed [CL 23569648 by Zousar Shaker in ue5-main branch]
2023-01-03 15:58:30 -05:00
void Initialize(const TCHAR* Namespace);
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
bool IsServiceReady();
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
static FCompositeBuffer SaveRpcPackage(const FCbPackage& Package);
THttpUniquePtr<IHttpRequest> CreateRpcRequest();
using FOnRpcComplete = TUniqueFunction<void(THttpUniquePtr<IHttpResponse>& HttpResponse, FCbPackage& Response)>;
void EnqueueAsyncRpc(IRequestOwner& Owner, FCbObject RequestObject, FOnRpcComplete&& OnComplete);
void EnqueueAsyncRpc(IRequestOwner& Owner, const FCbPackage& RequestPackage, FOnRpcComplete&& OnComplete);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
template <typename T, typename... ArgTypes>
static TRefCountPtr<T> MakeAsyncOp(ArgTypes&&... Args)
{
// TODO: This should in-place construct from a pre-allocated memory pool
return TRefCountPtr<T>(new T(Forward<ArgTypes>(Args)...));
}
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
private:
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
class FPutOp;
class FGetOp;
class FPutValueOp;
class FGetValueOp;
class FGetChunksOp;
class FCbPackageReceiver;
class FAsyncCbPackageReceiver;
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
FString Namespace;
UE::Zen::FScopeZenService ZenService;
mutable FDerivedDataCacheUsageStats UsageStats;
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
THttpUniquePtr<IHttpConnectionPool> ConnectionPool;
FHttpRequestQueue RequestQueue;
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
bool bIsUsable = false;
Allow support for receiving local file references for cache items Companion PR in zen: https://github.com/EpicGames/zen/pull/184 #rb stefan.boberg #preflight 6368ce57d0174259cc1e7b7a [CL 23009037 by dan engelbrecht in ue5-main branch]
2022-11-07 05:33:21 -05:00
bool bIsLocalConnection = false;
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
int32 BatchPutMaxBytes = 1024*1024;
int32 CacheRecordBatchSize = 8;
int32 CacheChunksBatchSize = 8;
FBackendDebugOptions DebugOptions;
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
TAnsiStringBuilder<256> RpcUri;
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
};
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
class FZenCacheStore::FPutOp final : public FThreadSafeRefCountedObject
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
public:
FPutOp(FZenCacheStore& InCacheStore,
IRequestOwner& InOwner,
const TConstArrayView<FCachePutRequest> InRequests,
FOnCachePutComplete&& InOnComplete)
: CacheStore(InCacheStore)
, Owner(InOwner)
, Requests(InRequests)
, Batches(Requests, [this](const FCachePutRequest& NextRequest) {return BatchGroupingFilter(NextRequest);})
, OnComplete(MoveTemp(InOnComplete))
{
Only report local stats for Zen DDC. ZenCacheStore now only tracks what the client sees, regardless if the zenserver instance gets a hit from a configured upstream. As we don't currently have any upstreams configured in the zen server instance this should give a more accurate number for hit/miss stats that matches the file cache. #preflight 63eabb1bb96a6b2718326d15 #rb zousar.shaker devin.doucette [CL 24206594 by dan engelbrecht in ue5-main branch]
2023-02-14 03:09:44 -05:00
TRACE_COUNTER_ADD(ZenDDC_Put, (int64)Requests.Num());
COOK_STAT(Timers.Reserve(Requests.Num()));
COOK_STAT(for (const FCachePutRequest& Request : Requests)
{
Timers.Add(Request.Record.GetKey(), CacheStore.UsageStats.TimePut());
})
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
void IssueRequests()
{
FRequestBarrier Barrier(Owner);
for (TArrayView<const FCachePutRequest> Batch : Batches)
{
FCbPackage BatchPackage;
FCbWriter BatchWriter;
BatchWriter.BeginObject();
{
BatchWriter << ANSITEXTVIEW("Method") << "PutCacheRecords";
Add "Accept" field in Zen RPC request to enable new style RPC responses. See https://github.com/EpicGames/zen/pull/180 for Zen side change. #rb stefan.boberg zousar.shaker #preflight 6347c2568a0a7b2adcbe4d71 [CL 22505182 by dan engelbrecht in ue5-main branch]
2022-10-13 11:17:53 -04:00
BatchWriter.AddInteger(ANSITEXTVIEW("Accept"), Zen::Http::kCbPkgMagic);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
BatchWriter.BeginObject(ANSITEXTVIEW("Params"));
{
ECachePolicy BatchDefaultPolicy = Batch[0].Policy.GetRecordPolicy();
BatchWriter << ANSITEXTVIEW("DefaultPolicy") << *WriteToString<128>(BatchDefaultPolicy);
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
BatchWriter.AddString(ANSITEXTVIEW("Namespace"), CacheStore.Namespace);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
BatchWriter.BeginArray(ANSITEXTVIEW("Requests"));
for (const FCachePutRequest& Request : Batch)
{
const FCacheRecord& Record = Request.Record;
BatchWriter.BeginObject();
{
BatchWriter.SetName(ANSITEXTVIEW("Record"));
Record.Save(BatchPackage, BatchWriter);
DDC: Fixed deprecation warnings in the Zen cache store #jira UE-157180 #rb none #rnx #ROBOMERGE-AUTHOR: devin.doucette #ROBOMERGE-SOURCE: CL 20745126 via CL 20745729 #ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v970-20704180) [CL 20751392 by devin doucette in ue5-main branch]
2022-06-20 23:40:25 -04:00
if (!Request.Policy.IsDefault())
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
DDC: Fixed deprecation warnings in the Zen cache store #jira UE-157180 #rb none #rnx #ROBOMERGE-AUTHOR: devin.doucette #ROBOMERGE-SOURCE: CL 20745126 via CL 20745729 #ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v970-20704180) [CL 20751392 by devin doucette in ue5-main branch]
2022-06-20 23:40:25 -04:00
BatchWriter << ANSITEXTVIEW("Policy") << Request.Policy;
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
}
BatchWriter.EndObject();
}
BatchWriter.EndArray();
}
BatchWriter.EndObject();
}
BatchWriter.EndObject();
BatchPackage.SetObject(BatchWriter.Save().AsObject());
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
auto OnRpcComplete = [this, OpRef = TRefCountPtr<FPutOp>(this), Batch](THttpUniquePtr<IHttpResponse>& HttpResponse, FCbPackage& Response)
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
int32 RequestIndex = 0;
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
if (HttpResponse->GetErrorCode() == EHttpErrorCode::None)
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
const FCbObject& ResponseObj = Response.GetObject();
for (FCbField ResponseField : ResponseObj[ANSITEXTVIEW("Result")])
{
if (RequestIndex >= Batch.Num())
{
++RequestIndex;
continue;
}
const FCachePutRequest& Request = Batch[RequestIndex++];
const FCacheKey& Key = Request.Record.GetKey();
bool bPutSucceeded = ResponseField.AsBool();
if (CacheStore.DebugOptions.ShouldSimulatePutMiss(Key))
{
UE_LOG(LogDerivedDataCache, Verbose, TEXT("%s: Simulated miss for put of %s from '%s'"),
*CacheStore.GetName(), *WriteToString<96>(Key), *Request.Name);
bPutSucceeded = false;
}
bPutSucceeded ? OnHit(Request) : OnMiss(Request);
}
if (RequestIndex != Batch.Num())
{
UE_LOG(LogDerivedDataCache, Warning,
DDC: Added more detail to warnings from the Zen cache client #preflight 62d9ced6d76ea4b503d3f487 #rb Zousar.Shaker #rnx [CL 21214276 by Devin Doucette in ue5-main branch]
2022-07-21 23:01:51 -04:00
TEXT("%s: Invalid response received from PutCacheRecords RPC: %d results expected, received %d, from %s"),
*CacheStore.GetName(), Batch.Num(), RequestIndex, *WriteToString<256>(*HttpResponse));
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
}
for (const FCachePutRequest& Request : Batch.RightChop(RequestIndex))
{
OnMiss(Request);
}
};
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
CacheStore.EnqueueAsyncRpc(Owner, BatchPackage, MoveTemp(OnRpcComplete));
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
}
private:
EBatchView BatchGroupingFilter(const FCachePutRequest& NextRequest)
{
const FCacheRecord& Record = NextRequest.Record;
uint64 RecordSize = sizeof(FCacheKey) + Record.GetMeta().GetSize();
for (const FValueWithId& Value : Record.GetValues())
{
RecordSize += Value.GetData().GetCompressedSize();
}
BatchSize += RecordSize;
if (BatchSize > CacheStore.BatchPutMaxBytes)
{
BatchSize = RecordSize;
return EBatchView::NewBatch;
}
return EBatchView::Continue;
}
void OnHit(const FCachePutRequest& Request)
{
const FCacheKey& Key = Request.Record.GetKey();
UE_LOG(LogDerivedDataCache, Verbose, TEXT("%s: Cache Put complete for %s from '%s'"),
*CacheStore.GetName(), *WriteToString<96>(Key), *Request.Name);
Only report local stats for Zen DDC. ZenCacheStore now only tracks what the client sees, regardless if the zenserver instance gets a hit from a configured upstream. As we don't currently have any upstreams configured in the zen server instance this should give a more accurate number for hit/miss stats that matches the file cache. #preflight 63eabb1bb96a6b2718326d15 #rb zousar.shaker devin.doucette [CL 24206594 by dan engelbrecht in ue5-main branch]
2023-02-14 03:09:44 -05:00
int64 SentSize = 0;
for (const FValueWithId& Value : Request.Record.GetValues())
{
SentSize += Value.GetData().GetCompressed().GetSize();
}
TRACE_COUNTER_ADD(ZenDDC_BytesSent, SentSize);
TRACE_COUNTER_INCREMENT(ZenDDC_PutHit);
COOK_STAT(Timers[Key].AddHit(SentSize));
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
OnComplete(Request.MakeResponse(EStatus::Ok));
};
void OnMiss(const FCachePutRequest& Request)
{
const FCacheKey& Key = Request.Record.GetKey();
UE_LOG(LogDerivedDataCache, Verbose, TEXT("%s: Cache Put miss for '%s' from '%s'"),
*CacheStore.GetName(), *WriteToString<96>(Key), *Request.Name);
Only report local stats for Zen DDC. ZenCacheStore now only tracks what the client sees, regardless if the zenserver instance gets a hit from a configured upstream. As we don't currently have any upstreams configured in the zen server instance this should give a more accurate number for hit/miss stats that matches the file cache. #preflight 63eabb1bb96a6b2718326d15 #rb zousar.shaker devin.doucette [CL 24206594 by dan engelbrecht in ue5-main branch]
2023-02-14 03:09:44 -05:00
int64 SentSize = 0;
for (const FValueWithId& Value : Request.Record.GetValues())
{
SentSize += Value.GetData().GetCompressed().GetSize();
}
COOK_STAT(Timers[Key].AddMiss(SentSize));
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
OnComplete(Request.MakeResponse(EStatus::Error));
};
FZenCacheStore& CacheStore;
IRequestOwner& Owner;
const TArray<FCachePutRequest, TInlineAllocator<1>> Requests;
uint64 BatchSize = 0;
TBatchView<const FCachePutRequest> Batches;
FOnCachePutComplete OnComplete;
Only report local stats for Zen DDC. ZenCacheStore now only tracks what the client sees, regardless if the zenserver instance gets a hit from a configured upstream. As we don't currently have any upstreams configured in the zen server instance this should give a more accurate number for hit/miss stats that matches the file cache. #preflight 63eabb1bb96a6b2718326d15 #rb zousar.shaker devin.doucette [CL 24206594 by dan engelbrecht in ue5-main branch]
2023-02-14 03:09:44 -05:00
COOK_STAT(TMap<FCacheKey, FCookStats::FScopedStatsCounter> Timers);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
};
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
class FZenCacheStore::FGetOp final : public FThreadSafeRefCountedObject
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
public:
FGetOp(FZenCacheStore& InCacheStore,
IRequestOwner& InOwner,
const TConstArrayView<FCacheGetRequest> InRequests,
FOnCacheGetComplete&& InOnComplete)
: CacheStore(InCacheStore)
, Owner(InOwner)
, Requests(InRequests)
, OnComplete(MoveTemp(InOnComplete))
{
TRACE_COUNTER_ADD(ZenDDC_Get, (int64)Requests.Num());
Only report local stats for Zen DDC. ZenCacheStore now only tracks what the client sees, regardless if the zenserver instance gets a hit from a configured upstream. As we don't currently have any upstreams configured in the zen server instance this should give a more accurate number for hit/miss stats that matches the file cache. #preflight 63eabb1bb96a6b2718326d15 #rb zousar.shaker devin.doucette [CL 24206594 by dan engelbrecht in ue5-main branch]
2023-02-14 03:09:44 -05:00
TRACE_COUNTER_ADD(ZenDDC_CacheRecordRequestCountInFlight, int64(Requests.Num()));
COOK_STAT(Timers.Reserve(Requests.Num()));
COOK_STAT(for (const FCacheGetRequest& Request : Requests)
{
Timers.Add(Request.Key, CacheStore.UsageStats.TimeGet());
})
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
virtual ~FGetOp()
{
Only report local stats for Zen DDC. ZenCacheStore now only tracks what the client sees, regardless if the zenserver instance gets a hit from a configured upstream. As we don't currently have any upstreams configured in the zen server instance this should give a more accurate number for hit/miss stats that matches the file cache. #preflight 63eabb1bb96a6b2718326d15 #rb zousar.shaker devin.doucette [CL 24206594 by dan engelbrecht in ue5-main branch]
2023-02-14 03:09:44 -05:00
TRACE_COUNTER_SUBTRACT(ZenDDC_CacheRecordRequestCountInFlight, int64(Requests.Num()));
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
void IssueRequests()
{
FRequestBarrier Barrier(Owner);
ForEachBatch(CacheStore.CacheRecordBatchSize, Requests.Num(),
[this](int32 BatchFirst, int32 BatchLast)
{
TArrayView<const FCacheGetRequest> Batch(Requests.GetData() + BatchFirst, BatchLast - BatchFirst + 1);
FCbWriter BatchRequest;
BatchRequest.BeginObject();
{
BatchRequest << ANSITEXTVIEW("Method") << ANSITEXTVIEW("GetCacheRecords");
Add "Accept" field in Zen RPC request to enable new style RPC responses. See https://github.com/EpicGames/zen/pull/180 for Zen side change. #rb stefan.boberg zousar.shaker #preflight 6347c2568a0a7b2adcbe4d71 [CL 22505182 by dan engelbrecht in ue5-main branch]
2022-10-13 11:17:53 -04:00
BatchRequest.AddInteger(ANSITEXTVIEW("Accept"), Zen::Http::kCbPkgMagic);
Allow support for receiving local file references for cache items Companion PR in zen: https://github.com/EpicGames/zen/pull/184 #rb stefan.boberg #preflight 6368ce57d0174259cc1e7b7a [CL 23009037 by dan engelbrecht in ue5-main branch]
2022-11-07 05:33:21 -05:00
if (CacheStore.bIsLocalConnection)
{
BatchRequest.AddInteger(ANSITEXTVIEW("AcceptFlags"), static_cast<uint32_t>(Zen::Http::RpcAcceptOptions::kAllowLocalReferences));
}
Add "Accept" field in Zen RPC request to enable new style RPC responses. See https://github.com/EpicGames/zen/pull/180 for Zen side change. #rb stefan.boberg zousar.shaker #preflight 6347c2568a0a7b2adcbe4d71 [CL 22505182 by dan engelbrecht in ue5-main branch]
2022-10-13 11:17:53 -04:00
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
BatchRequest.BeginObject(ANSITEXTVIEW("Params"));
{
ECachePolicy BatchDefaultPolicy = Batch[0].Policy.GetRecordPolicy();
BatchRequest << ANSITEXTVIEW("DefaultPolicy") << *WriteToString<128>(BatchDefaultPolicy);
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
BatchRequest.AddString(ANSITEXTVIEW("Namespace"), CacheStore.Namespace);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
BatchRequest.BeginArray(ANSITEXTVIEW("Requests"));
for (const FCacheGetRequest& Request : Batch)
{
BatchRequest.BeginObject();
{
BatchRequest << ANSITEXTVIEW("Key") << Request.Key;
DDC: Fixed deprecation warnings in the Zen cache store #jira UE-157180 #rb none #rnx #ROBOMERGE-AUTHOR: devin.doucette #ROBOMERGE-SOURCE: CL 20745126 via CL 20745729 #ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v970-20704180) [CL 20751392 by devin doucette in ue5-main branch]
2022-06-20 23:40:25 -04:00
if (!Request.Policy.IsDefault())
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
DDC: Fixed deprecation warnings in the Zen cache store #jira UE-157180 #rb none #rnx #ROBOMERGE-AUTHOR: devin.doucette #ROBOMERGE-SOURCE: CL 20745126 via CL 20745729 #ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v970-20704180) [CL 20751392 by devin doucette in ue5-main branch]
2022-06-20 23:40:25 -04:00
BatchRequest << ANSITEXTVIEW("Policy") << Request.Policy;
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
}
BatchRequest.EndObject();
}
BatchRequest.EndArray();
}
BatchRequest.EndObject();
}
BatchRequest.EndObject();
FGetOp* OriginalOp = this;
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
auto OnRpcComplete = [this, OpRef = TRefCountPtr<FGetOp>(OriginalOp), Batch](THttpUniquePtr<IHttpResponse>& HttpResponse, FCbPackage& Response)
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
int32 RequestIndex = 0;
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
if (HttpResponse->GetErrorCode() == EHttpErrorCode::None)
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
const FCbObject& ResponseObj = Response.GetObject();
for (FCbField RecordField : ResponseObj[ANSITEXTVIEW("Result")])
{
if (RequestIndex >= Batch.Num())
{
++RequestIndex;
continue;
}
const FCacheGetRequest& Request = Batch[RequestIndex++];
const FCacheKey& Key = Request.Key;
FOptionalCacheRecord Record;
if (CacheStore.DebugOptions.ShouldSimulateGetMiss(Key))
{
UE_LOG(LogDerivedDataCache, Verbose, TEXT("%s: Simulated miss for put of '%s' from '%s'"),
*CacheStore.GetName(), *WriteToString<96>(Key), *Request.Name);
}
else if (!RecordField.IsNull())
{
Record = FCacheRecord::Load(Response, RecordField.AsObject());
}
Record ? OnHit(Request, MoveTemp(Record).Get()) : OnMiss(Request);
}
if (RequestIndex != Batch.Num())
{
UE_LOG(LogDerivedDataCache, Warning,
DDC: Added more detail to warnings from the Zen cache client #preflight 62d9ced6d76ea4b503d3f487 #rb Zousar.Shaker #rnx [CL 21214276 by Devin Doucette in ue5-main branch]
2022-07-21 23:01:51 -04:00
TEXT("%s: Invalid response received from GetCacheRecords RPC: %d results expected, received %d, from %s"),
*CacheStore.GetName(), Batch.Num(), RequestIndex, *WriteToString<256>(*HttpResponse));
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
}
for (const FCacheGetRequest& Request : Batch.RightChop(RequestIndex))
{
OnMiss(Request);
}
};
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
CacheStore.EnqueueAsyncRpc(Owner, BatchRequest.Save().AsObject(), MoveTemp(OnRpcComplete));
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
});
}
private:
void OnHit(const FCacheGetRequest& Request, FCacheRecord&& Record)
{
UE_LOG(LogDerivedDataCache, Verbose, TEXT("%s: Cache hit for '%s' from '%s'"),
*CacheStore.GetName(), *WriteToString<96>(Request.Key), *Request.Name);
Only report local stats for Zen DDC. ZenCacheStore now only tracks what the client sees, regardless if the zenserver instance gets a hit from a configured upstream. As we don't currently have any upstreams configured in the zen server instance this should give a more accurate number for hit/miss stats that matches the file cache. #preflight 63eabb1bb96a6b2718326d15 #rb zousar.shaker devin.doucette [CL 24206594 by dan engelbrecht in ue5-main branch]
2023-02-14 03:09:44 -05:00
TRACE_COUNTER_INCREMENT(ZenDDC_GetHit);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
int64 ReceivedSize = Private::GetCacheRecordCompressedSize(Record);
TRACE_COUNTER_ADD(ZenDDC_BytesReceived, ReceivedSize);
Only report local stats for Zen DDC. ZenCacheStore now only tracks what the client sees, regardless if the zenserver instance gets a hit from a configured upstream. As we don't currently have any upstreams configured in the zen server instance this should give a more accurate number for hit/miss stats that matches the file cache. #preflight 63eabb1bb96a6b2718326d15 #rb zousar.shaker devin.doucette [CL 24206594 by dan engelbrecht in ue5-main branch]
2023-02-14 03:09:44 -05:00
COOK_STAT(Timers[Request.Key].AddHit(ReceivedSize));
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
OnComplete({Request.Name, MoveTemp(Record), Request.UserData, EStatus::Ok});
};
void OnMiss(const FCacheGetRequest& Request)
{
UE_LOG(LogDerivedDataCache, Verbose, TEXT("%s: Cache miss for '%s' from '%s'"),
*CacheStore.GetName(), *WriteToString<96>(Request.Key), *Request.Name);
Only report local stats for Zen DDC. ZenCacheStore now only tracks what the client sees, regardless if the zenserver instance gets a hit from a configured upstream. As we don't currently have any upstreams configured in the zen server instance this should give a more accurate number for hit/miss stats that matches the file cache. #preflight 63eabb1bb96a6b2718326d15 #rb zousar.shaker devin.doucette [CL 24206594 by dan engelbrecht in ue5-main branch]
2023-02-14 03:09:44 -05:00
COOK_STAT(Timers[Request.Key].AddMiss());
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
OnComplete(Request.MakeResponse(EStatus::Error));
};
FZenCacheStore& CacheStore;
IRequestOwner& Owner;
const TArray<FCacheGetRequest, TInlineAllocator<1>> Requests;
FOnCacheGetComplete OnComplete;
Only report local stats for Zen DDC. ZenCacheStore now only tracks what the client sees, regardless if the zenserver instance gets a hit from a configured upstream. As we don't currently have any upstreams configured in the zen server instance this should give a more accurate number for hit/miss stats that matches the file cache. #preflight 63eabb1bb96a6b2718326d15 #rb zousar.shaker devin.doucette [CL 24206594 by dan engelbrecht in ue5-main branch]
2023-02-14 03:09:44 -05:00
COOK_STAT(TMap<FCacheKey, FCookStats::FScopedStatsCounter> Timers);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
};
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
class FZenCacheStore::FPutValueOp final : public FThreadSafeRefCountedObject
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
public:
FPutValueOp(FZenCacheStore& InCacheStore,
IRequestOwner& InOwner,
const TConstArrayView<FCachePutValueRequest> InRequests,
FOnCachePutValueComplete&& InOnComplete)
: CacheStore(InCacheStore)
, Owner(InOwner)
, Requests(InRequests)
, Batches(Requests, [this](const FCachePutValueRequest& NextRequest) {return BatchGroupingFilter(NextRequest);})
, OnComplete(MoveTemp(InOnComplete))
{
Only report local stats for Zen DDC. ZenCacheStore now only tracks what the client sees, regardless if the zenserver instance gets a hit from a configured upstream. As we don't currently have any upstreams configured in the zen server instance this should give a more accurate number for hit/miss stats that matches the file cache. #preflight 63eabb1bb96a6b2718326d15 #rb zousar.shaker devin.doucette [CL 24206594 by dan engelbrecht in ue5-main branch]
2023-02-14 03:09:44 -05:00
TRACE_COUNTER_ADD(ZenDDC_Put, (int64)Requests.Num());
COOK_STAT(Timers.Reserve(Requests.Num()));
COOK_STAT(for (const FCachePutValueRequest& Request : Requests)
{
Timers.Add(Request.Key, CacheStore.UsageStats.TimePut());
})
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
void IssueRequests()
{
FRequestBarrier Barrier(Owner);
for (TArrayView<const FCachePutValueRequest> Batch : Batches)
{
FCbPackage BatchPackage;
FCbWriter BatchWriter;
BatchWriter.BeginObject();
{
BatchWriter << ANSITEXTVIEW("Method") << ANSITEXTVIEW("PutCacheValues");
Add "Accept" field in Zen RPC request to enable new style RPC responses. See https://github.com/EpicGames/zen/pull/180 for Zen side change. #rb stefan.boberg zousar.shaker #preflight 6347c2568a0a7b2adcbe4d71 [CL 22505182 by dan engelbrecht in ue5-main branch]
2022-10-13 11:17:53 -04:00
BatchWriter.AddInteger(ANSITEXTVIEW("Accept"), Zen::Http::kCbPkgMagic);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
BatchWriter.BeginObject(ANSITEXTVIEW("Params"));
{
ECachePolicy BatchDefaultPolicy = Batch[0].Policy;
BatchWriter << ANSITEXTVIEW("DefaultPolicy") << *WriteToString<128>(BatchDefaultPolicy);
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
BatchWriter.AddString(ANSITEXTVIEW("Namespace"), CacheStore.Namespace);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
BatchWriter.BeginArray("Requests");
for (const FCachePutValueRequest& Request : Batch)
{
BatchWriter.BeginObject();
{
BatchWriter << ANSITEXTVIEW("Key") << Request.Key;
const FValue& Value = Request.Value;
BatchWriter.AddBinaryAttachment("RawHash", Value.GetRawHash());
if (Value.HasData())
{
BatchPackage.AddAttachment(FCbAttachment(Value.GetData()));
}
if (Request.Policy != BatchDefaultPolicy)
{
BatchWriter << ANSITEXTVIEW("Policy") << WriteToString<128>(Request.Policy);
}
}
BatchWriter.EndObject();
}
BatchWriter.EndArray();
}
BatchWriter.EndObject();
}
BatchWriter.EndObject();
BatchPackage.SetObject(BatchWriter.Save().AsObject());
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
auto OnRpcComplete = [this, OpRef = TRefCountPtr<FPutValueOp>(this), Batch](THttpUniquePtr<IHttpResponse>& HttpResponse, FCbPackage& Response)
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
int32 RequestIndex = 0;
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
if (HttpResponse->GetErrorCode() == EHttpErrorCode::None)
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
const FCbObject& ResponseObj = Response.GetObject();
for (FCbField ResponseField : ResponseObj[ANSITEXTVIEW("Result")])
{
if (RequestIndex >= Batch.Num())
{
++RequestIndex;
continue;
}
const FCachePutValueRequest& Request = Batch[RequestIndex++];
bool bPutSucceeded = ResponseField.AsBool();
if (CacheStore.DebugOptions.ShouldSimulatePutMiss(Request.Key))
{
UE_LOG(LogDerivedDataCache, Verbose, TEXT("%s: Simulated miss for PutValue of %s from '%s'"),
*CacheStore.GetName(), *WriteToString<96>(Request.Key), *Request.Name);
bPutSucceeded = false;
}
bPutSucceeded ? OnHit(Request) : OnMiss(Request);
}
if (RequestIndex != Batch.Num())
{
UE_LOG(LogDerivedDataCache, Warning,
DDC: Added more detail to warnings from the Zen cache client #preflight 62d9ced6d76ea4b503d3f487 #rb Zousar.Shaker #rnx [CL 21214276 by Devin Doucette in ue5-main branch]
2022-07-21 23:01:51 -04:00
TEXT("%s: Invalid response received from PutCacheValues RPC: %d results expected, received %d, from %s"),
*CacheStore.GetName(), Batch.Num(), RequestIndex, *WriteToString<256>(*HttpResponse));
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
}
for (const FCachePutValueRequest& Request : Batch.RightChop(RequestIndex))
{
OnMiss(Request);
}
};
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
CacheStore.EnqueueAsyncRpc(Owner, BatchPackage, MoveTemp(OnRpcComplete));
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
}
private:
EBatchView BatchGroupingFilter(const FCachePutValueRequest& NextRequest)
{
uint64 ValueSize = sizeof(FCacheKey) + NextRequest.Value.GetData().GetCompressedSize();
BatchSize += ValueSize;
if (BatchSize > CacheStore.BatchPutMaxBytes)
{
BatchSize = ValueSize;
return EBatchView::NewBatch;
}
return EBatchView::Continue;
}
void OnHit(const FCachePutValueRequest& Request)
{
UE_LOG(LogDerivedDataCache, Verbose, TEXT("%s: Cache PutValue complete for %s from '%s'"),
*CacheStore.GetName(), *WriteToString<96>(Request.Key), *Request.Name);
Only report local stats for Zen DDC. ZenCacheStore now only tracks what the client sees, regardless if the zenserver instance gets a hit from a configured upstream. As we don't currently have any upstreams configured in the zen server instance this should give a more accurate number for hit/miss stats that matches the file cache. #preflight 63eabb1bb96a6b2718326d15 #rb zousar.shaker devin.doucette [CL 24206594 by dan engelbrecht in ue5-main branch]
2023-02-14 03:09:44 -05:00
TRACE_COUNTER_INCREMENT(ZenDDC_PutHit);
int64 SentSize = Request.Value.GetData().GetCompressedSize();
TRACE_COUNTER_ADD(ZenDDC_BytesSent, SentSize);
COOK_STAT(Timers[Request.Key].AddHit(SentSize));
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
OnComplete(Request.MakeResponse(EStatus::Ok));
};
void OnMiss(const FCachePutValueRequest& Request)
{
UE_LOG(LogDerivedDataCache, Verbose, TEXT("%s: Cache PutValue miss for '%s' from '%s'"),
*CacheStore.GetName(), *WriteToString<96>(Request.Key), *Request.Name);
Only report local stats for Zen DDC. ZenCacheStore now only tracks what the client sees, regardless if the zenserver instance gets a hit from a configured upstream. As we don't currently have any upstreams configured in the zen server instance this should give a more accurate number for hit/miss stats that matches the file cache. #preflight 63eabb1bb96a6b2718326d15 #rb zousar.shaker devin.doucette [CL 24206594 by dan engelbrecht in ue5-main branch]
2023-02-14 03:09:44 -05:00
COOK_STAT(Timers[Request.Key].AddMiss());
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
OnComplete(Request.MakeResponse(EStatus::Error));
};
FZenCacheStore& CacheStore;
IRequestOwner& Owner;
const TArray<FCachePutValueRequest, TInlineAllocator<1>> Requests;
uint64 BatchSize = 0;
TBatchView<const FCachePutValueRequest> Batches;
FOnCachePutValueComplete OnComplete;
Only report local stats for Zen DDC. ZenCacheStore now only tracks what the client sees, regardless if the zenserver instance gets a hit from a configured upstream. As we don't currently have any upstreams configured in the zen server instance this should give a more accurate number for hit/miss stats that matches the file cache. #preflight 63eabb1bb96a6b2718326d15 #rb zousar.shaker devin.doucette [CL 24206594 by dan engelbrecht in ue5-main branch]
2023-02-14 03:09:44 -05:00
COOK_STAT(TMap<FCacheKey, FCookStats::FScopedStatsCounter> Timers);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
};
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
class FZenCacheStore::FGetValueOp final : public FThreadSafeRefCountedObject
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
public:
FGetValueOp(FZenCacheStore& InCacheStore,
IRequestOwner& InOwner,
const TConstArrayView<FCacheGetValueRequest> InRequests,
FOnCacheGetValueComplete&& InOnComplete)
: CacheStore(InCacheStore)
, Owner(InOwner)
, Requests(InRequests)
, OnComplete(MoveTemp(InOnComplete))
{
TRACE_COUNTER_ADD(ZenDDC_Get, (int64)Requests.Num());
Only report local stats for Zen DDC. ZenCacheStore now only tracks what the client sees, regardless if the zenserver instance gets a hit from a configured upstream. As we don't currently have any upstreams configured in the zen server instance this should give a more accurate number for hit/miss stats that matches the file cache. #preflight 63eabb1bb96a6b2718326d15 #rb zousar.shaker devin.doucette [CL 24206594 by dan engelbrecht in ue5-main branch]
2023-02-14 03:09:44 -05:00
TRACE_COUNTER_ADD(ZenDDC_CacheRecordRequestCountInFlight, int64(Requests.Num()));
COOK_STAT(Timers.Reserve(Requests.Num()));
COOK_STAT(for (const FCacheGetValueRequest& Request : Requests)
{
Timers.Add(Request.Key, CacheStore.UsageStats.TimeGet());
})
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
virtual ~FGetValueOp()
{
Only report local stats for Zen DDC. ZenCacheStore now only tracks what the client sees, regardless if the zenserver instance gets a hit from a configured upstream. As we don't currently have any upstreams configured in the zen server instance this should give a more accurate number for hit/miss stats that matches the file cache. #preflight 63eabb1bb96a6b2718326d15 #rb zousar.shaker devin.doucette [CL 24206594 by dan engelbrecht in ue5-main branch]
2023-02-14 03:09:44 -05:00
TRACE_COUNTER_SUBTRACT(ZenDDC_CacheRecordRequestCountInFlight, int64(Requests.Num()));
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
void IssueRequests()
{
FRequestBarrier Barrier(Owner);
ForEachBatch(CacheStore.CacheRecordBatchSize, Requests.Num(),
[this](int32 BatchFirst, int32 BatchLast)
{
TArrayView<const FCacheGetValueRequest> Batch(Requests.GetData() + BatchFirst, BatchLast - BatchFirst + 1);
FCbWriter BatchRequest;
BatchRequest.BeginObject();
{
BatchRequest << ANSITEXTVIEW("Method") << ANSITEXTVIEW("GetCacheValues");
Add "Accept" field in Zen RPC request to enable new style RPC responses. See https://github.com/EpicGames/zen/pull/180 for Zen side change. #rb stefan.boberg zousar.shaker #preflight 6347c2568a0a7b2adcbe4d71 [CL 22505182 by dan engelbrecht in ue5-main branch]
2022-10-13 11:17:53 -04:00
BatchRequest.AddInteger(ANSITEXTVIEW("Accept"), Zen::Http::kCbPkgMagic);
Allow support for receiving local file references for cache items Companion PR in zen: https://github.com/EpicGames/zen/pull/184 #rb stefan.boberg #preflight 6368ce57d0174259cc1e7b7a [CL 23009037 by dan engelbrecht in ue5-main branch]
2022-11-07 05:33:21 -05:00
if (CacheStore.bIsLocalConnection)
{
BatchRequest.AddInteger(ANSITEXTVIEW("AcceptFlags"), static_cast<uint32_t>(Zen::Http::RpcAcceptOptions::kAllowLocalReferences));
}
Add "Accept" field in Zen RPC request to enable new style RPC responses. See https://github.com/EpicGames/zen/pull/180 for Zen side change. #rb stefan.boberg zousar.shaker #preflight 6347c2568a0a7b2adcbe4d71 [CL 22505182 by dan engelbrecht in ue5-main branch]
2022-10-13 11:17:53 -04:00
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
BatchRequest.BeginObject(ANSITEXTVIEW("Params"));
{
ECachePolicy BatchDefaultPolicy = Batch[0].Policy;
BatchRequest << ANSITEXTVIEW("DefaultPolicy") << *WriteToString<128>(BatchDefaultPolicy);
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
BatchRequest.AddString(ANSITEXTVIEW("Namespace"), CacheStore.Namespace);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
BatchRequest.BeginArray("Requests");
for (const FCacheGetValueRequest& Request : Batch)
{
BatchRequest.BeginObject();
{
BatchRequest << ANSITEXTVIEW("Key") << Request.Key;
if (Request.Policy != BatchDefaultPolicy)
{
BatchRequest << ANSITEXTVIEW("Policy") << WriteToString<128>(Request.Policy);
}
}
BatchRequest.EndObject();
}
BatchRequest.EndArray();
}
BatchRequest.EndObject();
}
BatchRequest.EndObject();
FGetValueOp* OriginalOp = this;
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
auto OnRpcComplete = [this, OpRef = TRefCountPtr<FGetValueOp>(OriginalOp), Batch](THttpUniquePtr<IHttpResponse>& HttpResponse, FCbPackage& Response)
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
int32 RequestIndex = 0;
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
if (HttpResponse->GetErrorCode() == EHttpErrorCode::None)
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
const FCbObject& ResponseObj = Response.GetObject();
for (FCbFieldView ResultField : ResponseObj[ANSITEXTVIEW("Result")])
{
if (RequestIndex >= Batch.Num())
{
++RequestIndex;
continue;
}
const FCacheGetValueRequest& Request = Batch[RequestIndex++];
FCbObjectView ResultObj = ResultField.AsObjectView();
TOptional<FValue> Value;
if (CacheStore.DebugOptions.ShouldSimulateGetMiss(Request.Key))
{
UE_LOG(LogDerivedDataCache, Verbose, TEXT("%s: Simulated miss for GetValue of '%s' from '%s'"),
*CacheStore.GetName(), *WriteToString<96>(Request.Key), *Request.Name);
}
else
{
FCbFieldView RawHashField = ResultObj["RawHash"];
FIoHash RawHash = RawHashField.AsHash();
if (const FCbAttachment* Attachment = Response.FindAttachment(RawHash))
{
Value.Emplace(Attachment->AsCompressedBinary());
}
else
{
FCbFieldView RawSizeField = ResultObj["RawSize"];
uint64 RawSize = RawSizeField.AsUInt64();
if (!RawSizeField.HasError() && !RawHashField.HasError())
{
Value.Emplace(RawHash, RawSize);
}
}
}
(bool)Value ? OnHit(Request, MoveTemp(*Value)) : OnMiss(Request);
}
if (RequestIndex != Batch.Num())
{
UE_LOG(LogDerivedDataCache, Warning,
DDC: Added more detail to warnings from the Zen cache client #preflight 62d9ced6d76ea4b503d3f487 #rb Zousar.Shaker #rnx [CL 21214276 by Devin Doucette in ue5-main branch]
2022-07-21 23:01:51 -04:00
TEXT("%s: Invalid response received from GetCacheValues RPC: %d results expected, received %d from %s"),
*CacheStore.GetName(), Batch.Num(), RequestIndex, *WriteToString<256>(*HttpResponse));
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
}
for (const FCacheGetValueRequest& Request : Batch.RightChop(RequestIndex))
{
OnMiss(Request);
}
};
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
CacheStore.EnqueueAsyncRpc(Owner, BatchRequest.Save().AsObject(), MoveTemp(OnRpcComplete));
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
});
}
private:
void OnHit(const FCacheGetValueRequest& Request, FValue&& Value)
{
UE_LOG(LogDerivedDataCache, Verbose, TEXT("%s: Cache hit for '%s' from '%s'"),
*CacheStore.GetName(), *WriteToString<96>(Request.Key), *Request.Name);
Only report local stats for Zen DDC. ZenCacheStore now only tracks what the client sees, regardless if the zenserver instance gets a hit from a configured upstream. As we don't currently have any upstreams configured in the zen server instance this should give a more accurate number for hit/miss stats that matches the file cache. #preflight 63eabb1bb96a6b2718326d15 #rb zousar.shaker devin.doucette [CL 24206594 by dan engelbrecht in ue5-main branch]
2023-02-14 03:09:44 -05:00
TRACE_COUNTER_INCREMENT(ZenDDC_GetHit);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
int64 ReceivedSize = Value.GetData().GetCompressedSize();
Only report local stats for Zen DDC. ZenCacheStore now only tracks what the client sees, regardless if the zenserver instance gets a hit from a configured upstream. As we don't currently have any upstreams configured in the zen server instance this should give a more accurate number for hit/miss stats that matches the file cache. #preflight 63eabb1bb96a6b2718326d15 #rb zousar.shaker devin.doucette [CL 24206594 by dan engelbrecht in ue5-main branch]
2023-02-14 03:09:44 -05:00
TRACE_COUNTER_ADD(ZenDDC_BytesReceived, ReceivedSize);
COOK_STAT(Timers[Request.Key].AddHit(ReceivedSize));
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
OnComplete({Request.Name, Request.Key, MoveTemp(Value), Request.UserData, EStatus::Ok});
};
void OnMiss(const FCacheGetValueRequest& Request)
{
UE_LOG(LogDerivedDataCache, Verbose, TEXT("%s: Cache miss for '%s' from '%s'"),
*CacheStore.GetName(), *WriteToString<96>(Request.Key), *Request.Name);
Only report local stats for Zen DDC. ZenCacheStore now only tracks what the client sees, regardless if the zenserver instance gets a hit from a configured upstream. As we don't currently have any upstreams configured in the zen server instance this should give a more accurate number for hit/miss stats that matches the file cache. #preflight 63eabb1bb96a6b2718326d15 #rb zousar.shaker devin.doucette [CL 24206594 by dan engelbrecht in ue5-main branch]
2023-02-14 03:09:44 -05:00
COOK_STAT(Timers[Request.Key].AddMiss());
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
OnComplete(Request.MakeResponse(EStatus::Error));
};
FZenCacheStore& CacheStore;
IRequestOwner& Owner;
const TArray<FCacheGetValueRequest, TInlineAllocator<1>> Requests;
FOnCacheGetValueComplete OnComplete;
Only report local stats for Zen DDC. ZenCacheStore now only tracks what the client sees, regardless if the zenserver instance gets a hit from a configured upstream. As we don't currently have any upstreams configured in the zen server instance this should give a more accurate number for hit/miss stats that matches the file cache. #preflight 63eabb1bb96a6b2718326d15 #rb zousar.shaker devin.doucette [CL 24206594 by dan engelbrecht in ue5-main branch]
2023-02-14 03:09:44 -05:00
COOK_STAT(TMap<FCacheKey, FCookStats::FScopedStatsCounter> Timers);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
};
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
class FZenCacheStore::FGetChunksOp final : public FThreadSafeRefCountedObject
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
public:
FGetChunksOp(FZenCacheStore& InCacheStore,
IRequestOwner& InOwner,
const TConstArrayView<FCacheGetChunkRequest> InRequests,
FOnCacheGetChunkComplete&& InOnComplete)
: CacheStore(InCacheStore)
, Owner(InOwner)
, Requests(InRequests)
, OnComplete(MoveTemp(InOnComplete))
{
Only report local stats for Zen DDC. ZenCacheStore now only tracks what the client sees, regardless if the zenserver instance gets a hit from a configured upstream. As we don't currently have any upstreams configured in the zen server instance this should give a more accurate number for hit/miss stats that matches the file cache. #preflight 63eabb1bb96a6b2718326d15 #rb zousar.shaker devin.doucette [CL 24206594 by dan engelbrecht in ue5-main branch]
2023-02-14 03:09:44 -05:00
TRACE_COUNTER_ADD(ZenDDC_ChunkRequestCountInFlight, int64(Requests.Num()));
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
TRACE_COUNTER_ADD(ZenDDC_Get, int64(Requests.Num()));
Requests.StableSort(TChunkLess());
Only report local stats for Zen DDC. ZenCacheStore now only tracks what the client sees, regardless if the zenserver instance gets a hit from a configured upstream. As we don't currently have any upstreams configured in the zen server instance this should give a more accurate number for hit/miss stats that matches the file cache. #preflight 63eabb1bb96a6b2718326d15 #rb zousar.shaker devin.doucette [CL 24206594 by dan engelbrecht in ue5-main branch]
2023-02-14 03:09:44 -05:00
COOK_STAT(Timers.Reserve(Requests.Num()));
COOK_STAT(for (const FCacheGetChunkRequest& Request : Requests)
{
Timers.Add(Request.Key, CacheStore.UsageStats.TimeGet());
})
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
virtual ~FGetChunksOp()
{
Only report local stats for Zen DDC. ZenCacheStore now only tracks what the client sees, regardless if the zenserver instance gets a hit from a configured upstream. As we don't currently have any upstreams configured in the zen server instance this should give a more accurate number for hit/miss stats that matches the file cache. #preflight 63eabb1bb96a6b2718326d15 #rb zousar.shaker devin.doucette [CL 24206594 by dan engelbrecht in ue5-main branch]
2023-02-14 03:09:44 -05:00
TRACE_COUNTER_SUBTRACT(ZenDDC_ChunkRequestCountInFlight, int64(Requests.Num()));
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
void IssueRequests()
{
FRequestBarrier Barrier(Owner);
ForEachBatch(CacheStore.CacheChunksBatchSize, Requests.Num(),
[this](int32 BatchFirst, int32 BatchLast)
{
TArrayView<const FCacheGetChunkRequest> Batch(Requests.GetData() + BatchFirst, BatchLast - BatchFirst + 1);
FCbWriter BatchRequest;
BatchRequest.BeginObject();
{
BatchRequest << ANSITEXTVIEW("Method") << "GetCacheChunks";
Add "Accept" field in Zen RPC request to enable new style RPC responses. See https://github.com/EpicGames/zen/pull/180 for Zen side change. #rb stefan.boberg zousar.shaker #preflight 6347c2568a0a7b2adcbe4d71 [CL 22505182 by dan engelbrecht in ue5-main branch]
2022-10-13 11:17:53 -04:00
BatchRequest.AddInteger(ANSITEXTVIEW("Accept"), Zen::Http::kCbPkgMagic);
Allow support for receiving local file references for cache items Companion PR in zen: https://github.com/EpicGames/zen/pull/184 #rb stefan.boberg #preflight 6368ce57d0174259cc1e7b7a [CL 23009037 by dan engelbrecht in ue5-main branch]
2022-11-07 05:33:21 -05:00
if (CacheStore.bIsLocalConnection)
{
BatchRequest.AddInteger(ANSITEXTVIEW("AcceptFlags"), static_cast<uint32_t>(Zen::Http::RpcAcceptOptions::kAllowLocalReferences));
}
Add "Accept" field in Zen RPC request to enable new style RPC responses. See https://github.com/EpicGames/zen/pull/180 for Zen side change. #rb stefan.boberg zousar.shaker #preflight 6347c2568a0a7b2adcbe4d71 [CL 22505182 by dan engelbrecht in ue5-main branch]
2022-10-13 11:17:53 -04:00
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
BatchRequest.BeginObject(ANSITEXTVIEW("Params"));
{
ECachePolicy DefaultPolicy = Batch[0].Policy;
BatchRequest << ANSITEXTVIEW("DefaultPolicy") << WriteToString<128>(DefaultPolicy);
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
BatchRequest.AddString(ANSITEXTVIEW("Namespace"), CacheStore.Namespace);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
BatchRequest.BeginArray(ANSITEXTVIEW("ChunkRequests"));
for (const FCacheGetChunkRequest& Request : Batch)
{
BatchRequest.BeginObject();
{
BatchRequest << ANSITEXTVIEW("Key") << Request.Key;
if (Request.Id.IsValid())
{
BatchRequest.AddObjectId(ANSITEXTVIEW("ValueId"), Request.Id);
}
if (Request.RawOffset != 0)
{
BatchRequest << ANSITEXTVIEW("RawOffset") << Request.RawOffset;
}
if (Request.RawSize != MAX_uint64)
{
BatchRequest << ANSITEXTVIEW("RawSize") << Request.RawSize;
}
if (!Request.RawHash.IsZero())
{
BatchRequest << ANSITEXTVIEW("ChunkId") << Request.RawHash;
}
if (Request.Policy != DefaultPolicy)
{
BatchRequest << ANSITEXTVIEW("Policy") << WriteToString<128>(Request.Policy);
}
}
BatchRequest.EndObject();
}
BatchRequest.EndArray();
}
BatchRequest.EndObject();
}
BatchRequest.EndObject();
Make ZenCacheStore::FGetChunksOp asynchronous using EnqueueAsyncRpc #preflight 63341e5a10030508062960fa #rb devin.doucette zousar.shaker [CL 22230515 by dan engelbrecht in ue5-main branch]
2022-09-28 16:39:24 -04:00
FGetChunksOp* OriginalOp = this;
auto OnRpcComplete = [this, OpRef = TRefCountPtr<FGetChunksOp>(OriginalOp), Batch](THttpUniquePtr<IHttpResponse>& HttpResponse, FCbPackage& Response)
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
Make ZenCacheStore::FGetChunksOp asynchronous using EnqueueAsyncRpc #preflight 63341e5a10030508062960fa #rb devin.doucette zousar.shaker [CL 22230515 by dan engelbrecht in ue5-main branch]
2022-09-28 16:39:24 -04:00
int32 RequestIndex = 0;
if (HttpResponse->GetErrorCode() == EHttpErrorCode::None)
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
Make ZenCacheStore::FGetChunksOp asynchronous using EnqueueAsyncRpc #preflight 63341e5a10030508062960fa #rb devin.doucette zousar.shaker [CL 22230515 by dan engelbrecht in ue5-main branch]
2022-09-28 16:39:24 -04:00
const FCbObject& ResponseObj = Response.GetObject();
for (FCbFieldView ResultView : ResponseObj[ANSITEXTVIEW("Result")])
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
Make ZenCacheStore::FGetChunksOp asynchronous using EnqueueAsyncRpc #preflight 63341e5a10030508062960fa #rb devin.doucette zousar.shaker [CL 22230515 by dan engelbrecht in ue5-main branch]
2022-09-28 16:39:24 -04:00
if (RequestIndex >= Batch.Num())
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
Make ZenCacheStore::FGetChunksOp asynchronous using EnqueueAsyncRpc #preflight 63341e5a10030508062960fa #rb devin.doucette zousar.shaker [CL 22230515 by dan engelbrecht in ue5-main branch]
2022-09-28 16:39:24 -04:00
++RequestIndex;
continue;
}
const FCacheGetChunkRequest& Request = Batch[RequestIndex++];
FIoHash RawHash;
bool Succeeded = false;
uint64 RawSize = 0;
FCbObjectView ResultObject = ResultView.AsObjectView();
FSharedBuffer RequestedBytes;
if (CacheStore.DebugOptions.ShouldSimulateGetMiss(Request.Key))
{
UE_LOG(LogDerivedDataCache, Verbose, TEXT("%s: Simulated miss for get of '%s' from '%s'"),
*CacheStore.GetName(), *WriteToString<96>(Request.Key, '/', Request.Id), *Request.Name);
}
else
{
FCbFieldView HashView = ResultObject[ANSITEXTVIEW("RawHash")];
RawHash = HashView.AsHash();
if (!HashView.HasError())
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
Make ZenCacheStore::FGetChunksOp asynchronous using EnqueueAsyncRpc #preflight 63341e5a10030508062960fa #rb devin.doucette zousar.shaker [CL 22230515 by dan engelbrecht in ue5-main branch]
2022-09-28 16:39:24 -04:00
if (const FCbAttachment* Attachment = Response.FindAttachment(HashView.AsHash()))
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
Make ZenCacheStore::FGetChunksOp asynchronous using EnqueueAsyncRpc #preflight 63341e5a10030508062960fa #rb devin.doucette zousar.shaker [CL 22230515 by dan engelbrecht in ue5-main branch]
2022-09-28 16:39:24 -04:00
FCompressedBuffer CompressedBuffer = Attachment->AsCompressedBinary();
if (CompressedBuffer)
{
TRACE_COUNTER_ADD(ZenDDC_BytesReceived, CompressedBuffer.GetCompressedSize());
RequestedBytes = FCompressedBufferReader(CompressedBuffer).Decompress(Request.RawOffset, Request.RawSize);
RawSize = RequestedBytes.GetSize();
Succeeded = true;
}
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
Make ZenCacheStore::FGetChunksOp asynchronous using EnqueueAsyncRpc #preflight 63341e5a10030508062960fa #rb devin.doucette zousar.shaker [CL 22230515 by dan engelbrecht in ue5-main branch]
2022-09-28 16:39:24 -04:00
else
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
Make ZenCacheStore::FGetChunksOp asynchronous using EnqueueAsyncRpc #preflight 63341e5a10030508062960fa #rb devin.doucette zousar.shaker [CL 22230515 by dan engelbrecht in ue5-main branch]
2022-09-28 16:39:24 -04:00
FCbFieldView RawSizeField = ResultObject[ANSITEXTVIEW("RawSize")];
uint64 TotalSize = RawSizeField.AsUInt64();
Succeeded = !RawSizeField.HasError();
if (Succeeded)
{
RawSize = FMath::Min(Request.RawSize, TotalSize - FMath::Min(Request.RawOffset, TotalSize));
}
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
}
}
Make ZenCacheStore::FGetChunksOp asynchronous using EnqueueAsyncRpc #preflight 63341e5a10030508062960fa #rb devin.doucette zousar.shaker [CL 22230515 by dan engelbrecht in ue5-main branch]
2022-09-28 16:39:24 -04:00
Succeeded ? OnHit(Request, MoveTemp(RawHash), RawSize, MoveTemp(RequestedBytes)) : OnMiss(Request);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
}
Make ZenCacheStore::FGetChunksOp asynchronous using EnqueueAsyncRpc #preflight 63341e5a10030508062960fa #rb devin.doucette zousar.shaker [CL 22230515 by dan engelbrecht in ue5-main branch]
2022-09-28 16:39:24 -04:00
for (const FCacheGetChunkRequest& Request : Batch.RightChop(RequestIndex))
{
OnMiss(Request);
}
};
CacheStore.EnqueueAsyncRpc(Owner, BatchRequest.Save().AsObject(), MoveTemp(OnRpcComplete));
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
});
}
private:
void OnHit(const FCacheGetChunkRequest& Request, FIoHash&& RawHash, uint64 RawSize, FSharedBuffer&& RequestedBytes)
{
UE_LOG(LogDerivedDataCache, Verbose, TEXT("%s: CacheChunk hit for '%s' from '%s'"),
*CacheStore.GetName(), *WriteToString<96>(Request.Key, '/', Request.Id), *Request.Name);
Only report local stats for Zen DDC. ZenCacheStore now only tracks what the client sees, regardless if the zenserver instance gets a hit from a configured upstream. As we don't currently have any upstreams configured in the zen server instance this should give a more accurate number for hit/miss stats that matches the file cache. #preflight 63eabb1bb96a6b2718326d15 #rb zousar.shaker devin.doucette [CL 24206594 by dan engelbrecht in ue5-main branch]
2023-02-14 03:09:44 -05:00
TRACE_COUNTER_INCREMENT(ZenDDC_GetHit);
int64 ReceivedSize = RequestedBytes.GetSize();
TRACE_COUNTER_ADD(ZenDDC_BytesReceived, ReceivedSize);
COOK_STAT(Timers[Request.Key].AddHit(RawSize));
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
OnComplete({Request.Name, Request.Key, Request.Id, Request.RawOffset,
RawSize, MoveTemp(RawHash), MoveTemp(RequestedBytes), Request.UserData, EStatus::Ok});
};
void OnMiss(const FCacheGetChunkRequest& Request)
{
DDC: Reduced Zen cache chunk miss log verbosity from Display to Verbose #preflight 631b942bd135b61bc57331c1 #rb none #rnx #fyi Tim.Smith [CL 21926517 by Devin Doucette in ue5-main branch]
2022-09-09 15:49:08 -04:00
UE_LOG(LogDerivedDataCache, Verbose, TEXT("%s: CacheChunk miss with missing value '%s' for '%s' from '%s'"),
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
*CacheStore.GetName(), *WriteToString<16>(Request.Id), *WriteToString<96>(Request.Key), *Request.Name);
Only report local stats for Zen DDC. ZenCacheStore now only tracks what the client sees, regardless if the zenserver instance gets a hit from a configured upstream. As we don't currently have any upstreams configured in the zen server instance this should give a more accurate number for hit/miss stats that matches the file cache. #preflight 63eabb1bb96a6b2718326d15 #rb zousar.shaker devin.doucette [CL 24206594 by dan engelbrecht in ue5-main branch]
2023-02-14 03:09:44 -05:00
COOK_STAT(Timers[Request.Key].AddMiss());
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
OnComplete(Request.MakeResponse(EStatus::Error));
};
FZenCacheStore& CacheStore;
IRequestOwner& Owner;
TArray<FCacheGetChunkRequest, TInlineAllocator<1>> Requests;
FOnCacheGetChunkComplete OnComplete;
Only report local stats for Zen DDC. ZenCacheStore now only tracks what the client sees, regardless if the zenserver instance gets a hit from a configured upstream. As we don't currently have any upstreams configured in the zen server instance this should give a more accurate number for hit/miss stats that matches the file cache. #preflight 63eabb1bb96a6b2718326d15 #rb zousar.shaker devin.doucette [CL 24206594 by dan engelbrecht in ue5-main branch]
2023-02-14 03:09:44 -05:00
COOK_STAT(TMap<FCacheKey, FCookStats::FScopedStatsCounter> Timers);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
};
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
class FZenCacheStore::FCbPackageReceiver final : public IHttpReceiver
{
public:
FCbPackageReceiver(const FCbPackageReceiver&) = delete;
FCbPackageReceiver& operator=(const FCbPackageReceiver&) = delete;
explicit FCbPackageReceiver(FCbPackage& OutPackage, IHttpReceiver* InNext = nullptr)
: Package(OutPackage)
, Next(InNext)
{
}
private:
IHttpReceiver* OnCreate(IHttpResponse& Response) final
{
return &BodyReceiver;
}
IHttpReceiver* OnComplete(IHttpResponse& Response) final
{
Use Http::TryLoadCbPackage as first option to parse packages from Zen. This will allow UE to talk to both old and new zen servers as we move to use the new package wire format in Zen responses. #rb devin.doucette #preflight 6334bf81691c0168b71bc0a9 [CL 22233160 by dan engelbrecht in ue5-main branch]
2022-09-28 18:22:23 -04:00
FMemoryView MemoryView = MakeMemoryView(BodyArray);
{
FMemoryReaderView Ar(MemoryView);
if (Zen::Http::TryLoadCbPackage(Package, Ar))
{
return Next;
}
}
FMemoryReaderView Ar(MemoryView);
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
Package.TryLoad(Ar);
return Next;
}
private:
FCbPackage& Package;
IHttpReceiver* Next;
TArray64<uint8> BodyArray;
FHttpByteArrayReceiver BodyReceiver{BodyArray, this};
};
class FZenCacheStore::FAsyncCbPackageReceiver final : public FRequestBase, public IHttpReceiver
{
public:
FAsyncCbPackageReceiver(const FAsyncCbPackageReceiver&) = delete;
FAsyncCbPackageReceiver& operator=(const FAsyncCbPackageReceiver&) = delete;
FAsyncCbPackageReceiver(
THttpUniquePtr<IHttpRequest>&& InRequest,
IRequestOwner* InOwner,
Add capability for things that depend on the Zen service to attempt to recover the service if there is an interruption. The Zen DDC cache store is using this to allow recovery and retry of a cache request that fails to connect or times out when communicating with a Zen service on the same machine. #rb devin.doucette #preflight 639cc26343330e63e5db1750 [CL 23541143 by Zousar Shaker in ue5-main branch]
2022-12-16 15:47:14 -05:00
Zen::FZenServiceInstance& InZenServiceInstance,
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
FOnRpcComplete&& InOnRpcComplete)
: Request(MoveTemp(InRequest))
, Owner(InOwner)
Add capability for things that depend on the Zen service to attempt to recover the service if there is an interruption. The Zen DDC cache store is using this to allow recovery and retry of a cache request that fails to connect or times out when communicating with a Zen service on the same machine. #rb devin.doucette #preflight 639cc26343330e63e5db1750 [CL 23541143 by Zousar Shaker in ue5-main branch]
2022-12-16 15:47:14 -05:00
, ZenServiceInstance(InZenServiceInstance)
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
, BaseReceiver(Package, this)
, OnRpcComplete(MoveTemp(InOnRpcComplete))
{
Request->SendAsync(this, Response);
}
private:
// IRequest Interface
void SetPriority(EPriority Priority) final {}
void Cancel() final { Monitor->Cancel(); }
void Wait() final { Monitor->Wait(); }
// IHttpReceiver Interface
IHttpReceiver* OnCreate(IHttpResponse& LocalResponse) final
{
Monitor = LocalResponse.GetMonitor();
Owner->Begin(this);
return &BaseReceiver;
}
Add capability for things that depend on the Zen service to attempt to recover the service if there is an interruption. The Zen DDC cache store is using this to allow recovery and retry of a cache request that fails to connect or times out when communicating with a Zen service on the same machine. #rb devin.doucette #preflight 639cc26343330e63e5db1750 [CL 23541143 by Zousar Shaker in ue5-main branch]
2022-12-16 15:47:14 -05:00
bool ShouldRecoverAndRetry(IHttpResponse& LocalResponse)
{
if (!ZenServiceInstance.IsServiceRunningLocally())
{
return false;
}
if ((LocalResponse.GetErrorCode() == EHttpErrorCode::Connect) ||
(LocalResponse.GetErrorCode() == EHttpErrorCode::TlsConnect) ||
(LocalResponse.GetErrorCode() == EHttpErrorCode::TimedOut))
{
return true;
}
return false;
}
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
IHttpReceiver* OnComplete(IHttpResponse& LocalResponse) final
{
Add capability for things that depend on the Zen service to attempt to recover the service if there is an interruption. The Zen DDC cache store is using this to allow recovery and retry of a cache request that fails to connect or times out when communicating with a Zen service on the same machine. #rb devin.doucette #preflight 639cc26343330e63e5db1750 [CL 23541143 by Zousar Shaker in ue5-main branch]
2022-12-16 15:47:14 -05:00
Owner->End(this, [Self = this, &LocalResponse]
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
{
Add capability for things that depend on the Zen service to attempt to recover the service if there is an interruption. The Zen DDC cache store is using this to allow recovery and retry of a cache request that fails to connect or times out when communicating with a Zen service on the same machine. #rb devin.doucette #preflight 639cc26343330e63e5db1750 [CL 23541143 by Zousar Shaker in ue5-main branch]
2022-12-16 15:47:14 -05:00
if (Self->ShouldRecoverAndRetry(LocalResponse) && Self->ZenServiceInstance.TryRecovery())
{
new FAsyncCbPackageReceiver(MoveTemp(Self->Request), Self->Owner, Self->ZenServiceInstance, MoveTemp(Self->OnRpcComplete));
return;
}
Self->Request.Reset();
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
if (Self->OnRpcComplete)
{
// Launch a task for the completion function since it can execute arbitrary code.
Self->Owner->LaunchTask(TEXT("ZenHttpComplete"), [Self = TRefCountPtr(Self)]
{
Self->OnRpcComplete(Self->Response, Self->Package);
});
}
});
return nullptr;
}
private:
THttpUniquePtr<IHttpRequest> Request;
THttpUniquePtr<IHttpResponse> Response;
TRefCountPtr<IHttpResponseMonitor> Monitor;
IRequestOwner* Owner;
Add capability for things that depend on the Zen service to attempt to recover the service if there is an interruption. The Zen DDC cache store is using this to allow recovery and retry of a cache request that fails to connect or times out when communicating with a Zen service on the same machine. #rb devin.doucette #preflight 639cc26343330e63e5db1750 [CL 23541143 by Zousar Shaker in ue5-main branch]
2022-12-16 15:47:14 -05:00
Zen::FZenServiceInstance& ZenServiceInstance;
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
FCbPackage Package;
FCbPackageReceiver BaseReceiver;
FOnRpcComplete OnRpcComplete;
};
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
FZenCacheStore::FZenCacheStore(
const TCHAR* InServiceUrl,
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
const TCHAR* InNamespace)
Add support for Zen DDC sandboxes which are separate instances of zenserver with separate data paths and can be optionally flushed from run-to-run. It is used to implement Warm and Cold DDC graphs using Zen sandboxes for projects/platforms that use Zen for local DDC by default, and to provide new ZenColdDDC and ZenWarmDDC as new graphs that have Cold/Warm behavior for projects that don't use Zen for local DDC by default. #rb devin.doucette #preflight 63b493d5202bee5e27a645ed [CL 23569648 by Zousar Shaker in ue5-main branch]
2023-01-03 15:58:30 -05:00
: ZenService(InServiceUrl)
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
{
Add support for Zen DDC sandboxes which are separate instances of zenserver with separate data paths and can be optionally flushed from run-to-run. It is used to implement Warm and Cold DDC graphs using Zen sandboxes for projects/platforms that use Zen for local DDC by default, and to provide new ZenColdDDC and ZenWarmDDC as new graphs that have Cold/Warm behavior for projects that don't use Zen for local DDC by default. #rb devin.doucette #preflight 63b493d5202bee5e27a645ed [CL 23569648 by Zousar Shaker in ue5-main branch]
2023-01-03 15:58:30 -05:00
Initialize(InNamespace);
}
FZenCacheStore::FZenCacheStore(
UE::Zen::FServiceSettings&& InSettings,
const TCHAR* InNamespace)
: ZenService(MoveTemp(InSettings))
{
Initialize(InNamespace);
}
void FZenCacheStore::Initialize(
const TCHAR* InNamespace)
{
Namespace = InNamespace;
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
if (IsServiceReady())
{
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
RpcUri << ZenService.GetInstance().GetURL() << ANSITEXTVIEW("/z$/$rpc");
DDC: Fixed unsafe type cast warnings #preflight 6317fc04a20b67673b1de7c3 #rb Zousar.Shaker #rnx [CL 21850770 by Devin Doucette in ue5-main branch]
2022-09-07 10:37:16 -04:00
const uint32 MaxConnections = uint32(FMath::Clamp(FPlatformMisc::NumberOfCoresIncludingHyperthreads(), 8, 64));
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
constexpr uint32 RequestPoolSize = 128;
constexpr uint32 RequestPoolOverflowSize = 128;
FHttpConnectionPoolParams ConnectionPoolParams;
ConnectionPoolParams.MaxConnections = MaxConnections;
ConnectionPoolParams.MinConnections = MaxConnections;
ConnectionPool = IHttpManager::Get().CreateConnectionPool(ConnectionPoolParams);
FHttpClientParams ClientParams;
ClientParams.MaxRequests = RequestPoolSize + RequestPoolOverflowSize;
ClientParams.MinRequests = RequestPoolSize;
RequestQueue = FHttpRequestQueue(*ConnectionPool, ClientParams);
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
bIsUsable = true;
Allow support for receiving local file references for cache items Companion PR in zen: https://github.com/EpicGames/zen/pull/184 #rb stefan.boberg #preflight 6368ce57d0174259cc1e7b7a [CL 23009037 by dan engelbrecht in ue5-main branch]
2022-11-07 05:33:21 -05:00
bIsLocalConnection = ZenService.GetInstance().IsServiceRunningLocally();
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
// Issue a request for stats as it will be fetched asynchronously and issuing now makes them available sooner for future callers.
Zen::FZenStats ZenStats;
ZenService.GetInstance().GetStats(ZenStats);
}
GConfig->GetInt(TEXT("Zen"), TEXT("BatchPutMaxBytes"), BatchPutMaxBytes, GEngineIni);
GConfig->GetInt(TEXT("Zen"), TEXT("CacheRecordBatchSize"), CacheRecordBatchSize, GEngineIni);
GConfig->GetInt(TEXT("Zen"), TEXT("CacheChunksBatchSize"), CacheChunksBatchSize, GEngineIni);
}
bool FZenCacheStore::IsServiceReady()
{
return ZenService.GetInstance().IsServiceReady();
}
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
FCompositeBuffer FZenCacheStore::SaveRpcPackage(const FCbPackage& Package)
2nd submit: Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 #preflight 629642a1be0ae0b33a46d012 [CL 20438549 by Zousar Shaker in ue5-main branch]
2022-05-31 12:42:17 -04:00
{
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
FLargeMemoryWriter Memory;
Zen::Http::SaveCbPackage(Package, Memory);
uint64 PackageMemorySize = Memory.TotalSize();
return FCompositeBuffer(FSharedBuffer::TakeOwnership(Memory.ReleaseOwnership(), PackageMemorySize, FMemory::Free));
2nd submit: Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 #preflight 629642a1be0ae0b33a46d012 [CL 20438549 by Zousar Shaker in ue5-main branch]
2022-05-31 12:42:17 -04:00
}
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
THttpUniquePtr<IHttpRequest> FZenCacheStore::CreateRpcRequest()
2nd submit: Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 #preflight 629642a1be0ae0b33a46d012 [CL 20438549 by Zousar Shaker in ue5-main branch]
2022-05-31 12:42:17 -04:00
{
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
THttpUniquePtr<IHttpRequest> Request = RequestQueue.CreateRequest({});
Request->SetUri(RpcUri);
Request->SetMethod(EHttpMethod::Post);
Request->AddAcceptType(EHttpMediaType::CbPackage);
return Request;
2nd submit: Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 #preflight 629642a1be0ae0b33a46d012 [CL 20438549 by Zousar Shaker in ue5-main branch]
2022-05-31 12:42:17 -04:00
}
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
void FZenCacheStore::EnqueueAsyncRpc(IRequestOwner& Owner, FCbObject RequestObject, FOnRpcComplete&& OnComplete)
2nd submit: Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 #preflight 629642a1be0ae0b33a46d012 [CL 20438549 by Zousar Shaker in ue5-main branch]
2022-05-31 12:42:17 -04:00
{
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
THttpUniquePtr<IHttpRequest> Request = CreateRpcRequest();
Request->SetContentType(EHttpMediaType::CbObject);
Request->SetBody(RequestObject.GetBuffer().MakeOwned());
Add capability for things that depend on the Zen service to attempt to recover the service if there is an interruption. The Zen DDC cache store is using this to allow recovery and retry of a cache request that fails to connect or times out when communicating with a Zen service on the same machine. #rb devin.doucette #preflight 639cc26343330e63e5db1750 [CL 23541143 by Zousar Shaker in ue5-main branch]
2022-12-16 15:47:14 -05:00
new FAsyncCbPackageReceiver(MoveTemp(Request), &Owner, ZenService.GetInstance(), MoveTemp(OnComplete));
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
}
void FZenCacheStore::EnqueueAsyncRpc(IRequestOwner& Owner, const FCbPackage& RequestPackage, FOnRpcComplete&& OnComplete)
{
THttpUniquePtr<IHttpRequest> Request = CreateRpcRequest();
Request->SetContentType(EHttpMediaType::CbPackage);
Request->SetBody(SaveRpcPackage(RequestPackage));
Add capability for things that depend on the Zen service to attempt to recover the service if there is an interruption. The Zen DDC cache store is using this to allow recovery and retry of a cache request that fails to connect or times out when communicating with a Zen service on the same machine. #rb devin.doucette #preflight 639cc26343330e63e5db1750 [CL 23541143 by Zousar Shaker in ue5-main branch]
2022-12-16 15:47:14 -05:00
new FAsyncCbPackageReceiver(MoveTemp(Request), &Owner, ZenService.GetInstance(), MoveTemp(OnComplete));
2nd submit: Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 #preflight 629642a1be0ae0b33a46d012 [CL 20438549 by Zousar Shaker in ue5-main branch]
2022-05-31 12:42:17 -04:00
}
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
void FZenCacheStore::LegacyStats(FDerivedDataCacheStatsNode& OutNode)
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
{
Only report local stats for Zen DDC. ZenCacheStore now only tracks what the client sees, regardless if the zenserver instance gets a hit from a configured upstream. As we don't currently have any upstreams configured in the zen server instance this should give a more accurate number for hit/miss stats that matches the file cache. #preflight 63eabb1bb96a6b2718326d15 #rb zousar.shaker devin.doucette [CL 24206594 by dan engelbrecht in ue5-main branch]
2023-02-14 03:09:44 -05:00
EDerivedDataCacheStatus CacheStatus = EDerivedDataCacheStatus::None;
OutNode = { TEXT("Zen"), ZenService.GetInstance().GetURL(), /*bIsLocal*/ ZenService.GetInstance().IsServiceRunningLocally()};
OutNode.UsageStats.Add(TEXT(""), UsageStats);
#if 0
// DE: 20230213 We might need to revisit this if we change so Zen handles upstream again
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
Zen::FZenStats ZenStats;
FDerivedDataCacheUsageStats LocalStats;
FDerivedDataCacheUsageStats RemoteStats;
#if ENABLE_COOK_STATS
using EHitOrMiss = FCookStats::CallStats::EHitOrMiss;
using ECacheStatType = FCookStats::CallStats::EStatType;
Add support for Zen DDC sandboxes which are separate instances of zenserver with separate data paths and can be optionally flushed from run-to-run. It is used to implement Warm and Cold DDC graphs using Zen sandboxes for projects/platforms that use Zen for local DDC by default, and to provide new ZenColdDDC and ZenWarmDDC as new graphs that have Cold/Warm behavior for projects that don't use Zen for local DDC by default. #rb devin.doucette #preflight 63b493d5202bee5e27a645ed [CL 23569648 by Zousar Shaker in ue5-main branch]
2023-01-03 15:58:30 -05:00
ZenService.GetInstance().GetStats(ZenStats);
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
const int64 RemotePutSize = int64(ZenStats.UpstreamStats.TotalUploadedMB * 1024 * 1024);
const int64 RemoteGetSize = int64(ZenStats.UpstreamStats.TotalDownloadedMB * 1024 * 1024);
const int64 LocalGetSize = FMath::Max<int64>(0, UsageStats.GetStats.GetAccumulatedValueAnyThread(EHitOrMiss::Hit, ECacheStatType::Bytes) - RemoteGetSize);
LocalStats.PutStats = UsageStats.PutStats;
LocalStats.ExistsStats = UsageStats.ExistsStats;
LocalStats.PrefetchStats = UsageStats.PrefetchStats;
LocalStats.GetStats.Accumulate(EHitOrMiss::Hit, ECacheStatType::Counter, ZenStats.CacheStats.Hits - ZenStats.CacheStats.UpstreamHits, /*bIsInGameThread*/ false);
LocalStats.GetStats.Accumulate(EHitOrMiss::Miss, ECacheStatType::Counter, ZenStats.CacheStats.Misses + ZenStats.CacheStats.UpstreamHits, /*bIsInGameThread*/ false);
RemoteStats.GetStats.Accumulate(EHitOrMiss::Hit, ECacheStatType::Counter, ZenStats.CacheStats.UpstreamHits, /*bIsInGameThread*/ false);
RemoteStats.GetStats.Accumulate(EHitOrMiss::Miss, ECacheStatType::Counter, ZenStats.CacheStats.Misses, /*bIsInGameThread*/ false);
LocalStats.GetStats.Accumulate(EHitOrMiss::Hit, ECacheStatType::Bytes, LocalGetSize, /*bIsInGameThread*/ false);
RemoteStats.GetStats.Accumulate(EHitOrMiss::Hit, ECacheStatType::Bytes, RemoteGetSize, /*bIsInGameThread*/ false);
RemoteStats.PutStats.Accumulate(EHitOrMiss::Hit, ECacheStatType::Bytes, RemotePutSize, /*bIsInGameThread*/ false);
#endif
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
if (ZenStats.UpstreamStats.EndPointStats.IsEmpty())
{
OutNode = {TEXT("Zen"), ZenService.GetInstance().GetURL(), /*bIsLocal*/ true};
OutNode.UsageStats.Add(TEXT(""), LocalStats);
return;
}
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
TSharedRef<FDerivedDataCacheStatsNode> LocalNode =
MakeShared<FDerivedDataCacheStatsNode>(TEXT("Zen"), ZenService.GetInstance().GetURL(), /*bIsLocal*/ true);
Added Cache Speed Stats to Analytics Added Zen Server Stats to Analytics #rb zousar.shaker, devin.doucette #jira none #preflight 6295d84bd57da28cc8ee9eab [CL 20434901 by mark lintott in ue5-main branch]
2022-05-31 05:11:05 -04:00
LocalNode->UsageStats.Add(TEXT(""), LocalStats);
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
TSharedRef<FDerivedDataCacheStatsNode> RemoteNode =
MakeShared<FDerivedDataCacheStatsNode>(ZenStats.UpstreamStats.EndPointStats[0].Name, ZenStats.UpstreamStats.EndPointStats[0].Url, /*bIsLocal*/ false);
Added Cache Speed Stats to Analytics Added Zen Server Stats to Analytics #rb zousar.shaker, devin.doucette #jira none #preflight 6295d84bd57da28cc8ee9eab [CL 20434901 by mark lintott in ue5-main branch]
2022-05-31 05:11:05 -04:00
RemoteNode->UsageStats.Add(TEXT(""), RemoteStats);
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
OutNode = {TEXT("Zen Group"), TEXT(""), /*bIsLocal*/ true};
OutNode.Children.Add(LocalNode);
OutNode.Children.Add(RemoteNode);
Only report local stats for Zen DDC. ZenCacheStore now only tracks what the client sees, regardless if the zenserver instance gets a hit from a configured upstream. As we don't currently have any upstreams configured in the zen server instance this should give a more accurate number for hit/miss stats that matches the file cache. #preflight 63eabb1bb96a6b2718326d15 #rb zousar.shaker devin.doucette [CL 24206594 by dan engelbrecht in ue5-main branch]
2023-02-14 03:09:44 -05:00
#endif // 0
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
}
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
bool FZenCacheStore::LegacyDebugOptions(FBackendDebugOptions& InOptions)
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
{
DebugOptions = InOptions;
return true;
}
void FZenCacheStore::Put(
const TConstArrayView<FCachePutRequest> Requests,
IRequestOwner& Owner,
FOnCachePutComplete&& OnComplete)
{
TRACE_CPUPROFILER_EVENT_SCOPE(ZenDDC::PutCachedRecord);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
TRefCountPtr<FPutOp> PutOp = MakeAsyncOp<FPutOp>(*this, Owner, Requests, MoveTemp(OnComplete));
PutOp->IssueRequests();
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
}
void FZenCacheStore::Get(
const TConstArrayView<FCacheGetRequest> Requests,
IRequestOwner& Owner,
FOnCacheGetComplete&& OnComplete)
{
TRACE_CPUPROFILER_EVENT_SCOPE(ZenDDC::GetCacheRecord);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
TRefCountPtr<FGetOp> GetOp = MakeAsyncOp<FGetOp>(*this, Owner, Requests, MoveTemp(OnComplete));
GetOp->IssueRequests();
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
}
void FZenCacheStore::PutValue(
TConstArrayView<FCachePutValueRequest> Requests,
IRequestOwner& Owner,
FOnCachePutValueComplete&& OnComplete)
{
TRACE_CPUPROFILER_EVENT_SCOPE(ZenDDC::PutValue);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
TRefCountPtr<FPutValueOp> PutValueOp = MakeAsyncOp<FPutValueOp>(*this, Owner, Requests, MoveTemp(OnComplete));
PutValueOp->IssueRequests();
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
}
void FZenCacheStore::GetValue(
TConstArrayView<FCacheGetValueRequest> Requests,
IRequestOwner& Owner,
FOnCacheGetValueComplete&& OnComplete)
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
TRACE_CPUPROFILER_EVENT_SCOPE(ZenDDC::GetValue);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
TRefCountPtr<FGetValueOp> GetValueOp = MakeAsyncOp<FGetValueOp>(*this, Owner, Requests, MoveTemp(OnComplete));
GetValueOp->IssueRequests();
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
}
void FZenCacheStore::GetChunks(
TConstArrayView<FCacheGetChunkRequest> Requests,
IRequestOwner& Owner,
FOnCacheGetChunkComplete&& OnComplete)
{
TRACE_CPUPROFILER_EVENT_SCOPE(ZenDDC::GetChunks);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
TRefCountPtr<FGetChunksOp> GetChunksOp = MakeAsyncOp<FGetChunksOp>(*this, Owner, Requests, MoveTemp(OnComplete));
GetChunksOp->IssueRequests();
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
}
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
TTuple<ILegacyCacheStore*, ECacheStoreFlags> CreateZenCacheStore(const TCHAR* NodeName, const TCHAR* Config)
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
{
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
FString ServiceUrl;
FParse::Value(Config, TEXT("Host="), ServiceUrl);
Add support for a Zen cache store Host to be None (implying that it is disabled) and to have both commandline and environment overrides (similar to the HTTP cache store). #rb devin.doucette [CL 23815887 by zousar shaker in ue5-main branch]
2023-01-23 12:48:38 -05:00
FString OverrideName;
if (FParse::Value(Config, TEXT("EnvHostOverride="), OverrideName))
{
FString ServiceUrlEnv = FPlatformMisc::GetEnvironmentVariable(*OverrideName);
if (!ServiceUrlEnv.IsEmpty())
{
ServiceUrl = ServiceUrlEnv;
UE_LOG(LogDerivedDataCache, Log, TEXT("%s: Found environment override for Host %s=%s"), NodeName, *OverrideName, *ServiceUrl);
}
}
if (FParse::Value(Config, TEXT("CommandLineHostOverride="), OverrideName))
{
if (FParse::Value(FCommandLine::Get(), *(OverrideName + TEXT("=")), ServiceUrl))
{
UE_LOG(LogDerivedDataCache, Log, TEXT("%s: Found command line override for Host %s=%s"), NodeName, *OverrideName, *ServiceUrl);
}
}
if (ServiceUrl == TEXT("None"))
{
UE_LOG(LogDerivedDataCache, Log, TEXT("Disabling %s data cache - host set to 'None'."), NodeName);
return MakeTuple<ILegacyCacheStore*, ECacheStoreFlags>(nullptr, ECacheStoreFlags::None);
}
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
FString Namespace;
Update ShooterGame, QAGame, and Lyra to use Zen for LOCAL DDC only. #rb devin.doucette #preflight 63643f6d882365b8596ce430 [CL 22970016 by Zousar Shaker in ue5-main branch]
2022-11-03 18:48:54 -04:00
if (!FParse::Value(Config, TEXT("StructuredNamespace="), Namespace) && !FParse::Value(Config, TEXT("Namespace="), Namespace))
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
{
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
Namespace = FApp::GetProjectName();
Update ShooterGame, QAGame, and Lyra to use Zen for LOCAL DDC only. #rb devin.doucette #preflight 63643f6d882365b8596ce430 [CL 22970016 by Zousar Shaker in ue5-main branch]
2022-11-03 18:48:54 -04:00
UE_LOG(LogDerivedDataCache, Warning, TEXT("%s: Missing required parameter 'Namespace', falling back to '%s'"), NodeName, *Namespace);
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
}
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
ZenCacheStore - ignore "None" as an override value for a local DDC path when using sandboxes (in line with past behavior for non-default local caches like Warm/Cold). #rb devin.doucette [CL 23658565 by zousar shaker in ue5-main branch]
2023-01-11 20:57:13 -05:00
FString Sandbox;
FParse::Value(Config, TEXT("Sandbox="), Sandbox);
bool bHasSandbox = !Sandbox.IsEmpty();
bool bUseLocalDataCachePathOverrides = !bHasSandbox;
Ensure that Zen as a local service can be configured to follow the same commandline overrides as a filesystem cache store. Also ensure that if the configured override mechanisms for cache store paths yield a path of "None" that we disable the Zen DDC cache. These are in support of using Zen as a drop-in replacement for an existing filesystem DDC store. #rb devin.doucette #preflight 6398ea4e680483bcb1057b0a [CL 23517045 by Zousar Shaker in ue5-main branch]
2022-12-14 15:06:30 -05:00
FString CachePathOverride;
ZenCacheStore - ignore "None" as an override value for a local DDC path when using sandboxes (in line with past behavior for non-default local caches like Warm/Cold). #rb devin.doucette [CL 23658565 by zousar shaker in ue5-main branch]
2023-01-11 20:57:13 -05:00
if (bUseLocalDataCachePathOverrides && UE::Zen::Private::IsLocalAutoLaunched(ServiceUrl) && UE::Zen::Private::GetLocalDataCachePathOverride(CachePathOverride))
Ensure that Zen as a local service can be configured to follow the same commandline overrides as a filesystem cache store. Also ensure that if the configured override mechanisms for cache store paths yield a path of "None" that we disable the Zen DDC cache. These are in support of using Zen as a drop-in replacement for an existing filesystem DDC store. #rb devin.doucette #preflight 6398ea4e680483bcb1057b0a [CL 23517045 by Zousar Shaker in ue5-main branch]
2022-12-14 15:06:30 -05:00
{
if (CachePathOverride == TEXT("None"))
{
Add support for Zen DDC sandboxes which are separate instances of zenserver with separate data paths and can be optionally flushed from run-to-run. It is used to implement Warm and Cold DDC graphs using Zen sandboxes for projects/platforms that use Zen for local DDC by default, and to provide new ZenColdDDC and ZenWarmDDC as new graphs that have Cold/Warm behavior for projects that don't use Zen for local DDC by default. #rb devin.doucette #preflight 63b493d5202bee5e27a645ed [CL 23569648 by Zousar Shaker in ue5-main branch]
2023-01-03 15:58:30 -05:00
UE_LOG(LogDerivedDataCache, Log, TEXT("Disabling %s data cache - path set to 'None'."), NodeName);
Ensure that Zen as a local service can be configured to follow the same commandline overrides as a filesystem cache store. Also ensure that if the configured override mechanisms for cache store paths yield a path of "None" that we disable the Zen DDC cache. These are in support of using Zen as a drop-in replacement for an existing filesystem DDC store. #rb devin.doucette #preflight 6398ea4e680483bcb1057b0a [CL 23517045 by Zousar Shaker in ue5-main branch]
2022-12-14 15:06:30 -05:00
return MakeTuple<ILegacyCacheStore*, ECacheStoreFlags>(nullptr, ECacheStoreFlags::None);
}
}
Add support for Zen DDC sandboxes which are separate instances of zenserver with separate data paths and can be optionally flushed from run-to-run. It is used to implement Warm and Cold DDC graphs using Zen sandboxes for projects/platforms that use Zen for local DDC by default, and to provide new ZenColdDDC and ZenWarmDDC as new graphs that have Cold/Warm behavior for projects that don't use Zen for local DDC by default. #rb devin.doucette #preflight 63b493d5202bee5e27a645ed [CL 23569648 by Zousar Shaker in ue5-main branch]
2023-01-03 15:58:30 -05:00
TUniquePtr<FZenCacheStore> Backend;
bool bFlush = false;
FParse::Bool(Config, TEXT("Flush="), bFlush);
ZenCacheStore - ignore "None" as an override value for a local DDC path when using sandboxes (in line with past behavior for non-default local caches like Warm/Cold). #rb devin.doucette [CL 23658565 by zousar shaker in ue5-main branch]
2023-01-11 20:57:13 -05:00
if (bHasSandbox)
Add support for Zen DDC sandboxes which are separate instances of zenserver with separate data paths and can be optionally flushed from run-to-run. It is used to implement Warm and Cold DDC graphs using Zen sandboxes for projects/platforms that use Zen for local DDC by default, and to provide new ZenColdDDC and ZenWarmDDC as new graphs that have Cold/Warm behavior for projects that don't use Zen for local DDC by default. #rb devin.doucette #preflight 63b493d5202bee5e27a645ed [CL 23569648 by Zousar Shaker in ue5-main branch]
2023-01-03 15:58:30 -05:00
{
Zen::FServiceSettings DefaultServiceSettings;
DefaultServiceSettings.ReadFromConfig();
if (!DefaultServiceSettings.IsAutoLaunch())
{
UE_LOG(LogDerivedDataCache, Warning, TEXT("%s: Attempting to use a sandbox when there is no default autolaunch configured to interhit settings from. Cache will be disabled."), NodeName);
return MakeTuple<ILegacyCacheStore*, ECacheStoreFlags>(nullptr, ECacheStoreFlags::None);
}
// Make a unique local instance (not the default local instance) of ZenServer
Zen::FServiceSettings ServiceSettings;
ServiceSettings.SettingsVariant.Emplace<Zen::FServiceAutoLaunchSettings>();
Zen::FServiceAutoLaunchSettings& AutoLaunchSettings = ServiceSettings.SettingsVariant.Get<Zen::FServiceAutoLaunchSettings>();
const Zen::FServiceAutoLaunchSettings& DefaultAutoLaunchSettings = DefaultServiceSettings.SettingsVariant.Get<Zen::FServiceAutoLaunchSettings>();
AutoLaunchSettings = DefaultAutoLaunchSettings;
// Default as one more than the default port to not collide. Multiple sandboxes will share a desired port, but will get differing effective ports.
AutoLaunchSettings.DesiredPort++;
FPaths::NormalizeDirectoryName(AutoLaunchSettings.DataPath);
AutoLaunchSettings.DataPath += TEXT("_");
AutoLaunchSettings.DataPath += Sandbox;
// The unique local instances will always limit process lifetime for now to avoid accumulating many of them
AutoLaunchSettings.bLimitProcessLifetime = true;
// Flush the cache if requested.
if (bFlush)
{
bool bStopped = true;
if (UE::Zen::IsLocalServiceRunning(*AutoLaunchSettings.DataPath))
{
bStopped = UE::Zen::StopLocalService(*AutoLaunchSettings.DataPath);
}
if (bStopped)
{
IFileManager::Get().DeleteDirectory(*(AutoLaunchSettings.DataPath / TEXT("")), /*bRequireExists*/ false, /*bTree*/ true);
}
else
{
UE_LOG(LogDerivedDataCache, Warning, TEXT("%s: Zen DDC could not be flushed due to an existing instance not shutting down when requested."), NodeName);
}
}
Backend = MakeUnique<FZenCacheStore>(MoveTemp(ServiceSettings), *Namespace);
}
else
{
Backend = MakeUnique<FZenCacheStore>(*ServiceUrl, *Namespace);
}
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
if (!Backend->IsUsable())
{
UE_LOG(LogDerivedDataCache, Warning, TEXT("%s: Failed to contact the service (%s), will not use it."), NodeName, *Backend->GetName());
Backend.Reset();
}
return MakeTuple(Backend.Release(), ECacheStoreFlags::Local | ECacheStoreFlags::Remote | ECacheStoreFlags::Query | ECacheStoreFlags::Store);
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
}
} // namespace UE::DerivedData
#else
namespace UE::DerivedData
{
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
TTuple<ILegacyCacheStore*, ECacheStoreFlags> CreateZenCacheStore(const TCHAR* NodeName, const TCHAR* Config)
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
{
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
UE_LOG(LogDerivedDataCache, Warning, TEXT("%s: Zen cache is not yet supported in the current build configuration."), NodeName);
return MakeTuple(nullptr, ECacheStoreFlags::None);
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
}
} // UE::DerivedData
ZenServer CachePolicy support. Add batched Put and PutValue. Update Batched Get to send the RecordPolicy with each key. Update GetChunks to send the CachePolicy as string and to handle SkipData in the response. Add more unit tests for upstream behavior and GetChunks. Update the CacheRecordPolicy Save/Load to match the new variable names on FCacheRecordPolicy. Deploy new zenserver binaries with the required matching server-side changes. #rb Per.Larsson,Devin.Doucette #rnx #preflight 61f94e5980608c7029b8fc5c #ROBOMERGE-AUTHOR: matt.peters #ROBOMERGE-SOURCE: CL 18808743 in //UE5/Release-5.0/... via CL 18809399 via CL 18822531 #ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v908-18788545) [CL 18823289 by matt peters in ue5-main branch]
2022-02-02 05:33:16 -05:00
#endif // UE_WITH_ZEN
Reference in New Issue Copy Permalink