izzy/UnrealEngineUWP
0
0
Fork 0
mirror of https://github.com/izzy2lost/UnrealEngineUWP.git synced 2026-03-26 18:15:20 -07:00
Code Issues Packages Projects Releases Wiki Activity
Files
d4e27237325cafa00b84dfc841e458cedadb90bc
UnrealEngineUWP/Engine/Source/Developer/DerivedDataCache/Private/ZenCacheStore.cpp

1628 lines
57 KiB
C++
Raw Normal View History

Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
// Copyright Epic Games, Inc. All Rights Reserved.
DDC: Temporary compile fix for platforms that should not include DerivedDataCache #preflight skip #rb none #rnx [CL 20445864 by Devin Doucette in ue5-main branch]
2022-05-31 23:25:37 -04:00
#include "DerivedDataLegacyCacheStore.h"
Move Zen module public headers into an "Experimental" subfolder to have them reflect the feature status. Fix locations that include the headers to use the "Experimental" subfolder and re-sort includes. No functional change, purely a move, search+replace, and sort of includes. #rb matt.peters #preflight 63939eebc415e8dc78d35d63 [CL 23468934 by Zousar Shaker in ue5-main branch]
2022-12-09 16:11:41 -05:00
#include "Experimental/ZenServerInterface.h"
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
#if UE_WITH_ZEN
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
#include "Async/UniqueLock.h"
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
#include "BatchView.h"
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
#include "DerivedDataBackendInterface.h"
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
#include "DerivedDataCachePrivate.h"
#include "DerivedDataCacheRecord.h"
#include "DerivedDataCacheUsageStats.h"
#include "DerivedDataChunk.h"
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
#include "DerivedDataRequest.h"
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
#include "DerivedDataRequestOwner.h"
Move Zen module public headers into an "Experimental" subfolder to have them reflect the feature status. Fix locations that include the headers to use the "Experimental" subfolder and re-sort includes. No functional change, purely a move, search+replace, and sort of includes. #rb matt.peters #preflight 63939eebc415e8dc78d35d63 [CL 23468934 by Zousar Shaker in ue5-main branch]
2022-12-09 16:11:41 -05:00
#include "Experimental/ZenStatistics.h"
Add support for Zen DDC sandboxes which are separate instances of zenserver with separate data paths and can be optionally flushed from run-to-run. It is used to implement Warm and Cold DDC graphs using Zen sandboxes for projects/platforms that use Zen for local DDC by default, and to provide new ZenColdDDC and ZenWarmDDC as new graphs that have Cold/Warm behavior for projects that don't use Zen for local DDC by default. #rb devin.doucette #preflight 63b493d5202bee5e27a645ed [CL 23569648 by Zousar Shaker in ue5-main branch]
2023-01-03 15:58:30 -05:00
#include "HAL/FileManager.h"
2nd submit: Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 #preflight 629642a1be0ae0b33a46d012 [CL 20438549 by Zousar Shaker in ue5-main branch]
2022-05-31 12:42:17 -04:00
#include "Http/HttpClient.h"
Select maximum HTTP connections for ZenCacheStore based on logical core count (clamped between 8 and 64). Speculative solution for farm cooks using Zen experiencing "couldn't connect errors". #rb trivial #fyi devin.doucette #rnx #preflight 62b00bd92405dc6102315616 [CL 20734103 by Zousar Shaker in ue5-main branch]
2022-06-20 02:08:49 -04:00
#include "Math/UnrealMathUtility.h"
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
#include "Misc/App.h"
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
#include "Misc/ConfigCacheIni.h"
#include "Misc/Optional.h"
#include "ProfilingDebugging/CountersTrace.h"
#include "ProfilingDebugging/CpuProfilerTrace.h"
#include "Serialization/CompactBinary.h"
#include "Serialization/CompactBinaryPackage.h"
#include "Serialization/CompactBinaryWriter.h"
2nd submit: Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 #preflight 629642a1be0ae0b33a46d012 [CL 20438549 by Zousar Shaker in ue5-main branch]
2022-05-31 12:42:17 -04:00
#include "Serialization/LargeMemoryWriter.h"
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
#include "Serialization/MemoryReader.h"
2nd submit: Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 #preflight 629642a1be0ae0b33a46d012 [CL 20438549 by Zousar Shaker in ue5-main branch]
2022-05-31 12:42:17 -04:00
#include "Templates/Function.h"
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
#include "ZenBackendUtils.h"
#include "ZenSerialization.h"
Only report local stats for Zen DDC. ZenCacheStore now only tracks what the client sees, regardless if the zenserver instance gets a hit from a configured upstream. As we don't currently have any upstreams configured in the zen server instance this should give a more accurate number for hit/miss stats that matches the file cache. #preflight 63eabb1bb96a6b2718326d15 #rb zousar.shaker devin.doucette [CL 24206594 by dan engelbrecht in ue5-main branch]
2023-02-14 03:09:44 -05:00
TRACE_DECLARE_INT_COUNTER(ZenDDC_Get, TEXT("ZenDDC Get"));
TRACE_DECLARE_INT_COUNTER(ZenDDC_GetHit, TEXT("ZenDDC Get Hit"));
TRACE_DECLARE_INT_COUNTER(ZenDDC_Put, TEXT("ZenDDC Put"));
TRACE_DECLARE_INT_COUNTER(ZenDDC_PutHit, TEXT("ZenDDC Put Hit"));
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
TRACE_DECLARE_INT_COUNTER(ZenDDC_BytesReceived, TEXT("ZenDDC Bytes Received"));
Only report local stats for Zen DDC. ZenCacheStore now only tracks what the client sees, regardless if the zenserver instance gets a hit from a configured upstream. As we don't currently have any upstreams configured in the zen server instance this should give a more accurate number for hit/miss stats that matches the file cache. #preflight 63eabb1bb96a6b2718326d15 #rb zousar.shaker devin.doucette [CL 24206594 by dan engelbrecht in ue5-main branch]
2023-02-14 03:09:44 -05:00
TRACE_DECLARE_INT_COUNTER(ZenDDC_BytesSent, TEXT("ZenDDC Bytes Sent"));
TRACE_DECLARE_INT_COUNTER(ZenDDC_CacheRecordRequestCountInFlight, TEXT("ZenDDC CacheRecord Request Count"));
TRACE_DECLARE_INT_COUNTER(ZenDDC_ChunkRequestCountInFlight, TEXT("ZenDDC Chunk Request Count"));
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
namespace UE::DerivedData
{
template<typename T>
void ForEachBatch(const int32 BatchSize, const int32 TotalCount, T&& Fn)
{
check(BatchSize > 0);
if (TotalCount > 0)
{
const int32 BatchCount = FMath::DivideAndRoundUp(TotalCount, BatchSize);
const int32 Last = TotalCount - 1;
for (int32 BatchIndex = 0; BatchIndex < BatchCount; BatchIndex++)
{
const int32 BatchFirstIndex = BatchIndex * BatchSize;
const int32 BatchLastIndex = FMath::Min(BatchFirstIndex + BatchSize - 1, Last);
Fn(BatchFirstIndex, BatchLastIndex);
}
}
}
/**
* Backend for a HTTP based caching service (Zen)
*/
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
class FZenCacheStore final : public ILegacyCacheStore
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
{
public:
/**
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
* Creates the cache store client, checks health status and attempts to acquire an access token.
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
*
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
* @param ServiceUrl Base url to the service including scheme.
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
* @param Namespace Namespace to use.
*/
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
FZenCacheStore(
const TCHAR* ServiceUrl,
const TCHAR* Namespace,
const TCHAR* Name,
const TCHAR* Config,
ICacheStoreOwner* Owner);
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
FZenCacheStore(
UE::Zen::FServiceSettings&& Settings,
const TCHAR* Namespace,
const TCHAR* Name,
const TCHAR* Config,
ICacheStoreOwner* Owner);
Add support for Zen DDC sandboxes which are separate instances of zenserver with separate data paths and can be optionally flushed from run-to-run. It is used to implement Warm and Cold DDC graphs using Zen sandboxes for projects/platforms that use Zen for local DDC by default, and to provide new ZenColdDDC and ZenWarmDDC as new graphs that have Cold/Warm behavior for projects that don't use Zen for local DDC by default. #rb devin.doucette #preflight 63b493d5202bee5e27a645ed [CL 23569648 by Zousar Shaker in ue5-main branch]
2023-01-03 15:58:30 -05:00
DDC: Fixed leaking cache store stats in the Zen cache #jira UE-133395 #rb none #rnx [CL 25977512 by devin doucette in ue5-main branch]
2023-06-14 10:12:47 -04:00
~FZenCacheStore() final;
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
inline FString GetName() const { return ZenService.GetInstance().GetURL(); }
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
/**
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
* Checks if cache service is usable (reachable and accessible).
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
* @return true if usable
*/
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
inline bool IsUsable() const { return bIsUsable; }
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
// ICacheStore
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
void Put(
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
TConstArrayView<FCachePutRequest> Requests,
IRequestOwner& Owner,
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
FOnCachePutComplete&& OnComplete = FOnCachePutComplete()) final;
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
void Get(
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
TConstArrayView<FCacheGetRequest> Requests,
IRequestOwner& Owner,
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
FOnCacheGetComplete&& OnComplete) final;
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
void PutValue(
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
TConstArrayView<FCachePutValueRequest> Requests,
IRequestOwner& Owner,
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
FOnCachePutValueComplete&& OnComplete = FOnCachePutValueComplete()) final;
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
void GetValue(
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
TConstArrayView<FCacheGetValueRequest> Requests,
IRequestOwner& Owner,
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
FOnCacheGetValueComplete&& OnComplete = FOnCacheGetValueComplete()) final;
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
void GetChunks(
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
TConstArrayView<FCacheGetChunkRequest> Requests,
IRequestOwner& Owner,
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
FOnCacheGetChunkComplete&& OnComplete) final;
// ILegacyCacheStore
void LegacyStats(FDerivedDataCacheStatsNode& OutNode) final;
bool LegacyDebugOptions(FBackendDebugOptions& Options) final;
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
Change Zen DDC cache store to allow: - Local storage flag to be inferred by whether the service is run locally or set explicitly in config. - Remote storage flag to be inferred by whether the service is NOT run locally or set explicitly in config. - ReadOnly storage flag to be set explicitly in config (defaults to true). #rb devin.doucette [CL 24822351 by zousar shaker in ue5-main branch]
2023-03-28 13:43:50 -04:00
const Zen::FZenServiceInstance& GetServiceInstance() const { return ZenService.GetInstance(); }
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
private:
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
void Initialize(const TCHAR* Namespace, const TCHAR* Name, const TCHAR* Config);
Add support for Zen DDC sandboxes which are separate instances of zenserver with separate data paths and can be optionally flushed from run-to-run. It is used to implement Warm and Cold DDC graphs using Zen sandboxes for projects/platforms that use Zen for local DDC by default, and to provide new ZenColdDDC and ZenWarmDDC as new graphs that have Cold/Warm behavior for projects that don't use Zen for local DDC by default. #rb devin.doucette #preflight 63b493d5202bee5e27a645ed [CL 23569648 by Zousar Shaker in ue5-main branch]
2023-01-03 15:58:30 -05:00
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
bool IsServiceReady();
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
static FCompositeBuffer SaveRpcPackage(const FCbPackage& Package);
THttpUniquePtr<IHttpRequest> CreateRpcRequest();
using FOnRpcComplete = TUniqueFunction<void(THttpUniquePtr<IHttpResponse>& HttpResponse, FCbPackage& Response)>;
void EnqueueAsyncRpc(IRequestOwner& Owner, FCbObject RequestObject, FOnRpcComplete&& OnComplete);
void EnqueueAsyncRpc(IRequestOwner& Owner, const FCbPackage& RequestPackage, FOnRpcComplete&& OnComplete);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
template <typename T, typename... ArgTypes>
static TRefCountPtr<T> MakeAsyncOp(ArgTypes&&... Args)
{
// TODO: This should in-place construct from a pre-allocated memory pool
return TRefCountPtr<T>(new T(Forward<ArgTypes>(Args)...));
}
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
private:
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
template <typename RequestType>
struct TRequestWithStats;
template <typename OutContainerType, typename InContainerType, typename BucketAccessorType, typename RequestTypeAccessorType>
static void StartRequests(
OutContainerType& Out,
const InContainerType& In,
BucketAccessorType BucketAccessor,
RequestTypeAccessorType TypeAccessor,
const ERequestOp Op);
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
class FPutOp;
class FGetOp;
class FPutValueOp;
class FGetValueOp;
class FGetChunksOp;
class FCbPackageReceiver;
class FAsyncCbPackageReceiver;
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
FString Namespace;
UE::Zen::FScopeZenService ZenService;
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
ICacheStoreOwner* StoreOwner = nullptr;
ICacheStoreStats* StoreStats = nullptr;
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
THttpUniquePtr<IHttpConnectionPool> ConnectionPool;
FHttpRequestQueue RequestQueue;
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
bool bIsUsable = false;
Allow support for receiving local file references for cache items Companion PR in zen: https://github.com/EpicGames/zen/pull/184 #rb stefan.boberg #preflight 6368ce57d0174259cc1e7b7a [CL 23009037 by dan engelbrecht in ue5-main branch]
2022-11-07 05:33:21 -05:00
bool bIsLocalConnection = false;
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
int32 BatchPutMaxBytes = 1024*1024;
int32 CacheRecordBatchSize = 8;
int32 CacheChunksBatchSize = 8;
FBackendDebugOptions DebugOptions;
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
TAnsiStringBuilder<256> RpcUri;
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
};
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
template <typename RequestType>
struct FZenCacheStore::TRequestWithStats
{
RequestType Request;
mutable FRequestStats Stats;
explicit TRequestWithStats(const RequestType& InRequest)
: Request(InRequest)
{
}
void EndRequest(FZenCacheStore& Outer, const EStatus Status) const
{
{
TUniqueLock Lock(Stats.Mutex);
Stats.EndTime = FMonotonicTimePoint::Now();
Stats.Status = Status;
}
if (Outer.StoreStats)
{
Outer.StoreStats->AddRequest(Stats);
}
}
};
template <typename OutContainerType, typename InContainerType, typename BucketAccessorType, typename RequestTypeAccessorType>
void FZenCacheStore::StartRequests(
OutContainerType& Out,
const InContainerType& In,
BucketAccessorType BucketAccessor,
RequestTypeAccessorType TypeAccessor,
const ERequestOp Op)
{
const FMonotonicTimePoint Now = FMonotonicTimePoint::Now();
Out.Reserve(In.Num());
for (const auto& Request : In)
{
auto& RequestWithStats = Out[Out.Emplace(Request)];
DDC: Added request name to FCacheStoreRequestStats #jira UE-133395 #rb Zousar.Shaker #rnx [CL 26210120 by devin doucette in ue5-main branch]
2023-06-23 10:51:52 -04:00
RequestWithStats.Stats.Name = Request.Name;
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
RequestWithStats.Stats.Bucket = BucketAccessor(Request);
RequestWithStats.Stats.Type = TypeAccessor(Request);
RequestWithStats.Stats.Op = Op;
RequestWithStats.Stats.StartTime = Now;
}
}
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
class FZenCacheStore::FPutOp final : public FThreadSafeRefCountedObject
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
public:
FPutOp(FZenCacheStore& InCacheStore,
IRequestOwner& InOwner,
const TConstArrayView<FCachePutRequest> InRequests,
FOnCachePutComplete&& InOnComplete)
: CacheStore(InCacheStore)
, Owner(InOwner)
, OnComplete(MoveTemp(InOnComplete))
{
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
StartRequests(Requests, InRequests, [](const FCachePutRequest& Request) { return Request.Record.GetKey().Bucket; },
[](auto&) { return ERequestType::Record; }, ERequestOp::Put);
Batches = TBatchView<const TRequestWithStats<FCachePutRequest>>(Requests,
[this](const TRequestWithStats<FCachePutRequest>& NextRequest) { return BatchGroupingFilter(NextRequest.Request); });
TRACE_COUNTER_ADD(ZenDDC_Put, int64(Requests.Num()));
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
void IssueRequests()
{
FRequestBarrier Barrier(Owner);
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
for (TArrayView<const TRequestWithStats<FCachePutRequest>> Batch : Batches)
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
FCbPackage BatchPackage;
FCbWriter BatchWriter;
BatchWriter.BeginObject();
{
BatchWriter << ANSITEXTVIEW("Method") << "PutCacheRecords";
Add "Accept" field in Zen RPC request to enable new style RPC responses. See https://github.com/EpicGames/zen/pull/180 for Zen side change. #rb stefan.boberg zousar.shaker #preflight 6347c2568a0a7b2adcbe4d71 [CL 22505182 by dan engelbrecht in ue5-main branch]
2022-10-13 11:17:53 -04:00
BatchWriter.AddInteger(ANSITEXTVIEW("Accept"), Zen::Http::kCbPkgMagic);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
BatchWriter.BeginObject(ANSITEXTVIEW("Params"));
{
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
ECachePolicy BatchDefaultPolicy = Batch[0].Request.Policy.GetRecordPolicy();
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
BatchWriter << ANSITEXTVIEW("DefaultPolicy") << *WriteToString<128>(BatchDefaultPolicy);
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
BatchWriter.AddString(ANSITEXTVIEW("Namespace"), CacheStore.Namespace);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
BatchWriter.BeginArray(ANSITEXTVIEW("Requests"));
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
for (const TRequestWithStats<FCachePutRequest>& RequestWithStats : Batch)
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
const FCachePutRequest& Request = RequestWithStats.Request;
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
const FCacheRecord& Record = Request.Record;
BatchWriter.BeginObject();
{
BatchWriter.SetName(ANSITEXTVIEW("Record"));
Record.Save(BatchPackage, BatchWriter);
Make sure record policy is set up correctly in batch requests to Zen #rb devin.doucette [CL 25982752 by dan engelbrecht in ue5-main branch]
2023-06-14 12:31:13 -04:00
if ((!Request.Policy.IsUniform()) || (Request.Policy.GetRecordPolicy() != BatchDefaultPolicy))
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
DDC: Fixed deprecation warnings in the Zen cache store #jira UE-157180 #rb none #rnx #ROBOMERGE-AUTHOR: devin.doucette #ROBOMERGE-SOURCE: CL 20745126 via CL 20745729 #ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v970-20704180) [CL 20751392 by devin doucette in ue5-main branch]
2022-06-20 23:40:25 -04:00
BatchWriter << ANSITEXTVIEW("Policy") << Request.Policy;
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
}
BatchWriter.EndObject();
}
BatchWriter.EndArray();
}
BatchWriter.EndObject();
}
BatchWriter.EndObject();
BatchPackage.SetObject(BatchWriter.Save().AsObject());
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
auto OnRpcComplete = [this, OpRef = TRefCountPtr<FPutOp>(this), Batch](THttpUniquePtr<IHttpResponse>& HttpResponse, FCbPackage& Response)
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
int32 RequestIndex = 0;
Check status code for responses received from zenserver DDC requests. #rb stefan.boberg #preflight 645a3339013f3751584a39ed [CL 25385641 by dan engelbrecht in ue5-main branch]
2023-05-09 08:08:07 -04:00
if (HttpResponse->GetErrorCode() == EHttpErrorCode::None && HttpResponse->GetStatusCode() >= 200 && HttpResponse->GetStatusCode() <= 299)
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
const FCbObject& ResponseObj = Response.GetObject();
for (FCbField ResponseField : ResponseObj[ANSITEXTVIEW("Result")])
{
if (RequestIndex >= Batch.Num())
{
++RequestIndex;
continue;
}
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
const TRequestWithStats<FCachePutRequest>& RequestWithStats = Batch[RequestIndex++];
RequestWithStats.Stats.AddLatency(FMonotonicTimeSpan::FromSeconds(HttpResponse->GetStats().StartTransferTime));
const FCacheKey& Key = RequestWithStats.Request.Record.GetKey();
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
bool bPutSucceeded = ResponseField.AsBool();
if (CacheStore.DebugOptions.ShouldSimulatePutMiss(Key))
{
UE_LOG(LogDerivedDataCache, Verbose, TEXT("%s: Simulated miss for put of %s from '%s'"),
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
*CacheStore.GetName(), *WriteToString<96>(Key), *RequestWithStats.Request.Name);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
bPutSucceeded = false;
}
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
bPutSucceeded ? OnHit(RequestWithStats) : OnMiss(RequestWithStats);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
if (RequestIndex != Batch.Num())
{
UE_LOG(LogDerivedDataCache, Warning,
DDC: Added more detail to warnings from the Zen cache client #preflight 62d9ced6d76ea4b503d3f487 #rb Zousar.Shaker #rnx [CL 21214276 by Devin Doucette in ue5-main branch]
2022-07-21 23:01:51 -04:00
TEXT("%s: Invalid response received from PutCacheRecords RPC: %d results expected, received %d, from %s"),
*CacheStore.GetName(), Batch.Num(), RequestIndex, *WriteToString<256>(*HttpResponse));
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
}
Fix handling of cancelation in Zen DDC client to avoid having it treated as equivalent to an unexpected error. #rb devin.doucette [CL 27969813 by zousar shaker in ue5-main branch]
2023-09-18 15:22:34 -04:00
else if ((HttpResponse->GetErrorCode() != EHttpErrorCode::Canceled) && (HttpResponse->GetStatusCode() != 404))
Check status code for responses received from zenserver DDC requests. #rb stefan.boberg #preflight 645a3339013f3751584a39ed [CL 25385641 by dan engelbrecht in ue5-main branch]
2023-05-09 08:08:07 -04:00
{
UE_LOG(LogDerivedDataCache, Warning,
TEXT("%s: Error response received from PutCacheRecords RPC: from %s"),
*CacheStore.GetName(), *WriteToString<256>(*HttpResponse));
}
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
for (const TRequestWithStats<FCachePutRequest>& RequestWithStats : Batch.RightChop(RequestIndex))
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
Fix handling of cancelation in Zen DDC client to avoid having it treated as equivalent to an unexpected error. #rb devin.doucette [CL 27969813 by zousar shaker in ue5-main branch]
2023-09-18 15:22:34 -04:00
if (HttpResponse->GetErrorCode() == EHttpErrorCode::Canceled)
{
OnCanceled(RequestWithStats);
}
else
{
OnMiss(RequestWithStats);
}
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
};
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
CacheStore.EnqueueAsyncRpc(Owner, BatchPackage, MoveTemp(OnRpcComplete));
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
}
private:
EBatchView BatchGroupingFilter(const FCachePutRequest& NextRequest)
{
const FCacheRecord& Record = NextRequest.Record;
uint64 RecordSize = sizeof(FCacheKey) + Record.GetMeta().GetSize();
for (const FValueWithId& Value : Record.GetValues())
{
RecordSize += Value.GetData().GetCompressedSize();
}
BatchSize += RecordSize;
if (BatchSize > CacheStore.BatchPutMaxBytes)
{
BatchSize = RecordSize;
return EBatchView::NewBatch;
}
return EBatchView::Continue;
}
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
void OnHit(const TRequestWithStats<FCachePutRequest>& RequestWithStats)
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
Fix handling of cancelation in Zen DDC client to avoid having it treated as equivalent to an unexpected error. #rb devin.doucette [CL 27969813 by zousar shaker in ue5-main branch]
2023-09-18 15:22:34 -04:00
UE_LOG(LogDerivedDataCache, Verbose, TEXT("%s: Cache put complete for %s from '%s'"),
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
*CacheStore.GetName(), *WriteToString<96>(RequestWithStats.Request.Record.GetKey()), *RequestWithStats.Request.Name);
if (const FCbObject& Meta = RequestWithStats.Request.Record.GetMeta())
Only report local stats for Zen DDC. ZenCacheStore now only tracks what the client sees, regardless if the zenserver instance gets a hit from a configured upstream. As we don't currently have any upstreams configured in the zen server instance this should give a more accurate number for hit/miss stats that matches the file cache. #preflight 63eabb1bb96a6b2718326d15 #rb zousar.shaker devin.doucette [CL 24206594 by dan engelbrecht in ue5-main branch]
2023-02-14 03:09:44 -05:00
{
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
RequestWithStats.Stats.PhysicalWriteSize += Meta.GetSize();
Only report local stats for Zen DDC. ZenCacheStore now only tracks what the client sees, regardless if the zenserver instance gets a hit from a configured upstream. As we don't currently have any upstreams configured in the zen server instance this should give a more accurate number for hit/miss stats that matches the file cache. #preflight 63eabb1bb96a6b2718326d15 #rb zousar.shaker devin.doucette [CL 24206594 by dan engelbrecht in ue5-main branch]
2023-02-14 03:09:44 -05:00
}
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
for (const FValueWithId& Value : RequestWithStats.Request.Record.GetValues())
{
RequestWithStats.Stats.AddLogicalWrite(Value);
RequestWithStats.Stats.PhysicalWriteSize += Value.GetData().GetCompressedSize();
}
RequestWithStats.EndRequest(CacheStore, EStatus::Ok);
TRACE_COUNTER_ADD(ZenDDC_BytesSent, int64(RequestWithStats.Stats.PhysicalWriteSize));
Only report local stats for Zen DDC. ZenCacheStore now only tracks what the client sees, regardless if the zenserver instance gets a hit from a configured upstream. As we don't currently have any upstreams configured in the zen server instance this should give a more accurate number for hit/miss stats that matches the file cache. #preflight 63eabb1bb96a6b2718326d15 #rb zousar.shaker devin.doucette [CL 24206594 by dan engelbrecht in ue5-main branch]
2023-02-14 03:09:44 -05:00
TRACE_COUNTER_INCREMENT(ZenDDC_PutHit);
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
OnComplete(RequestWithStats.Request.MakeResponse(EStatus::Ok));
}
Only report local stats for Zen DDC. ZenCacheStore now only tracks what the client sees, regardless if the zenserver instance gets a hit from a configured upstream. As we don't currently have any upstreams configured in the zen server instance this should give a more accurate number for hit/miss stats that matches the file cache. #preflight 63eabb1bb96a6b2718326d15 #rb zousar.shaker devin.doucette [CL 24206594 by dan engelbrecht in ue5-main branch]
2023-02-14 03:09:44 -05:00
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
void OnMiss(const TRequestWithStats<FCachePutRequest>& RequestWithStats)
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
Fix handling of cancelation in Zen DDC client to avoid having it treated as equivalent to an unexpected error. #rb devin.doucette [CL 27969813 by zousar shaker in ue5-main branch]
2023-09-18 15:22:34 -04:00
UE_LOG(LogDerivedDataCache, Verbose, TEXT("%s: Cache put failed for '%s' from '%s'"),
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
*CacheStore.GetName(), *WriteToString<96>(RequestWithStats.Request.Record.GetKey()), *RequestWithStats.Request.Name);
RequestWithStats.EndRequest(CacheStore, EStatus::Error);
OnComplete(RequestWithStats.Request.MakeResponse(EStatus::Error));
}
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
Fix handling of cancelation in Zen DDC client to avoid having it treated as equivalent to an unexpected error. #rb devin.doucette [CL 27969813 by zousar shaker in ue5-main branch]
2023-09-18 15:22:34 -04:00
void OnCanceled(const TRequestWithStats<FCachePutRequest>& RequestWithStats)
{
UE_LOG(LogDerivedDataCache, Verbose, TEXT("%s: Cache put failed with canceled request for '%s' from '%s'"),
*CacheStore.GetName(), *WriteToString<96>(RequestWithStats.Request.Record.GetKey()), *RequestWithStats.Request.Name);
RequestWithStats.EndRequest(CacheStore, EStatus::Canceled);
OnComplete(RequestWithStats.Request.MakeResponse(EStatus::Canceled));
}
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
FZenCacheStore& CacheStore;
IRequestOwner& Owner;
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
TArray<TRequestWithStats<FCachePutRequest>, TInlineAllocator<1>> Requests;
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
uint64 BatchSize = 0;
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
TBatchView<const TRequestWithStats<FCachePutRequest>> Batches;
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
FOnCachePutComplete OnComplete;
};
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
class FZenCacheStore::FGetOp final : public FThreadSafeRefCountedObject
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
public:
FGetOp(FZenCacheStore& InCacheStore,
IRequestOwner& InOwner,
const TConstArrayView<FCacheGetRequest> InRequests,
FOnCacheGetComplete&& InOnComplete)
: CacheStore(InCacheStore)
, Owner(InOwner)
, OnComplete(MoveTemp(InOnComplete))
{
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
StartRequests(Requests, InRequests, [](const FCacheGetRequest& Request) { return Request.Key.Bucket; },
[](auto&) { return ERequestType::Record; }, ERequestOp::Get);
TRACE_COUNTER_ADD(ZenDDC_Get, int64(Requests.Num()));
Only report local stats for Zen DDC. ZenCacheStore now only tracks what the client sees, regardless if the zenserver instance gets a hit from a configured upstream. As we don't currently have any upstreams configured in the zen server instance this should give a more accurate number for hit/miss stats that matches the file cache. #preflight 63eabb1bb96a6b2718326d15 #rb zousar.shaker devin.doucette [CL 24206594 by dan engelbrecht in ue5-main branch]
2023-02-14 03:09:44 -05:00
TRACE_COUNTER_ADD(ZenDDC_CacheRecordRequestCountInFlight, int64(Requests.Num()));
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
virtual ~FGetOp()
{
Only report local stats for Zen DDC. ZenCacheStore now only tracks what the client sees, regardless if the zenserver instance gets a hit from a configured upstream. As we don't currently have any upstreams configured in the zen server instance this should give a more accurate number for hit/miss stats that matches the file cache. #preflight 63eabb1bb96a6b2718326d15 #rb zousar.shaker devin.doucette [CL 24206594 by dan engelbrecht in ue5-main branch]
2023-02-14 03:09:44 -05:00
TRACE_COUNTER_SUBTRACT(ZenDDC_CacheRecordRequestCountInFlight, int64(Requests.Num()));
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
void IssueRequests()
{
FRequestBarrier Barrier(Owner);
ForEachBatch(CacheStore.CacheRecordBatchSize, Requests.Num(),
[this](int32 BatchFirst, int32 BatchLast)
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
{
TConstArrayView<TRequestWithStats<FCacheGetRequest>> Batch(Requests.GetData() + BatchFirst, BatchLast - BatchFirst + 1);
FCbWriter BatchRequest;
BatchRequest.BeginObject();
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
BatchRequest << ANSITEXTVIEW("Method") << ANSITEXTVIEW("GetCacheRecords");
BatchRequest.AddInteger(ANSITEXTVIEW("Accept"), Zen::Http::kCbPkgMagic);
if (CacheStore.bIsLocalConnection)
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
BatchRequest.AddInteger(ANSITEXTVIEW("AcceptFlags"), static_cast<uint32_t>(Zen::Http::RpcAcceptOptions::kAllowLocalReferences));
BatchRequest.AddInteger(ANSITEXTVIEW("Pid"), FPlatformProcess::GetCurrentProcessId());
}
Add "Accept" field in Zen RPC request to enable new style RPC responses. See https://github.com/EpicGames/zen/pull/180 for Zen side change. #rb stefan.boberg zousar.shaker #preflight 6347c2568a0a7b2adcbe4d71 [CL 22505182 by dan engelbrecht in ue5-main branch]
2022-10-13 11:17:53 -04:00
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
BatchRequest.BeginObject(ANSITEXTVIEW("Params"));
{
ECachePolicy BatchDefaultPolicy = Batch[0].Request.Policy.GetRecordPolicy();
BatchRequest << ANSITEXTVIEW("DefaultPolicy") << *WriteToString<128>(BatchDefaultPolicy);
BatchRequest.AddString(ANSITEXTVIEW("Namespace"), CacheStore.Namespace);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
BatchRequest.BeginArray(ANSITEXTVIEW("Requests"));
for (const TRequestWithStats<FCacheGetRequest>& RequestWithStats : Batch)
{
BatchRequest.BeginObject();
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
BatchRequest << ANSITEXTVIEW("Key") << RequestWithStats.Request.Key;
Make sure record policy is set up correctly in batch requests to Zen #rb devin.doucette [CL 25982752 by dan engelbrecht in ue5-main branch]
2023-06-14 12:31:13 -04:00
if ((!RequestWithStats.Request.Policy.IsUniform()) || (RequestWithStats.Request.Policy.GetRecordPolicy() != BatchDefaultPolicy))
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
BatchRequest << ANSITEXTVIEW("Policy") << RequestWithStats.Request.Policy;
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
}
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
BatchRequest.EndObject();
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
BatchRequest.EndArray();
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
BatchRequest.EndObject();
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
}
BatchRequest.EndObject();
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
FGetOp* OriginalOp = this;
auto OnRpcComplete = [this, OpRef = TRefCountPtr<FGetOp>(OriginalOp), Batch](THttpUniquePtr<IHttpResponse>& HttpResponse, FCbPackage& Response)
{
int32 RequestIndex = 0;
if (HttpResponse->GetErrorCode() == EHttpErrorCode::None && HttpResponse->GetStatusCode() >= 200 && HttpResponse->GetStatusCode() <= 299)
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
const FCbObject& ResponseObj = Response.GetObject();
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
for (FCbField RecordField : ResponseObj[ANSITEXTVIEW("Result")])
{
if (RequestIndex >= Batch.Num())
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
++RequestIndex;
continue;
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
const TRequestWithStats<FCacheGetRequest>& RequestWithStats = Batch[RequestIndex++];
RequestWithStats.Stats.AddLatency(FMonotonicTimeSpan::FromSeconds(HttpResponse->GetStats().StartTransferTime));
const FCacheKey& Key = RequestWithStats.Request.Key;
FOptionalCacheRecord Record;
if (CacheStore.DebugOptions.ShouldSimulateGetMiss(Key))
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
UE_LOG(LogDerivedDataCache, Verbose, TEXT("%s: Simulated miss for put of '%s' from '%s'"),
*CacheStore.GetName(), *WriteToString<96>(Key), *RequestWithStats.Request.Name);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
else if (!RecordField.IsNull())
{
Record = FCacheRecord::Load(Response, RecordField.AsObject());
}
Record ? OnHit(RequestWithStats, MoveTemp(Record).Get()) : OnMiss(RequestWithStats);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
if (RequestIndex != Batch.Num())
Check status code for responses received from zenserver DDC requests. #rb stefan.boberg #preflight 645a3339013f3751584a39ed [CL 25385641 by dan engelbrecht in ue5-main branch]
2023-05-09 08:08:07 -04:00
{
UE_LOG(LogDerivedDataCache, Warning,
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
TEXT("%s: Invalid response received from GetCacheRecords RPC: %d results expected, received %d, from %s"),
*CacheStore.GetName(), Batch.Num(), RequestIndex, *WriteToString<256>(*HttpResponse));
Check status code for responses received from zenserver DDC requests. #rb stefan.boberg #preflight 645a3339013f3751584a39ed [CL 25385641 by dan engelbrecht in ue5-main branch]
2023-05-09 08:08:07 -04:00
}
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
}
Fix handling of cancelation in Zen DDC client to avoid having it treated as equivalent to an unexpected error. #rb devin.doucette [CL 27969813 by zousar shaker in ue5-main branch]
2023-09-18 15:22:34 -04:00
else if ((HttpResponse->GetErrorCode() != EHttpErrorCode::Canceled) && (HttpResponse->GetStatusCode() != 404))
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
{
UE_LOG(LogDerivedDataCache, Warning,
TEXT("%s: Error response received from GetCacheRecords RPC: from %s"),
*CacheStore.GetName(), *WriteToString<256>(*HttpResponse));
}
Check status code for responses received from zenserver DDC requests. #rb stefan.boberg #preflight 645a3339013f3751584a39ed [CL 25385641 by dan engelbrecht in ue5-main branch]
2023-05-09 08:08:07 -04:00
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
for (const TRequestWithStats<FCacheGetRequest>& RequestWithStats : Batch.RightChop(RequestIndex))
{
Fix handling of cancelation in Zen DDC client to avoid having it treated as equivalent to an unexpected error. #rb devin.doucette [CL 27969813 by zousar shaker in ue5-main branch]
2023-09-18 15:22:34 -04:00
if (HttpResponse->GetErrorCode() == EHttpErrorCode::Canceled)
{
OnCanceled(RequestWithStats);
}
else
{
OnMiss(RequestWithStats);
}
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
}
};
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
CacheStore.EnqueueAsyncRpc(Owner, BatchRequest.Save().AsObject(), MoveTemp(OnRpcComplete));
});
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
private:
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
void OnHit(const TRequestWithStats<FCacheGetRequest>& RequestWithStats, FCacheRecord&& Record)
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
UE_LOG(LogDerivedDataCache, Verbose, TEXT("%s: Cache hit for '%s' from '%s'"),
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
*CacheStore.GetName(), *WriteToString<96>(RequestWithStats.Request.Key), *RequestWithStats.Request.Name);
Return error if we receive a partial record from zenserver #rb devin.doucette #prefligth 648a3d2f88385dd88cfdda4a [CL 26006460 by dan engelbrecht in ue5-main branch]
2023-06-15 04:54:40 -04:00
bool bInComplete = false;
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
if (const FCbObject& Meta = Record.GetMeta())
{
RequestWithStats.Stats.PhysicalReadSize += Meta.GetSize();
}
for (const FValueWithId& Value : Record.GetValues())
{
RequestWithStats.Stats.AddLogicalRead(Value);
RequestWithStats.Stats.PhysicalReadSize += Value.GetData().GetCompressedSize();
Return error if we receive a partial record from zenserver #rb devin.doucette #prefligth 648a3d2f88385dd88cfdda4a [CL 26006460 by dan engelbrecht in ue5-main branch]
2023-06-15 04:54:40 -04:00
ECachePolicy ValuePolicy = RequestWithStats.Request.Policy.GetValuePolicy(Value.GetId());
if (EnumHasAnyFlags(ValuePolicy, ECachePolicy::SkipData))
{
continue;
}
if (Value.HasData())
{
continue;
}
if (EnumHasAnyFlags(ValuePolicy, ECachePolicy::Query))
{
bInComplete = true;
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
}
Return error if we receive a partial record from zenserver #rb devin.doucette #prefligth 648a3d2f88385dd88cfdda4a [CL 26006460 by dan engelbrecht in ue5-main branch]
2023-06-15 04:54:40 -04:00
}
EStatus Status = bInComplete ? EStatus::Error : EStatus::Ok;
RequestWithStats.EndRequest(CacheStore, Status);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
Only report local stats for Zen DDC. ZenCacheStore now only tracks what the client sees, regardless if the zenserver instance gets a hit from a configured upstream. As we don't currently have any upstreams configured in the zen server instance this should give a more accurate number for hit/miss stats that matches the file cache. #preflight 63eabb1bb96a6b2718326d15 #rb zousar.shaker devin.doucette [CL 24206594 by dan engelbrecht in ue5-main branch]
2023-02-14 03:09:44 -05:00
TRACE_COUNTER_INCREMENT(ZenDDC_GetHit);
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
TRACE_COUNTER_ADD(ZenDDC_BytesReceived, int64(RequestWithStats.Stats.PhysicalReadSize));
Return error if we receive a partial record from zenserver #rb devin.doucette #prefligth 648a3d2f88385dd88cfdda4a [CL 26006460 by dan engelbrecht in ue5-main branch]
2023-06-15 04:54:40 -04:00
OnComplete({RequestWithStats.Request.Name, MoveTemp(Record), RequestWithStats.Request.UserData, Status});
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
}
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
void OnMiss(const TRequestWithStats<FCacheGetRequest>& RequestWithStats)
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
UE_LOG(LogDerivedDataCache, Verbose, TEXT("%s: Cache miss for '%s' from '%s'"),
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
*CacheStore.GetName(), *WriteToString<96>(RequestWithStats.Request.Key), *RequestWithStats.Request.Name);
RequestWithStats.EndRequest(CacheStore, EStatus::Error);
OnComplete(RequestWithStats.Request.MakeResponse(EStatus::Error));
}
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
Fix handling of cancelation in Zen DDC client to avoid having it treated as equivalent to an unexpected error. #rb devin.doucette [CL 27969813 by zousar shaker in ue5-main branch]
2023-09-18 15:22:34 -04:00
void OnCanceled(const TRequestWithStats<FCacheGetRequest>& RequestWithStats)
{
UE_LOG(LogDerivedDataCache, Verbose, TEXT("%s: Cache miss with canceled request for '%s' from '%s'"),
*CacheStore.GetName(), *WriteToString<96>(RequestWithStats.Request.Key), *RequestWithStats.Request.Name);
RequestWithStats.EndRequest(CacheStore, EStatus::Canceled);
OnComplete(RequestWithStats.Request.MakeResponse(EStatus::Canceled));
}
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
FZenCacheStore& CacheStore;
IRequestOwner& Owner;
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
TArray<TRequestWithStats<FCacheGetRequest>, TInlineAllocator<1>> Requests;
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
FOnCacheGetComplete OnComplete;
};
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
class FZenCacheStore::FPutValueOp final : public FThreadSafeRefCountedObject
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
public:
FPutValueOp(FZenCacheStore& InCacheStore,
IRequestOwner& InOwner,
const TConstArrayView<FCachePutValueRequest> InRequests,
FOnCachePutValueComplete&& InOnComplete)
: CacheStore(InCacheStore)
, Owner(InOwner)
, OnComplete(MoveTemp(InOnComplete))
{
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
StartRequests(Requests, InRequests, [](const FCachePutValueRequest& Request) { return Request.Key.Bucket; },
[](auto&) { return ERequestType::Value; }, ERequestOp::Put);
TRACE_COUNTER_ADD(ZenDDC_Put, int64(Requests.Num()));
Batches = TBatchView<const TRequestWithStats<FCachePutValueRequest>>(Requests,
[this](const TRequestWithStats<FCachePutValueRequest>& NextRequest) { return BatchGroupingFilter(NextRequest.Request); });
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
void IssueRequests()
{
FRequestBarrier Barrier(Owner);
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
for (TArrayView<const TRequestWithStats<FCachePutValueRequest>> Batch : Batches)
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
FCbPackage BatchPackage;
FCbWriter BatchWriter;
BatchWriter.BeginObject();
{
BatchWriter << ANSITEXTVIEW("Method") << ANSITEXTVIEW("PutCacheValues");
Add "Accept" field in Zen RPC request to enable new style RPC responses. See https://github.com/EpicGames/zen/pull/180 for Zen side change. #rb stefan.boberg zousar.shaker #preflight 6347c2568a0a7b2adcbe4d71 [CL 22505182 by dan engelbrecht in ue5-main branch]
2022-10-13 11:17:53 -04:00
BatchWriter.AddInteger(ANSITEXTVIEW("Accept"), Zen::Http::kCbPkgMagic);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
BatchWriter.BeginObject(ANSITEXTVIEW("Params"));
{
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
ECachePolicy BatchDefaultPolicy = Batch[0].Request.Policy;
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
BatchWriter << ANSITEXTVIEW("DefaultPolicy") << *WriteToString<128>(BatchDefaultPolicy);
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
BatchWriter.AddString(ANSITEXTVIEW("Namespace"), CacheStore.Namespace);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
BatchWriter.BeginArray("Requests");
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
for (const TRequestWithStats<FCachePutValueRequest>& RequestWithStats : Batch)
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
BatchWriter.BeginObject();
{
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
BatchWriter << ANSITEXTVIEW("Key") << RequestWithStats.Request.Key;
const FValue& Value = RequestWithStats.Request.Value;
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
BatchWriter.AddBinaryAttachment("RawHash", Value.GetRawHash());
if (Value.HasData())
{
BatchPackage.AddAttachment(FCbAttachment(Value.GetData()));
}
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
if (RequestWithStats.Request.Policy != BatchDefaultPolicy)
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
BatchWriter << ANSITEXTVIEW("Policy") << WriteToString<128>(RequestWithStats.Request.Policy);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
}
BatchWriter.EndObject();
}
BatchWriter.EndArray();
}
BatchWriter.EndObject();
}
BatchWriter.EndObject();
BatchPackage.SetObject(BatchWriter.Save().AsObject());
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
auto OnRpcComplete = [this, OpRef = TRefCountPtr<FPutValueOp>(this), Batch](THttpUniquePtr<IHttpResponse>& HttpResponse, FCbPackage& Response)
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
int32 RequestIndex = 0;
Check status code for responses received from zenserver DDC requests. #rb stefan.boberg #preflight 645a3339013f3751584a39ed [CL 25385641 by dan engelbrecht in ue5-main branch]
2023-05-09 08:08:07 -04:00
if (HttpResponse->GetErrorCode() == EHttpErrorCode::None && HttpResponse->GetStatusCode() >= 200 && HttpResponse->GetStatusCode() <= 299)
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
const FCbObject& ResponseObj = Response.GetObject();
for (FCbField ResponseField : ResponseObj[ANSITEXTVIEW("Result")])
{
if (RequestIndex >= Batch.Num())
{
++RequestIndex;
continue;
}
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
const TRequestWithStats<FCachePutValueRequest>& RequestWithStats = Batch[RequestIndex++];
RequestWithStats.Stats.AddLatency(FMonotonicTimeSpan::FromSeconds(HttpResponse->GetStats().StartTransferTime));
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
bool bPutSucceeded = ResponseField.AsBool();
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
if (CacheStore.DebugOptions.ShouldSimulatePutMiss(RequestWithStats.Request.Key))
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
UE_LOG(LogDerivedDataCache, Verbose, TEXT("%s: Simulated miss for PutValue of %s from '%s'"),
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
*CacheStore.GetName(), *WriteToString<96>(RequestWithStats.Request.Key), *RequestWithStats.Request.Name);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
bPutSucceeded = false;
}
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
bPutSucceeded ? OnHit(RequestWithStats) : OnMiss(RequestWithStats);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
if (RequestIndex != Batch.Num())
{
UE_LOG(LogDerivedDataCache, Warning,
DDC: Added more detail to warnings from the Zen cache client #preflight 62d9ced6d76ea4b503d3f487 #rb Zousar.Shaker #rnx [CL 21214276 by Devin Doucette in ue5-main branch]
2022-07-21 23:01:51 -04:00
TEXT("%s: Invalid response received from PutCacheValues RPC: %d results expected, received %d, from %s"),
*CacheStore.GetName(), Batch.Num(), RequestIndex, *WriteToString<256>(*HttpResponse));
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
}
Fix handling of cancelation in Zen DDC client to avoid having it treated as equivalent to an unexpected error. #rb devin.doucette [CL 27969813 by zousar shaker in ue5-main branch]
2023-09-18 15:22:34 -04:00
else if ((HttpResponse->GetErrorCode() != EHttpErrorCode::Canceled) && (HttpResponse->GetStatusCode() != 404))
Check status code for responses received from zenserver DDC requests. #rb stefan.boberg #preflight 645a3339013f3751584a39ed [CL 25385641 by dan engelbrecht in ue5-main branch]
2023-05-09 08:08:07 -04:00
{
UE_LOG(LogDerivedDataCache, Warning,
TEXT("%s: Error response received from PutCacheValues RPC: from %s"),
*CacheStore.GetName(), *WriteToString<256>(*HttpResponse));
}
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
for (const TRequestWithStats<FCachePutValueRequest>& RequestWithStats : Batch.RightChop(RequestIndex))
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
Fix handling of cancelation in Zen DDC client to avoid having it treated as equivalent to an unexpected error. #rb devin.doucette [CL 27969813 by zousar shaker in ue5-main branch]
2023-09-18 15:22:34 -04:00
if (HttpResponse->GetErrorCode() == EHttpErrorCode::Canceled)
{
OnCanceled(RequestWithStats);
}
else
{
OnMiss(RequestWithStats);
}
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
};
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
CacheStore.EnqueueAsyncRpc(Owner, BatchPackage, MoveTemp(OnRpcComplete));
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
}
private:
EBatchView BatchGroupingFilter(const FCachePutValueRequest& NextRequest)
{
uint64 ValueSize = sizeof(FCacheKey) + NextRequest.Value.GetData().GetCompressedSize();
BatchSize += ValueSize;
if (BatchSize > CacheStore.BatchPutMaxBytes)
{
BatchSize = ValueSize;
return EBatchView::NewBatch;
}
return EBatchView::Continue;
}
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
void OnHit(const TRequestWithStats<FCachePutValueRequest>& RequestWithStats)
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
Fix handling of cancelation in Zen DDC client to avoid having it treated as equivalent to an unexpected error. #rb devin.doucette [CL 27969813 by zousar shaker in ue5-main branch]
2023-09-18 15:22:34 -04:00
UE_LOG(LogDerivedDataCache, Verbose, TEXT("%s: Cache put complete for %s from '%s'"),
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
*CacheStore.GetName(), *WriteToString<96>(RequestWithStats.Request.Key), *RequestWithStats.Request.Name);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
RequestWithStats.Stats.AddLogicalWrite(RequestWithStats.Request.Value);
RequestWithStats.Stats.PhysicalWriteSize += RequestWithStats.Request.Value.GetData().GetCompressedSize();
RequestWithStats.EndRequest(CacheStore, EStatus::Ok);
TRACE_COUNTER_INCREMENT(ZenDDC_PutHit);
TRACE_COUNTER_ADD(ZenDDC_BytesSent, RequestWithStats.Stats.PhysicalWriteSize);
OnComplete(RequestWithStats.Request.MakeResponse(EStatus::Ok));
}
void OnMiss(const TRequestWithStats<FCachePutValueRequest>& RequestWithStats)
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
Fix handling of cancelation in Zen DDC client to avoid having it treated as equivalent to an unexpected error. #rb devin.doucette [CL 27969813 by zousar shaker in ue5-main branch]
2023-09-18 15:22:34 -04:00
UE_LOG(LogDerivedDataCache, Verbose, TEXT("%s: Cache put failed for '%s' from '%s'"),
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
*CacheStore.GetName(), *WriteToString<96>(RequestWithStats.Request.Key), *RequestWithStats.Request.Name);
RequestWithStats.EndRequest(CacheStore, EStatus::Error);
OnComplete(RequestWithStats.Request.MakeResponse(EStatus::Error));
}
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
Fix handling of cancelation in Zen DDC client to avoid having it treated as equivalent to an unexpected error. #rb devin.doucette [CL 27969813 by zousar shaker in ue5-main branch]
2023-09-18 15:22:34 -04:00
void OnCanceled(const TRequestWithStats<FCachePutValueRequest>& RequestWithStats)
{
UE_LOG(LogDerivedDataCache, Verbose, TEXT("%s: Cache put failed with canceled request for '%s' from '%s'"),
*CacheStore.GetName(), *WriteToString<96>(RequestWithStats.Request.Key), *RequestWithStats.Request.Name);
RequestWithStats.EndRequest(CacheStore, EStatus::Canceled);
OnComplete(RequestWithStats.Request.MakeResponse(EStatus::Canceled));
}
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
FZenCacheStore& CacheStore;
IRequestOwner& Owner;
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
TArray<TRequestWithStats<FCachePutValueRequest>, TInlineAllocator<1>> Requests;
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
uint64 BatchSize = 0;
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
TBatchView<const TRequestWithStats<FCachePutValueRequest>> Batches;
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
FOnCachePutValueComplete OnComplete;
};
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
class FZenCacheStore::FGetValueOp final : public FThreadSafeRefCountedObject
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
public:
FGetValueOp(FZenCacheStore& InCacheStore,
IRequestOwner& InOwner,
const TConstArrayView<FCacheGetValueRequest> InRequests,
FOnCacheGetValueComplete&& InOnComplete)
: CacheStore(InCacheStore)
, Owner(InOwner)
, OnComplete(MoveTemp(InOnComplete))
{
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
StartRequests(Requests, InRequests, [](const FCacheGetValueRequest& Request) { return Request.Key.Bucket; },
[](auto&) { return ERequestType::Value; }, ERequestOp::Get);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
TRACE_COUNTER_ADD(ZenDDC_Get, (int64)Requests.Num());
Only report local stats for Zen DDC. ZenCacheStore now only tracks what the client sees, regardless if the zenserver instance gets a hit from a configured upstream. As we don't currently have any upstreams configured in the zen server instance this should give a more accurate number for hit/miss stats that matches the file cache. #preflight 63eabb1bb96a6b2718326d15 #rb zousar.shaker devin.doucette [CL 24206594 by dan engelbrecht in ue5-main branch]
2023-02-14 03:09:44 -05:00
TRACE_COUNTER_ADD(ZenDDC_CacheRecordRequestCountInFlight, int64(Requests.Num()));
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
virtual ~FGetValueOp()
{
Only report local stats for Zen DDC. ZenCacheStore now only tracks what the client sees, regardless if the zenserver instance gets a hit from a configured upstream. As we don't currently have any upstreams configured in the zen server instance this should give a more accurate number for hit/miss stats that matches the file cache. #preflight 63eabb1bb96a6b2718326d15 #rb zousar.shaker devin.doucette [CL 24206594 by dan engelbrecht in ue5-main branch]
2023-02-14 03:09:44 -05:00
TRACE_COUNTER_SUBTRACT(ZenDDC_CacheRecordRequestCountInFlight, int64(Requests.Num()));
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
void IssueRequests()
{
FRequestBarrier Barrier(Owner);
ForEachBatch(CacheStore.CacheRecordBatchSize, Requests.Num(),
[this](int32 BatchFirst, int32 BatchLast)
{
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
TConstArrayView<TRequestWithStats<FCacheGetValueRequest>> Batch(Requests.GetData() + BatchFirst, BatchLast - BatchFirst + 1);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
FCbWriter BatchRequest;
BatchRequest.BeginObject();
{
BatchRequest << ANSITEXTVIEW("Method") << ANSITEXTVIEW("GetCacheValues");
Add "Accept" field in Zen RPC request to enable new style RPC responses. See https://github.com/EpicGames/zen/pull/180 for Zen side change. #rb stefan.boberg zousar.shaker #preflight 6347c2568a0a7b2adcbe4d71 [CL 22505182 by dan engelbrecht in ue5-main branch]
2022-10-13 11:17:53 -04:00
BatchRequest.AddInteger(ANSITEXTVIEW("Accept"), Zen::Http::kCbPkgMagic);
Allow support for receiving local file references for cache items Companion PR in zen: https://github.com/EpicGames/zen/pull/184 #rb stefan.boberg #preflight 6368ce57d0174259cc1e7b7a [CL 23009037 by dan engelbrecht in ue5-main branch]
2022-11-07 05:33:21 -05:00
if (CacheStore.bIsLocalConnection)
{
BatchRequest.AddInteger(ANSITEXTVIEW("AcceptFlags"), static_cast<uint32_t>(Zen::Http::RpcAcceptOptions::kAllowLocalReferences));
Accept duplicated file handles in Zen RPC responses #preflight 641c11bf76461c460b8eabb4 #rb stefan.boberg per.larsson [CL 24759274 by dan engelbrecht in ue5-main branch]
2023-03-23 05:15:22 -04:00
BatchRequest.AddInteger(ANSITEXTVIEW("Pid"), FPlatformProcess::GetCurrentProcessId());
Allow support for receiving local file references for cache items Companion PR in zen: https://github.com/EpicGames/zen/pull/184 #rb stefan.boberg #preflight 6368ce57d0174259cc1e7b7a [CL 23009037 by dan engelbrecht in ue5-main branch]
2022-11-07 05:33:21 -05:00
}
Add "Accept" field in Zen RPC request to enable new style RPC responses. See https://github.com/EpicGames/zen/pull/180 for Zen side change. #rb stefan.boberg zousar.shaker #preflight 6347c2568a0a7b2adcbe4d71 [CL 22505182 by dan engelbrecht in ue5-main branch]
2022-10-13 11:17:53 -04:00
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
BatchRequest.BeginObject(ANSITEXTVIEW("Params"));
{
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
ECachePolicy BatchDefaultPolicy = Batch[0].Request.Policy;
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
BatchRequest << ANSITEXTVIEW("DefaultPolicy") << *WriteToString<128>(BatchDefaultPolicy);
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
BatchRequest.AddString(ANSITEXTVIEW("Namespace"), CacheStore.Namespace);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
BatchRequest.BeginArray("Requests");
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
for (const TRequestWithStats<FCacheGetValueRequest>& RequestWithStats : Batch)
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
BatchRequest.BeginObject();
{
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
BatchRequest << ANSITEXTVIEW("Key") << RequestWithStats.Request.Key;
if (RequestWithStats.Request.Policy != BatchDefaultPolicy)
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
BatchRequest << ANSITEXTVIEW("Policy") << WriteToString<128>(RequestWithStats.Request.Policy);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
}
BatchRequest.EndObject();
}
BatchRequest.EndArray();
}
BatchRequest.EndObject();
}
BatchRequest.EndObject();
FGetValueOp* OriginalOp = this;
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
auto OnRpcComplete = [this, OpRef = TRefCountPtr<FGetValueOp>(OriginalOp), Batch](THttpUniquePtr<IHttpResponse>& HttpResponse, FCbPackage& Response)
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
int32 RequestIndex = 0;
Check status code for responses received from zenserver DDC requests. #rb stefan.boberg #preflight 645a3339013f3751584a39ed [CL 25385641 by dan engelbrecht in ue5-main branch]
2023-05-09 08:08:07 -04:00
if (HttpResponse->GetErrorCode() == EHttpErrorCode::None && HttpResponse->GetStatusCode() >= 200 && HttpResponse->GetStatusCode() <= 299)
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
const FCbObject& ResponseObj = Response.GetObject();
for (FCbFieldView ResultField : ResponseObj[ANSITEXTVIEW("Result")])
{
if (RequestIndex >= Batch.Num())
{
++RequestIndex;
continue;
}
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
const TRequestWithStats<FCacheGetValueRequest>& RequestWithStats = Batch[RequestIndex++];
RequestWithStats.Stats.AddLatency(FMonotonicTimeSpan::FromSeconds(HttpResponse->GetStats().StartTransferTime));
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
FCbObjectView ResultObj = ResultField.AsObjectView();
TOptional<FValue> Value;
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
if (CacheStore.DebugOptions.ShouldSimulateGetMiss(RequestWithStats.Request.Key))
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
UE_LOG(LogDerivedDataCache, Verbose, TEXT("%s: Simulated miss for GetValue of '%s' from '%s'"),
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
*CacheStore.GetName(), *WriteToString<96>(RequestWithStats.Request.Key), *RequestWithStats.Request.Name);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
else
{
FCbFieldView RawHashField = ResultObj["RawHash"];
FIoHash RawHash = RawHashField.AsHash();
if (const FCbAttachment* Attachment = Response.FindAttachment(RawHash))
{
Value.Emplace(Attachment->AsCompressedBinary());
}
else
{
FCbFieldView RawSizeField = ResultObj["RawSize"];
uint64 RawSize = RawSizeField.AsUInt64();
if (!RawSizeField.HasError() && !RawHashField.HasError())
{
Value.Emplace(RawHash, RawSize);
}
}
}
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
(bool)Value ? OnHit(RequestWithStats, MoveTemp(*Value)) : OnMiss(RequestWithStats);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
if (RequestIndex != Batch.Num())
{
UE_LOG(LogDerivedDataCache, Warning,
DDC: Added more detail to warnings from the Zen cache client #preflight 62d9ced6d76ea4b503d3f487 #rb Zousar.Shaker #rnx [CL 21214276 by Devin Doucette in ue5-main branch]
2022-07-21 23:01:51 -04:00
TEXT("%s: Invalid response received from GetCacheValues RPC: %d results expected, received %d from %s"),
*CacheStore.GetName(), Batch.Num(), RequestIndex, *WriteToString<256>(*HttpResponse));
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
}
Fix handling of cancelation in Zen DDC client to avoid having it treated as equivalent to an unexpected error. #rb devin.doucette [CL 27969813 by zousar shaker in ue5-main branch]
2023-09-18 15:22:34 -04:00
else if ((HttpResponse->GetErrorCode() != EHttpErrorCode::Canceled) && (HttpResponse->GetStatusCode() != 404))
Check status code for responses received from zenserver DDC requests. #rb stefan.boberg #preflight 645a3339013f3751584a39ed [CL 25385641 by dan engelbrecht in ue5-main branch]
2023-05-09 08:08:07 -04:00
{
UE_LOG(LogDerivedDataCache, Warning,
TEXT("%s: Error response received from GetCacheValues RPC: from %s"),
*CacheStore.GetName(), *WriteToString<256>(*HttpResponse));
}
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
for (const TRequestWithStats<FCacheGetValueRequest>& RequestWithStats : Batch.RightChop(RequestIndex))
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
Fix handling of cancelation in Zen DDC client to avoid having it treated as equivalent to an unexpected error. #rb devin.doucette [CL 27969813 by zousar shaker in ue5-main branch]
2023-09-18 15:22:34 -04:00
if (HttpResponse->GetErrorCode() == EHttpErrorCode::Canceled)
{
OnCanceled(RequestWithStats);
}
else
{
OnMiss(RequestWithStats);
}
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
};
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
CacheStore.EnqueueAsyncRpc(Owner, BatchRequest.Save().AsObject(), MoveTemp(OnRpcComplete));
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
});
}
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
private:
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
void OnHit(const TRequestWithStats<FCacheGetValueRequest>& RequestWithStats, FValue&& Value)
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
UE_LOG(LogDerivedDataCache, Verbose, TEXT("%s: Cache hit for '%s' from '%s'"),
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
*CacheStore.GetName(), *WriteToString<96>(RequestWithStats.Request.Key), *RequestWithStats.Request.Name);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
RequestWithStats.Stats.AddLogicalRead(Value);
RequestWithStats.Stats.PhysicalReadSize += Value.GetData().GetCompressedSize();
RequestWithStats.EndRequest(CacheStore, EStatus::Ok);
TRACE_COUNTER_INCREMENT(ZenDDC_GetHit);
TRACE_COUNTER_ADD(ZenDDC_BytesReceived, int64(RequestWithStats.Stats.PhysicalReadSize));
OnComplete({RequestWithStats.Request.Name, RequestWithStats.Request.Key, MoveTemp(Value), RequestWithStats.Request.UserData, EStatus::Ok});
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
};
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
void OnMiss(const TRequestWithStats<FCacheGetValueRequest>& RequestWithStats)
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
UE_LOG(LogDerivedDataCache, Verbose, TEXT("%s: Cache miss for '%s' from '%s'"),
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
*CacheStore.GetName(), *WriteToString<96>(RequestWithStats.Request.Key), *RequestWithStats.Request.Name);
RequestWithStats.EndRequest(CacheStore, EStatus::Error);
OnComplete(RequestWithStats.Request.MakeResponse(EStatus::Error));
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
};
Fix handling of cancelation in Zen DDC client to avoid having it treated as equivalent to an unexpected error. #rb devin.doucette [CL 27969813 by zousar shaker in ue5-main branch]
2023-09-18 15:22:34 -04:00
void OnCanceled(const TRequestWithStats<FCacheGetValueRequest>& RequestWithStats)
{
UE_LOG(LogDerivedDataCache, Verbose, TEXT("%s: Cache miss with canceled request for '%s' from '%s'"),
*CacheStore.GetName(), *WriteToString<96>(RequestWithStats.Request.Key), *RequestWithStats.Request.Name);
RequestWithStats.EndRequest(CacheStore, EStatus::Canceled);
OnComplete(RequestWithStats.Request.MakeResponse(EStatus::Canceled));
};
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
FZenCacheStore& CacheStore;
IRequestOwner& Owner;
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
TArray<TRequestWithStats<FCacheGetValueRequest>, TInlineAllocator<1>> Requests;
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
FOnCacheGetValueComplete OnComplete;
};
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
class FZenCacheStore::FGetChunksOp final : public FThreadSafeRefCountedObject
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
public:
FGetChunksOp(FZenCacheStore& InCacheStore,
IRequestOwner& InOwner,
const TConstArrayView<FCacheGetChunkRequest> InRequests,
FOnCacheGetChunkComplete&& InOnComplete)
: CacheStore(InCacheStore)
, Owner(InOwner)
, OnComplete(MoveTemp(InOnComplete))
{
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
StartRequests(Requests, InRequests, [](const FCacheGetChunkRequest& Request) { return Request.Key.Bucket; },
[](const FCacheGetChunkRequest& Request) { return Request.Id.IsNull() ? ERequestType::Value : ERequestType::Record; },
ERequestOp::GetChunk);
Algo::StableSortBy(Requests, &TRequestWithStats<FCacheGetChunkRequest>::Request, TChunkLess());
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
TRACE_COUNTER_ADD(ZenDDC_Get, int64(Requests.Num()));
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
TRACE_COUNTER_ADD(ZenDDC_ChunkRequestCountInFlight, int64(Requests.Num()));
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
virtual ~FGetChunksOp()
{
Only report local stats for Zen DDC. ZenCacheStore now only tracks what the client sees, regardless if the zenserver instance gets a hit from a configured upstream. As we don't currently have any upstreams configured in the zen server instance this should give a more accurate number for hit/miss stats that matches the file cache. #preflight 63eabb1bb96a6b2718326d15 #rb zousar.shaker devin.doucette [CL 24206594 by dan engelbrecht in ue5-main branch]
2023-02-14 03:09:44 -05:00
TRACE_COUNTER_SUBTRACT(ZenDDC_ChunkRequestCountInFlight, int64(Requests.Num()));
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
void IssueRequests()
{
FRequestBarrier Barrier(Owner);
ForEachBatch(CacheStore.CacheChunksBatchSize, Requests.Num(),
[this](int32 BatchFirst, int32 BatchLast)
{
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
TConstArrayView<TRequestWithStats<FCacheGetChunkRequest>> Batch(Requests.GetData() + BatchFirst, BatchLast - BatchFirst + 1);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
FCbWriter BatchRequest;
BatchRequest.BeginObject();
{
BatchRequest << ANSITEXTVIEW("Method") << "GetCacheChunks";
Add "Accept" field in Zen RPC request to enable new style RPC responses. See https://github.com/EpicGames/zen/pull/180 for Zen side change. #rb stefan.boberg zousar.shaker #preflight 6347c2568a0a7b2adcbe4d71 [CL 22505182 by dan engelbrecht in ue5-main branch]
2022-10-13 11:17:53 -04:00
BatchRequest.AddInteger(ANSITEXTVIEW("Accept"), Zen::Http::kCbPkgMagic);
Allow support for receiving local file references for cache items Companion PR in zen: https://github.com/EpicGames/zen/pull/184 #rb stefan.boberg #preflight 6368ce57d0174259cc1e7b7a [CL 23009037 by dan engelbrecht in ue5-main branch]
2022-11-07 05:33:21 -05:00
if (CacheStore.bIsLocalConnection)
{
BatchRequest.AddInteger(ANSITEXTVIEW("AcceptFlags"), static_cast<uint32_t>(Zen::Http::RpcAcceptOptions::kAllowLocalReferences));
Accept duplicated file handles in Zen RPC responses #preflight 641c11bf76461c460b8eabb4 #rb stefan.boberg per.larsson [CL 24759274 by dan engelbrecht in ue5-main branch]
2023-03-23 05:15:22 -04:00
BatchRequest.AddInteger(ANSITEXTVIEW("Pid"), FPlatformProcess::GetCurrentProcessId());
Allow support for receiving local file references for cache items Companion PR in zen: https://github.com/EpicGames/zen/pull/184 #rb stefan.boberg #preflight 6368ce57d0174259cc1e7b7a [CL 23009037 by dan engelbrecht in ue5-main branch]
2022-11-07 05:33:21 -05:00
}
Add "Accept" field in Zen RPC request to enable new style RPC responses. See https://github.com/EpicGames/zen/pull/180 for Zen side change. #rb stefan.boberg zousar.shaker #preflight 6347c2568a0a7b2adcbe4d71 [CL 22505182 by dan engelbrecht in ue5-main branch]
2022-10-13 11:17:53 -04:00
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
BatchRequest.BeginObject(ANSITEXTVIEW("Params"));
{
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
ECachePolicy DefaultPolicy = Batch[0].Request.Policy;
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
BatchRequest << ANSITEXTVIEW("DefaultPolicy") << WriteToString<128>(DefaultPolicy);
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
BatchRequest.AddString(ANSITEXTVIEW("Namespace"), CacheStore.Namespace);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
BatchRequest.BeginArray(ANSITEXTVIEW("ChunkRequests"));
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
for (const TRequestWithStats<FCacheGetChunkRequest>& RequestWithStats : Batch)
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
const FCacheGetChunkRequest& Request = RequestWithStats.Request;
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
BatchRequest.BeginObject();
{
BatchRequest << ANSITEXTVIEW("Key") << Request.Key;
if (Request.Id.IsValid())
{
BatchRequest.AddObjectId(ANSITEXTVIEW("ValueId"), Request.Id);
}
if (Request.RawOffset != 0)
{
BatchRequest << ANSITEXTVIEW("RawOffset") << Request.RawOffset;
}
if (Request.RawSize != MAX_uint64)
{
BatchRequest << ANSITEXTVIEW("RawSize") << Request.RawSize;
}
if (!Request.RawHash.IsZero())
{
BatchRequest << ANSITEXTVIEW("ChunkId") << Request.RawHash;
}
if (Request.Policy != DefaultPolicy)
{
BatchRequest << ANSITEXTVIEW("Policy") << WriteToString<128>(Request.Policy);
}
}
BatchRequest.EndObject();
}
BatchRequest.EndArray();
}
BatchRequest.EndObject();
}
BatchRequest.EndObject();
Make ZenCacheStore::FGetChunksOp asynchronous using EnqueueAsyncRpc #preflight 63341e5a10030508062960fa #rb devin.doucette zousar.shaker [CL 22230515 by dan engelbrecht in ue5-main branch]
2022-09-28 16:39:24 -04:00
FGetChunksOp* OriginalOp = this;
auto OnRpcComplete = [this, OpRef = TRefCountPtr<FGetChunksOp>(OriginalOp), Batch](THttpUniquePtr<IHttpResponse>& HttpResponse, FCbPackage& Response)
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
Make ZenCacheStore::FGetChunksOp asynchronous using EnqueueAsyncRpc #preflight 63341e5a10030508062960fa #rb devin.doucette zousar.shaker [CL 22230515 by dan engelbrecht in ue5-main branch]
2022-09-28 16:39:24 -04:00
int32 RequestIndex = 0;
Check status code for responses received from zenserver DDC requests. #rb stefan.boberg #preflight 645a3339013f3751584a39ed [CL 25385641 by dan engelbrecht in ue5-main branch]
2023-05-09 08:08:07 -04:00
if (HttpResponse->GetErrorCode() == EHttpErrorCode::None && HttpResponse->GetStatusCode() >= 200 && HttpResponse->GetStatusCode() <= 299)
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
Make ZenCacheStore::FGetChunksOp asynchronous using EnqueueAsyncRpc #preflight 63341e5a10030508062960fa #rb devin.doucette zousar.shaker [CL 22230515 by dan engelbrecht in ue5-main branch]
2022-09-28 16:39:24 -04:00
const FCbObject& ResponseObj = Response.GetObject();
for (FCbFieldView ResultView : ResponseObj[ANSITEXTVIEW("Result")])
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
Make ZenCacheStore::FGetChunksOp asynchronous using EnqueueAsyncRpc #preflight 63341e5a10030508062960fa #rb devin.doucette zousar.shaker [CL 22230515 by dan engelbrecht in ue5-main branch]
2022-09-28 16:39:24 -04:00
if (RequestIndex >= Batch.Num())
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
Make ZenCacheStore::FGetChunksOp asynchronous using EnqueueAsyncRpc #preflight 63341e5a10030508062960fa #rb devin.doucette zousar.shaker [CL 22230515 by dan engelbrecht in ue5-main branch]
2022-09-28 16:39:24 -04:00
++RequestIndex;
continue;
}
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
const TRequestWithStats<FCacheGetChunkRequest>& RequestWithStats = Batch[RequestIndex++];
RequestWithStats.Stats.AddLatency(FMonotonicTimeSpan::FromSeconds(HttpResponse->GetStats().StartTransferTime));
Make ZenCacheStore::FGetChunksOp asynchronous using EnqueueAsyncRpc #preflight 63341e5a10030508062960fa #rb devin.doucette zousar.shaker [CL 22230515 by dan engelbrecht in ue5-main branch]
2022-09-28 16:39:24 -04:00
FIoHash RawHash;
bool Succeeded = false;
uint64 RawSize = 0;
FCbObjectView ResultObject = ResultView.AsObjectView();
FSharedBuffer RequestedBytes;
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
const FCacheGetChunkRequest& Request = RequestWithStats.Request;
Make ZenCacheStore::FGetChunksOp asynchronous using EnqueueAsyncRpc #preflight 63341e5a10030508062960fa #rb devin.doucette zousar.shaker [CL 22230515 by dan engelbrecht in ue5-main branch]
2022-09-28 16:39:24 -04:00
if (CacheStore.DebugOptions.ShouldSimulateGetMiss(Request.Key))
{
UE_LOG(LogDerivedDataCache, Verbose, TEXT("%s: Simulated miss for get of '%s' from '%s'"),
*CacheStore.GetName(), *WriteToString<96>(Request.Key, '/', Request.Id), *Request.Name);
}
else
{
FCbFieldView HashView = ResultObject[ANSITEXTVIEW("RawHash")];
RawHash = HashView.AsHash();
if (!HashView.HasError())
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
Make ZenCacheStore::FGetChunksOp asynchronous using EnqueueAsyncRpc #preflight 63341e5a10030508062960fa #rb devin.doucette zousar.shaker [CL 22230515 by dan engelbrecht in ue5-main branch]
2022-09-28 16:39:24 -04:00
if (const FCbAttachment* Attachment = Response.FindAttachment(HashView.AsHash()))
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
Make ZenCacheStore::FGetChunksOp asynchronous using EnqueueAsyncRpc #preflight 63341e5a10030508062960fa #rb devin.doucette zousar.shaker [CL 22230515 by dan engelbrecht in ue5-main branch]
2022-09-28 16:39:24 -04:00
FCompressedBuffer CompressedBuffer = Attachment->AsCompressedBinary();
if (CompressedBuffer)
{
TRACE_COUNTER_ADD(ZenDDC_BytesReceived, CompressedBuffer.GetCompressedSize());
RequestedBytes = FCompressedBufferReader(CompressedBuffer).Decompress(Request.RawOffset, Request.RawSize);
RawSize = RequestedBytes.GetSize();
Succeeded = true;
}
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
Make ZenCacheStore::FGetChunksOp asynchronous using EnqueueAsyncRpc #preflight 63341e5a10030508062960fa #rb devin.doucette zousar.shaker [CL 22230515 by dan engelbrecht in ue5-main branch]
2022-09-28 16:39:24 -04:00
else
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
Make ZenCacheStore::FGetChunksOp asynchronous using EnqueueAsyncRpc #preflight 63341e5a10030508062960fa #rb devin.doucette zousar.shaker [CL 22230515 by dan engelbrecht in ue5-main branch]
2022-09-28 16:39:24 -04:00
FCbFieldView RawSizeField = ResultObject[ANSITEXTVIEW("RawSize")];
uint64 TotalSize = RawSizeField.AsUInt64();
Succeeded = !RawSizeField.HasError();
if (Succeeded)
{
RawSize = FMath::Min(Request.RawSize, TotalSize - FMath::Min(Request.RawOffset, TotalSize));
}
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
}
}
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
Succeeded ? OnHit(RequestWithStats, MoveTemp(RawHash), RawSize, MoveTemp(RequestedBytes)) : OnMiss(RequestWithStats);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
}
}
Fix handling of cancelation in Zen DDC client to avoid having it treated as equivalent to an unexpected error. #rb devin.doucette [CL 27969813 by zousar shaker in ue5-main branch]
2023-09-18 15:22:34 -04:00
else if ((HttpResponse->GetErrorCode() != EHttpErrorCode::Canceled) && (HttpResponse->GetStatusCode() != 404))
Check status code for responses received from zenserver DDC requests. #rb stefan.boberg #preflight 645a3339013f3751584a39ed [CL 25385641 by dan engelbrecht in ue5-main branch]
2023-05-09 08:08:07 -04:00
{
UE_LOG(LogDerivedDataCache, Warning,
TEXT("%s: Error response received from GetChunks RPC: from %s"),
*CacheStore.GetName(), *WriteToString<256>(*HttpResponse));
}
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
for (const TRequestWithStats<FCacheGetChunkRequest>& RequestWithStats : Batch.RightChop(RequestIndex))
Make ZenCacheStore::FGetChunksOp asynchronous using EnqueueAsyncRpc #preflight 63341e5a10030508062960fa #rb devin.doucette zousar.shaker [CL 22230515 by dan engelbrecht in ue5-main branch]
2022-09-28 16:39:24 -04:00
{
Fix handling of cancelation in Zen DDC client to avoid having it treated as equivalent to an unexpected error. #rb devin.doucette [CL 27969813 by zousar shaker in ue5-main branch]
2023-09-18 15:22:34 -04:00
if (HttpResponse->GetErrorCode() == EHttpErrorCode::Canceled)
{
OnCanceled(RequestWithStats);
}
else
{
OnMiss(RequestWithStats);
}
Make ZenCacheStore::FGetChunksOp asynchronous using EnqueueAsyncRpc #preflight 63341e5a10030508062960fa #rb devin.doucette zousar.shaker [CL 22230515 by dan engelbrecht in ue5-main branch]
2022-09-28 16:39:24 -04:00
}
};
CacheStore.EnqueueAsyncRpc(Owner, BatchRequest.Save().AsObject(), MoveTemp(OnRpcComplete));
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
});
}
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
private:
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
void OnHit(const TRequestWithStats<FCacheGetChunkRequest>& RequestWithStats, FIoHash&& RawHash, uint64 RawSize, FSharedBuffer&& RequestedBytes)
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
const FCacheGetChunkRequest& Request = RequestWithStats.Request;
Fix handling of cancelation in Zen DDC client to avoid having it treated as equivalent to an unexpected error. #rb devin.doucette [CL 27969813 by zousar shaker in ue5-main branch]
2023-09-18 15:22:34 -04:00
UE_LOG(LogDerivedDataCache, Verbose, TEXT("%s: Cache hit for '%s' from '%s'"),
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
*CacheStore.GetName(), *WriteToString<96>(Request.Key, '/', Request.Id), *Request.Name);
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
// This is a rough estimate of physical read size until Zen communicates stats with each response.
RequestWithStats.Stats.LogicalReadSize += RequestedBytes.GetSize();
RequestWithStats.Stats.PhysicalReadSize += RequestedBytes.GetSize();
RequestWithStats.EndRequest(CacheStore, EStatus::Ok);
Only report local stats for Zen DDC. ZenCacheStore now only tracks what the client sees, regardless if the zenserver instance gets a hit from a configured upstream. As we don't currently have any upstreams configured in the zen server instance this should give a more accurate number for hit/miss stats that matches the file cache. #preflight 63eabb1bb96a6b2718326d15 #rb zousar.shaker devin.doucette [CL 24206594 by dan engelbrecht in ue5-main branch]
2023-02-14 03:09:44 -05:00
TRACE_COUNTER_INCREMENT(ZenDDC_GetHit);
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
TRACE_COUNTER_ADD(ZenDDC_BytesReceived, int64(RequestWithStats.Stats.PhysicalReadSize));
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
OnComplete({Request.Name, Request.Key, Request.Id, Request.RawOffset,
RawSize, MoveTemp(RawHash), MoveTemp(RequestedBytes), Request.UserData, EStatus::Ok});
};
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
void OnMiss(const TRequestWithStats<FCacheGetChunkRequest>& RequestWithStats)
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
const FCacheGetChunkRequest& Request = RequestWithStats.Request;
Fix handling of cancelation in Zen DDC client to avoid having it treated as equivalent to an unexpected error. #rb devin.doucette [CL 27969813 by zousar shaker in ue5-main branch]
2023-09-18 15:22:34 -04:00
UE_LOG(LogDerivedDataCache, Verbose, TEXT("%s: Cache miss with missing value '%s' for '%s' from '%s'"),
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
*CacheStore.GetName(), *WriteToString<16>(Request.Id), *WriteToString<96>(Request.Key), *Request.Name);
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
RequestWithStats.EndRequest(CacheStore, EStatus::Error);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
OnComplete(Request.MakeResponse(EStatus::Error));
};
Fix handling of cancelation in Zen DDC client to avoid having it treated as equivalent to an unexpected error. #rb devin.doucette [CL 27969813 by zousar shaker in ue5-main branch]
2023-09-18 15:22:34 -04:00
void OnCanceled(const TRequestWithStats<FCacheGetChunkRequest>& RequestWithStats)
{
const FCacheGetChunkRequest& Request = RequestWithStats.Request;
UE_LOG(LogDerivedDataCache, Verbose, TEXT("%s: Cache miss with canceled request for '%s' from '%s'"),
*CacheStore.GetName(), *WriteToString<96>(Request.Key, '/', Request.Id), *Request.Name);
RequestWithStats.EndRequest(CacheStore, EStatus::Canceled);
OnComplete(Request.MakeResponse(EStatus::Canceled));
};
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
FZenCacheStore& CacheStore;
IRequestOwner& Owner;
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
TArray<TRequestWithStats<FCacheGetChunkRequest>, TInlineAllocator<1>> Requests;
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
FOnCacheGetChunkComplete OnComplete;
};
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
class FZenCacheStore::FCbPackageReceiver final : public IHttpReceiver
{
public:
FCbPackageReceiver(const FCbPackageReceiver&) = delete;
FCbPackageReceiver& operator=(const FCbPackageReceiver&) = delete;
explicit FCbPackageReceiver(FCbPackage& OutPackage, IHttpReceiver* InNext = nullptr)
: Package(OutPackage)
, Next(InNext)
{
}
private:
IHttpReceiver* OnCreate(IHttpResponse& Response) final
{
return &BodyReceiver;
}
IHttpReceiver* OnComplete(IHttpResponse& Response) final
{
Use Http::TryLoadCbPackage as first option to parse packages from Zen. This will allow UE to talk to both old and new zen servers as we move to use the new package wire format in Zen responses. #rb devin.doucette #preflight 6334bf81691c0168b71bc0a9 [CL 22233160 by dan engelbrecht in ue5-main branch]
2022-09-28 18:22:23 -04:00
FMemoryView MemoryView = MakeMemoryView(BodyArray);
{
FMemoryReaderView Ar(MemoryView);
if (Zen::Http::TryLoadCbPackage(Package, Ar))
{
return Next;
}
}
FMemoryReaderView Ar(MemoryView);
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
Package.TryLoad(Ar);
return Next;
}
private:
FCbPackage& Package;
IHttpReceiver* Next;
TArray64<uint8> BodyArray;
FHttpByteArrayReceiver BodyReceiver{BodyArray, this};
};
class FZenCacheStore::FAsyncCbPackageReceiver final : public FRequestBase, public IHttpReceiver
{
public:
FAsyncCbPackageReceiver(const FAsyncCbPackageReceiver&) = delete;
FAsyncCbPackageReceiver& operator=(const FAsyncCbPackageReceiver&) = delete;
FAsyncCbPackageReceiver(
THttpUniquePtr<IHttpRequest>&& InRequest,
IRequestOwner* InOwner,
Add capability for things that depend on the Zen service to attempt to recover the service if there is an interruption. The Zen DDC cache store is using this to allow recovery and retry of a cache request that fails to connect or times out when communicating with a Zen service on the same machine. #rb devin.doucette #preflight 639cc26343330e63e5db1750 [CL 23541143 by Zousar Shaker in ue5-main branch]
2022-12-16 15:47:14 -05:00
Zen::FZenServiceInstance& InZenServiceInstance,
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
FOnRpcComplete&& InOnRpcComplete)
: Request(MoveTemp(InRequest))
, Owner(InOwner)
Add capability for things that depend on the Zen service to attempt to recover the service if there is an interruption. The Zen DDC cache store is using this to allow recovery and retry of a cache request that fails to connect or times out when communicating with a Zen service on the same machine. #rb devin.doucette #preflight 639cc26343330e63e5db1750 [CL 23541143 by Zousar Shaker in ue5-main branch]
2022-12-16 15:47:14 -05:00
, ZenServiceInstance(InZenServiceInstance)
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
, BaseReceiver(Package, this)
, OnRpcComplete(MoveTemp(InOnRpcComplete))
{
Request->SendAsync(this, Response);
}
private:
// IRequest Interface
void SetPriority(EPriority Priority) final {}
void Cancel() final { Monitor->Cancel(); }
void Wait() final { Monitor->Wait(); }
// IHttpReceiver Interface
IHttpReceiver* OnCreate(IHttpResponse& LocalResponse) final
{
Monitor = LocalResponse.GetMonitor();
Owner->Begin(this);
return &BaseReceiver;
}
Add capability for things that depend on the Zen service to attempt to recover the service if there is an interruption. The Zen DDC cache store is using this to allow recovery and retry of a cache request that fails to connect or times out when communicating with a Zen service on the same machine. #rb devin.doucette #preflight 639cc26343330e63e5db1750 [CL 23541143 by Zousar Shaker in ue5-main branch]
2022-12-16 15:47:14 -05:00
bool ShouldRecoverAndRetry(IHttpResponse& LocalResponse)
{
if (!ZenServiceInstance.IsServiceRunningLocally())
{
return false;
}
if ((LocalResponse.GetErrorCode() == EHttpErrorCode::Connect) ||
(LocalResponse.GetErrorCode() == EHttpErrorCode::TlsConnect) ||
(LocalResponse.GetErrorCode() == EHttpErrorCode::TimedOut))
{
return true;
}
return false;
}
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
IHttpReceiver* OnComplete(IHttpResponse& LocalResponse) final
{
Add capability for things that depend on the Zen service to attempt to recover the service if there is an interruption. The Zen DDC cache store is using this to allow recovery and retry of a cache request that fails to connect or times out when communicating with a Zen service on the same machine. #rb devin.doucette #preflight 639cc26343330e63e5db1750 [CL 23541143 by Zousar Shaker in ue5-main branch]
2022-12-16 15:47:14 -05:00
Owner->End(this, [Self = this, &LocalResponse]
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
{
Add capability for things that depend on the Zen service to attempt to recover the service if there is an interruption. The Zen DDC cache store is using this to allow recovery and retry of a cache request that fails to connect or times out when communicating with a Zen service on the same machine. #rb devin.doucette #preflight 639cc26343330e63e5db1750 [CL 23541143 by Zousar Shaker in ue5-main branch]
2022-12-16 15:47:14 -05:00
if (Self->ShouldRecoverAndRetry(LocalResponse) && Self->ZenServiceInstance.TryRecovery())
{
new FAsyncCbPackageReceiver(MoveTemp(Self->Request), Self->Owner, Self->ZenServiceInstance, MoveTemp(Self->OnRpcComplete));
return;
}
Self->Request.Reset();
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
if (Self->OnRpcComplete)
{
// Launch a task for the completion function since it can execute arbitrary code.
Self->Owner->LaunchTask(TEXT("ZenHttpComplete"), [Self = TRefCountPtr(Self)]
{
Self->OnRpcComplete(Self->Response, Self->Package);
});
}
});
return nullptr;
}
private:
THttpUniquePtr<IHttpRequest> Request;
THttpUniquePtr<IHttpResponse> Response;
TRefCountPtr<IHttpResponseMonitor> Monitor;
IRequestOwner* Owner;
Add capability for things that depend on the Zen service to attempt to recover the service if there is an interruption. The Zen DDC cache store is using this to allow recovery and retry of a cache request that fails to connect or times out when communicating with a Zen service on the same machine. #rb devin.doucette #preflight 639cc26343330e63e5db1750 [CL 23541143 by Zousar Shaker in ue5-main branch]
2022-12-16 15:47:14 -05:00
Zen::FZenServiceInstance& ZenServiceInstance;
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
FCbPackage Package;
FCbPackageReceiver BaseReceiver;
FOnRpcComplete OnRpcComplete;
};
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
FZenCacheStore::FZenCacheStore(
const TCHAR* InServiceUrl,
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
const TCHAR* InNamespace,
const TCHAR* InName,
const TCHAR* InConfig,
ICacheStoreOwner* InStoreOwner)
Add support for Zen DDC sandboxes which are separate instances of zenserver with separate data paths and can be optionally flushed from run-to-run. It is used to implement Warm and Cold DDC graphs using Zen sandboxes for projects/platforms that use Zen for local DDC by default, and to provide new ZenColdDDC and ZenWarmDDC as new graphs that have Cold/Warm behavior for projects that don't use Zen for local DDC by default. #rb devin.doucette #preflight 63b493d5202bee5e27a645ed [CL 23569648 by Zousar Shaker in ue5-main branch]
2023-01-03 15:58:30 -05:00
: ZenService(InServiceUrl)
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
, StoreOwner(InStoreOwner)
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
{
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
Initialize(InNamespace, InName, InConfig);
Add support for Zen DDC sandboxes which are separate instances of zenserver with separate data paths and can be optionally flushed from run-to-run. It is used to implement Warm and Cold DDC graphs using Zen sandboxes for projects/platforms that use Zen for local DDC by default, and to provide new ZenColdDDC and ZenWarmDDC as new graphs that have Cold/Warm behavior for projects that don't use Zen for local DDC by default. #rb devin.doucette #preflight 63b493d5202bee5e27a645ed [CL 23569648 by Zousar Shaker in ue5-main branch]
2023-01-03 15:58:30 -05:00
}
FZenCacheStore::FZenCacheStore(
UE::Zen::FServiceSettings&& InSettings,
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
const TCHAR* InNamespace,
const TCHAR* InName,
const TCHAR* InConfig,
ICacheStoreOwner* InStoreOwner)
Add support for Zen DDC sandboxes which are separate instances of zenserver with separate data paths and can be optionally flushed from run-to-run. It is used to implement Warm and Cold DDC graphs using Zen sandboxes for projects/platforms that use Zen for local DDC by default, and to provide new ZenColdDDC and ZenWarmDDC as new graphs that have Cold/Warm behavior for projects that don't use Zen for local DDC by default. #rb devin.doucette #preflight 63b493d5202bee5e27a645ed [CL 23569648 by Zousar Shaker in ue5-main branch]
2023-01-03 15:58:30 -05:00
: ZenService(MoveTemp(InSettings))
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
, StoreOwner(InStoreOwner)
Add support for Zen DDC sandboxes which are separate instances of zenserver with separate data paths and can be optionally flushed from run-to-run. It is used to implement Warm and Cold DDC graphs using Zen sandboxes for projects/platforms that use Zen for local DDC by default, and to provide new ZenColdDDC and ZenWarmDDC as new graphs that have Cold/Warm behavior for projects that don't use Zen for local DDC by default. #rb devin.doucette #preflight 63b493d5202bee5e27a645ed [CL 23569648 by Zousar Shaker in ue5-main branch]
2023-01-03 15:58:30 -05:00
{
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
Initialize(InNamespace, InName, InConfig);
Add support for Zen DDC sandboxes which are separate instances of zenserver with separate data paths and can be optionally flushed from run-to-run. It is used to implement Warm and Cold DDC graphs using Zen sandboxes for projects/platforms that use Zen for local DDC by default, and to provide new ZenColdDDC and ZenWarmDDC as new graphs that have Cold/Warm behavior for projects that don't use Zen for local DDC by default. #rb devin.doucette #preflight 63b493d5202bee5e27a645ed [CL 23569648 by Zousar Shaker in ue5-main branch]
2023-01-03 15:58:30 -05:00
}
DDC: Fixed leaking cache store stats in the Zen cache #jira UE-133395 #rb none #rnx [CL 25977512 by devin doucette in ue5-main branch]
2023-06-14 10:12:47 -04:00
FZenCacheStore::~FZenCacheStore()
{
if (StoreStats)
{
StoreOwner->DestroyStats(StoreStats);
}
}
Add support for Zen DDC sandboxes which are separate instances of zenserver with separate data paths and can be optionally flushed from run-to-run. It is used to implement Warm and Cold DDC graphs using Zen sandboxes for projects/platforms that use Zen for local DDC by default, and to provide new ZenColdDDC and ZenWarmDDC as new graphs that have Cold/Warm behavior for projects that don't use Zen for local DDC by default. #rb devin.doucette #preflight 63b493d5202bee5e27a645ed [CL 23569648 by Zousar Shaker in ue5-main branch]
2023-01-03 15:58:30 -05:00
void FZenCacheStore::Initialize(
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
const TCHAR* InNamespace,
const TCHAR* InName,
const TCHAR* InConfig)
Add support for Zen DDC sandboxes which are separate instances of zenserver with separate data paths and can be optionally flushed from run-to-run. It is used to implement Warm and Cold DDC graphs using Zen sandboxes for projects/platforms that use Zen for local DDC by default, and to provide new ZenColdDDC and ZenWarmDDC as new graphs that have Cold/Warm behavior for projects that don't use Zen for local DDC by default. #rb devin.doucette #preflight 63b493d5202bee5e27a645ed [CL 23569648 by Zousar Shaker in ue5-main branch]
2023-01-03 15:58:30 -05:00
{
Namespace = InNamespace;
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
if (IsServiceReady())
{
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
RpcUri << ZenService.GetInstance().GetURL() << ANSITEXTVIEW("/z$/$rpc");
DDC: Fixed unsafe type cast warnings #preflight 6317fc04a20b67673b1de7c3 #rb Zousar.Shaker #rnx [CL 21850770 by Devin Doucette in ue5-main branch]
2022-09-07 10:37:16 -04:00
const uint32 MaxConnections = uint32(FMath::Clamp(FPlatformMisc::NumberOfCoresIncludingHyperthreads(), 8, 64));
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
constexpr uint32 RequestPoolSize = 128;
constexpr uint32 RequestPoolOverflowSize = 128;
FHttpConnectionPoolParams ConnectionPoolParams;
ConnectionPoolParams.MaxConnections = MaxConnections;
ConnectionPoolParams.MinConnections = MaxConnections;
ConnectionPool = IHttpManager::Get().CreateConnectionPool(ConnectionPoolParams);
FHttpClientParams ClientParams;
DDC: Added EHttpVersion to HttpClient #rb Zousar.Shaker #rnx [CL 25184555 by devin doucette in ue5-main branch]
2023-04-25 13:54:42 -04:00
ClientParams.Version = EHttpVersion::V2;
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
ClientParams.MaxRequests = RequestPoolSize + RequestPoolOverflowSize;
ClientParams.MinRequests = RequestPoolSize;
RequestQueue = FHttpRequestQueue(*ConnectionPool, ClientParams);
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
bIsUsable = true;
Allow support for receiving local file references for cache items Companion PR in zen: https://github.com/EpicGames/zen/pull/184 #rb stefan.boberg #preflight 6368ce57d0174259cc1e7b7a [CL 23009037 by dan engelbrecht in ue5-main branch]
2022-11-07 05:33:21 -05:00
bIsLocalConnection = ZenService.GetInstance().IsServiceRunningLocally();
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
if (StoreOwner)
{
bool bReadOnly = false;
FParse::Bool(InConfig, TEXT("ReadOnly="), bReadOnly);
// Default to locally launched service getting the Local cache store flag. Can be overridden by explicit value in config.
bool bLocal = bIsLocalConnection;
FParse::Bool(InConfig, TEXT("Local="), bLocal);
// Default to non-locally launched service getting the Remote cache store flag. Can be overridden by explicit value in config.
// In the future this could be extended to allow the Remote flag by default (even for locally launched instances) if they have upstreams configured.
bool bRemote = !bIsLocalConnection;
FParse::Bool(InConfig, TEXT("Remote="), bRemote);
ECacheStoreFlags Flags = ECacheStoreFlags::Query;
Flags |= bReadOnly ? ECacheStoreFlags::None : ECacheStoreFlags::Store;
Flags |= bLocal ? ECacheStoreFlags::Local : ECacheStoreFlags::None;
Flags |= bRemote ? ECacheStoreFlags::Remote : ECacheStoreFlags::None;
StoreOwner->Add(this, Flags);
StoreStats = StoreOwner->CreateStats(this, Flags, TEXT("Zen"), InName, ZenService.GetInstance().GetURL());
}
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
// Issue a request for stats as it will be fetched asynchronously and issuing now makes them available sooner for future callers.
ZenDashboard refresh to add project stats and clean up reporting of disk size. #rb dan.engelbrecht [CL 27546624 by zousar shaker in ue5-main branch]
2023-08-31 22:30:53 -04:00
Zen::FZenCacheStats ZenStats;
ZenService.GetInstance().GetCacheStats(ZenStats);
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
}
GConfig->GetInt(TEXT("Zen"), TEXT("BatchPutMaxBytes"), BatchPutMaxBytes, GEngineIni);
GConfig->GetInt(TEXT("Zen"), TEXT("CacheRecordBatchSize"), CacheRecordBatchSize, GEngineIni);
GConfig->GetInt(TEXT("Zen"), TEXT("CacheChunksBatchSize"), CacheChunksBatchSize, GEngineIni);
}
bool FZenCacheStore::IsServiceReady()
{
return ZenService.GetInstance().IsServiceReady();
}
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
FCompositeBuffer FZenCacheStore::SaveRpcPackage(const FCbPackage& Package)
2nd submit: Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 #preflight 629642a1be0ae0b33a46d012 [CL 20438549 by Zousar Shaker in ue5-main branch]
2022-05-31 12:42:17 -04:00
{
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
FLargeMemoryWriter Memory;
Zen::Http::SaveCbPackage(Package, Memory);
uint64 PackageMemorySize = Memory.TotalSize();
return FCompositeBuffer(FSharedBuffer::TakeOwnership(Memory.ReleaseOwnership(), PackageMemorySize, FMemory::Free));
2nd submit: Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 #preflight 629642a1be0ae0b33a46d012 [CL 20438549 by Zousar Shaker in ue5-main branch]
2022-05-31 12:42:17 -04:00
}
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
THttpUniquePtr<IHttpRequest> FZenCacheStore::CreateRpcRequest()
2nd submit: Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 #preflight 629642a1be0ae0b33a46d012 [CL 20438549 by Zousar Shaker in ue5-main branch]
2022-05-31 12:42:17 -04:00
{
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
THttpUniquePtr<IHttpRequest> Request = RequestQueue.CreateRequest({});
Request->SetUri(RpcUri);
Request->SetMethod(EHttpMethod::Post);
Request->AddAcceptType(EHttpMediaType::CbPackage);
return Request;
2nd submit: Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 #preflight 629642a1be0ae0b33a46d012 [CL 20438549 by Zousar Shaker in ue5-main branch]
2022-05-31 12:42:17 -04:00
}
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
void FZenCacheStore::EnqueueAsyncRpc(IRequestOwner& Owner, FCbObject RequestObject, FOnRpcComplete&& OnComplete)
2nd submit: Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 #preflight 629642a1be0ae0b33a46d012 [CL 20438549 by Zousar Shaker in ue5-main branch]
2022-05-31 12:42:17 -04:00
{
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
THttpUniquePtr<IHttpRequest> Request = CreateRpcRequest();
Request->SetContentType(EHttpMediaType::CbObject);
Request->SetBody(RequestObject.GetBuffer().MakeOwned());
Add capability for things that depend on the Zen service to attempt to recover the service if there is an interruption. The Zen DDC cache store is using this to allow recovery and retry of a cache request that fails to connect or times out when communicating with a Zen service on the same machine. #rb devin.doucette #preflight 639cc26343330e63e5db1750 [CL 23541143 by Zousar Shaker in ue5-main branch]
2022-12-16 15:47:14 -05:00
new FAsyncCbPackageReceiver(MoveTemp(Request), &Owner, ZenService.GetInstance(), MoveTemp(OnComplete));
DDC: Refactored the HTTP API to add configurability and remove reliance on types from DDC - FHttpSharedData is now IHttpConnectionPool. It is a connection pool of configurable size. - FHttpRequestPool is now IHttpClient. It is a request pool of configurable size and the point at which most behaviors are configured. - FHttpRequest is now split into IHttpRequest and IHttpResponse. This separation allows requests to be released back to the client for reuse while the response is still being processed. - FScopedHttpPoolRequestPtr is now THttpUniquePtr. It is used uniformly with the pool, client, request, and response. - The response body is now handled by IHttpReceiver, with a basic FHttpByteArrayReceiver provided to capture the whole body into an array. - The response now includes stats about how much time was spent on various parts of the request, as well as how much data was sent and received and at what rate. - The response supports cancellation at any point before it is complete, and IHttpResponseMonitor is provided to support thread-safe cancellation and waiting without requiring locks around the unique owner of the response. - The response will be canceled automatically when destroyed through its THttpUniquePtr. - The client does not directly support blocking on request creation. Blocking creation of requests is now provided by FHttpRequestQueue. - The HTTP thread now relies on curl_multi_poll, available in newer libcurl versions, to allow immediate waking when a handle needs to be added or removed. #preflight 62d580dd2e3e5993c3b678dd #rb Zousar.Shaker [CL 21149626 by Devin Doucette in ue5-main branch]
2022-07-18 12:23:16 -04:00
}
void FZenCacheStore::EnqueueAsyncRpc(IRequestOwner& Owner, const FCbPackage& RequestPackage, FOnRpcComplete&& OnComplete)
{
THttpUniquePtr<IHttpRequest> Request = CreateRpcRequest();
Request->SetContentType(EHttpMediaType::CbPackage);
Request->SetBody(SaveRpcPackage(RequestPackage));
Add capability for things that depend on the Zen service to attempt to recover the service if there is an interruption. The Zen DDC cache store is using this to allow recovery and retry of a cache request that fails to connect or times out when communicating with a Zen service on the same machine. #rb devin.doucette #preflight 639cc26343330e63e5db1750 [CL 23541143 by Zousar Shaker in ue5-main branch]
2022-12-16 15:47:14 -05:00
new FAsyncCbPackageReceiver(MoveTemp(Request), &Owner, ZenService.GetInstance(), MoveTemp(OnComplete));
2nd submit: Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 #preflight 629642a1be0ae0b33a46d012 [CL 20438549 by Zousar Shaker in ue5-main branch]
2022-05-31 12:42:17 -04:00
}
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
void FZenCacheStore::LegacyStats(FDerivedDataCacheStatsNode& OutNode)
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
{
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
checkNoEntry();
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
}
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
bool FZenCacheStore::LegacyDebugOptions(FBackendDebugOptions& InOptions)
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
{
DebugOptions = InOptions;
return true;
}
void FZenCacheStore::Put(
const TConstArrayView<FCachePutRequest> Requests,
IRequestOwner& Owner,
FOnCachePutComplete&& OnComplete)
{
TRACE_CPUPROFILER_EVENT_SCOPE(ZenDDC::PutCachedRecord);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
TRefCountPtr<FPutOp> PutOp = MakeAsyncOp<FPutOp>(*this, Owner, Requests, MoveTemp(OnComplete));
PutOp->IssueRequests();
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
}
void FZenCacheStore::Get(
const TConstArrayView<FCacheGetRequest> Requests,
IRequestOwner& Owner,
FOnCacheGetComplete&& OnComplete)
{
TRACE_CPUPROFILER_EVENT_SCOPE(ZenDDC::GetCacheRecord);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
TRefCountPtr<FGetOp> GetOp = MakeAsyncOp<FGetOp>(*this, Owner, Requests, MoveTemp(OnComplete));
GetOp->IssueRequests();
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
}
void FZenCacheStore::PutValue(
TConstArrayView<FCachePutValueRequest> Requests,
IRequestOwner& Owner,
FOnCachePutValueComplete&& OnComplete)
{
TRACE_CPUPROFILER_EVENT_SCOPE(ZenDDC::PutValue);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
TRefCountPtr<FPutValueOp> PutValueOp = MakeAsyncOp<FPutValueOp>(*this, Owner, Requests, MoveTemp(OnComplete));
PutValueOp->IssueRequests();
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
}
void FZenCacheStore::GetValue(
TConstArrayView<FCacheGetValueRequest> Requests,
IRequestOwner& Owner,
FOnCacheGetValueComplete&& OnComplete)
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
{
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
TRACE_CPUPROFILER_EVENT_SCOPE(ZenDDC::GetValue);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
TRefCountPtr<FGetValueOp> GetValueOp = MakeAsyncOp<FGetValueOp>(*this, Owner, Requests, MoveTemp(OnComplete));
GetValueOp->IssueRequests();
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
}
void FZenCacheStore::GetChunks(
TConstArrayView<FCacheGetChunkRequest> Requests,
IRequestOwner& Owner,
FOnCacheGetChunkComplete&& OnComplete)
{
TRACE_CPUPROFILER_EVENT_SCOPE(ZenDDC::GetChunks);
Change Zen cache store to issue non-blocking HTTP requests. GetChunks remains blocking because of the sequential nature of the way it consumes results. This can be changed in the future. Fixed a bug in the code for async POST operations to ensure it sets the post size appropriately. This change requires a fix (in progress) to EditorDomain to avoid a deadlock. #rb devin.doucette #fyi stefan.boberg #preflight 62a3619670680d6143dc5425 [CL 20598472 by Zousar Shaker in ue5-main branch]
2022-06-10 12:56:08 -04:00
TRefCountPtr<FGetChunksOp> GetChunksOp = MakeAsyncOp<FGetChunksOp>(*this, Owner, Requests, MoveTemp(OnComplete));
GetChunksOp->IssueRequests();
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
}
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
ILegacyCacheStore* CreateZenCacheStore(const TCHAR* NodeName, const TCHAR* Config, ICacheStoreOwner* Owner)
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
{
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
FString ServiceUrl;
FParse::Value(Config, TEXT("Host="), ServiceUrl);
Add support for a Zen cache store Host to be None (implying that it is disabled) and to have both commandline and environment overrides (similar to the HTTP cache store). #rb devin.doucette [CL 23815887 by zousar shaker in ue5-main branch]
2023-01-23 12:48:38 -05:00
FString OverrideName;
if (FParse::Value(Config, TEXT("EnvHostOverride="), OverrideName))
{
FString ServiceUrlEnv = FPlatformMisc::GetEnvironmentVariable(*OverrideName);
if (!ServiceUrlEnv.IsEmpty())
{
ServiceUrl = ServiceUrlEnv;
UE_LOG(LogDerivedDataCache, Log, TEXT("%s: Found environment override for Host %s=%s"), NodeName, *OverrideName, *ServiceUrl);
}
}
if (FParse::Value(Config, TEXT("CommandLineHostOverride="), OverrideName))
{
if (FParse::Value(FCommandLine::Get(), *(OverrideName + TEXT("=")), ServiceUrl))
{
UE_LOG(LogDerivedDataCache, Log, TEXT("%s: Found command line override for Host %s=%s"), NodeName, *OverrideName, *ServiceUrl);
}
}
if (ServiceUrl == TEXT("None"))
{
UE_LOG(LogDerivedDataCache, Log, TEXT("Disabling %s data cache - host set to 'None'."), NodeName);
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
return nullptr;
Add support for a Zen cache store Host to be None (implying that it is disabled) and to have both commandline and environment overrides (similar to the HTTP cache store). #rb devin.doucette [CL 23815887 by zousar shaker in ue5-main branch]
2023-01-23 12:48:38 -05:00
}
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
FString Namespace;
Update ShooterGame, QAGame, and Lyra to use Zen for LOCAL DDC only. #rb devin.doucette #preflight 63643f6d882365b8596ce430 [CL 22970016 by Zousar Shaker in ue5-main branch]
2022-11-03 18:48:54 -04:00
if (!FParse::Value(Config, TEXT("StructuredNamespace="), Namespace) && !FParse::Value(Config, TEXT("Namespace="), Namespace))
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
{
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
Namespace = FApp::GetProjectName();
Update ShooterGame, QAGame, and Lyra to use Zen for LOCAL DDC only. #rb devin.doucette #preflight 63643f6d882365b8596ce430 [CL 22970016 by Zousar Shaker in ue5-main branch]
2022-11-03 18:48:54 -04:00
UE_LOG(LogDerivedDataCache, Warning, TEXT("%s: Missing required parameter 'Namespace', falling back to '%s'"), NodeName, *Namespace);
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
}
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
ZenCacheStore - ignore "None" as an override value for a local DDC path when using sandboxes (in line with past behavior for non-default local caches like Warm/Cold). #rb devin.doucette [CL 23658565 by zousar shaker in ue5-main branch]
2023-01-11 20:57:13 -05:00
FString Sandbox;
FParse::Value(Config, TEXT("Sandbox="), Sandbox);
bool bHasSandbox = !Sandbox.IsEmpty();
bool bUseLocalDataCachePathOverrides = !bHasSandbox;
Ensure that Zen as a local service can be configured to follow the same commandline overrides as a filesystem cache store. Also ensure that if the configured override mechanisms for cache store paths yield a path of "None" that we disable the Zen DDC cache. These are in support of using Zen as a drop-in replacement for an existing filesystem DDC store. #rb devin.doucette #preflight 6398ea4e680483bcb1057b0a [CL 23517045 by Zousar Shaker in ue5-main branch]
2022-12-14 15:06:30 -05:00
FString CachePathOverride;
ZenCacheStore - ignore "None" as an override value for a local DDC path when using sandboxes (in line with past behavior for non-default local caches like Warm/Cold). #rb devin.doucette [CL 23658565 by zousar shaker in ue5-main branch]
2023-01-11 20:57:13 -05:00
if (bUseLocalDataCachePathOverrides && UE::Zen::Private::IsLocalAutoLaunched(ServiceUrl) && UE::Zen::Private::GetLocalDataCachePathOverride(CachePathOverride))
Ensure that Zen as a local service can be configured to follow the same commandline overrides as a filesystem cache store. Also ensure that if the configured override mechanisms for cache store paths yield a path of "None" that we disable the Zen DDC cache. These are in support of using Zen as a drop-in replacement for an existing filesystem DDC store. #rb devin.doucette #preflight 6398ea4e680483bcb1057b0a [CL 23517045 by Zousar Shaker in ue5-main branch]
2022-12-14 15:06:30 -05:00
{
if (CachePathOverride == TEXT("None"))
{
Add support for Zen DDC sandboxes which are separate instances of zenserver with separate data paths and can be optionally flushed from run-to-run. It is used to implement Warm and Cold DDC graphs using Zen sandboxes for projects/platforms that use Zen for local DDC by default, and to provide new ZenColdDDC and ZenWarmDDC as new graphs that have Cold/Warm behavior for projects that don't use Zen for local DDC by default. #rb devin.doucette #preflight 63b493d5202bee5e27a645ed [CL 23569648 by Zousar Shaker in ue5-main branch]
2023-01-03 15:58:30 -05:00
UE_LOG(LogDerivedDataCache, Log, TEXT("Disabling %s data cache - path set to 'None'."), NodeName);
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
return nullptr;
Ensure that Zen as a local service can be configured to follow the same commandline overrides as a filesystem cache store. Also ensure that if the configured override mechanisms for cache store paths yield a path of "None" that we disable the Zen DDC cache. These are in support of using Zen as a drop-in replacement for an existing filesystem DDC store. #rb devin.doucette #preflight 6398ea4e680483bcb1057b0a [CL 23517045 by Zousar Shaker in ue5-main branch]
2022-12-14 15:06:30 -05:00
}
}
Add support for Zen DDC sandboxes which are separate instances of zenserver with separate data paths and can be optionally flushed from run-to-run. It is used to implement Warm and Cold DDC graphs using Zen sandboxes for projects/platforms that use Zen for local DDC by default, and to provide new ZenColdDDC and ZenWarmDDC as new graphs that have Cold/Warm behavior for projects that don't use Zen for local DDC by default. #rb devin.doucette #preflight 63b493d5202bee5e27a645ed [CL 23569648 by Zousar Shaker in ue5-main branch]
2023-01-03 15:58:30 -05:00
TUniquePtr<FZenCacheStore> Backend;
bool bFlush = false;
FParse::Bool(Config, TEXT("Flush="), bFlush);
ZenCacheStore - ignore "None" as an override value for a local DDC path when using sandboxes (in line with past behavior for non-default local caches like Warm/Cold). #rb devin.doucette [CL 23658565 by zousar shaker in ue5-main branch]
2023-01-11 20:57:13 -05:00
if (bHasSandbox)
Add support for Zen DDC sandboxes which are separate instances of zenserver with separate data paths and can be optionally flushed from run-to-run. It is used to implement Warm and Cold DDC graphs using Zen sandboxes for projects/platforms that use Zen for local DDC by default, and to provide new ZenColdDDC and ZenWarmDDC as new graphs that have Cold/Warm behavior for projects that don't use Zen for local DDC by default. #rb devin.doucette #preflight 63b493d5202bee5e27a645ed [CL 23569648 by Zousar Shaker in ue5-main branch]
2023-01-03 15:58:30 -05:00
{
Zen::FServiceSettings DefaultServiceSettings;
DefaultServiceSettings.ReadFromConfig();
if (!DefaultServiceSettings.IsAutoLaunch())
{
UE_LOG(LogDerivedDataCache, Warning, TEXT("%s: Attempting to use a sandbox when there is no default autolaunch configured to interhit settings from. Cache will be disabled."), NodeName);
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
return nullptr;
Add support for Zen DDC sandboxes which are separate instances of zenserver with separate data paths and can be optionally flushed from run-to-run. It is used to implement Warm and Cold DDC graphs using Zen sandboxes for projects/platforms that use Zen for local DDC by default, and to provide new ZenColdDDC and ZenWarmDDC as new graphs that have Cold/Warm behavior for projects that don't use Zen for local DDC by default. #rb devin.doucette #preflight 63b493d5202bee5e27a645ed [CL 23569648 by Zousar Shaker in ue5-main branch]
2023-01-03 15:58:30 -05:00
}
// Make a unique local instance (not the default local instance) of ZenServer
Zen::FServiceSettings ServiceSettings;
ServiceSettings.SettingsVariant.Emplace<Zen::FServiceAutoLaunchSettings>();
Zen::FServiceAutoLaunchSettings& AutoLaunchSettings = ServiceSettings.SettingsVariant.Get<Zen::FServiceAutoLaunchSettings>();
const Zen::FServiceAutoLaunchSettings& DefaultAutoLaunchSettings = DefaultServiceSettings.SettingsVariant.Get<Zen::FServiceAutoLaunchSettings>();
AutoLaunchSettings = DefaultAutoLaunchSettings;
// Default as one more than the default port to not collide. Multiple sandboxes will share a desired port, but will get differing effective ports.
AutoLaunchSettings.DesiredPort++;
FPaths::NormalizeDirectoryName(AutoLaunchSettings.DataPath);
AutoLaunchSettings.DataPath += TEXT("_");
AutoLaunchSettings.DataPath += Sandbox;
Fix a collection of bugs related to ZenServer launching: - Fix a message that had the text and caption fields inverted. - Ensure that sandbox instances don't overwrite the .runcontext file used by non-sandbox instances. - Fix a bug where the Zen data path when following DDC path settings wasn't normalizing the path leading to needless shutdown and startup of zenserver. #rb matt.peters [CL 26205402 by zousar shaker in ue5-main branch]
2023-06-23 00:49:34 -04:00
AutoLaunchSettings.bIsDefaultSharedRunContext = false;
Add support for Zen DDC sandboxes which are separate instances of zenserver with separate data paths and can be optionally flushed from run-to-run. It is used to implement Warm and Cold DDC graphs using Zen sandboxes for projects/platforms that use Zen for local DDC by default, and to provide new ZenColdDDC and ZenWarmDDC as new graphs that have Cold/Warm behavior for projects that don't use Zen for local DDC by default. #rb devin.doucette #preflight 63b493d5202bee5e27a645ed [CL 23569648 by Zousar Shaker in ue5-main branch]
2023-01-03 15:58:30 -05:00
// The unique local instances will always limit process lifetime for now to avoid accumulating many of them
AutoLaunchSettings.bLimitProcessLifetime = true;
// Flush the cache if requested.
Fix crash when using -ddc=Cold on projects using ZenServer as Local DDC and Multi-Process Cook simultaneously. #rb devin.doucette [CL 25649292 by zousar shaker in ue5-main branch]
2023-05-26 13:52:11 -04:00
uint32 MultiprocessId = 0;
FParse::Value(FCommandLine::Get(), TEXT("-MultiprocessId="), MultiprocessId);
if (bFlush && (MultiprocessId == 0))
Add support for Zen DDC sandboxes which are separate instances of zenserver with separate data paths and can be optionally flushed from run-to-run. It is used to implement Warm and Cold DDC graphs using Zen sandboxes for projects/platforms that use Zen for local DDC by default, and to provide new ZenColdDDC and ZenWarmDDC as new graphs that have Cold/Warm behavior for projects that don't use Zen for local DDC by default. #rb devin.doucette #preflight 63b493d5202bee5e27a645ed [CL 23569648 by Zousar Shaker in ue5-main branch]
2023-01-03 15:58:30 -05:00
{
bool bStopped = true;
if (UE::Zen::IsLocalServiceRunning(*AutoLaunchSettings.DataPath))
{
bStopped = UE::Zen::StopLocalService(*AutoLaunchSettings.DataPath);
}
if (bStopped)
{
IFileManager::Get().DeleteDirectory(*(AutoLaunchSettings.DataPath / TEXT("")), /*bRequireExists*/ false, /*bTree*/ true);
}
else
{
UE_LOG(LogDerivedDataCache, Warning, TEXT("%s: Zen DDC could not be flushed due to an existing instance not shutting down when requested."), NodeName);
}
}
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
Backend = MakeUnique<FZenCacheStore>(MoveTemp(ServiceSettings), *Namespace, NodeName, Config, Owner);
Add support for Zen DDC sandboxes which are separate instances of zenserver with separate data paths and can be optionally flushed from run-to-run. It is used to implement Warm and Cold DDC graphs using Zen sandboxes for projects/platforms that use Zen for local DDC by default, and to provide new ZenColdDDC and ZenWarmDDC as new graphs that have Cold/Warm behavior for projects that don't use Zen for local DDC by default. #rb devin.doucette #preflight 63b493d5202bee5e27a645ed [CL 23569648 by Zousar Shaker in ue5-main branch]
2023-01-03 15:58:30 -05:00
}
else
{
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
Backend = MakeUnique<FZenCacheStore>(*ServiceUrl, *Namespace, NodeName, Config, Owner);
Add support for Zen DDC sandboxes which are separate instances of zenserver with separate data paths and can be optionally flushed from run-to-run. It is used to implement Warm and Cold DDC graphs using Zen sandboxes for projects/platforms that use Zen for local DDC by default, and to provide new ZenColdDDC and ZenWarmDDC as new graphs that have Cold/Warm behavior for projects that don't use Zen for local DDC by default. #rb devin.doucette #preflight 63b493d5202bee5e27a645ed [CL 23569648 by Zousar Shaker in ue5-main branch]
2023-01-03 15:58:30 -05:00
}
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
if (!Backend->IsUsable())
{
UE_LOG(LogDerivedDataCache, Warning, TEXT("%s: Failed to contact the service (%s), will not use it."), NodeName, *Backend->GetName());
Backend.Reset();
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
return nullptr;
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
}
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
return Backend.Release();
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
}
} // namespace UE::DerivedData
#else
namespace UE::DerivedData
{
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
ILegacyCacheStore* CreateZenCacheStore(const TCHAR* NodeName, const TCHAR* Config, ICacheStoreOwner* Owner)
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
{
DDC: Removed the legacy cache implementation from the clients for Horde Storage and Zen #preflight 62967fd60c151996e13038ec #rb Zousar.Shaker [CL 20445538 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:53 -04:00
UE_LOG(LogDerivedDataCache, Warning, TEXT("%s: Zen cache is not yet supported in the current build configuration."), NodeName);
DDC: Added ICacheStoreStats to the Zen cache #jira UE-133395 #rb Zousar.Shaker #rnx [CL 25964028 by devin doucette in ue5-main branch]
2023-06-13 17:14:59 -04:00
return nullptr;
Switch Zen cache store to use the HTTP request codepath in the Jupiter cache store while taking some of the improvements that had been made when that code was branched off to make the Zen HTTP codepath. This is a precursor for getting Zen cache store using non-blocking operations like Jupiter cache store - this has started to be stubbed in with the EnqueueAsyncRpc methods that are stubbed in but not used from the Zen cache store. Eventually I'd like this HTTP code to move out of the DerivedDataCache module and be usable elsewhere and supplant the HTTP code currently in the Zen module, but this will require some decisions about whether IRequestOwner should move out of the DerivedDataCache module or not. Tested Zen operation using ShooterGame editor. Tested Jupiter operation using CitySample editor (empty local cache). #rb devin.doucette #fyi stefan.boberg #preflight 62953e65401169f78c57c298 [CL 20433163 by Zousar Shaker in ue5-main branch]
2022-05-30 18:14:02 -04:00
}
} // UE::DerivedData
ZenServer CachePolicy support. Add batched Put and PutValue. Update Batched Get to send the RecordPolicy with each key. Update GetChunks to send the CachePolicy as string and to handle SkipData in the response. Add more unit tests for upstream behavior and GetChunks. Update the CacheRecordPolicy Save/Load to match the new variable names on FCacheRecordPolicy. Deploy new zenserver binaries with the required matching server-side changes. #rb Per.Larsson,Devin.Doucette #rnx #preflight 61f94e5980608c7029b8fc5c #ROBOMERGE-AUTHOR: matt.peters #ROBOMERGE-SOURCE: CL 18808743 in //UE5/Release-5.0/... via CL 18809399 via CL 18822531 #ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v908-18788545) [CL 18823289 by matt peters in ue5-main branch]
2022-02-02 05:33:16 -05:00
#endif // UE_WITH_ZEN
Reference in New Issue Copy Permalink