Commit Graph

31 Commits

Author SHA1 Message Date
devin doucette
35393bbb2b DDC: Split FPayload into separate FValue and FValueId types
A payload was conceptually a value with an ID. That has been formalized by removing the ID from the payload and having separate FValue and FValueId types. This separation cleans up the API in a few areas, and provides a more natural path to providing a basic key/value API.

#rb Zousar.Shaker
#rnx
#preflight 61d704c04c252480ca284d61

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18531844 in //UE5/Release-5.0/... via CL 18531856
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)

[CL 18531864 by devin doucette in ue5-release-engine-test branch]
2022-01-06 11:05:57 -05:00
devin doucette
f0dd93d7cf DDC: Changed FCacheRecord::Load to take the record as an object instead of a view
#rb Zousar.Shaker
#rnx
#preflight 61d489feec35d1b940d21bbf

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18510417 in //UE5/Release-5.0/... via CL 18510464
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)

[CL 18510496 by devin doucette in ue5-release-engine-test branch]
2022-01-04 13:28:36 -05:00
devin doucette
8e3ba57e91 DDC: Moved the ICacheStore Context params to be a Name member of the request types and return them in the completion params
#rb Zousar.Shaker
#rnx
#preflight 61d3729eaf44b43628f4bdc9

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18507435 in //UE5/Release-5.0/... via CL 18507440
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)

[CL 18507451 by devin doucette in ue5-release-engine-test branch]
2022-01-04 09:26:33 -05:00
devin doucette
54235555db DDC: Moved FDerivedDataBackendInterface from Public to Private
The interface was in the public folder before 5.0, but was not usable from there in a meaningful way.

#rb Zousar.Shaker
#rnx
#preflight 61bbac388a62de8385c132c7

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18479761 in //UE5/Release-5.0/... via CL 18481541
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)

[CL 18481817 by devin doucette in ue5-release-engine-test branch]
2021-12-16 19:56:56 -05:00
devin doucette
14bcbbe4c6 DDC: Updated the stat dialog layout and split Zen to include its local stats and upstream stats
#rb Zousar.Shaker
#rnx
#preflight 61bba91ac32d830955820a47

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18479360 in //UE5/Release-5.0/... via CL 18481538
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)

[CL 18481812 by devin doucette in ue5-release-engine-test branch]
2021-12-16 19:56:35 -05:00
devin doucette
4d164bdfd5 DDC: Added a uint64 UserData to ICacheStore requests to make completion callbacks more efficient
Requests to Put and Get now specify the policy independently for each record or key, but the implementations have not yet been updated to take this into account.

The existing Put and Get API has been maintained temporarily to allow the conversion to happen separately, and will be removed after callers have been converted to use the new API.

#rb Zousar.Shaker
#rnx
#preflight 61b2e550ee0de9822e2e84dc

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18443052 in //UE5/Release-5.0/... via CL 18445037
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v897-18405271)

[CL 18445201 by devin doucette in ue5-release-engine-test branch]
2021-12-13 13:32:28 -05:00
devin doucette
27c1393427 CompressedBuffer: Removed partial decompression from FCompressedBuffer now that FCompressedBufferReader is available
Requiring the use of a separate reader type makes it more likely that readers will be reused, and makes it easier to audit reader usage going forward. Reusing readers is desirable to reduce the number of large temporary allocations made during partial decompression of a buffer.

- Added FCompressedBuffer::Save(FArchive&) and renamed FromCompressed(FArchive&) to Load(FArchive&).
- Added FCompressedBufferReaderSourceScope to set a buffer source within a scope.
- Added proper bounds checks to FNoneDecoder.
- Store the header checksum on the decoder context to allow raw blocks to be reused across sources.
- Decode the header on the fly to avoid a temporary header allocation when the header is in contiguous memory.

#rb Zousar.Shaker
#rnx
#preflight 61a98d53800738dbfbc84c73

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18382211 in //UE5/Release-5.0/... via CL 18382310
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v896-18170469)

[CL 18382377 by devin doucette in ue5-release-engine-test branch]
2021-12-06 10:16:05 -05:00
mark lintott
a90eaa1e87 Optimise FZenDerivedDataBackend::IsRemote() by checking the once on backend creation and caching the result
#jira UE-135791
#rb trivial
[FYI] zousar.shaker
#preflight 619e27c3aa4521f9e7b6b567

#ROBOMERGE-AUTHOR: mark.lintott
#ROBOMERGE-SOURCE: CL 18282332 in //UE5/Release-5.0/... via CL 18282340
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)

[CL 18282346 by mark lintott in ue5-release-engine-test branch]
2021-11-24 07:17:42 -05:00
Marc Audy
0c3be2b6ad Merge Release-Engine-Staging to Test @ CL# 18240298
[CL 18241953 by Marc Audy in ue5-release-engine-test branch]
2021-11-18 14:37:34 -05:00
aurel cordonnier
a6e741e007 Merge from Release-Engine-Staging @ 17915896 to Release-Engine-Test
This represents UE4/Main @17911760, Release-5.0 @17915875 and Dev-PerfTest @17914035

[CL 17918595 by aurel cordonnier in ue5-release-engine-test branch]
2021-10-25 20:05:28 -04:00
aurel cordonnier
a12d56ff31 Merge from Release-Engine-Staging @ 17791557 to Release-Engine-Test
This represents UE4/Main @17774255, Release-5.0 @17791557 and Dev-PerfTest @17789485

[CL 17794212 by aurel cordonnier in ue5-release-engine-test branch]
2021-10-12 21:21:22 -04:00
Devin Doucette
a077feffbd DDC: Reworked ICacheStore to allow partial records, filtering of payloads, and loading parts of payloads
- ICacheStore::Put() has updated documentation to reflect the requirements of partial records.
- ICacheStore::Get() now takes a FCacheRecordPolicy, which is implicitly constructible from ECachePolicy, and allows setting the policy by payload.
- ICacheStore::GetPayload() is replaced by ICacheStore::GetChunks(), which allows loading parts of payloads.
- ICacheStore::CancelAll() is moved to ICache::CancelAll() because the cache can track requests at the top level and cancel them without exposing cancellation on individual cache stores.
- ECachePolicy::SkipLocalCopy has been removed because it is difficult to reason about.
- ECachePolicy::SkipData flags now have a documented meaning for put requests, to hint that record existence implies payload existence.
- The filesystem and memory cache stores have been updated to support partial records, filtering of payloads, and loading parts of payloads.
- Requesting part of a payload will decompress the entire payload for now, until compressed buffers expose a way to decompress only part.
- Fixed a bug in FTexturePlatformData::AreDerivedMipsAvailable() that caused it to return false for structured cache keys.

#rb Zousar.Shaker
#rnx
#preflight 615e03241ed62f0001b95454

#ROBOMERGE-OWNER: Devin.Doucette
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 17748550 in //UE5/Release-5.0/... via CL 17748555
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v879-17706426)
#ROBOMERGE-CONFLICT from-shelf
#ROBOMERGE[STARSHIP]: UE5-Main

[CL 17748602 by Devin Doucette in ue5-release-engine-test branch]
2021-10-07 09:11:32 -04:00
per larsson
fa26f9c012 Make content-type/accept headers mandatory
#rb Stefan.Boberg
#rnx

#ROBOMERGE-AUTHOR: per.larsson
#ROBOMERGE-SOURCE: CL 17645289 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v875-17642767)

[CL 17645303 by per larsson in ue5-release-engine-test branch]
2021-09-28 09:53:06 -04:00
zousar shaker
4da5e948f4 Fix missing ZenDerivedDataBackend stats.
#rb trivial
[FYI] devin.doucette
#preflight 614c0c0203e28a00011fc1d3

#ROBOMERGE-AUTHOR: zousar.shaker
#ROBOMERGE-SOURCE: CL 17604816 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v871-17566257)

[CL 17604821 by zousar shaker in ue5-release-engine-test branch]
2021-09-23 02:00:09 -04:00
zousar shaker
96d0784963 Allow ZenServer to be autolaunched by editor use cases that require its presence without insisting that it always be autolaunched.
#rb matt.peters
#preflight 614b4150ee16e20001b146e3

#ROBOMERGE-AUTHOR: zousar.shaker
#ROBOMERGE-SOURCE: CL 17597162 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v871-17566257)

[CL 17597204 by zousar shaker in ue5-release-engine-test branch]
2021-09-22 13:00:17 -04:00
per larsson
ba0ff4ff28 ZenDDC: Remove the use of Save/LoadCbPackage
#rb Stefan.Boberg
#rnx

#ROBOMERGE-AUTHOR: per.larsson
#ROBOMERGE-SOURCE: CL 17569390 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v871-17566257)

[CL 17569391 by per larsson in ue5-release-engine-test branch]
2021-09-20 07:13:37 -04:00
zousar shaker
8c80e5a866 Change Compact Binary Package API to represent binary attachments as compressed buffers identified by their raw hash.
Change Compact Binary Package serialization for binary attachments, compressed binary attachments and objects.

#rb devin.doucette

#ROBOMERGE-AUTHOR: zousar.shaker
#ROBOMERGE-SOURCE: CL 17450533 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v870-17433530)

[CL 17450587 by zousar shaker in ue5-release-engine-test branch]
2021-09-07 18:07:00 -04:00
devin doucette
c39e30cfd7 Added missing includes of DerivedDataPayload.h found by the non-unity build
#rb none
#rnx

#ROBOMERGE-SOURCE: CL 17426590 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v865-17346139)

[CL 17426595 by devin doucette in ue5-release-engine-test branch]
2021-09-03 17:11:57 -04:00
matt peters
30ac4d4b22 Fix error in ZenDerivedDataBackend::Put. It needs to make payloads with missing data references to existing data in the CacheStore, and reject the put if any don't exist. This error was causing population of editordomain packages in a Zen store from a shareddatacache to fail.
Change GetZenData to only take TArray64, and to use PerformBlockingHead instead of PerformBlockingDownload if outputdata is null.
#rb Devin.Doucette
#rnx
[FYI] Stefan.Boberg

#ROBOMERGE-SOURCE: CL 17388568 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v865-17346139)

[CL 17388571 by matt peters in ue5-release-engine-test branch]
2021-09-01 12:26:46 -04:00
devin doucette
a066ec0c49 DDC: Implemented existing cache debug options for the new cache stores
#rb Zousar.Shaker
#rnx

#ROBOMERGE-SOURCE: CL 17308400 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v861-17282326)

[CL 17308416 by devin doucette in ue5-release-engine-test branch]
2021-08-25 16:08:33 -04:00
stefan boberg
e549b0b12d Updated Zen HTTP interfaces to accept large payloads as a preparation for unifying the cache/store client implementations
#ROBOMERGE-SOURCE: CL 17218879 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v855-17104924)

[CL 17218897 by stefan boberg in ue5-release-engine-test branch]
2021-08-18 14:12:33 -04:00
stefan boberg
2d5b5e3409 DLL-exported functions for Zen HTTP interfaces to allow them to be consumed from other modules, and renamed them to make it easier to search for them (previous names were derived from the Jupiter HTTP back-end)
#rb trivial

#ROBOMERGE-SOURCE: CL 17187385 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v855-17104924)

[CL 17187388 by stefan boberg in ue5-release-engine-test branch]
2021-08-17 03:32:28 -04:00
devin doucette
bd60f3ef1b DDC: Removed ICacheFactory in favor of exported functions
FCacheBucket is now always exposed as FAnsiStringView.

#rb Zousar.Shaker
#rnx

#ROBOMERGE-SOURCE: CL 17084521 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)

[CL 17084549 by devin doucette in ue5-release-engine-test branch]
2021-08-06 12:53:08 -04:00
devin doucette
e3b266c064 DDC: Fixed non-unity compile errors
#rb trivial
#rnx

#ROBOMERGE-SOURCE: CL 17079642 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)

[CL 17079646 by devin doucette in ue5-release-engine-test branch]
2021-08-05 22:30:50 -04:00
devin doucette
733d9c1a97 DDC: Added conversions between FPayloadId and FCbObjectId
#rb Zousar.Shaker
#rnx

#ROBOMERGE-SOURCE: CL 17071877 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)

[CL 17071890 by devin doucette in ue5-release-engine-test branch]
2021-08-05 13:37:13 -04:00