Commit Graph

25 Commits

Author SHA1 Message Date
devin doucette
27c1393427 CompressedBuffer: Removed partial decompression from FCompressedBuffer now that FCompressedBufferReader is available
Requiring the use of a separate reader type makes it more likely that readers will be reused, and makes it easier to audit reader usage going forward. Reusing readers is desirable to reduce the number of large temporary allocations made during partial decompression of a buffer.

- Added FCompressedBuffer::Save(FArchive&) and renamed FromCompressed(FArchive&) to Load(FArchive&).
- Added FCompressedBufferReaderSourceScope to set a buffer source within a scope.
- Added proper bounds checks to FNoneDecoder.
- Store the header checksum on the decoder context to allow raw blocks to be reused across sources.
- Decode the header on the fly to avoid a temporary header allocation when the header is in contiguous memory.

#rb Zousar.Shaker
#rnx
#preflight 61a98d53800738dbfbc84c73

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18382211 in //UE5/Release-5.0/... via CL 18382310
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v896-18170469)

[CL 18382377 by devin doucette in ue5-release-engine-test branch]
2021-12-06 10:16:05 -05:00
mark lintott
a90eaa1e87 Optimise FZenDerivedDataBackend::IsRemote() by checking the once on backend creation and caching the result
#jira UE-135791
#rb trivial
[FYI] zousar.shaker
#preflight 619e27c3aa4521f9e7b6b567

#ROBOMERGE-AUTHOR: mark.lintott
#ROBOMERGE-SOURCE: CL 18282332 in //UE5/Release-5.0/... via CL 18282340
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)

[CL 18282346 by mark lintott in ue5-release-engine-test branch]
2021-11-24 07:17:42 -05:00
Marc Audy
0c3be2b6ad Merge Release-Engine-Staging to Test @ CL# 18240298
[CL 18241953 by Marc Audy in ue5-release-engine-test branch]
2021-11-18 14:37:34 -05:00
aurel cordonnier
a6e741e007 Merge from Release-Engine-Staging @ 17915896 to Release-Engine-Test
This represents UE4/Main @17911760, Release-5.0 @17915875 and Dev-PerfTest @17914035

[CL 17918595 by aurel cordonnier in ue5-release-engine-test branch]
2021-10-25 20:05:28 -04:00
aurel cordonnier
a12d56ff31 Merge from Release-Engine-Staging @ 17791557 to Release-Engine-Test
This represents UE4/Main @17774255, Release-5.0 @17791557 and Dev-PerfTest @17789485

[CL 17794212 by aurel cordonnier in ue5-release-engine-test branch]
2021-10-12 21:21:22 -04:00
Devin Doucette
a077feffbd DDC: Reworked ICacheStore to allow partial records, filtering of payloads, and loading parts of payloads
- ICacheStore::Put() has updated documentation to reflect the requirements of partial records.
- ICacheStore::Get() now takes a FCacheRecordPolicy, which is implicitly constructible from ECachePolicy, and allows setting the policy by payload.
- ICacheStore::GetPayload() is replaced by ICacheStore::GetChunks(), which allows loading parts of payloads.
- ICacheStore::CancelAll() is moved to ICache::CancelAll() because the cache can track requests at the top level and cancel them without exposing cancellation on individual cache stores.
- ECachePolicy::SkipLocalCopy has been removed because it is difficult to reason about.
- ECachePolicy::SkipData flags now have a documented meaning for put requests, to hint that record existence implies payload existence.
- The filesystem and memory cache stores have been updated to support partial records, filtering of payloads, and loading parts of payloads.
- Requesting part of a payload will decompress the entire payload for now, until compressed buffers expose a way to decompress only part.
- Fixed a bug in FTexturePlatformData::AreDerivedMipsAvailable() that caused it to return false for structured cache keys.

#rb Zousar.Shaker
#rnx
#preflight 615e03241ed62f0001b95454

#ROBOMERGE-OWNER: Devin.Doucette
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 17748550 in //UE5/Release-5.0/... via CL 17748555
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v879-17706426)
#ROBOMERGE-CONFLICT from-shelf
#ROBOMERGE[STARSHIP]: UE5-Main

[CL 17748602 by Devin Doucette in ue5-release-engine-test branch]
2021-10-07 09:11:32 -04:00
per larsson
fa26f9c012 Make content-type/accept headers mandatory
#rb Stefan.Boberg
#rnx

#ROBOMERGE-AUTHOR: per.larsson
#ROBOMERGE-SOURCE: CL 17645289 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v875-17642767)

[CL 17645303 by per larsson in ue5-release-engine-test branch]
2021-09-28 09:53:06 -04:00
zousar shaker
4da5e948f4 Fix missing ZenDerivedDataBackend stats.
#rb trivial
[FYI] devin.doucette
#preflight 614c0c0203e28a00011fc1d3

#ROBOMERGE-AUTHOR: zousar.shaker
#ROBOMERGE-SOURCE: CL 17604816 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v871-17566257)

[CL 17604821 by zousar shaker in ue5-release-engine-test branch]
2021-09-23 02:00:09 -04:00
zousar shaker
96d0784963 Allow ZenServer to be autolaunched by editor use cases that require its presence without insisting that it always be autolaunched.
#rb matt.peters
#preflight 614b4150ee16e20001b146e3

#ROBOMERGE-AUTHOR: zousar.shaker
#ROBOMERGE-SOURCE: CL 17597162 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v871-17566257)

[CL 17597204 by zousar shaker in ue5-release-engine-test branch]
2021-09-22 13:00:17 -04:00
per larsson
ba0ff4ff28 ZenDDC: Remove the use of Save/LoadCbPackage
#rb Stefan.Boberg
#rnx

#ROBOMERGE-AUTHOR: per.larsson
#ROBOMERGE-SOURCE: CL 17569390 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v871-17566257)

[CL 17569391 by per larsson in ue5-release-engine-test branch]
2021-09-20 07:13:37 -04:00
zousar shaker
8c80e5a866 Change Compact Binary Package API to represent binary attachments as compressed buffers identified by their raw hash.
Change Compact Binary Package serialization for binary attachments, compressed binary attachments and objects.

#rb devin.doucette

#ROBOMERGE-AUTHOR: zousar.shaker
#ROBOMERGE-SOURCE: CL 17450533 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v870-17433530)

[CL 17450587 by zousar shaker in ue5-release-engine-test branch]
2021-09-07 18:07:00 -04:00
devin doucette
c39e30cfd7 Added missing includes of DerivedDataPayload.h found by the non-unity build
#rb none
#rnx

#ROBOMERGE-SOURCE: CL 17426590 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v865-17346139)

[CL 17426595 by devin doucette in ue5-release-engine-test branch]
2021-09-03 17:11:57 -04:00
matt peters
30ac4d4b22 Fix error in ZenDerivedDataBackend::Put. It needs to make payloads with missing data references to existing data in the CacheStore, and reject the put if any don't exist. This error was causing population of editordomain packages in a Zen store from a shareddatacache to fail.
Change GetZenData to only take TArray64, and to use PerformBlockingHead instead of PerformBlockingDownload if outputdata is null.
#rb Devin.Doucette
#rnx
[FYI] Stefan.Boberg

#ROBOMERGE-SOURCE: CL 17388568 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v865-17346139)

[CL 17388571 by matt peters in ue5-release-engine-test branch]
2021-09-01 12:26:46 -04:00
devin doucette
a066ec0c49 DDC: Implemented existing cache debug options for the new cache stores
#rb Zousar.Shaker
#rnx

#ROBOMERGE-SOURCE: CL 17308400 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v861-17282326)

[CL 17308416 by devin doucette in ue5-release-engine-test branch]
2021-08-25 16:08:33 -04:00
stefan boberg
e549b0b12d Updated Zen HTTP interfaces to accept large payloads as a preparation for unifying the cache/store client implementations
#ROBOMERGE-SOURCE: CL 17218879 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v855-17104924)

[CL 17218897 by stefan boberg in ue5-release-engine-test branch]
2021-08-18 14:12:33 -04:00
stefan boberg
2d5b5e3409 DLL-exported functions for Zen HTTP interfaces to allow them to be consumed from other modules, and renamed them to make it easier to search for them (previous names were derived from the Jupiter HTTP back-end)
#rb trivial

#ROBOMERGE-SOURCE: CL 17187385 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v855-17104924)

[CL 17187388 by stefan boberg in ue5-release-engine-test branch]
2021-08-17 03:32:28 -04:00
devin doucette
bd60f3ef1b DDC: Removed ICacheFactory in favor of exported functions
FCacheBucket is now always exposed as FAnsiStringView.

#rb Zousar.Shaker
#rnx

#ROBOMERGE-SOURCE: CL 17084521 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)

[CL 17084549 by devin doucette in ue5-release-engine-test branch]
2021-08-06 12:53:08 -04:00
devin doucette
e3b266c064 DDC: Fixed non-unity compile errors
#rb trivial
#rnx

#ROBOMERGE-SOURCE: CL 17079642 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)

[CL 17079646 by devin doucette in ue5-release-engine-test branch]
2021-08-05 22:30:50 -04:00
devin doucette
733d9c1a97 DDC: Added conversions between FPayloadId and FCbObjectId
#rb Zousar.Shaker
#rnx

#ROBOMERGE-SOURCE: CL 17071877 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)

[CL 17071890 by devin doucette in ue5-release-engine-test branch]
2021-08-05 13:37:13 -04:00
stefan boberg
fdffa5c30d Zen HTTP interface refactored into separate header/source files
#rb none

#ROBOMERGE-SOURCE: CL 17066037 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v852-17065384)

[CL 17066041 by stefan boberg in ue5-release-engine-test branch]
2021-08-05 05:31:06 -04:00
devin doucette
bfda02d384 DDC: Replaced FRequest with FRequestGroup to handle nested and chained requests more robustly
Any function that may create a request now has a IRequestOwner& parameter, and uses the Begin and End functions on the owner to manage the lifetime of any requests that it creates, as well as using End to invoke the completion callback for any request which has one.

The new FRequestBarrier may be used to block a group from being considered complete in a scope where more requests may be added to it.

#rb Matt.Peters
#rnx
#preflight 6109b5c403d303000144cce5
#preflight 610acf7103d30300016fda94

#ROBOMERGE-SOURCE: CL 17060470 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v850-17047176)

[CL 17060649 by devin doucette in ue5-release-engine-test branch]
2021-08-04 18:08:50 -04:00
stefan boberg
01f3343c82 Refactored ZenDDC communication. HTTP interface code is now simplified and separated out from the ZenDDC client code as a step towards a shared communication layer for the oplog/cache/etc as well as instance lifetime management. There should be no functional differences.
#rb devin.doucette
#jira none

#ROBOMERGE-SOURCE: CL 17056895 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v850-17047176)

[CL 17056908 by stefan boberg in ue5-release-engine-test branch]
2021-08-04 14:56:02 -04:00
stefan boberg
de2a508d79 General HTTP zen client implementation simplification
* Rename FRequest -> FZenHttpRequest to make it easier to move the implementation elsewhere
* Simplified implementation of PUT/HEAD/DELETE operations by turning template functions into plain member functions to make it easier to hide the implementation details when the class is moved into a header

#rb none
#jira none

#ROBOMERGE-SOURCE: CL 17048397 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v850-17047176)

[CL 17049901 by stefan boberg in ue5-release-engine-test branch]
2021-08-04 09:33:43 -04:00
sebastien lussier
8c31cfedfd DDC backends - removed speed class tests in implementations of TryToPrefetch()
#rb danny.couture, devin.doucette

#ROBOMERGE-SOURCE: CL 16900182 via CL 16917763
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v836-16769935)

[CL 16917911 by sebastien lussier in ue5-release-engine-test branch]
2021-07-21 19:12:55 -04:00
zousar shaker
c27625b056 Integrating //UE5/Dev-Cooker [at] 16678003 to //UE5/Main (Zousar.Shaker-YEG-0943-Quaternary)
Non-DevIteration_ShooterGame


#ROBOMERGE-SOURCE: CL 16678907 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v835-16672529)

[CL 16678928 by zousar shaker in ue5-release-engine-test branch]
2021-06-15 16:38:03 -04:00