Commit Graph

75 Commits

Author SHA1 Message Date
Devin Doucette
28ef6b6386 DDC: Iteration on ICache
- Identify payloads by the hash of the raw data.
- Added FCacheKey::Empty and FCachePayloadKey::Empty for convenience.
- Changed FCacheRecord and FCacheRecordBuilder constructors to be private.
- Changed storage in the legacy cache to store values inline for records with no attachments.
- Changed storage in the legacy cache to use CAS within each bucket.
- Improved wording and details in assert messages.

#rb Zousar.Shaker
#rnx

[CL 15984411 by Devin Doucette in ue5-main branch]
2021-04-12 18:25:18 -04:00
Zousar Shaker
699ad048c3 Ensure that at the time the FDerivedDataCache singleton is shutdown that we have waited on all internal backends to complete in-flight work.
#jira UE-110943
#rb devin.doucette

[CL 15930084 by Zousar Shaker in ue5-main branch]
2021-04-06 10:20:39 -04:00
Devin Doucette
ab4a43d5db CompressedBuffer: Implemented a stable format for compressed buffers with support for no compression and LZ4
#rb Tim.Smith

[CL 15895927 by Devin Doucette in ue5-main branch]
2021-04-01 16:31:55 -04:00
Marc Audy
0cbbc781ca Merge UE5/Release-Engine-Staging @ 15740152 to UE5/Main
This represents UE4/Main @ 15709114

[CL 15740605 by Marc Audy in ue5-main branch]
2021-03-18 15:20:03 -04:00
Devin Doucette
490d684d40 DDC: Clean up a few details in the new Cache API
#rb Zousar.Shaker
#rnx

[CL 15681766 by Devin Doucette in ue5-main branch]
2021-03-11 18:12:08 -04:00
Devin Doucette
7d5d9f3798 DDC: Clean up a few details in the new Cache API and remove the last exported type from DDC
#rb Zousar.Shaker
#rnx

[CL 15670275 by Devin Doucette in ue5-main branch]
2021-03-10 17:33:38 -04:00
Devin Doucette
017b41eb57 DDC: Refactored the new Cache API to support its use through only an interface dependency
#rb Zousar.Shaker
#rnx

[CL 15666872 by Devin Doucette in ue5-main branch]
2021-03-10 12:23:29 -04:00
Devin Doucette
2440bc5fbd DDC: Add generic EStatus and replace ECacheStatus with it
#rb Zousar.Shaker
#rnx

[CL 15583574 by Devin Doucette in ue5-main branch]
2021-03-03 11:55:38 -04:00
devin doucette
0f5201b8a1 DDC: Added CreateOrGetDDC to the module and changed creation of DDC to occur there instead of on module startup
#rb Arciel.Rekman, Zousar.Shaker

#ROBOMERGE-OWNER: devin.doucette
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 15530385 via CL 15530408 via CL 15530432 via CL 15530444 via CL 15531592
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Main) (v771-15082668)
#ROBOMERGE[STARSHIP]: Release-5.0-EarlyAccess

[CL 15531598 by devin doucette in ue5-main branch]
2021-02-25 14:50:04 -04:00
Devin Doucette
7e2e44ebf4 DDC: Refactored the Cache API to prepare for the Build API
- Removed the "Cache" prefix from ECachePriority, FCacheRequest, ICacheRequest, FCachePayload, FCachePayloadId to allow reuse in the Build API.
- Added FCompressedBuffer as the starting point for the API for creating and accessing compressed buffers.

#rb Zousar.Shaker
#rnx

[CL 15511723 by Devin Doucette in ue5-main branch]
2021-02-24 01:37:45 -04:00
Devin Doucette
7c5e0f99fa DDC: Capture cache value arrays into shared buffers instead of copying them
#rb Zousar.Shaker
#rnx

[CL 15501496 by Devin Doucette in ue5-main branch]
2021-02-23 12:32:38 -04:00
Devin Doucette
b5ae234c7c CompactBinary: Renamed view types to use a View suffix, and remove the Ref suffix from types that hold a SharedBuffer
#rb Zousar.Shaker
#rnx

[CL 15493834 by Devin Doucette in ue5-main branch]
2021-02-22 16:34:05 -04:00
Devin Doucette
c13b0dbecf DDC: Updated the new cache interface to expose compression of cache records
- FCacheRecord must now be created by FCacheRecordBuilder.
- FCacheRecord values are always an opaque buffer now, and ECacheRecordType has been removed.
- FCachePayload has been added as a common type for the cache record value and attachments.
- FCachePayloadId has been added as a 12-byte identifier for cache payloads.
- ICache::GetAttachments is now GetPayloads and can fetch both values and attachments.
- ECachePriority::Blocking has been added for requests that the caller will block on.

#rb Zousar.Shaker
#rnx

[CL 15471951 by Devin Doucette in ue5-main branch]
2021-02-19 11:26:34 -04:00
Devin Doucette
faf23d828f DDC: Fixed TryToPrefetch of one key, which was skipped due to an incorrect copy from CachedDataProbablyExistsBatch
#rb trivial
#rnx

[CL 15452343 by Devin Doucette in ue5-main branch]
2021-02-18 10:13:14 -04:00
will damon
7fc8e3eee0 Move static object out of global namespace.
#rb arne.shober, devin.doucette
#jira UE-107840
#rnx

#ROBOMERGE-SOURCE: CL 15426173 in //UE5/Release-5.0-EarlyAccess/...
#ROBOMERGE-BOT: STARSHIP (Release-5.0-EarlyAccess -> Main) (v771-15082668)

[CL 15426403 by will damon in ue5-main branch]
2021-02-16 20:04:45 -04:00
Devin Doucette
6394ccc1cd DDC: Added TryToPrefetch to check multiple keys for existence and prefetch the data into a fast cache backend
#rb Zousar.Shaker

[CL 15422719 by Devin Doucette in ue5-main branch]
2021-02-16 17:24:48 -04:00
Devin Doucette
a05f939c6b CompactBinary: Renamed and added overloads to FCbWriter to clarify behavior and allow specifying name and value in the same function
#rb Zousar.Shaker
#rnx

[CL 15303766 by Devin Doucette in ue5-main branch]
2021-02-03 18:37:36 -04:00
Devin Doucette
d273616103 CompactBinary: Renamed Reference to Attachment to clarify the expected relationship between reference fields and package attachments
#rb trivial
#rnx

[CL 15299118 by Devin Doucette in ue5-main branch]
2021-02-03 14:33:20 -04:00
Devin Doucette
90298de5c2 DDC: Added the value hash to FCacheRecord and made it easier to access object values
#rb Matt.Peters
#rnx

[CL 15237973 by Devin Doucette in ue5-main branch]
2021-01-28 09:50:58 -04:00
Devin Doucette
e9b798ca9c CompactBinary: Converted from FBlake3Hash to FIoHash
#rb Zousar.Shaker
#rnx

[CL 15212418 by Devin Doucette in ue5-main branch]
2021-01-26 18:17:02 -04:00
danny couture
0790ec6eb0 Create a separate thread pool for the I/O portion of DDC requests to avoid clogging the workers with long I/O waits
Add queued work priority support to auto delete async tasks and schedule DDC Puts as Low priority
ThreadPool threads are now honoring the names they've been given on creation
Move the I/O only async DDC Get into the DDC ThreadPool with higher priority (Normal) than DCC Puts

#rb Devin.Doucette

[CL 15177897 by danny couture in ue5-main branch]
2021-01-25 10:55:26 -04:00
Devin Doucette
3fdb9bc4e5 CompactBinary: Rename Reference to CompactBinaryReference for clarity and consistency
#rb Zousar.Shaker
#rnx

[CL 15169906 by Devin Doucette in ue5-main branch]
2021-01-23 02:45:14 -04:00
mark lintott
6c3abefe2f #rb Devin.Doucette
Added stats for Cloud based DCC ( ie Jupiter ) to DDC.Summary
Added BackEnd name to DDC.Summary
Added HasLocalDDC, HasSharedDDC, HasCloudDDC to DDC.Summary

[CL 15164347 by mark lintott in ue5-main branch]
2021-01-22 12:20:09 -04:00
Marc Audy
bc88b73a29 Merge Release-Engine-Staging to Main @ CL# 15151250
Represents UE4/Main @ 15133763

[CL 15158774 by Marc Audy in ue5-main branch]
2021-01-21 16:22:06 -04:00
Devin Doucette
0c7d307086 DDC: Removed ECachePolicy::Overwrite because there is not a valid use case for it
#rb Zousar.Shaker
#rnx

[CL 15154520 by Devin Doucette in ue5-main branch]
2021-01-21 11:37:24 -04:00