Commit Graph

86 Commits

Author SHA1 Message Date
aurel cordonnier
02c0f425e8 Copy up from Release-Engine-Staging @ 16738359
This represents UE4/Main @ 16738161 and Dev-PerfTest @ 16737719

[CL 16738582 by aurel cordonnier in ue5-release-engine-test branch]
2021-06-22 00:27:54 -04:00
Devin Doucette
600ee5377a DDC: Added serialization of FCacheRecord to/from FCbPackage
#rb Devin.Doucette
#rnx

#ushell-cherrypick of 16476127 by Matt.Peters

[CL 16483925 by Devin Doucette in ue5-main branch]
2021-05-27 10:16:47 -04:00
Devin Doucette
e6d6d110aa DDC: Moved the module and IBuild implementation to their own files
#rb Zousar.Shaker
#rnx

[CL 16420332 by Devin Doucette in ue5-main branch]
2021-05-21 12:24:43 -04:00
Devin Doucette
e3600df4ee DDC: Added UE::DerivedData::FBuildInput
FBuildInput and the associated builder type collect the inputs for a build.

#rb Zousar.Shaker
#rnx

[CL 16406497 by Devin Doucette in ue5-main branch]
2021-05-20 13:53:48 -04:00
Devin Doucette
8ce8977949 DDC: Added optional versions of BuildAction, BuildDefinition, BuildOutput, CacheRecord
- Optional versions are now returned from operations that may fail.
- Fixed a bug in the cache where a null record could be dereferenced.
- Improved validation when loading build types.

#rb Zousar.Shaker
#rnx

[CL 16332909 by Devin Doucette in ue5-main branch]
2021-05-14 14:47:11 -04:00
Devin Doucette
31e8388cb0 DDC: Added UE::DerivedData::FBuildAction
FBuildAction and the associated builder type define the exact action to be executed, including the function and its version, and the inputs referenced by hash.

#rb Zousar.Shaker
#rnx
#preflight 609cbaed7ebade00018f29c9

[CL 16317116 by Devin Doucette in ue5-main branch]
2021-05-13 15:36:50 -04:00
Devin Doucette
cf09b53604 DDC: Added UE::DerivedData::FBuildOutput
FBuildOutput and the associated builder type define the output of a derived data build as a collection of payloads and diagnostic messages.

#rb Zousar.Shaker
#rnx
#preflight 609c0fc05b35c60001cbc2e3

[CL 16304655 by Devin Doucette in ue5-main branch]
2021-05-12 15:46:19 -04:00
Devin Doucette
3f40f504cf DDC: Changed Put to take a const array of cache records now that records are copyable
#rb Zousar.Shaker
#rnx

[CL 16184736 by Devin Doucette in ue5-main branch]
2021-05-03 12:05:18 -04:00
Devin Doucette
0baafa125a DDC: Added the start of the new build interface UE::DerivedData::IBuild
This only includes the functionality required to create a build definition, and does not include the functionalty required to execute a build from the build definition.

#rb Zousar.Shaker
#rnx
#preflight 608cf5ef7343ea00016e58f4

[CL 16183533 by Devin Doucette in ue5-main branch]
2021-05-03 09:25:48 -04:00
Devin Doucette
e9c43a79ae DDC: Implemented ICache on FDerivedDataCacheInterface
Moved IDerivedDataCacheModule to its own header and renamed CreateOrGetDDC to CreateOrGetCache.

#rb Zousar.Shaker
#rnx
#preflight 6089d5a036afba00015c0348

[CL 16153121 by Devin Doucette in ue5-main branch]
2021-04-28 23:25:28 -04:00
Devin Doucette
e42bbbedab DDC: Implement ICacheStore on FDerivedDataBackendInterface
This allows the new cache to be implemented more efficiently using the old cache backends because functionality like the corruption wrapper and key length limiter can be bypassed and the individual backends can store cache records in whatever way is most efficient for them.

The hierarchical backend may request payloads when they are not required due to incomplete tracking of backend state, and GetPayload will never fill other backends due to the inefficiency of the existing backend framework.

The filesystem backend does not cache any state in memory, which makes requests for individual payloads less efficient than if it cached a mapping of cache payload key to raw hash after the first request for a cache key.

The HTTP, S3, and pak file backends are not implemented for the new interface.

The backends do not implement ICacheStore::CancelAll() because the existing backend framework provides WaitForQuiescence to wait for completion of async requests, and the implementation of ICacheStore by those backends works with that mechanism.

The non-leaf backends (hierarchical, async put, etc.) do not update stats from the ICacheStore functions.

#rb Zousar.Shaker
#rnx
#preflight 60899f35d324590001b47517

[CL 16148296 by Devin Doucette in ue5-main branch]
2021-04-28 16:22:18 -04:00
Devin Doucette
28ef6b6386 DDC: Iteration on ICache
- Identify payloads by the hash of the raw data.
- Added FCacheKey::Empty and FCachePayloadKey::Empty for convenience.
- Changed FCacheRecord and FCacheRecordBuilder constructors to be private.
- Changed storage in the legacy cache to store values inline for records with no attachments.
- Changed storage in the legacy cache to use CAS within each bucket.
- Improved wording and details in assert messages.

#rb Zousar.Shaker
#rnx

[CL 15984411 by Devin Doucette in ue5-main branch]
2021-04-12 18:25:18 -04:00
Zousar Shaker
699ad048c3 Ensure that at the time the FDerivedDataCache singleton is shutdown that we have waited on all internal backends to complete in-flight work.
#jira UE-110943
#rb devin.doucette

[CL 15930084 by Zousar Shaker in ue5-main branch]
2021-04-06 10:20:39 -04:00
Devin Doucette
ab4a43d5db CompressedBuffer: Implemented a stable format for compressed buffers with support for no compression and LZ4
#rb Tim.Smith

[CL 15895927 by Devin Doucette in ue5-main branch]
2021-04-01 16:31:55 -04:00
Marc Audy
0cbbc781ca Merge UE5/Release-Engine-Staging @ 15740152 to UE5/Main
This represents UE4/Main @ 15709114

[CL 15740605 by Marc Audy in ue5-main branch]
2021-03-18 15:20:03 -04:00
Devin Doucette
490d684d40 DDC: Clean up a few details in the new Cache API
#rb Zousar.Shaker
#rnx

[CL 15681766 by Devin Doucette in ue5-main branch]
2021-03-11 18:12:08 -04:00
Devin Doucette
7d5d9f3798 DDC: Clean up a few details in the new Cache API and remove the last exported type from DDC
#rb Zousar.Shaker
#rnx

[CL 15670275 by Devin Doucette in ue5-main branch]
2021-03-10 17:33:38 -04:00
Devin Doucette
017b41eb57 DDC: Refactored the new Cache API to support its use through only an interface dependency
#rb Zousar.Shaker
#rnx

[CL 15666872 by Devin Doucette in ue5-main branch]
2021-03-10 12:23:29 -04:00
Devin Doucette
2440bc5fbd DDC: Add generic EStatus and replace ECacheStatus with it
#rb Zousar.Shaker
#rnx

[CL 15583574 by Devin Doucette in ue5-main branch]
2021-03-03 11:55:38 -04:00
devin doucette
0f5201b8a1 DDC: Added CreateOrGetDDC to the module and changed creation of DDC to occur there instead of on module startup
#rb Arciel.Rekman, Zousar.Shaker

#ROBOMERGE-OWNER: devin.doucette
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 15530385 via CL 15530408 via CL 15530432 via CL 15530444 via CL 15531592
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Main) (v771-15082668)
#ROBOMERGE[STARSHIP]: Release-5.0-EarlyAccess

[CL 15531598 by devin doucette in ue5-main branch]
2021-02-25 14:50:04 -04:00
Devin Doucette
7e2e44ebf4 DDC: Refactored the Cache API to prepare for the Build API
- Removed the "Cache" prefix from ECachePriority, FCacheRequest, ICacheRequest, FCachePayload, FCachePayloadId to allow reuse in the Build API.
- Added FCompressedBuffer as the starting point for the API for creating and accessing compressed buffers.

#rb Zousar.Shaker
#rnx

[CL 15511723 by Devin Doucette in ue5-main branch]
2021-02-24 01:37:45 -04:00
Devin Doucette
7c5e0f99fa DDC: Capture cache value arrays into shared buffers instead of copying them
#rb Zousar.Shaker
#rnx

[CL 15501496 by Devin Doucette in ue5-main branch]
2021-02-23 12:32:38 -04:00
Devin Doucette
b5ae234c7c CompactBinary: Renamed view types to use a View suffix, and remove the Ref suffix from types that hold a SharedBuffer
#rb Zousar.Shaker
#rnx

[CL 15493834 by Devin Doucette in ue5-main branch]
2021-02-22 16:34:05 -04:00
Devin Doucette
c13b0dbecf DDC: Updated the new cache interface to expose compression of cache records
- FCacheRecord must now be created by FCacheRecordBuilder.
- FCacheRecord values are always an opaque buffer now, and ECacheRecordType has been removed.
- FCachePayload has been added as a common type for the cache record value and attachments.
- FCachePayloadId has been added as a 12-byte identifier for cache payloads.
- ICache::GetAttachments is now GetPayloads and can fetch both values and attachments.
- ECachePriority::Blocking has been added for requests that the caller will block on.

#rb Zousar.Shaker
#rnx

[CL 15471951 by Devin Doucette in ue5-main branch]
2021-02-19 11:26:34 -04:00
Devin Doucette
faf23d828f DDC: Fixed TryToPrefetch of one key, which was skipped due to an incorrect copy from CachedDataProbablyExistsBatch
#rb trivial
#rnx

[CL 15452343 by Devin Doucette in ue5-main branch]
2021-02-18 10:13:14 -04:00