Commit Graph

82 Commits

Author SHA1 Message Date
devin doucette
3331f68391 DDC: Added ECachePolicy::PartialOnError and implemented partial cache records in the filesystem and memory backends
#rb Zousar.Shaker
#rnx
#preflight 615f703dcec44a000146cf80

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 17759092 in //UE5/Release-5.0/... via CL 17760688
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v879-17706426)
#ROBOMERGE[STARSHIP]: UE5-Main

[CL 17760906 by devin doucette in ue5-release-engine-test branch]
2021-10-08 11:01:21 -04:00
Devin Doucette
a077feffbd DDC: Reworked ICacheStore to allow partial records, filtering of payloads, and loading parts of payloads
- ICacheStore::Put() has updated documentation to reflect the requirements of partial records.
- ICacheStore::Get() now takes a FCacheRecordPolicy, which is implicitly constructible from ECachePolicy, and allows setting the policy by payload.
- ICacheStore::GetPayload() is replaced by ICacheStore::GetChunks(), which allows loading parts of payloads.
- ICacheStore::CancelAll() is moved to ICache::CancelAll() because the cache can track requests at the top level and cancel them without exposing cancellation on individual cache stores.
- ECachePolicy::SkipLocalCopy has been removed because it is difficult to reason about.
- ECachePolicy::SkipData flags now have a documented meaning for put requests, to hint that record existence implies payload existence.
- The filesystem and memory cache stores have been updated to support partial records, filtering of payloads, and loading parts of payloads.
- Requesting part of a payload will decompress the entire payload for now, until compressed buffers expose a way to decompress only part.
- Fixed a bug in FTexturePlatformData::AreDerivedMipsAvailable() that caused it to return false for structured cache keys.

#rb Zousar.Shaker
#rnx
#preflight 615e03241ed62f0001b95454

#ROBOMERGE-OWNER: Devin.Doucette
#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 17748550 in //UE5/Release-5.0/... via CL 17748555
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v879-17706426)
#ROBOMERGE-CONFLICT from-shelf
#ROBOMERGE[STARSHIP]: UE5-Main

[CL 17748602 by Devin Doucette in ue5-release-engine-test branch]
2021-10-07 09:11:32 -04:00
devin doucette
c39e30cfd7 Added missing includes of DerivedDataPayload.h found by the non-unity build
#rb none
#rnx

#ROBOMERGE-SOURCE: CL 17426590 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v865-17346139)

[CL 17426595 by devin doucette in ue5-release-engine-test branch]
2021-09-03 17:11:57 -04:00
devin doucette
a066ec0c49 DDC: Implemented existing cache debug options for the new cache stores
#rb Zousar.Shaker
#rnx

#ROBOMERGE-SOURCE: CL 17308400 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v861-17282326)

[CL 17308416 by devin doucette in ue5-release-engine-test branch]
2021-08-25 16:08:33 -04:00
devin doucette
bd60f3ef1b DDC: Removed ICacheFactory in favor of exported functions
FCacheBucket is now always exposed as FAnsiStringView.

#rb Zousar.Shaker
#rnx

#ROBOMERGE-SOURCE: CL 17084521 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)

[CL 17084549 by devin doucette in ue5-release-engine-test branch]
2021-08-06 12:53:08 -04:00
devin doucette
733d9c1a97 DDC: Added conversions between FPayloadId and FCbObjectId
#rb Zousar.Shaker
#rnx

#ROBOMERGE-SOURCE: CL 17071877 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)

[CL 17071890 by devin doucette in ue5-release-engine-test branch]
2021-08-05 13:37:13 -04:00
devin doucette
bfda02d384 DDC: Replaced FRequest with FRequestGroup to handle nested and chained requests more robustly
Any function that may create a request now has a IRequestOwner& parameter, and uses the Begin and End functions on the owner to manage the lifetime of any requests that it creates, as well as using End to invoke the completion callback for any request which has one.

The new FRequestBarrier may be used to block a group from being considered complete in a scope where more requests may be added to it.

#rb Matt.Peters
#rnx
#preflight 6109b5c403d303000144cce5
#preflight 610acf7103d30300016fda94

#ROBOMERGE-SOURCE: CL 17060470 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v850-17047176)

[CL 17060649 by devin doucette in ue5-release-engine-test branch]
2021-08-04 18:08:50 -04:00
geoff evans
e9a24937a3 Add trace counter increments for more cache interaction prototypes in FileSystemDDC.
This fixes bytes read not matching the reported get hit count, and likely the same for put hits.

#jira none
#rb francis.hurteau
#preflight 60fad90ee885f200013f6939

#ROBOMERGE-SOURCE: CL 16939581 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v838-16927207)

[CL 16939595 by geoff evans in ue5-release-engine-test branch]
2021-07-23 14:21:32 -04:00
sebastien lussier
8c31cfedfd DDC backends - removed speed class tests in implementations of TryToPrefetch()
#rb danny.couture, devin.doucette

#ROBOMERGE-SOURCE: CL 16900182 via CL 16917763
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v836-16769935)

[CL 16917911 by sebastien lussier in ue5-release-engine-test branch]
2021-07-21 19:12:55 -04:00
devin doucette
bed1f486f8 DDC: Added EPutStatus::Skipped to avoid slow retry when simulating cache misses
#rb Zousar.Shaker
#preflight 60f12443a34bf50001654df1

#ROBOMERGE-SOURCE: CL 16883247 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v836-16769935)

[CL 16883252 by devin doucette in ue5-release-engine-test branch]
2021-07-19 10:12:20 -04:00
Devin Doucette
7e392be53b DDC: Fixed MeasureCacheRecord to only count payloads with data
#rb trivial
#rnx

[CL 16553211 by Devin Doucette in ue5-main branch]
2021-06-03 17:06:13 -04:00
Devin Doucette
8e3e3cc389 DDC: Fixed TryToPrefetch to work when there is only one backend
#rb Zousar.Shaker
#rnx

[CL 16549112 by Devin Doucette in ue5-main branch]
2021-06-03 12:57:44 -04:00
Devin Doucette
eb2cda6fba Added conversions and constructors to/from 12-byte arrays for FCbObjectId and DerivedData::FPayloadId
#rb Zousar.Shaker
#rnx

[CL 16536271 by Devin Doucette in ue5-main branch]
2021-06-02 13:06:45 -04:00
geoff evans
b1c54a1182 Add Trace Cpu Events and Counters to FileSystem DDC backend.
#jira none
#rb devin.doucette
#preflight 60a4337db7ccb000013e0fd6

[CL 16378052 by geoff evans in ue5-main branch]
2021-05-18 18:50:43 -04:00
Devin Doucette
8ce8977949 DDC: Added optional versions of BuildAction, BuildDefinition, BuildOutput, CacheRecord
- Optional versions are now returned from operations that may fail.
- Fixed a bug in the cache where a null record could be dereferenced.
- Improved validation when loading build types.

#rb Zousar.Shaker
#rnx

[CL 16332909 by Devin Doucette in ue5-main branch]
2021-05-14 14:47:11 -04:00
Devin Doucette
dd9e1ec58b DDC: Added support for cache records with null data
#rb Zousar.Shaker
#rnx
#preflight 609b6698b16d45000155cb04

[CL 16298475 by Devin Doucette in ue5-main branch]
2021-05-12 09:28:05 -04:00
Devin Doucette
70a7cb25a8 DDC: Added raw size to the payload
#rb Zousar.Shaker
#rnx
#preflight 609229cb58c4790001a048f0

[CL 16207035 by Devin Doucette in ue5-main branch]
2021-05-05 10:15:26 -04:00
Devin Doucette
3f40f504cf DDC: Changed Put to take a const array of cache records now that records are copyable
#rb Zousar.Shaker
#rnx

[CL 16184736 by Devin Doucette in ue5-main branch]
2021-05-03 12:05:18 -04:00
Devin Doucette
e42bbbedab DDC: Implement ICacheStore on FDerivedDataBackendInterface
This allows the new cache to be implemented more efficiently using the old cache backends because functionality like the corruption wrapper and key length limiter can be bypassed and the individual backends can store cache records in whatever way is most efficient for them.

The hierarchical backend may request payloads when they are not required due to incomplete tracking of backend state, and GetPayload will never fill other backends due to the inefficiency of the existing backend framework.

The filesystem backend does not cache any state in memory, which makes requests for individual payloads less efficient than if it cached a mapping of cache payload key to raw hash after the first request for a cache key.

The HTTP, S3, and pak file backends are not implemented for the new interface.

The backends do not implement ICacheStore::CancelAll() because the existing backend framework provides WaitForQuiescence to wait for completion of async requests, and the implementation of ICacheStore by those backends works with that mechanism.

The non-leaf backends (hierarchical, async put, etc.) do not update stats from the ICacheStore functions.

#rb Zousar.Shaker
#rnx
#preflight 60899f35d324590001b47517

[CL 16148296 by Devin Doucette in ue5-main branch]
2021-04-28 16:22:18 -04:00
Zousar Shaker
ce91cce583 Ensure that the DDC filesystem backend updates timestamps on files in GetCachedData (if within the last quarter of its cleanup lifetime), not solely from CachedDataProbablyExists.
#jira UE-112593
#rb devin.doucette
#preflight 6085a7400bdfbc0001c61a68

[CL 16110598 by Zousar Shaker in ue5-main branch]
2021-04-25 13:59:44 -04:00
Marc Audy
01b7c9f4f5 Merge UE5/RES @ 15958325 to UE5/Main
This represents UE4/Main @ 15913390 and Dev-PerfTest @ 15913304

[CL 15958515 by Marc Audy in ue5-main branch]
2021-04-08 14:32:07 -04:00
Marc Audy
0cbbc781ca Merge UE5/Release-Engine-Staging @ 15740152 to UE5/Main
This represents UE4/Main @ 15709114

[CL 15740605 by Marc Audy in ue5-main branch]
2021-03-18 15:20:03 -04:00
Ben Marsh
4f78759ae2 Disable intermittent DDC warning when running on build machines.
#rb none
#rnx

[CL 15478823 by Ben Marsh in ue5-main branch]
2021-02-19 18:09:28 -04:00
Devin Doucette
6394ccc1cd DDC: Added TryToPrefetch to check multiple keys for existence and prefetch the data into a fast cache backend
#rb Zousar.Shaker

[CL 15422719 by Devin Doucette in ue5-main branch]
2021-02-16 17:24:48 -04:00
Marc Audy
bc88b73a29 Merge Release-Engine-Staging to Main @ CL# 15151250
Represents UE4/Main @ 15133763

[CL 15158774 by Marc Audy in ue5-main branch]
2021-01-21 16:22:06 -04:00