Commit Graph

247 Commits

Author SHA1 Message Date
Devin Doucette
536b5fa4dc DDC: Updated documentation and removed unnecessary explicit hash construction
#rb trivial
#rnx

[CL 16207037 by Devin Doucette in ue5-main branch]
2021-05-05 10:15:34 -04:00
Devin Doucette
70a7cb25a8 DDC: Added raw size to the payload
#rb Zousar.Shaker
#rnx
#preflight 609229cb58c4790001a048f0

[CL 16207035 by Devin Doucette in ue5-main branch]
2021-05-05 10:15:26 -04:00
Devin Doucette
3f40f504cf DDC: Changed Put to take a const array of cache records now that records are copyable
#rb Zousar.Shaker
#rnx

[CL 16184736 by Devin Doucette in ue5-main branch]
2021-05-03 12:05:18 -04:00
Devin Doucette
e39324b347 DDC: Changed FCacheRecord to be reference-counted instead of uniquely owned
#rb Zousar.Shaker
#rnx

[CL 16184511 by Devin Doucette in ue5-main branch]
2021-05-03 11:41:14 -04:00
Devin Doucette
0a560db3a0 DDC: Added FPayload::Null and FPayloadId::Null
#rb Zousar.Shaker
#rnx

[CL 16184503 by Devin Doucette in ue5-main branch]
2021-05-03 11:40:06 -04:00
Devin Doucette
7149637a27 DDC: Moved Private::CreateBuildDefinition[Builder] out of the header to avoid them being used externally
#rb Zousar.Shaker
#rnx

[CL 16184493 by Devin Doucette in ue5-main branch]
2021-05-03 11:38:45 -04:00
Devin Doucette
0baafa125a DDC: Added the start of the new build interface UE::DerivedData::IBuild
This only includes the functionality required to create a build definition, and does not include the functionalty required to execute a build from the build definition.

#rb Zousar.Shaker
#rnx
#preflight 608cf5ef7343ea00016e58f4

[CL 16183533 by Devin Doucette in ue5-main branch]
2021-05-03 09:25:48 -04:00
Andriy Tylychko
9f1cf3eabf re-deprecated FPlatformProcess::CreateSynchEvent(), now on all platforms. Replaced FEvent* by FEventRef where it made sense. Deprecated FEvent::Create() as another method to bypass the event pool
#rb steve.robb

[CL 16163267 by Andriy Tylychko in ue5-main branch]
2021-04-29 18:52:48 -04:00
Devin Doucette
0fc6e341c1 DDC: Display pak file size when opening a pak cache
#jira UE-114516
#rb Zousar.Shaker
#rnx

[CL 16157309 by Devin Doucette in ue5-main branch]
2021-04-29 10:42:42 -04:00
Devin Doucette
e9c43a79ae DDC: Implemented ICache on FDerivedDataCacheInterface
Moved IDerivedDataCacheModule to its own header and renamed CreateOrGetDDC to CreateOrGetCache.

#rb Zousar.Shaker
#rnx
#preflight 6089d5a036afba00015c0348

[CL 16153121 by Devin Doucette in ue5-main branch]
2021-04-28 23:25:28 -04:00
Devin Doucette
e42bbbedab DDC: Implement ICacheStore on FDerivedDataBackendInterface
This allows the new cache to be implemented more efficiently using the old cache backends because functionality like the corruption wrapper and key length limiter can be bypassed and the individual backends can store cache records in whatever way is most efficient for them.

The hierarchical backend may request payloads when they are not required due to incomplete tracking of backend state, and GetPayload will never fill other backends due to the inefficiency of the existing backend framework.

The filesystem backend does not cache any state in memory, which makes requests for individual payloads less efficient than if it cached a mapping of cache payload key to raw hash after the first request for a cache key.

The HTTP, S3, and pak file backends are not implemented for the new interface.

The backends do not implement ICacheStore::CancelAll() because the existing backend framework provides WaitForQuiescence to wait for completion of async requests, and the implementation of ICacheStore by those backends works with that mechanism.

The non-leaf backends (hierarchical, async put, etc.) do not update stats from the ICacheStore functions.

#rb Zousar.Shaker
#rnx
#preflight 60899f35d324590001b47517

[CL 16148296 by Devin Doucette in ue5-main branch]
2021-04-28 16:22:18 -04:00
Devin Doucette
84641e44fb DDC: Added an assert to FCacheRecordBuilder::BuildAsync to require a callback
#rb trivial
#rnx

[CL 16148043 by Devin Doucette in ue5-main branch]
2021-04-28 16:05:08 -04:00
Devin Doucette
6566793856 DDC: Split ICache into ICacheFactory and ICacheStore, expand FRequest into TRequest
#rb Zousar.Shaker
#rnx
#preflight 6089849fe1c5d10001273917

[CL 16145619 by Devin Doucette in ue5-main branch]
2021-04-28 12:55:00 -04:00
Zousar Shaker
ce91cce583 Ensure that the DDC filesystem backend updates timestamps on files in GetCachedData (if within the last quarter of its cleanup lifetime), not solely from CachedDataProbablyExists.
#jira UE-112593
#rb devin.doucette
#preflight 6085a7400bdfbc0001c61a68

[CL 16110598 by Zousar Shaker in ue5-main branch]
2021-04-25 13:59:44 -04:00
Zousar Shaker
94bcce4d3e Put a minimum "transfer speed over duration" limit on curl operations done by the HTTP DDC backend.
#rb devin.doucette
#preflight 60808a0ca698b3000172743c

[CL 16081589 by Zousar Shaker in ue5-main branch]
2021-04-21 17:14:57 -04:00
Devin Doucette
e7b98e5582 DDC: Added FCacheRecord KeyFuncs, FCacheRecord::GetPayload, and FPayload(FPayloadId)
#rb Zousar.Shaker
#rnx
#preflight 608056e0d4e67000017ac573

[CL 16078733 by Devin Doucette in ue5-main branch]
2021-04-21 14:56:45 -04:00
Zousar Shaker
8b5a796e2a Add support for ReadOnly={true|false} and EnvHostOverride={string} as configurable parameters on an HTTP DDC backend.
#rb devin.doucette

[CL 16076564 by Zousar Shaker in ue5-main branch]
2021-04-21 13:19:11 -04:00
Zousar Shaker
3b9ff68e09 Fix bug in the HttpDerivedDataBackend where an FRequest may not be returned to the FRequestPool if we're aborting queued PUTs due to a shutdown request.
#jira UE-113612
#rb devin.doucette
#preflight 607e38a842dbff0001a4dc20

[CL 16058184 by Zousar Shaker in ue5-main branch]
2021-04-19 22:53:53 -04:00
Devin Doucette
95aa03cd0d DDC: Simplify FCacheBucket
#rb Zousar.Shaker
#rnx
#preflight 607d0e59099fea0001460d34

[CL 16053217 by Devin Doucette in ue5-main branch]
2021-04-19 14:28:07 -04:00
Devin Doucette
cf7d6505f3 DDC: Tweak names and namespace formatting for consistency
#rb trivial
#rnx
#preflight 607d0e236e83200001593a45

[CL 16050392 by Devin Doucette in ue5-main branch]
2021-04-19 09:56:34 -04:00
Zousar Shaker
a7de618da5 Avoid stalling during shutdown due to queued async PUTs to non-local backends in DDC.
#rb devin.doucette

[CL 16016623 by Zousar Shaker in ue5-main branch]
2021-04-14 23:37:27 -04:00
Johan Berg
ce27da1f98 Remove UE4 strings
#jira UE-111642
#rb none

[CL 16002173 by Johan Berg in ue5-main branch]
2021-04-14 04:25:18 -04:00
Zousar Shaker
8bbc201340 Ensure that the pak writing DDC backend doesn't silently ignore overwrites of keys that are already in the pak file. The writes can happen, but they will emit a warning and leave the old copy in the serialized file as an abandoned blob. If the overwrite happens to be the same size as the original blob, the overwrite can successfully be done without wasting space.
#rb matt.peters

[CL 16001194 by Zousar Shaker in ue5-main branch]
2021-04-14 01:07:34 -04:00
Devin Doucette
28ef6b6386 DDC: Iteration on ICache
- Identify payloads by the hash of the raw data.
- Added FCacheKey::Empty and FCachePayloadKey::Empty for convenience.
- Changed FCacheRecord and FCacheRecordBuilder constructors to be private.
- Changed storage in the legacy cache to store values inline for records with no attachments.
- Changed storage in the legacy cache to use CAS within each bucket.
- Improved wording and details in assert messages.

#rb Zousar.Shaker
#rnx

[CL 15984411 by Devin Doucette in ue5-main branch]
2021-04-12 18:25:18 -04:00
Marc Audy
01b7c9f4f5 Merge UE5/RES @ 15958325 to UE5/Main
This represents UE4/Main @ 15913390 and Dev-PerfTest @ 15913304

[CL 15958515 by Marc Audy in ue5-main branch]
2021-04-08 14:32:07 -04:00