Commit Graph

234 Commits

Author SHA1 Message Date
Zousar Shaker
ce91cce583 Ensure that the DDC filesystem backend updates timestamps on files in GetCachedData (if within the last quarter of its cleanup lifetime), not solely from CachedDataProbablyExists.
#jira UE-112593
#rb devin.doucette
#preflight 6085a7400bdfbc0001c61a68

[CL 16110598 by Zousar Shaker in ue5-main branch]
2021-04-25 13:59:44 -04:00
Zousar Shaker
94bcce4d3e Put a minimum "transfer speed over duration" limit on curl operations done by the HTTP DDC backend.
#rb devin.doucette
#preflight 60808a0ca698b3000172743c

[CL 16081589 by Zousar Shaker in ue5-main branch]
2021-04-21 17:14:57 -04:00
Devin Doucette
e7b98e5582 DDC: Added FCacheRecord KeyFuncs, FCacheRecord::GetPayload, and FPayload(FPayloadId)
#rb Zousar.Shaker
#rnx
#preflight 608056e0d4e67000017ac573

[CL 16078733 by Devin Doucette in ue5-main branch]
2021-04-21 14:56:45 -04:00
Zousar Shaker
8b5a796e2a Add support for ReadOnly={true|false} and EnvHostOverride={string} as configurable parameters on an HTTP DDC backend.
#rb devin.doucette

[CL 16076564 by Zousar Shaker in ue5-main branch]
2021-04-21 13:19:11 -04:00
Zousar Shaker
3b9ff68e09 Fix bug in the HttpDerivedDataBackend where an FRequest may not be returned to the FRequestPool if we're aborting queued PUTs due to a shutdown request.
#jira UE-113612
#rb devin.doucette
#preflight 607e38a842dbff0001a4dc20

[CL 16058184 by Zousar Shaker in ue5-main branch]
2021-04-19 22:53:53 -04:00
Devin Doucette
95aa03cd0d DDC: Simplify FCacheBucket
#rb Zousar.Shaker
#rnx
#preflight 607d0e59099fea0001460d34

[CL 16053217 by Devin Doucette in ue5-main branch]
2021-04-19 14:28:07 -04:00
Devin Doucette
cf7d6505f3 DDC: Tweak names and namespace formatting for consistency
#rb trivial
#rnx
#preflight 607d0e236e83200001593a45

[CL 16050392 by Devin Doucette in ue5-main branch]
2021-04-19 09:56:34 -04:00
Zousar Shaker
a7de618da5 Avoid stalling during shutdown due to queued async PUTs to non-local backends in DDC.
#rb devin.doucette

[CL 16016623 by Zousar Shaker in ue5-main branch]
2021-04-14 23:37:27 -04:00
Johan Berg
ce27da1f98 Remove UE4 strings
#jira UE-111642
#rb none

[CL 16002173 by Johan Berg in ue5-main branch]
2021-04-14 04:25:18 -04:00
Zousar Shaker
8bbc201340 Ensure that the pak writing DDC backend doesn't silently ignore overwrites of keys that are already in the pak file. The writes can happen, but they will emit a warning and leave the old copy in the serialized file as an abandoned blob. If the overwrite happens to be the same size as the original blob, the overwrite can successfully be done without wasting space.
#rb matt.peters

[CL 16001194 by Zousar Shaker in ue5-main branch]
2021-04-14 01:07:34 -04:00
Devin Doucette
28ef6b6386 DDC: Iteration on ICache
- Identify payloads by the hash of the raw data.
- Added FCacheKey::Empty and FCachePayloadKey::Empty for convenience.
- Changed FCacheRecord and FCacheRecordBuilder constructors to be private.
- Changed storage in the legacy cache to store values inline for records with no attachments.
- Changed storage in the legacy cache to use CAS within each bucket.
- Improved wording and details in assert messages.

#rb Zousar.Shaker
#rnx

[CL 15984411 by Devin Doucette in ue5-main branch]
2021-04-12 18:25:18 -04:00
Marc Audy
01b7c9f4f5 Merge UE5/RES @ 15958325 to UE5/Main
This represents UE4/Main @ 15913390 and Dev-PerfTest @ 15913304

[CL 15958515 by Marc Audy in ue5-main branch]
2021-04-08 14:32:07 -04:00
Devin Doucette
5d93c9c31e CompressedBuffer: Added FHeader::TotalCompressedSize and FCompressedBuffer::FromCompressed(FArchive&)
#rb Tim.Smith
#rnx

[CL 15946194 by Devin Doucette in ue5-main branch]
2021-04-07 17:13:31 -04:00
Zousar Shaker
3149d95ca3 Since we are now waiting for quiescence from DDC backends during shutdown even in interactive sessions, reduce the sleep time before checking for task completion from 1s to 0.1s.
#rb trivial

[CL 15934332 by Zousar Shaker in ue5-main branch]
2021-04-06 16:13:44 -04:00
Zousar Shaker
699ad048c3 Ensure that at the time the FDerivedDataCache singleton is shutdown that we have waited on all internal backends to complete in-flight work.
#jira UE-110943
#rb devin.doucette

[CL 15930084 by Zousar Shaker in ue5-main branch]
2021-04-06 10:20:39 -04:00
Zousar Shaker
d014867e9f Ensure that the non-DataRequest-Helper codepath for putting DDC data into the cache also tears-down promptly during process shutdown if not running on a build machine.
#jira UE-112179
#rb devin.doucette

[CL 15930076 by Zousar Shaker in ue5-main branch]
2021-04-06 10:20:06 -04:00
Zousar Shaker
dc68fab37b Ensure HTTP DDC backend doesn't delay process shutdown due to a queue of pending PUTs for normal users.
Still allows build machine runs to be thorough about completing PUTs.
Doesn't interrupt any in-flight PUTs, just prevents processing of any remaining queued PUTs.

#jira UE-112179
#rb devin.doucette

[CL 15920539 by Zousar Shaker in ue5-main branch]
2021-04-05 14:36:58 -04:00
Devin Doucette
ab4a43d5db CompressedBuffer: Implemented a stable format for compressed buffers with support for no compression and LZ4
#rb Tim.Smith

[CL 15895927 by Devin Doucette in ue5-main branch]
2021-04-01 16:31:55 -04:00
halfdan ingvarsson
5f84b5227e Fix new errors/warnings that are no longer being accidentally suppressed in clang on Windows (see CL 15880121).
- Member variable shadowing inside lambdas
- Deprecation warnings due to implicit copy constructors/operator being generated because of dllexports (this is more of a clang bug than an actual issue in our code).

#jira none
#fyi jurre.debaare, emil.persson, ian.fox, devin,doucette
#rnx

[CL 15882629 by halfdan ingvarsson in ue5-main branch]
2021-03-31 17:59:50 -04:00
Devin Doucette
89eb2eef0f DDC: Fixed a compile error inside a checkfSlow
#rb trivial
#rnx

[CL 15830267 by Devin Doucette in ue5-main branch]
2021-03-25 15:52:40 -04:00
Devin Doucette
d87eb29d07 DDC: Switched to WriteToString in the Jupiter backend
#rb Zousar.Shaker
#rnx

[CL 15829627 by Devin Doucette in ue5-main branch]
2021-03-25 15:34:31 -04:00
Zousar Shaker
0ee734bf61 Fix HTTP/Jupiter DDC backend to account for the fact that Jupiter will return batched results in arbitrary order, not the same order that the requests/keys were listed.
#rb devin.doucette

[CL 15771001 by Zousar Shaker in ue5-main branch]
2021-03-22 18:23:46 -04:00
nick darnell
832cc08a58 DDC - Don't look up the store for DDC paths that don't have environment variable overrides.
#jira UE-110702

#ROBOMERGE-OWNER: marc.audy
#ROBOMERGE-AUTHOR: nick.darnell
#ROBOMERGE-COMMAND: _robomerge UE5-Main
#ROBOMERGE-SOURCE: CL 15715366 via CL 15715370 via CL 15715374 via CL 15715407 via CL 15743064
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Main) (v781-15675533)

[CL 15747819 by nick darnell in ue5-main branch]
2021-03-19 11:39:18 -04:00
Marc Audy
0cbbc781ca Merge UE5/Release-Engine-Staging @ 15740152 to UE5/Main
This represents UE4/Main @ 15709114

[CL 15740605 by Marc Audy in ue5-main branch]
2021-03-18 15:20:03 -04:00
Joakim Lindqvist
5957a9c659 Added option to override Jupiter speed class
#rb devin.doucette

[CL 15736124 by Joakim Lindqvist in ue5-main branch]
2021-03-18 05:51:42 -04:00