Commit Graph

44 Commits

Author SHA1 Message Date
devin doucette
169318850a DDC: Masked record-only cache policy flags out of the value policy
Fixing backends to respect the masked flags includes a few fixes to handling of ~Query by Put and PutValue.

#rb Matt.Peters
#rnx
#preflight 61f7f7ede55232619f887977

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18790481 in //UE5/Release-5.0/... via CL 18790769 via CL 18791304
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v908-18788545)

[CL 18791515 by devin doucette in ue5-main branch]
2022-01-31 11:52:13 -05:00
devin doucette
517947112b DDC: Reorganized cache policy code into its own files
- FCacheRecordPolicyBuilder now disallows adding value policies with flags that only make sense for records.
- FCacheRecordPolicyBuilder now disallows adding multiple value policies for the same value.
- FCacheRecordPolicyBuilder now skips addition of value policies that match the default policy.

#rb Matt.Peters
#rnx
#preflight 61f471cc6b5aea38e5c64cb6

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18782116 in //UE5/Release-5.0/... via CL 18782123 via CL 18782168
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v903-18687472)

[CL 18782178 by devin doucette in ue5-main branch]
2022-01-28 20:34:31 -05:00
devin doucette
cdedb37d66 DDC: Converted the Build API to UTF-8 where appropriate
#jira UE-133382
#rb Zousar.Shaker
#rnx
#preflight 61e798ee3f00a0a23ef78c49

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18655307 in //UE5/Release-5.0/... via CL 18655320 via CL 18655322
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v900-18638592)

[CL 18655349 by devin doucette in ue5-main branch]
2022-01-19 00:27:48 -05:00
devin doucette
1883382ae7 DDC: Added functions to transform a cache policy and build policy
#rb Zousar.Shaker
#rnx
#preflight 61e0aa6e341d372424e948f1

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18617338 in //UE5/Release-5.0/... via CL 18617349 via CL 18617359
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v899-18417669)

[CL 18617376 by devin doucette in ue5-main branch]
2022-01-14 09:48:40 -05:00
devin doucette
b46e70ea4e DDC: Rename CompleteParams to Response and use FSharedString with no alias
#rb Zousar.Shaker
#rnx
#preflight 61ddf9e0f10add4574e6cf59

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18577467 in //UE5/Release-5.0/... via CL 18577471
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)

[CL 18577481 by devin doucette in ue5-release-engine-test branch]
2022-01-11 17:09:20 -05:00
devin doucette
b4e6fce55b DDC: Changed FCacheRecord to be a collection of values
The structure with a value and attachments is an artifact of an early design where the record *was* a compact binary package. That design changed months ago and it now simplifies the cache record significantly to make it a collection of values without specific "value" and "attachment" types.

#rb Zousar.Shaker
#rnx
#preflight 61d8b94ed17842e547def37a

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18553971 in //UE5/Release-5.0/... via CL 18553976
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)

[CL 18553977 by devin doucette in ue5-release-engine-test branch]
2022-01-07 23:22:29 -05:00
devin doucette
35393bbb2b DDC: Split FPayload into separate FValue and FValueId types
A payload was conceptually a value with an ID. That has been formalized by removing the ID from the payload and having separate FValue and FValueId types. This separation cleans up the API in a few areas, and provides a more natural path to providing a basic key/value API.

#rb Zousar.Shaker
#rnx
#preflight 61d704c04c252480ca284d61

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18531844 in //UE5/Release-5.0/... via CL 18531856
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)

[CL 18531864 by devin doucette in ue5-release-engine-test branch]
2022-01-06 11:05:57 -05:00
devin doucette
48f68493ef DDC: Removed the Put/Get compatibility functions from ICache
#rb Zousar.Shaker
#rnx
#preflight 61d49a971a3fd09dcbc87978

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18511490 in //UE5/Release-5.0/... via CL 18511542
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)

[CL 18511622 by devin doucette in ue5-release-engine-test branch]
2022-01-04 14:38:48 -05:00
devin doucette
27c1393427 CompressedBuffer: Removed partial decompression from FCompressedBuffer now that FCompressedBufferReader is available
Requiring the use of a separate reader type makes it more likely that readers will be reused, and makes it easier to audit reader usage going forward. Reusing readers is desirable to reduce the number of large temporary allocations made during partial decompression of a buffer.

- Added FCompressedBuffer::Save(FArchive&) and renamed FromCompressed(FArchive&) to Load(FArchive&).
- Added FCompressedBufferReaderSourceScope to set a buffer source within a scope.
- Added proper bounds checks to FNoneDecoder.
- Store the header checksum on the decoder context to allow raw blocks to be reused across sources.
- Decode the header on the fly to avoid a temporary header allocation when the header is in contiguous memory.

#rb Zousar.Shaker
#rnx
#preflight 61a98d53800738dbfbc84c73

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18382211 in //UE5/Release-5.0/... via CL 18382310
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v896-18170469)

[CL 18382377 by devin doucette in ue5-release-engine-test branch]
2021-12-06 10:16:05 -05:00
devin doucette
cea98e6ab4 DDC: Split build diagnostics into messages and logs
Messages are deterministic output recorded explicitly by the build function. Logs will be automatically captured from the output log and are not necessarily deterministic. The presence of logs in a build output automatically disables caching of the build output.

#rb Zousar.Shaker
#rnx

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18286155 in //UE5/Release-5.0/... via CL 18286173
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)

[CL 18286201 by devin doucette in ue5-release-engine-test branch]
2021-11-24 13:25:25 -05:00
devin doucette
dc4658fa91 DDC: Updated IBuildWorkerExecutor to allow filtering of payloads
#rb Zousar.Shaker
#rnx

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 17876434 in //UE5/Release-5.0/... via CL 17876436
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v883-17842818)
#ROBOMERGE[STARSHIP]: UE5-Main

[CL 17876451 by devin doucette in ue5-release-engine-test branch]
2021-10-20 14:50:22 -04:00
devin doucette
01c33e6702 DDC: Reworked FBuildSession to allow filtering of payloads
- Build now takes optional inputs for consistency with BuildAction.
- BuildAction has been renamed to Build and works the same as Build except it starts from an action.
- Build now takes a FBuildPolicy, which is implicitly constructible from EBuildPolicy, and allows setting the policy by payload.
- FBuildContext and FBuildConfigContext now have policy masks that allow clearing flags across every payload in the build.
- EBuildPolicy has CacheStore split into CacheStoreOnQuery and CacheStoreOnBuild for more control of cache behavior.
- Builds now use consistent completion callback parameters, FBuildCompleteParams.
- Build types now have debugger visualization.

#rb Zousar.Shaker
#rnx

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 17874670 in //UE5/Release-5.0/... via CL 17874724
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v883-17842818)
#ROBOMERGE[STARSHIP]: UE5-Main

[CL 17874740 by devin doucette in ue5-release-engine-test branch]
2021-10-20 12:38:01 -04:00
devin doucette
750419503e DDC: Added ECachePolicy::KeepAlive and EBuildPolicy::CacheKeepAlive
The new flags are meant to prevent a cache store from evicting the associated cache records for the duration of the session.

#rb Zousar.Shaker
#rnx
#preflight 614954b611925a00017c525f

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 17579887 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v871-17566257)

[CL 17579899 by devin doucette in ue5-release-engine-test branch]
2021-09-21 01:10:55 -04:00
devin doucette
a6226e2f2e DDC: Improved handling of build cancellation
#rb Zousar.Shaker
#rnx
#preflight 6143d4423c7c670001e9aa66

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 17551334 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v870-17433530)

[CL 17551348 by devin doucette in ue5-release-engine-test branch]
2021-09-17 09:32:55 -04:00
devin doucette
55ac73b651 DDC: Renamed Dispatch[Op] to Schedule[Op] and tweaked formatting for consistency
#rb none
#rnx
#preflight 613a336abf5a710001674089

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 17474273 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v870-17433530)

[CL 17474278 by devin doucette in ue5-release-engine-test branch]
2021-09-09 13:36:48 -04:00
johan torp
4afc055300 Add DDC2 memory scheduler to limit peak memory consumption
Also:
* Implemented rough memory usage estimate for texture cooking
* Fixed build action -> constant copying bug

#rb devin.doucette

#ROBOMERGE-AUTHOR: johan.torp
#ROBOMERGE-SOURCE: CL 17470303 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v870-17433530)

[CL 17470308 by johan torp in ue5-release-engine-test branch]
2021-09-09 06:04:13 -04:00
devin doucette
ce01b1fb44 DDC: Stopped loading build inputs for local execution when execution is disabled
#rb Zousar.Shaker
#rnx

#ROBOMERGE-SOURCE: CL 17358468 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v865-17346139)

[CL 17358490 by devin doucette in ue5-release-engine-test branch]
2021-08-30 16:38:36 -04:00
devin doucette
b575156c6f DDC: Added request barriers to the build job constructors
The barriers are needed to handle re-entrant AdvanceToState calls within the constructor of jobs with Blocking priority.

#rb Zousar.Shaker
#rnx

#ROBOMERGE-SOURCE: CL 17302241 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v861-17282326)

[CL 17302256 by devin doucette in ue5-release-engine-test branch]
2021-08-25 10:15:39 -04:00
devin doucette
275765055a DDC: Fixed build callback status when the policy allows querying the cache but not building
#rb Zousar.Shaker
#rnx

#ROBOMERGE-SOURCE: CL 17291662 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v861-17282326)

[CL 17291681 by devin doucette in ue5-release-engine-test branch]
2021-08-24 15:25:59 -04:00
devin doucette
ae7825b29c DDC: Added EBuildStatus::CacheKey and return EStatus::Ok from a build when no build or cache was requested
#rb Zousar.Shaker
#rnx
#preflight 611f304c008be900010bf006

#ROBOMERGE-SOURCE: CL 17247202 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v855-17104924)

[CL 17247215 by devin doucette in ue5-release-engine-test branch]
2021-08-20 10:04:41 -04:00
devin doucette
d5a632673b DDC: Fixed a build job assert caused by canceling at the wrong time
#rb Zousar.Shaker
#rnx

#ROBOMERGE-SOURCE: CL 17108375 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v854-17104634)

[CL 17108399 by devin doucette in ue5-release-engine-test branch]
2021-08-09 17:56:38 -04:00
devin doucette
d55593526e DDC: Add FBuildConfigContext::SetRequiredMemory() and forward the estimate to the build scheduler
#rb Zousar.Shaker
#rnx

#ROBOMERGE-SOURCE: CL 17088443 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)

[CL 17089009 by devin doucette in ue5-release-engine-test branch]
2021-08-06 17:53:47 -04:00
devin doucette
c67fcf1702 DDC: Split DerivedDataRequest.h and rename DerivedDataBuildPolicy.h
#rb none
#rnx

#ROBOMERGE-SOURCE: CL 17086960 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)

[CL 17086977 by devin doucette in ue5-release-engine-test branch]
2021-08-06 15:50:00 -04:00
devin doucette
bd60f3ef1b DDC: Removed ICacheFactory in favor of exported functions
FCacheBucket is now always exposed as FAnsiStringView.

#rb Zousar.Shaker
#rnx

#ROBOMERGE-SOURCE: CL 17084521 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)

[CL 17084549 by devin doucette in ue5-release-engine-test branch]
2021-08-06 12:53:08 -04:00
devin doucette
7ae4b64cd6 DDC: Removed IBuild::Load[Type] in favor of exported functions
#rb Zousar.Shaker
#rnx

#ROBOMERGE-SOURCE: CL 17083958 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)

[CL 17083970 by devin doucette in ue5-release-engine-test branch]
2021-08-06 11:58:24 -04:00