Commit Graph

38 Commits

Author SHA1 Message Date
devin doucette
35393bbb2b DDC: Split FPayload into separate FValue and FValueId types
A payload was conceptually a value with an ID. That has been formalized by removing the ID from the payload and having separate FValue and FValueId types. This separation cleans up the API in a few areas, and provides a more natural path to providing a basic key/value API.

#rb Zousar.Shaker
#rnx
#preflight 61d704c04c252480ca284d61

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18531844 in //UE5/Release-5.0/... via CL 18531856
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)

[CL 18531864 by devin doucette in ue5-release-engine-test branch]
2022-01-06 11:05:57 -05:00
devin doucette
48f68493ef DDC: Removed the Put/Get compatibility functions from ICache
#rb Zousar.Shaker
#rnx
#preflight 61d49a971a3fd09dcbc87978

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18511490 in //UE5/Release-5.0/... via CL 18511542
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)

[CL 18511622 by devin doucette in ue5-release-engine-test branch]
2022-01-04 14:38:48 -05:00
devin doucette
27c1393427 CompressedBuffer: Removed partial decompression from FCompressedBuffer now that FCompressedBufferReader is available
Requiring the use of a separate reader type makes it more likely that readers will be reused, and makes it easier to audit reader usage going forward. Reusing readers is desirable to reduce the number of large temporary allocations made during partial decompression of a buffer.

- Added FCompressedBuffer::Save(FArchive&) and renamed FromCompressed(FArchive&) to Load(FArchive&).
- Added FCompressedBufferReaderSourceScope to set a buffer source within a scope.
- Added proper bounds checks to FNoneDecoder.
- Store the header checksum on the decoder context to allow raw blocks to be reused across sources.
- Decode the header on the fly to avoid a temporary header allocation when the header is in contiguous memory.

#rb Zousar.Shaker
#rnx
#preflight 61a98d53800738dbfbc84c73

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18382211 in //UE5/Release-5.0/... via CL 18382310
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v896-18170469)

[CL 18382377 by devin doucette in ue5-release-engine-test branch]
2021-12-06 10:16:05 -05:00
devin doucette
cea98e6ab4 DDC: Split build diagnostics into messages and logs
Messages are deterministic output recorded explicitly by the build function. Logs will be automatically captured from the output log and are not necessarily deterministic. The presence of logs in a build output automatically disables caching of the build output.

#rb Zousar.Shaker
#rnx

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18286155 in //UE5/Release-5.0/... via CL 18286173
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)

[CL 18286201 by devin doucette in ue5-release-engine-test branch]
2021-11-24 13:25:25 -05:00
devin doucette
dc4658fa91 DDC: Updated IBuildWorkerExecutor to allow filtering of payloads
#rb Zousar.Shaker
#rnx

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 17876434 in //UE5/Release-5.0/... via CL 17876436
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v883-17842818)
#ROBOMERGE[STARSHIP]: UE5-Main

[CL 17876451 by devin doucette in ue5-release-engine-test branch]
2021-10-20 14:50:22 -04:00
devin doucette
01c33e6702 DDC: Reworked FBuildSession to allow filtering of payloads
- Build now takes optional inputs for consistency with BuildAction.
- BuildAction has been renamed to Build and works the same as Build except it starts from an action.
- Build now takes a FBuildPolicy, which is implicitly constructible from EBuildPolicy, and allows setting the policy by payload.
- FBuildContext and FBuildConfigContext now have policy masks that allow clearing flags across every payload in the build.
- EBuildPolicy has CacheStore split into CacheStoreOnQuery and CacheStoreOnBuild for more control of cache behavior.
- Builds now use consistent completion callback parameters, FBuildCompleteParams.
- Build types now have debugger visualization.

#rb Zousar.Shaker
#rnx

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 17874670 in //UE5/Release-5.0/... via CL 17874724
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v883-17842818)
#ROBOMERGE[STARSHIP]: UE5-Main

[CL 17874740 by devin doucette in ue5-release-engine-test branch]
2021-10-20 12:38:01 -04:00
devin doucette
750419503e DDC: Added ECachePolicy::KeepAlive and EBuildPolicy::CacheKeepAlive
The new flags are meant to prevent a cache store from evicting the associated cache records for the duration of the session.

#rb Zousar.Shaker
#rnx
#preflight 614954b611925a00017c525f

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 17579887 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v871-17566257)

[CL 17579899 by devin doucette in ue5-release-engine-test branch]
2021-09-21 01:10:55 -04:00
devin doucette
a6226e2f2e DDC: Improved handling of build cancellation
#rb Zousar.Shaker
#rnx
#preflight 6143d4423c7c670001e9aa66

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 17551334 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v870-17433530)

[CL 17551348 by devin doucette in ue5-release-engine-test branch]
2021-09-17 09:32:55 -04:00
devin doucette
55ac73b651 DDC: Renamed Dispatch[Op] to Schedule[Op] and tweaked formatting for consistency
#rb none
#rnx
#preflight 613a336abf5a710001674089

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 17474273 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v870-17433530)

[CL 17474278 by devin doucette in ue5-release-engine-test branch]
2021-09-09 13:36:48 -04:00
johan torp
4afc055300 Add DDC2 memory scheduler to limit peak memory consumption
Also:
* Implemented rough memory usage estimate for texture cooking
* Fixed build action -> constant copying bug

#rb devin.doucette

#ROBOMERGE-AUTHOR: johan.torp
#ROBOMERGE-SOURCE: CL 17470303 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v870-17433530)

[CL 17470308 by johan torp in ue5-release-engine-test branch]
2021-09-09 06:04:13 -04:00
devin doucette
ce01b1fb44 DDC: Stopped loading build inputs for local execution when execution is disabled
#rb Zousar.Shaker
#rnx

#ROBOMERGE-SOURCE: CL 17358468 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v865-17346139)

[CL 17358490 by devin doucette in ue5-release-engine-test branch]
2021-08-30 16:38:36 -04:00
devin doucette
b575156c6f DDC: Added request barriers to the build job constructors
The barriers are needed to handle re-entrant AdvanceToState calls within the constructor of jobs with Blocking priority.

#rb Zousar.Shaker
#rnx

#ROBOMERGE-SOURCE: CL 17302241 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v861-17282326)

[CL 17302256 by devin doucette in ue5-release-engine-test branch]
2021-08-25 10:15:39 -04:00
devin doucette
275765055a DDC: Fixed build callback status when the policy allows querying the cache but not building
#rb Zousar.Shaker
#rnx

#ROBOMERGE-SOURCE: CL 17291662 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v861-17282326)

[CL 17291681 by devin doucette in ue5-release-engine-test branch]
2021-08-24 15:25:59 -04:00
devin doucette
ae7825b29c DDC: Added EBuildStatus::CacheKey and return EStatus::Ok from a build when no build or cache was requested
#rb Zousar.Shaker
#rnx
#preflight 611f304c008be900010bf006

#ROBOMERGE-SOURCE: CL 17247202 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v855-17104924)

[CL 17247215 by devin doucette in ue5-release-engine-test branch]
2021-08-20 10:04:41 -04:00
devin doucette
d5a632673b DDC: Fixed a build job assert caused by canceling at the wrong time
#rb Zousar.Shaker
#rnx

#ROBOMERGE-SOURCE: CL 17108375 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v854-17104634)

[CL 17108399 by devin doucette in ue5-release-engine-test branch]
2021-08-09 17:56:38 -04:00
devin doucette
d55593526e DDC: Add FBuildConfigContext::SetRequiredMemory() and forward the estimate to the build scheduler
#rb Zousar.Shaker
#rnx

#ROBOMERGE-SOURCE: CL 17088443 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)

[CL 17089009 by devin doucette in ue5-release-engine-test branch]
2021-08-06 17:53:47 -04:00
devin doucette
c67fcf1702 DDC: Split DerivedDataRequest.h and rename DerivedDataBuildPolicy.h
#rb none
#rnx

#ROBOMERGE-SOURCE: CL 17086960 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)

[CL 17086977 by devin doucette in ue5-release-engine-test branch]
2021-08-06 15:50:00 -04:00
devin doucette
bd60f3ef1b DDC: Removed ICacheFactory in favor of exported functions
FCacheBucket is now always exposed as FAnsiStringView.

#rb Zousar.Shaker
#rnx

#ROBOMERGE-SOURCE: CL 17084521 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)

[CL 17084549 by devin doucette in ue5-release-engine-test branch]
2021-08-06 12:53:08 -04:00
devin doucette
7ae4b64cd6 DDC: Removed IBuild::Load[Type] in favor of exported functions
#rb Zousar.Shaker
#rnx

#ROBOMERGE-SOURCE: CL 17083958 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)

[CL 17083970 by devin doucette in ue5-release-engine-test branch]
2021-08-06 11:58:24 -04:00
devin doucette
bfda02d384 DDC: Replaced FRequest with FRequestGroup to handle nested and chained requests more robustly
Any function that may create a request now has a IRequestOwner& parameter, and uses the Begin and End functions on the owner to manage the lifetime of any requests that it creates, as well as using End to invoke the completion callback for any request which has one.

The new FRequestBarrier may be used to block a group from being considered complete in a scope where more requests may be added to it.

#rb Matt.Peters
#rnx
#preflight 6109b5c403d303000144cce5
#preflight 610acf7103d30300016fda94

#ROBOMERGE-SOURCE: CL 17060470 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v850-17047176)

[CL 17060649 by devin doucette in ue5-release-engine-test branch]
2021-08-04 18:08:50 -04:00
devin doucette
58774329d7 DDC: Exposed the cache key and build status in the build complete callback params
#rb Zousar.Shaker
#rnx
#preflight 60f5c7a78da1560001c68086

#ROBOMERGE-SOURCE: CL 16888763 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v836-16769935)

[CL 16888787 by devin doucette in ue5-release-engine-test branch]
2021-07-19 16:49:25 -04:00
devin doucette
dd8df40e87 DDC: Fixed a crash when creating the build context fails
#rb Zousar.Shaker
#rnx

#ROBOMERGE-SOURCE: CL 16883576 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v836-16769935)

[CL 16883585 by devin doucette in ue5-release-engine-test branch]
2021-07-19 10:51:53 -04:00
devin doucette
0af9901545 DDC: Fixed race conditions in the build job
#rb Zousar.Shaker
#rnx

#ROBOMERGE-SOURCE: CL 16870676 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v836-16769935)

[CL 16870677 by devin doucette in ue5-release-engine-test branch]
2021-07-16 01:11:34 -04:00
devin doucette
449072c4e6 DDC: Added cpu trace scopes to the build job
#rb Zousar.Shaker
#rnx

#ROBOMERGE-SOURCE: CL 16864034 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v836-16769935)

[CL 16864068 by devin doucette in ue5-release-engine-test branch]
2021-07-15 12:58:07 -04:00
devin doucette
540937d6f0 DDC: Fixed errors with NO_LOGGING
#rb trivial
#rnx

#ROBOMERGE-SOURCE: CL 16863666 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v836-16769935)

[CL 16863682 by devin doucette in ue5-release-engine-test branch]
2021-07-15 12:30:05 -04:00