Commit Graph

27 Commits

Author SHA1 Message Date
Matt Peters
81a3ae5b81 Cooker: Multiprocess support: Remove some deprecated code and move some out of place functions that should not be executed on CookWorker into the proper calling function that will be called on the director only.
#rb Zousar.Shaker
#rnx
#preflight 629135e11b686da53f41b463

[CL 20398652 by Matt Peters in ue5-main branch]
2022-05-27 16:48:59 -04:00
devin doucette
b2a07ea03e DDC: Merge from UE5/Main
#preflight 6288ff678828ea88c8af7034
#preflight 628ab5d93246d5019db76ed2
#rb none
#rnx

#ROBOMERGE-OWNER: devin.doucette
#ROBOMERGE-AUTHOR: Devin.Doucette
#ROBOMERGE-SOURCE: CL 20353148 via CL 20353832 via CL 20353839
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v948-20297126)

[CL 20355348 by devin doucette in ue5-main branch]
2022-05-24 16:40:25 -04:00
Matt Peters
5e4fdf3eaf CookOnTheFlyServer: Combine duplicated variables from cook modes to reduce complexity, avoid the need to check for null, and prepare for the addition of a CookWorker mode.
#rb Zousar.Shaker
#rnx
#fyi CarlMagnus.Nordin, PJ.Kack
#preflight 6287f0582c34da686af615eb

[CL 20301986 by Matt Peters in ue5-main branch]
2022-05-20 16:12:27 -04:00
matt peters
51e802ef74 EditorDomain: Reduce costs from TrySavePackage into the EditorDomain on iterative cooks on the buildfarm (which empties its local ZenServer before each iterative cook). Do this by skipping the save of any package that is loaded before we have heard back from the upstream about whether the upstream has the package.
#preflight 61dca80e8f44ab439b3dafe8
#rb Zousar.Shaker
#rnx

#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 18566222 in //UE5/Release-5.0/... via CL 18566238
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)

[CL 18566251 by matt peters in ue5-release-engine-test branch]
2022-01-10 17:35:14 -05:00
matt peters
409316ded9 #jira UE-138657
CookRequestClusters: suppress the timeout warning on reading vertex data if we're not reading vertexes from the oplog. In the non-oplog case, the vertex data is empty and is just passed through on an async thread rather than waiting for the oplog. It's unknown why that async thread could take up to 10 seconds (hitting a breakpoint maybe), but we don't need a warning about it.
#rb None, Trivial
#rnx
#preflight 61d89964c65e66487b381d9c

#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 18549727 in //UE5/Release-5.0/... via CL 18549743
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)

[CL 18549758 by matt peters in ue5-release-engine-test branch]
2022-01-07 15:00:30 -05:00
devin doucette
48f68493ef DDC: Removed the Put/Get compatibility functions from ICache
#rb Zousar.Shaker
#rnx
#preflight 61d49a971a3fd09dcbc87978

#ROBOMERGE-AUTHOR: devin.doucette
#ROBOMERGE-SOURCE: CL 18511490 in //UE5/Release-5.0/... via CL 18511542
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)

[CL 18511622 by devin doucette in ue5-release-engine-test branch]
2022-01-04 14:38:48 -05:00
matt peters
1d007d263b #jira UE-137758
Cooker: Fix FRequestCluster's search of dependencies to consider CoreRedirects, so that it checks for existence of the redirected file rather than the original reference which will likely no longer exist.
#preflight 61d3af1c5d522c9b407bbc11

#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 18505900 in //UE5/Release-5.0/... via CL 18505902
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)

[CL 18505903 by matt peters in ue5-release-engine-test branch]
2022-01-04 00:39:31 -05:00
matt peters
8d87746df1 EditorDomain: Use the EditorDomain's cache of EditorDomainHash instead of recalculating it every time it is needed for BulkData and TargetDomain keys.
#rb Zousar.Shaker
#rnx

#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 18396652 in //UE5/Release-5.0/... via CL 18396659
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v896-18170469)

[CL 18396670 by matt peters in ue5-release-engine-test branch]
2021-12-07 13:12:55 -05:00
matt peters
4ccf1ddd5f CookRequestCluster: Fix TryAddPackageDataByStandardFileName not handling missing extension; handling the missing extension is needed in CookOnTheFly.
#rb Devin.Doucette
#rnx

#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 18384903 in //UE5/Release-5.0/... via CL 18384907
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v896-18170469)

[CL 18384919 by matt peters in ue5-release-engine-test branch]
2021-12-06 13:24:26 -05:00
matt peters
61c3178b58 Cooker: Delete PackageNameCache; FPackageDatas now covers its functionality and removing it saves cputime and memory from the duplicate maps of packagename to filename.
Make construction of new FPackageData threadsafe.
#rb Zousar.Shaker
#rnx

#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 18377408 in //UE5/Release-5.0/... via CL 18377410
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)

[CL 18377411 by matt peters in ue5-release-engine-test branch]
2021-12-04 16:14:37 -05:00
matt peters
6e65166ead #jira UE-124623
EDLCookChecker: suppress missing export warnings for packages that were iteratively skipped and thus do have the exports but did not declare them.

#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 18288814 in //UE5/Release-5.0/... via CL 18288852
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)

[CL 18288863 by matt peters in ue5-release-engine-test branch]
2021-11-24 16:20:12 -05:00
aurel cordonnier
fc542f6cfd Merge from Release-Engine-Staging @ 18081189 to Release-Engine-Test
This represents UE4/Main @18073326, Release-5.0 @18081140 and Dev-PerfTest @18045971

[CL 18081471 by aurel cordonnier in ue5-release-engine-test branch]
2021-11-07 23:43:01 -05:00
aurel cordonnier
a6e741e007 Merge from Release-Engine-Staging @ 17915896 to Release-Engine-Test
This represents UE4/Main @17911760, Release-5.0 @17915875 and Dev-PerfTest @17914035

[CL 17918595 by aurel cordonnier in ue5-release-engine-test branch]
2021-10-25 20:05:28 -04:00
aurel cordonnier
a12d56ff31 Merge from Release-Engine-Staging @ 17791557 to Release-Engine-Test
This represents UE4/Main @17774255, Release-5.0 @17791557 and Dev-PerfTest @17789485

[CL 17794212 by aurel cordonnier in ue5-release-engine-test branch]
2021-10-12 21:21:22 -04:00
matt peters
54295195da CookOnTheFly - handle already-cooked packages from iterative builds.
#rb Per.Larsson
#rnx

#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 17508341 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v870-17433530)

[CL 17508358 by matt peters in ue5-release-engine-test branch]
2021-09-14 15:15:17 -04:00
matt peters
ead28b557b Load/Save BuildDefinitionList to the TargetDomain and EditorDomain.
#rb Zousar.Shaker
#rnx

#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 17458074 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v870-17433530)

[CL 17458084 by matt peters in ue5-release-engine-test branch]
2021-09-08 10:01:09 -04:00
matt peters
f2207992cf Cooker - DLC - Fix spurious cook error when DLC is cooked that references a nevercook package. The test for uncooked referenced engine content needs to come after removing packages that were skipped in the earlier base cook and are still skipped now in the DLC cook.
#rb None, Trivial
#rnx

#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 17456719 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v870-17433530)

[CL 17456757 by matt peters in ue5-release-engine-test branch]
2021-09-08 07:44:37 -04:00
matt peters
c530e3a294 EditorDomain: Allow EditorDomain packages to propagate upstream. Bulk download EditorDomain packages from remote. Make EditorDomain archive opens a blocking get.
#rb Devin.Doucette
#rnx

#ROBOMERGE-SOURCE: CL 17407292 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v865-17346139)

[CL 17407301 by matt peters in ue5-release-engine-test branch]
2021-09-02 15:22:21 -04:00
matt peters
8b3738fbd8 #jira UE-125064
Silence false positive static analysis warning C6011: Dereferencing NULL pointer 'Variable'
#rb None, trivial
#rnx

#ROBOMERGE-SOURCE: CL 17401651 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v865-17346139)

[CL 17401663 by matt peters in ue5-release-engine-test branch]
2021-09-02 08:40:51 -04:00
matt peters
daba8c3dd9 Cooker: Fix softlock in the cooker when cooking with no hard references, i.e. -cooksinglepackagenoref.
#rb Zousar.Shaker
#rnx

#ROBOMERGE-SOURCE: CL 17393153 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v865-17346139)

[CL 17393165 by matt peters in ue5-release-engine-test branch]
2021-09-01 17:47:33 -04:00
matt peters
f00e13c26c Change cleanbuild -> fullbuild to match our public-facing terminology for clean aka full aka noniterative builds.
#rb Zousar.Shaker
#rnx

#ROBOMERGE-SOURCE: CL 17378218 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v865-17346139)

[CL 17378244 by matt peters in ue5-release-engine-test branch]
2021-08-31 17:28:15 -04:00
matt peters
79b61a9d91 Cooker: Hybrid Iterative cooks
Add the concept of pre-exploration of dependencies - searching assetregistry or target domain dependencies of the set of requested packages up front, to find all packages that will be cooked and allow us to do asynchronous and batch operations on them.
Convert FRequestCluster into an FPackageData container, holding packages in a substate of EPackageState::Request. This allows us to create a new RequestClusters when we encounter a package that was not found in the initial discovery.
Add hybriditerative mode for iterative cooks; this uses targetdomain keys to check for packages that exist from the previous cook, rather tracing the dependency graph to find all packages that transitively reference packages with modified guids.
Modify AssetRegistryGenerator to use a TMap of iteratively loaded packages that can be edied as packages cook, rather than using a const previous state; this supports hybrid iterative where we do not calculate the list of iterative packages until we encounter them in a request cluster.

#ROBOMERGE-SOURCE: CL 17373295 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v865-17346139)

[CL 17373304 by matt peters in ue5-release-engine-test branch]
2021-08-31 14:20:48 -04:00
matt peters
d09fe9658b CookOnTheFlyServer: Create FLooseCookedPackageWriter and refactor UCookOnTheFlyServer to use ICookedPackageWriter in all cases rather than having a separate path for writing cooked files to disk.
#rb CarlMagnus.Nordin
#rnx

#ROBOMERGE-SOURCE: CL 17247861 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v855-17104924)

[CL 17247875 by matt peters in ue5-release-engine-test branch]
2021-08-20 10:57:45 -04:00
matt peters
dc0db90119 PackageStoreWriter: Split IPackageStoreWriter into IPackageWriter, IPackageCookWrite, IPackageStoreWriter.
We plan to implement an IPackageWriter for workspace domain editor files to reduce duplication in SavePackage, and implementations for that purpose do not need to handle cook functionality.
IOStore-specific functionality is not needed by the cooker, and is pushed into IPackageStoreWriter.
#rb Per.Larsson, CarlMagnus.Nordin
#rnx

#ROBOMERGE-SOURCE: CL 17247188 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v855-17104924)

[CL 17247191 by matt peters in ue5-release-engine-test branch]
2021-08-20 10:02:58 -04:00
matt peters
65926948cb Cooking: Create TargetDomain dependencies and key from the PackageBuildDependencyTracker and the AssetRegistry.
#rb Zousar.Shaker
#rnx

#ROBOMERGE-SOURCE: CL 17175833 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v855-17104924)

[CL 17175851 by matt peters in ue5-release-engine-test branch]
2021-08-16 10:13:06 -04:00