Commit Graph

34 Commits

Author SHA1 Message Date
CarlMagnus Nordin
210d67b30e Promoted Zen project id generation to FApp
#rb pj.kack
#preflight 628769032c34da686ac82442

[CL 20316798 by CarlMagnus Nordin in ue5-main branch]
2022-05-23 01:49:25 -04:00
Matt Peters
02ba0b5b70 ZenStoreWriter: Fix assertion "Trying to add package that is already pending" if the cooker commits a package a second time while the first commit of the package is queued for asynchronous completion aand has not yet reached CommitPackageInternal.
This is necessary because as of CL 20151503, the cooker can commit the same package multiple times if the first save failed due to a timeout.

#fyi CarlMagnus.Nordin
#rnx
#rb Per.Larsson
#preflight 62853ba90f0d5076dee74ced

[CL 20265738 by Matt Peters in ue5-main branch]
2022-05-18 14:49:11 -04:00
Matt Peters
ed761b4332 ZenStoreWriter: Remove spurious warnings when packages are committed twice during the cook because of reconsidering only-editoronly-referenced packages.
#rb Dan.Thompson
#rnx
#preflight 626863c5853fdb6fddd845a6

[CL 19929495 by Matt Peters in ue5-main branch]
2022-04-26 17:36:49 -04:00
CarlMagnus Nordin
63b8cbc9dd AsyncLoading2: Implemented loading of optional package segments
#rnx
#preflight 62627548ac29fcf6a213fe33
#rb francis.hurteau

[CL 19900368 by CarlMagnus Nordin in ue5-main branch]
2022-04-25 07:37:07 -04:00
Per Larsson
ddbf9eb5d2 Zen - cook to custom project/oplog ID with -ZenStoreProject=<ProjectId> and -ZenStorePlatform=<OplogId>
#rb none
#rnx
#preflight 19755222

[CL 19804798 by Per Larsson in ue5-main branch]
2022-04-19 02:37:06 -04:00
Dan Thompson
cd73707429 Update package writers to gather package hashes internally, then have CookByTheBookFinished copy them to asset package datas.
Add iostore chunk hashes to the asset registry during cook.
Remove old hashing code, which was extensive. This moves the async writes in to UE::Task from the old AsyncWorkSequence().
#rb Matt.Peters
#rb Jeff.Roberts
#preflight 624b24a5f73c316f68303946

[CL 19611812 by Dan Thompson in ue5-main branch]
2022-04-04 13:26:27 -04:00
wouter burgers
3217b35e31 Fixed 'files(s) -> file(s)' typo's in code and comments.
#jira none
#review-19507050
#preflight 623dbe0b9f11d3c739504676

[CL 19524850 by wouter burgers in ue5-main branch]
2022-03-28 05:50:13 -04:00
CarlMagnus Nordin
d4a06b766c ZenCOTF:
*Let the game be in full control of what is getting cooked
*Removed artificial delays when waiting for DDC operations
*No longer relies on all requests being urgent
#rnx
#preflight 620d2f80f8655681e051eded
#rb pj.kack

[CL 19032506 by CarlMagnus Nordin in ue5-main branch]
2022-02-17 04:54:33 -05:00
CarlMagnus Nordin
2abbea75a6 Remove Flush/WaitForPendingFlush calls when running COTF, they are no longer needed
#rb per.larsson
#preflight 61fbac05176256ec4f6d4e4b

[CL 18841496 by CarlMagnus Nordin in ue5-main branch]
2022-02-03 06:34:31 -05:00
francis hurteau
d262daac13 Added support to Loose File Package Writer to write multiple package output per input
Added similar support to the package store manifest
Zen Store support will be added separately
Added some extra data to linker object resources to be used by the iostore package optimizer to properly support optional package chunks.

Multiple cooking outputs is used to support "optional objects" which are editor data package exports that would be put in separate iostore containers than the normal cooked game content.

#jira UE-129801
#rb Matt.Peters, CarlMagnus.Nordin
#preflight 61e5d3bb873f2ea48f48166c

#ROBOMERGE-AUTHOR: francis.hurteau
#ROBOMERGE-SOURCE: CL 18639742 in //UE5/Release-5.0/... via CL 18639751 via CL 18639758
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v900-18638592)

[CL 18639762 by francis hurteau in ue5-main branch]
2022-01-18 09:56:23 -05:00
carlmagnus nordin
fe4d9e9c89 Fixed package store manifest not working correctly for iterative cooks
#rnx
#rb per.larsson

#ROBOMERGE-AUTHOR: carlmagnus.nordin
#ROBOMERGE-SOURCE: CL 18465551 in //UE5/Release-5.0/... via CL 18465553
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)

[CL 18465554 by carlmagnus nordin in ue5-release-engine-test branch]
2021-12-15 03:42:08 -05:00
zousar shaker
0832797d28 Ensure that Zen ProjectIds are unique for projects with the same name in differing roots (or in the same root) while attempting to maintain some easily recognizable desriptors for the project.
Staged builds are explicitly assigned a project id via the persistent UECommandline.txt file.  This works on mobile/consoles, but isn't implemented for Windows/Linux/Mac, so staged builds for those platforms don't have a persistent assignment of project id and instead rely on the launch by the editor including the appropriate commandlines.
Unstaged builds on Windows/Mac/Linux now determine project id independently after they have determined the project file path.  This happens after the project file has been determined and has had case corrected.  This ensures it is different/unique for multiple blueprint projects that share the same executable.
Fix a bug where second cook in editor to zen would stall because the queue thinks all additions are finished due to a boolean that isn't reset.

#rb devin.doucette
#rb per.larsson
#preflight 61b39a35a2562c8b1c40f81e

#ROBOMERGE-AUTHOR: zousar.shaker
#ROBOMERGE-SOURCE: CL 18432317 in //UE5/Release-5.0/... via CL 18435404
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v897-18405271)

[CL 18435895 by zousar shaker in ue5-release-engine-test branch]
2021-12-10 18:06:39 -05:00
Marc Audy
0c3be2b6ad Merge Release-Engine-Staging to Test @ CL# 18240298
[CL 18241953 by Marc Audy in ue5-release-engine-test branch]
2021-11-18 14:37:34 -05:00
aurel cordonnier
fc542f6cfd Merge from Release-Engine-Staging @ 18081189 to Release-Engine-Test
This represents UE4/Main @18073326, Release-5.0 @18081140 and Dev-PerfTest @18045971

[CL 18081471 by aurel cordonnier in ue5-release-engine-test branch]
2021-11-07 23:43:01 -05:00
matt peters
b9be0036df #jira UE-132874
#jira UE-132939
LargeMemoryArchive::ReleaseOwnership() sets TotalSize() to 0, so TotalSize must be called before calling ReleaseOwnership. Calling them both as arguments into a function was succeeding on some compilers and failing on others due to compiler optimization differences.
#rb Zousar.Shaker
#rnx
#[fyi] Francis.Hurteau

#ushell-cherrypick of 17947411 by Matt.Peters

#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 18008712 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v885-17909292)

[CL 18008728 by matt peters in ue5-release-engine-test branch]
2021-11-01 17:20:22 -04:00
aurel cordonnier
a6e741e007 Merge from Release-Engine-Staging @ 17915896 to Release-Engine-Test
This represents UE4/Main @17911760, Release-5.0 @17915875 and Dev-PerfTest @17914035

[CL 17918595 by aurel cordonnier in ue5-release-engine-test branch]
2021-10-25 20:05:28 -04:00
aurel cordonnier
a12d56ff31 Merge from Release-Engine-Staging @ 17791557 to Release-Engine-Test
This represents UE4/Main @17774255, Release-5.0 @17791557 and Dev-PerfTest @17789485

[CL 17794212 by aurel cordonnier in ue5-release-engine-test branch]
2021-10-12 21:21:22 -04:00
matt peters
763b1ad106 ZenStoreHttpClient: Expose connection failure to the Oplog owner to handle.
#rb Per.Larsson
#rnx

#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 17661528 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v875-17642767)

[CL 17661553 by matt peters in ue5-release-engine-test branch]
2021-09-29 10:51:17 -04:00
carlmagnus nordin
51c9794dea PackageStore refactor
* Decouple container concept from IoDispatcher
* Decoiuple PackageStore implementation from AsyncLoading2
* Restore ucas unmount fix that got kist when merrging from UE4
* Fix packages being left in the PackageStiore even after unmounting contaiiners

#rnx
#rb pj.kack, per.larsson
#preflight 61520cc52afc2d0001146ce7

#ROBOMERGE-AUTHOR: carlmagnus.nordin
#ROBOMERGE-SOURCE: CL 17641845 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v874-17637634)

[CL 17642353 by carlmagnus nordin in ue5-release-engine-test branch]
2021-09-28 04:00:33 -04:00
zousar shaker
96d0784963 Allow ZenServer to be autolaunched by editor use cases that require its presence without insisting that it always be autolaunched.
#rb matt.peters
#preflight 614b4150ee16e20001b146e3

#ROBOMERGE-AUTHOR: zousar.shaker
#ROBOMERGE-SOURCE: CL 17597162 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v871-17566257)

[CL 17597204 by zousar shaker in ue5-release-engine-test branch]
2021-09-22 13:00:17 -04:00
matt peters
e27fb01763 CookedPackageWriter iterative builds - change to use FAssetRegistryState as the list of previous files. Load the AssetRegistry from the AssetRegistry.bin file even when using ZenStore for the package storage.
#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 17551332 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v870-17433530)

[CL 17551347 by matt peters in ue5-release-engine-test branch]
2021-09-17 09:32:48 -04:00
stefan boberg
e3eeb47216 Trivial logging fix: 'packges' -> 'packages'
#ROBOMERGE-AUTHOR: stefan.boberg
#ROBOMERGE-SOURCE: CL 17534852 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v870-17433530)

[CL 17534855 by stefan boberg in ue5-release-engine-test branch]
2021-09-16 03:18:06 -04:00
carlmagnus nordin
a627591ffa IoStore: Changes to allow IoStore to run as part of UnrealPak instead of as a cmdlet
#rb per.larsson
#rnx

#ROBOMERGE-AUTHOR: carlmagnus.nordin
#ROBOMERGE-SOURCE: CL 17517167 via CL 17517189 via CL 17517191 via CL 17519703
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v870-17433530)
#ROBOMERGE[STARSHIP]: UE5-Main

[CL 17519714 by carlmagnus nordin in ue5-release-engine-test branch]
2021-09-15 10:39:23 -04:00
matt peters
54295195da CookOnTheFly - handle already-cooked packages from iterative builds.
#rb Per.Larsson
#rnx

#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 17508341 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v870-17433530)

[CL 17508358 by matt peters in ue5-release-engine-test branch]
2021-09-14 15:15:17 -04:00
matt peters
bd9ec84b79 ZenStoreWriter: Make oplog keys case insensitive, to remove the possiblity of two oplog entries for the same packagename.
#rb CarlMagnus.Nordin
#rnx

#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 17458021 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v870-17433530)

[CL 17458033 by matt peters in ue5-release-engine-test branch]
2021-09-08 09:51:13 -04:00