Commit Graph

27 Commits

Author SHA1 Message Date
Matt Peters
02ba0b5b70 ZenStoreWriter: Fix assertion "Trying to add package that is already pending" if the cooker commits a package a second time while the first commit of the package is queued for asynchronous completion aand has not yet reached CommitPackageInternal.
This is necessary because as of CL 20151503, the cooker can commit the same package multiple times if the first save failed due to a timeout.

#fyi CarlMagnus.Nordin
#rnx
#rb Per.Larsson
#preflight 62853ba90f0d5076dee74ced

[CL 20265738 by Matt Peters in ue5-main branch]
2022-05-18 14:49:11 -04:00
CarlMagnus Nordin
63b8cbc9dd AsyncLoading2: Implemented loading of optional package segments
#rnx
#preflight 62627548ac29fcf6a213fe33
#rb francis.hurteau

[CL 19900368 by CarlMagnus Nordin in ue5-main branch]
2022-04-25 07:37:07 -04:00
Per Larsson
ddbf9eb5d2 Zen - cook to custom project/oplog ID with -ZenStoreProject=<ProjectId> and -ZenStorePlatform=<OplogId>
#rb none
#rnx
#preflight 19755222

[CL 19804798 by Per Larsson in ue5-main branch]
2022-04-19 02:37:06 -04:00
Dan Thompson
cd73707429 Update package writers to gather package hashes internally, then have CookByTheBookFinished copy them to asset package datas.
Add iostore chunk hashes to the asset registry during cook.
Remove old hashing code, which was extensive. This moves the async writes in to UE::Task from the old AsyncWorkSequence().
#rb Matt.Peters
#rb Jeff.Roberts
#preflight 624b24a5f73c316f68303946

[CL 19611812 by Dan Thompson in ue5-main branch]
2022-04-04 13:26:27 -04:00
CarlMagnus Nordin
d4a06b766c ZenCOTF:
*Let the game be in full control of what is getting cooked
*Removed artificial delays when waiting for DDC operations
*No longer relies on all requests being urgent
#rnx
#preflight 620d2f80f8655681e051eded
#rb pj.kack

[CL 19032506 by CarlMagnus Nordin in ue5-main branch]
2022-02-17 04:54:33 -05:00
CarlMagnus Nordin
2abbea75a6 Remove Flush/WaitForPendingFlush calls when running COTF, they are no longer needed
#rb per.larsson
#preflight 61fbac05176256ec4f6d4e4b

[CL 18841496 by CarlMagnus Nordin in ue5-main branch]
2022-02-03 06:34:31 -05:00
francis hurteau
d262daac13 Added support to Loose File Package Writer to write multiple package output per input
Added similar support to the package store manifest
Zen Store support will be added separately
Added some extra data to linker object resources to be used by the iostore package optimizer to properly support optional package chunks.

Multiple cooking outputs is used to support "optional objects" which are editor data package exports that would be put in separate iostore containers than the normal cooked game content.

#jira UE-129801
#rb Matt.Peters, CarlMagnus.Nordin
#preflight 61e5d3bb873f2ea48f48166c

#ROBOMERGE-AUTHOR: francis.hurteau
#ROBOMERGE-SOURCE: CL 18639742 in //UE5/Release-5.0/... via CL 18639751 via CL 18639758
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v900-18638592)

[CL 18639762 by francis hurteau in ue5-main branch]
2022-01-18 09:56:23 -05:00
Marc Audy
0c3be2b6ad Merge Release-Engine-Staging to Test @ CL# 18240298
[CL 18241953 by Marc Audy in ue5-release-engine-test branch]
2021-11-18 14:37:34 -05:00
aurel cordonnier
fc542f6cfd Merge from Release-Engine-Staging @ 18081189 to Release-Engine-Test
This represents UE4/Main @18073326, Release-5.0 @18081140 and Dev-PerfTest @18045971

[CL 18081471 by aurel cordonnier in ue5-release-engine-test branch]
2021-11-07 23:43:01 -05:00
aurel cordonnier
a6e741e007 Merge from Release-Engine-Staging @ 17915896 to Release-Engine-Test
This represents UE4/Main @17911760, Release-5.0 @17915875 and Dev-PerfTest @17914035

[CL 17918595 by aurel cordonnier in ue5-release-engine-test branch]
2021-10-25 20:05:28 -04:00
aurel cordonnier
a12d56ff31 Merge from Release-Engine-Staging @ 17791557 to Release-Engine-Test
This represents UE4/Main @17774255, Release-5.0 @17791557 and Dev-PerfTest @17789485

[CL 17794212 by aurel cordonnier in ue5-release-engine-test branch]
2021-10-12 21:21:22 -04:00
matt peters
e27fb01763 CookedPackageWriter iterative builds - change to use FAssetRegistryState as the list of previous files. Load the AssetRegistry from the AssetRegistry.bin file even when using ZenStore for the package storage.
#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 17551332 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v870-17433530)

[CL 17551347 by matt peters in ue5-release-engine-test branch]
2021-09-17 09:32:48 -04:00
carlmagnus nordin
9f118fe8a4 Compile fixes for building UnrealPak in shipping config
#rnx
#rb pj.kack

#ROBOMERGE-OWNER: carlmagnus.nordin
#ROBOMERGE-AUTHOR: carlmagnus.nordin
#ROBOMERGE-SOURCE: CL 17536937 via CL 17536972 via CL 17536975 via CL 17536976
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v870-17433530)
#ROBOMERGE-CONFLICT from-shelf
#ROBOMERGE[STARSHIP]: UE5-Main

[CL 17537001 by carlmagnus nordin in ue5-release-engine-test branch]
2021-09-16 07:48:26 -04:00
carlmagnus nordin
a627591ffa IoStore: Changes to allow IoStore to run as part of UnrealPak instead of as a cmdlet
#rb per.larsson
#rnx

#ROBOMERGE-AUTHOR: carlmagnus.nordin
#ROBOMERGE-SOURCE: CL 17517167 via CL 17517189 via CL 17517191 via CL 17519703
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v870-17433530)
#ROBOMERGE[STARSHIP]: UE5-Main

[CL 17519714 by carlmagnus nordin in ue5-release-engine-test branch]
2021-09-15 10:39:23 -04:00
matt peters
54295195da CookOnTheFly - handle already-cooked packages from iterative builds.
#rb Per.Larsson
#rnx

#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 17508341 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v870-17433530)

[CL 17508358 by matt peters in ue5-release-engine-test branch]
2021-09-14 15:15:17 -04:00
matt peters
56274a7640 ZenStoreWriter - Make CommitPackage take an array of attachments; remove separate variables for TargetDomainDependencies and send them in as an attachment.
ZenStoreHttpClient - Add IsConnected.
#rb CarlMagnus.Nordin, Stefan.Boberg
#rnx

#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 17456396 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v870-17433530)
#ROBOMERGE[bot1]: dev-enginemerge-test

[CL 17456510 by matt peters in ue5-release-engine-test branch]
2021-09-08 07:04:39 -04:00
carlmagnus nordin
0a58a09ea1 Removed functionality to cook to an IoStore container file, it has been superseded by cooking to Zen.
#rb pj.kack
#rnx

#ROBOMERGE-SOURCE: CL 17401028 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v865-17346139)

[CL 17401029 by carlmagnus nordin in ue5-release-engine-test branch]
2021-09-02 07:53:29 -04:00
matt peters
d09fe9658b CookOnTheFlyServer: Create FLooseCookedPackageWriter and refactor UCookOnTheFlyServer to use ICookedPackageWriter in all cases rather than having a separate path for writing cooked files to disk.
#rb CarlMagnus.Nordin
#rnx

#ROBOMERGE-SOURCE: CL 17247861 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v855-17104924)

[CL 17247875 by matt peters in ue5-release-engine-test branch]
2021-08-20 10:57:45 -04:00
matt peters
dc0db90119 PackageStoreWriter: Split IPackageStoreWriter into IPackageWriter, IPackageCookWrite, IPackageStoreWriter.
We plan to implement an IPackageWriter for workspace domain editor files to reduce duplication in SavePackage, and implementations for that purpose do not need to handle cook functionality.
IOStore-specific functionality is not needed by the cooker, and is pushed into IPackageStoreWriter.
#rb Per.Larsson, CarlMagnus.Nordin
#rnx

#ROBOMERGE-SOURCE: CL 17247188 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v855-17104924)

[CL 17247191 by matt peters in ue5-release-engine-test branch]
2021-08-20 10:02:58 -04:00
matt peters
eb280403be PackageStore: Add IPackageStoreWriter::GetTargetDomainDependencies and FZenStoreHttpClient::ReadOplogAttachment.
#rb CarlMagnus.Nordin
#rnx

#ROBOMERGE-SOURCE: CL 17152232 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v855-17104924)

[CL 17152248 by matt peters in ue5-release-engine-test branch]
2021-08-12 10:51:31 -04:00
matt peters
64eb24ccec SavePackage PackageStoreWriter with EditorDomain: fix offsets written into the BulkDatas to be relative to the end of the Linker's archive as expected, rather than relative to some number of bytes before the end of the archive because we were copying Linker->Tell at a point before AppendAdditionalData and the Package Post Tag were written to the archive.
Also add the writing of AppendAdditionalData to a callback on the PackageStoreWriter, so that FVirtualizedUntypedBulkDatas can be separated out into separate archives in the same way FUntypedBaseData are.

#rnx
#rb Paul.Chipchase

#ROBOMERGE-SOURCE: CL 17015164 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v839-17012307)

[CL 17015166 by matt peters in ue5-release-engine-test branch]
2021-08-02 08:36:21 -04:00
per larsson
998e1d47b2 Iterative cooking support when using Zen loader/storage server
Zen: af7ff3f1c6

#rb CarlMagnus.Nordin, Stefan.Boberg
#jira none
#rnx

#ROBOMERGE-SOURCE: CL 16755977 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v835-16672529)

[CL 16755985 by per larsson in ue5-release-engine-test branch]
2021-06-23 09:01:23 -04:00
zousar shaker
c27625b056 Integrating //UE5/Dev-Cooker [at] 16678003 to //UE5/Main (Zousar.Shaker-YEG-0943-Quaternary)
Non-DevIteration_ShooterGame


#ROBOMERGE-SOURCE: CL 16678907 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v835-16672529)

[CL 16678928 by zousar shaker in ue5-release-engine-test branch]
2021-06-15 16:38:03 -04:00
aurel cordonnier
50944fd712 Merge UE5/RES @ 16162155 to UE5/Main
This represents UE4/Main @ 16130047 and Dev-PerfTest @ 16126156

[CL 16163576 by aurel cordonnier in ue5-main branch]
2021-04-29 19:32:06 -04:00
CarlMagnus Nordin
528602295d Added option to cook for iostore, allowing the new loader to be used with cooked data without having to stage first
#rnx
#rb pj.kack
#jira none

[CL 15387057 by CarlMagnus Nordin in ue5-main branch]
2021-02-11 08:52:34 -04:00