60 Commits

Author SHA1 Message Date
dan engelbrecht
02d1ce55e7 Store cooker computer remote ips in .projectstore
#rb zousar.shaker

[CL 26035338 by dan engelbrecht in 5.3 branch]
2023-06-16 01:10:08 -04:00
dan engelbrecht
790e6a8b46 Make sure we can gracefully destroy a FZenStoreWriter instance even if EndCook() is not called.
#jira UE-188177
#rb zousar.shaker

[CL 25888492 by dan engelbrecht in ue5-main branch]
2023-06-09 03:24:09 -04:00
Matt Peters
88d33c5f37 Cooker and SavePackage: Move calls to BeginCacheForCookedPlatformData and ClearCachedCookedPlatformData from SavePackage into the cooker. Moving the Clear calls allows us to clear only after repeated SavePackage attempts from the diffonly cooker have been completed, preventing a break in contract by having objects being saved after Clear is called. Moving the Begin calls is necessary to track which objects need Clear called on them.
#jira FORT-613920
#rn Minor, Cooking
#rb Zousar.Shaker

[CL 25851419 by Matt Peters in ue5-main branch]
2023-06-07 13:56:57 -04:00
zousar shaker
a3bf9945d7 Ensure that multi-process cook doesn't cause workers to delete the oplog created by the director when using ZenStoreWriter.
#rb dan.engelbrecht

[CL 25644245 by zousar shaker in ue5-main branch]
2023-05-26 12:01:58 -04:00
zousar shaker
b0d5721dde Initial implementation of DiffOnly mode for ZenServer as Cooked Output Store. Lacking detailed header diffing, but stubbed in for future addition. When using ZenServer as Cooked Output Store the packages use the ZenPackageSummary header format instead of the PackageFileSummary header format. This is how data is stored in ZenServer and the format it is when fetched. Data produced in process is transformed into this format ahead of diffing.
The code for parsing a view of the ZenPackageSummary has been extracted from AsyncLoading2.cpp into a private h/cpp and is possible to use from the header diffing code when printing out detailed diffs, but is not used yet.

#rb matt.peters
#rb pj.kack
#jira UE-175237

[CL 25549055 by zousar shaker in ue5-main branch]
2023-05-19 14:43:02 -04:00
dan engelbrecht
96500c25d1 Automatically detect a if a staged target is using zenstore without pak and configure zenserver host and project id.
#jira UE-144425
#rb zousar.shaker pj.kack
#preflight 64427d5b4944ef78923c85b6

[CL 25144415 by dan engelbrecht in ue5-main branch]
2023-04-21 08:56:22 -04:00
carlmagnus nordin
ecfc01c258 ZenLoader: Include preload dependencies in package headers instead of relying on predetermined export bundle dependencies. This means that each package can be treated in isolation when staging. Also removes the need to keep track of export and bundle counts in the package store which saves some memory.
#preflight 642e8838441e8b2f5506dbb4
#rb pk.kack,per.larsson,francis.hurteau

[CL 24944233 by carlmagnus nordin in ue5-main branch]
2023-04-06 10:50:30 -04:00
dan engelbrecht
db5e568754 Game client no longer needs -zenstoreproject or -zenstorehost arguments to correctly run with a COTF server which uses Zen.
Game client now accepts only -filehostip for connection to a CTOF server, -cookonthefly is no longer necessary and is ignored.
Game client will now receive ProjectName (ProjectId), Platform (OplogId), Zen server host name and port from COTF server if running with zenstore.
Fixed issues with generating the ProjectId from a path where drive letters where lower case.

#rb zousar.shaker pj.kack
#preflight

[CL 24889513 by dan engelbrecht in ue5-main branch]
2023-04-03 01:51:40 -04:00
carlmagnus nordin
024df6d65c ZenCook: Added support for file regions
#preflight 64269c8491589478cdeb2a6f
#rb pj.kack
#rnx

[CL 24869761 by carlmagnus nordin in ue5-main branch]
2023-03-31 08:36:14 -04:00
carlmagnus nordin
58b7c9404b IoStore: Refactor to prepare for staging without a global optimization pass
* Moved package redirection out of the package store optimizer
* Moved iocontainer header creation out of the package store optimizer
* Moved package load order sorting out of the package store optimizer

#preflight 641d8dab803cb466e83a05dc
#rb pj.kack
#rnx

[CL 24778887 by carlmagnus nordin in ue5-main branch]
2023-03-24 10:06:23 -04:00
carlmagnus nordin
b99a74ba54 ZenCook: Workaround for not being able to reliably start the zen server from UAT
#preflight 641ac64f691c5ebc152d9ed9
#rb dan.engelbrecht
#rnx

[CL 24745405 by carlmagnus nordin in ue5-main branch]
2023-03-22 07:02:26 -04:00
carlmagnus nordin
30c0f28fc0 ZenCook: Merge the oplog and package store manifest. Write an oplog instead of a package store manifest when cooking to loose files and read the oplog instead of the package store manifest to find cooked files in UAT when staging from Zen.
#rb per.larsson
#preflight 640efc9bc41a0a2a7799d470
#rnx

[CL 24632735 by carlmagnus nordin in ue5-main branch]
2023-03-14 06:43:23 -04:00
carlmagnus nordin
641eb5b626 ZenCook: Added support for optional package segments
#jira UE-148021
#preflight 63f365ace832ea950a6217e4
#rb per.larsson

[CL 24343344 by carlmagnus nordin in ue5-main branch]
2023-02-21 13:03:12 -05:00
Matt Peters
fa59e5fba4 MPCook: ZenStoreWriter: Replicate ZenStoreWriter artifacts data.
#rnx
#rb Zousar.Shaker
#preflight 63dd7dd85c0ce8f11b71f405

[CL 23999102 by Matt Peters in ue5-main branch]
2023-02-03 17:38:06 -05:00
pj kack
d3da578b56 ZenCook: Add async write of additional files like .ufont to the sandbox directory before compressing for Zen.
Fix problem of missing additional files in .pak files when packaging based on a ZenCook.

#jira UE-175144
#rb carlmagnus.nordin,per.larsson,zousar.shaker
#rnx
#preflight 63d92560f626715201d687ce

[CL 23945129 by pj kack in ue5-main branch]
2023-02-01 03:19:39 -05:00
carlmagnus nordin
9f028b8744 ZenCook: Remove unused fields "guid" and "disksize" from oplog
#preflight 63d8ede38505ea6b1f7e96b6
#rb per.larsson

[CL 23927486 by carlmagnus nordin in ue5-main branch]
2023-01-31 10:39:54 -05:00
Matt Peters
5d2f838486 Cooker: MPCook: Replicate package hashes in LooseCookedPackageWriter.
#rb Zousar.Shaker
#rnx
#preflight 63d431bc5428dc67b168e481

[CL 23899261 by Matt Peters in ue5-main branch]
2023-01-28 14:33:10 -05:00
Matt Peters
178d5856dc MPCook - PackageWriter and IOStore API changes to support MPCook.
Add FCookInfo argument to BeginCook and EndCook, the same data passed to initialize. It is provided in Begin/End as well since Writers need to consider it for Begin/End actions.
Add WriteMPCookMessageForPackage/TryReadMPCookMessageForPackage functions for replicating information from PackageWriters from CookWorker to CookDirector.
Add WritePackage/TryReadPackage to Manifest. Some extra information is collected when MPCook is active to support this replication.
Add CompactBinary serialization functions to FIoChunkId.
#rbx
#preflight 63d1c62a45795e7d5c164b0e

[CL 23872901 by Matt Peters in ue5-main branch]
2023-01-26 14:32:12 -05:00
Dan Thompson
9630f87172 Removing storing the hashes of chunk types other than bulk and external file, because Zen stores packed data before its hit by the PackageOptimizer under the same chunk id, so it doesn't match the data that ends up in the iostore containers.
#rb matt.peters
#preflight 63cad462243c2d811953ae52

[CL 23793142 by Dan Thompson in ue5-main branch]
2023-01-20 13:06:32 -05:00
dan engelbrecht
e018c47697 Add marker file for project store oplog in Saved/Cooked/<platform> folder so Zen can check if an oplog should be garbage collected.
Companion Zen PR: https://github.com/EpicGames/zen/pull/209

#preflight 63971e55cb2317695e180344

[CL 23510788 by dan engelbrecht in ue5-main branch]
2022-12-14 07:18:38 -05:00
matt peters
daff9f8c5d EditorDomain: Implement support for PackageTrailers in the same package as BulkData in a regular BulkData section for PackageWriters, so that EditorDomain can have both references to BulkData in a WorkspaceDomain PackageTrailer and embedded BulkDatas that needed to be resaved.
Because the PackageFooter comes before the PackageTrailer, we need to change accounting for the PackageFooter in PackageWriters to include adjusting the VirtualExportsFileOffset that is passed into the PackageTrailer.
#rb Paul.Chipchase
#rnx
#preflight 638cbb1135192facc13f83e6

[CL 23386395 by matt peters in ue5-main branch]
2022-12-04 12:54:00 -05:00
carlmagnus nordin
ffd1e2abfb ZenCook: Use the oplog for package store entries instead of writing a container header
#preflight 63720da8ee4d25f90aacf4dd
#rb per.larsson

[CL 23116204 by carlmagnus nordin in ue5-main branch]
2022-11-14 06:35:53 -05:00
dan engelbrecht
2f8823f7dc Pass along path to project file when zen server is running locally so we can detect when it is removed and garbage-collect any projectstore entries on the zen side.
Allow Cook to remote Zen, if we detect that we are targetting a zenservice on a different machine we change the local file references to attachment based references.
#rb stefan.boberg
#preflight 63491c31ad0f7e2f20230b39

[CL 22525021 by dan engelbrecht in ue5-main branch]
2022-10-14 08:17:43 -04:00
dan engelbrecht
502ded405a Make sure we wait for any async commits in ZenStoreWriter to complete before sending EndCook.
Previous implementation only waited for the commit to be picked up by the worker thread.
Simplified CommitQueue event logic.
#rb zousar.shaker per.larsson
#preflight none

[CL 22364055 by dan engelbrecht in ue5-main branch]
2022-10-05 17:41:51 -04:00
Matt Peters
ec493e4052 Multiprocess Cook - Add CookWorkerMode and IWorkerRequests interface.
#rb Zousar.Shaker
#rn Cooking, Minor
#preflight 62b4ff53970a8db3373f6e3c
#robomerge EngineMerge

[CL 20807998 by Matt Peters in ue5-main branch]
2022-06-24 08:14:11 -04:00