67 Commits

Author SHA1 Message Date
zousar shaker
b957ae9353 Convert ue.projectstore file to json.
#rb dan.engelbrecht

[CL 32513882 by zousar shaker in 5.4 branch]
2024-03-26 12:46:12 -04:00
zousar shaker
775e6495fc Change naming of .projectstore to ue.projectstore to avoid having hidden by default status on Mac/Linux.
#rb dan.engelbrecht

[CL 31054882 by zousar shaker in 5.4 branch]
2024-01-31 13:00:16 -05:00
zousar shaker
98800ff6a4 When writing file manifest to zenstore, don't attempt to set both the "data" field on a file entry and the "serverpath" field on the file entry, as zenserver will treat the two as mutually exclusive, and setting the "data" field will mean we no loger get "serverpath" out of zenserver. This causes issues when doing an iterative cook and trying to read the file manifest back from zenserver.
#rb matt.peters

[CL 26980097 by zousar shaker in ue5-main branch]
2023-08-09 19:55:19 -04:00
bob tellez
de0ac0a5c5 #UE Only assert that you have package data to write when you are intending to write data or computehash. The new SkipSave cook option does not write data intentionally
[FYI] Matt.Peters

[CL 26837731 by bob tellez in ue5-main branch]
2023-08-03 23:00:11 -04:00
Matt Peters
13643c4f10 ZenWriter: Log the fatal assertion about invalid CommitPackage call on the CommitPackage thread in addition to the async thread so it is easier to debug.
#rb Zousar.Shaker
#rnx

[CL 26355924 by Matt Peters in ue5-main branch]
2023-06-30 17:07:55 -04:00
Matt Peters
ec9fef914b Cooking: Iterative: Add -iterativevalidate mode that diffs iteratively unmodified previous packages against their resaved version.
#rb Zousar.Shaker
#rnx

[CL 26327591 by Matt Peters in ue5-main branch]
2023-06-29 16:40:47 -04:00
zousar shaker
edaa3a7983 Ensure that when a package is saved multiple times (eg: when doing determinism testing) that we don't get into a situation where we attempt to save it a second time while it is still being asynchronously written the first time.
#rb matt.peters

[CL 26153970 by zousar shaker in ue5-main branch]
2023-06-21 13:06:10 -04:00
dan engelbrecht
20799c9f48 Store cooker computer remote ips in .projectstore
#rb zousar.shaker

[CL 26006400 by dan engelbrecht in ue5-main branch]
2023-06-15 04:51:13 -04:00
dan engelbrecht
790e6a8b46 Make sure we can gracefully destroy a FZenStoreWriter instance even if EndCook() is not called.
#jira UE-188177
#rb zousar.shaker

[CL 25888492 by dan engelbrecht in ue5-main branch]
2023-06-09 03:24:09 -04:00
Matt Peters
88d33c5f37 Cooker and SavePackage: Move calls to BeginCacheForCookedPlatformData and ClearCachedCookedPlatformData from SavePackage into the cooker. Moving the Clear calls allows us to clear only after repeated SavePackage attempts from the diffonly cooker have been completed, preventing a break in contract by having objects being saved after Clear is called. Moving the Begin calls is necessary to track which objects need Clear called on them.
#jira FORT-613920
#rn Minor, Cooking
#rb Zousar.Shaker

[CL 25851419 by Matt Peters in ue5-main branch]
2023-06-07 13:56:57 -04:00
zousar shaker
a3bf9945d7 Ensure that multi-process cook doesn't cause workers to delete the oplog created by the director when using ZenStoreWriter.
#rb dan.engelbrecht

[CL 25644245 by zousar shaker in ue5-main branch]
2023-05-26 12:01:58 -04:00
zousar shaker
b0d5721dde Initial implementation of DiffOnly mode for ZenServer as Cooked Output Store. Lacking detailed header diffing, but stubbed in for future addition. When using ZenServer as Cooked Output Store the packages use the ZenPackageSummary header format instead of the PackageFileSummary header format. This is how data is stored in ZenServer and the format it is when fetched. Data produced in process is transformed into this format ahead of diffing.
The code for parsing a view of the ZenPackageSummary has been extracted from AsyncLoading2.cpp into a private h/cpp and is possible to use from the header diffing code when printing out detailed diffs, but is not used yet.

#rb matt.peters
#rb pj.kack
#jira UE-175237

[CL 25549055 by zousar shaker in ue5-main branch]
2023-05-19 14:43:02 -04:00
dan engelbrecht
96500c25d1 Automatically detect a if a staged target is using zenstore without pak and configure zenserver host and project id.
#jira UE-144425
#rb zousar.shaker pj.kack
#preflight 64427d5b4944ef78923c85b6

[CL 25144415 by dan engelbrecht in ue5-main branch]
2023-04-21 08:56:22 -04:00
carlmagnus nordin
ecfc01c258 ZenLoader: Include preload dependencies in package headers instead of relying on predetermined export bundle dependencies. This means that each package can be treated in isolation when staging. Also removes the need to keep track of export and bundle counts in the package store which saves some memory.
#preflight 642e8838441e8b2f5506dbb4
#rb pk.kack,per.larsson,francis.hurteau

[CL 24944233 by carlmagnus nordin in ue5-main branch]
2023-04-06 10:50:30 -04:00
dan engelbrecht
db5e568754 Game client no longer needs -zenstoreproject or -zenstorehost arguments to correctly run with a COTF server which uses Zen.
Game client now accepts only -filehostip for connection to a CTOF server, -cookonthefly is no longer necessary and is ignored.
Game client will now receive ProjectName (ProjectId), Platform (OplogId), Zen server host name and port from COTF server if running with zenstore.
Fixed issues with generating the ProjectId from a path where drive letters where lower case.

#rb zousar.shaker pj.kack
#preflight

[CL 24889513 by dan engelbrecht in ue5-main branch]
2023-04-03 01:51:40 -04:00
carlmagnus nordin
024df6d65c ZenCook: Added support for file regions
#preflight 64269c8491589478cdeb2a6f
#rb pj.kack
#rnx

[CL 24869761 by carlmagnus nordin in ue5-main branch]
2023-03-31 08:36:14 -04:00
carlmagnus nordin
58b7c9404b IoStore: Refactor to prepare for staging without a global optimization pass
* Moved package redirection out of the package store optimizer
* Moved iocontainer header creation out of the package store optimizer
* Moved package load order sorting out of the package store optimizer

#preflight 641d8dab803cb466e83a05dc
#rb pj.kack
#rnx

[CL 24778887 by carlmagnus nordin in ue5-main branch]
2023-03-24 10:06:23 -04:00
carlmagnus nordin
b99a74ba54 ZenCook: Workaround for not being able to reliably start the zen server from UAT
#preflight 641ac64f691c5ebc152d9ed9
#rb dan.engelbrecht
#rnx

[CL 24745405 by carlmagnus nordin in ue5-main branch]
2023-03-22 07:02:26 -04:00
carlmagnus nordin
30c0f28fc0 ZenCook: Merge the oplog and package store manifest. Write an oplog instead of a package store manifest when cooking to loose files and read the oplog instead of the package store manifest to find cooked files in UAT when staging from Zen.
#rb per.larsson
#preflight 640efc9bc41a0a2a7799d470
#rnx

[CL 24632735 by carlmagnus nordin in ue5-main branch]
2023-03-14 06:43:23 -04:00
carlmagnus nordin
641eb5b626 ZenCook: Added support for optional package segments
#jira UE-148021
#preflight 63f365ace832ea950a6217e4
#rb per.larsson

[CL 24343344 by carlmagnus nordin in ue5-main branch]
2023-02-21 13:03:12 -05:00
Matt Peters
fa59e5fba4 MPCook: ZenStoreWriter: Replicate ZenStoreWriter artifacts data.
#rnx
#rb Zousar.Shaker
#preflight 63dd7dd85c0ce8f11b71f405

[CL 23999102 by Matt Peters in ue5-main branch]
2023-02-03 17:38:06 -05:00
pj kack
d3da578b56 ZenCook: Add async write of additional files like .ufont to the sandbox directory before compressing for Zen.
Fix problem of missing additional files in .pak files when packaging based on a ZenCook.

#jira UE-175144
#rb carlmagnus.nordin,per.larsson,zousar.shaker
#rnx
#preflight 63d92560f626715201d687ce

[CL 23945129 by pj kack in ue5-main branch]
2023-02-01 03:19:39 -05:00
carlmagnus nordin
9f028b8744 ZenCook: Remove unused fields "guid" and "disksize" from oplog
#preflight 63d8ede38505ea6b1f7e96b6
#rb per.larsson

[CL 23927486 by carlmagnus nordin in ue5-main branch]
2023-01-31 10:39:54 -05:00
Matt Peters
5d2f838486 Cooker: MPCook: Replicate package hashes in LooseCookedPackageWriter.
#rb Zousar.Shaker
#rnx
#preflight 63d431bc5428dc67b168e481

[CL 23899261 by Matt Peters in ue5-main branch]
2023-01-28 14:33:10 -05:00
Matt Peters
178d5856dc MPCook - PackageWriter and IOStore API changes to support MPCook.
Add FCookInfo argument to BeginCook and EndCook, the same data passed to initialize. It is provided in Begin/End as well since Writers need to consider it for Begin/End actions.
Add WriteMPCookMessageForPackage/TryReadMPCookMessageForPackage functions for replicating information from PackageWriters from CookWorker to CookDirector.
Add WritePackage/TryReadPackage to Manifest. Some extra information is collected when MPCook is active to support this replication.
Add CompactBinary serialization functions to FIoChunkId.
#rbx
#preflight 63d1c62a45795e7d5c164b0e

[CL 23872901 by Matt Peters in ue5-main branch]
2023-01-26 14:32:12 -05:00