77 Commits

Author SHA1 Message Date
zousar shaker
5cabc11d45 Introduce an interface (ICookArtifactReader) for reading data back from a package store used at cook time, with implementations to either read artifacts from zen (if not present on the filesystem), or read artifacts from the filesystem alone. Also ensured that the reader is used in the places that currently expect to read data for incremental cook. Specifically:
- CookSettings.txt
- DevelopmentAssetRegistry.bin
- Shader code libraries

#rb Matt.Peters

[CL 35461438 by zousar shaker in ue5-main branch]
2024-08-12 14:45:14 -04:00
zousar shaker
839606e1e9 2nd submit: Incremental step towards being able to stage to both a pak build as well as a nopak (streamng) build from a snapshot entirely stored in zenserver (no loose files on the filesystem except a ue.projectstore file).
Key changes:
- Removed the use of cookedfiles.manifest
- Changed IoStore mode of UnrealPak to  be capable of getting zenserver launch data from either a package store manifest (cbobject, metadata) OR a project store marker file (json)
- Ensured that the UAT and IoStore mode of UnrealPak can launch zenserver reliably by passing along the SponsorProcessId when launching zenserver
- Ensured that shader archives (and their accompanying json metadata files) can be read either as loose files on disk or directly from zenserver if it the data for them is internal to zenserver (ie: they have a valid  chunk id)
Remaining work:
- Pak mode of UnrealPak must be able to launch zenserver and pull data from it.

#rb PJ.Kack

[CL 34565129 by zousar shaker in ue5-main branch]
2024-06-21 11:48:33 -04:00
zousar shaker
008dcac8a3 Ensure DLC cook and pak via zenstore don't interfere with the base project cooked output oplogs by segregating their cooked data into its own project id. Tested using EngineTestDLC. EditorDomain oplog is allowed to be shared between the DLC and the base project.
#rb Matt.Peters

[CL 34555257 by zousar shaker in ue5-main branch]
2024-06-21 01:06:39 -04:00
zousar shaker
082e9969db [Backout] - CL34481610
[FYI] Zousar.Shaker
Original CL Desc
-----------------------------------------------------------------
Incremental step towards being able to stage to both a pak build as well as a nopak (streamng) build from a snapshot entirely stored in zenserver (no loose files on the filesystem except a ue.projectstore file).
Key changes:
- Removed the use of cookedfiles.manifest
- Changed IoStore mode of UnrealPak to  be capable of getting zenserver launch data from either a package store manifest (cbobject, metadata) OR a project store marker file (json)
- Ensured that the UAT and IoStore mode of UnrealPak can launch zenserver reliably by passing along the SponsorProcessId when launching zenserver
- Ensured that shader archives (and their accompanying json metadata files) can be read either as loose files on disk or directly from zenserver if it the data for them is internal to zenserver (ie: they have a valid  chunk id)
Remaining work:
- Pak mode of UnrealPak must be able to launch zenserver and pull data from it.

#rb PJ.Kack

[CL 34498668 by zousar shaker in ue5-main branch]
2024-06-19 10:15:01 -04:00
zousar shaker
a7b5faeaf0 Incremental step towards being able to stage to both a pak build as well as a nopak (streamng) build from a snapshot entirely stored in zenserver (no loose files on the filesystem except a ue.projectstore file).
Key changes:
- Removed the use of cookedfiles.manifest
- Changed IoStore mode of UnrealPak to  be capable of getting zenserver launch data from either a package store manifest (cbobject, metadata) OR a project store marker file (json)
- Ensured that the UAT and IoStore mode of UnrealPak can launch zenserver reliably by passing along the SponsorProcessId when launching zenserver
- Ensured that shader archives (and their accompanying json metadata files) can be read either as loose files on disk or directly from zenserver if it the data for them is internal to zenserver (ie: they have a valid  chunk id)
Remaining work:
- Pak mode of UnrealPak must be able to launch zenserver and pull data from it.

#rb PJ.Kack

[CL 34481636 by zousar shaker in ue5-main branch]
2024-06-18 17:33:46 -04:00
dan engelbrecht
c0bf74a232 Make sure we can read large files when saving loose files to non-local remote zenserver
#rb Zousar.Shaker
#jira UE-217522

[CL 34267114 by dan engelbrecht in ue5-main branch]
2024-06-11 04:25:36 -04:00
matt peters
1a04a479f2 IncrementalCook: Add UE::Cook::IDeterminismHelper to write diagnostics for cook indeterminism and false positive iterative skips.
[RN] Minor, Cooking
#rb Zousar.Shaker

[CL 34153566 by matt peters in ue5-main branch]
2024-06-06 07:46:50 -04:00
matt peters
3fbe2177d9 Cooker: ZenStore oplog attachments: Fix corrupted attachments in incremental cooks when we have more than 1 attachment. Part of the attachment data from the previous cook was not cleared when recording the results from the new version of the package.
#rnx
#rb Zousar.Shaker

[CL 34111244 by matt peters in ue5-main branch]
2024-06-04 19:06:31 -04:00
matt peters
2971248963 IncrementalCook: Add support for iterative and incremental cooks of CookPackageSplitters.
#rnx
#rb Zousar.Shaker

[CL 33710894 by matt peters in ue5-main branch]
2024-05-16 23:22:12 -04:00
matt peters
66f0b9950e Cooker:
Bugfix for incremental ZenStoreWriter: it was broadcasting iterativelyunmodified for every package whether it was modified or not.
MPCook Performance bugfix for GenerationHelper on cookdirector: it was initializing them (and loading the generator package) when it did not need to in response to discovery and savepackage messages from cookworkers.
Fix some coding standard issues.
Readability improvements for some save code that is about to be extended.
#rnx
#rb Zousar.Shaker

[CL 33554042 by matt peters in ue5-main branch]
2024-05-09 14:40:00 -04:00
zousar shaker
106b9499e0 Convert ue.projectstore file to json.
#rb dan.engelbrecht

[CL 32464544 by zousar shaker in ue5-main branch]
2024-03-24 22:03:42 -04:00
zousar shaker
9f1b48b917 Change naming of .projectstore to ue.projectstore to avoid having hidden by default status on Mac/Linux.
#rb dan.engelbrecht

[CL 31054932 by zousar shaker in ue5-main branch]
2024-01-31 13:01:29 -05:00
zousar shaker
98800ff6a4 When writing file manifest to zenstore, don't attempt to set both the "data" field on a file entry and the "serverpath" field on the file entry, as zenserver will treat the two as mutually exclusive, and setting the "data" field will mean we no loger get "serverpath" out of zenserver. This causes issues when doing an iterative cook and trying to read the file manifest back from zenserver.
#rb matt.peters

[CL 26980097 by zousar shaker in ue5-main branch]
2023-08-09 19:55:19 -04:00
bob tellez
de0ac0a5c5 #UE Only assert that you have package data to write when you are intending to write data or computehash. The new SkipSave cook option does not write data intentionally
[FYI] Matt.Peters

[CL 26837731 by bob tellez in ue5-main branch]
2023-08-03 23:00:11 -04:00
Matt Peters
13643c4f10 ZenWriter: Log the fatal assertion about invalid CommitPackage call on the CommitPackage thread in addition to the async thread so it is easier to debug.
#rb Zousar.Shaker
#rnx

[CL 26355924 by Matt Peters in ue5-main branch]
2023-06-30 17:07:55 -04:00
Matt Peters
ec9fef914b Cooking: Iterative: Add -iterativevalidate mode that diffs iteratively unmodified previous packages against their resaved version.
#rb Zousar.Shaker
#rnx

[CL 26327591 by Matt Peters in ue5-main branch]
2023-06-29 16:40:47 -04:00
zousar shaker
edaa3a7983 Ensure that when a package is saved multiple times (eg: when doing determinism testing) that we don't get into a situation where we attempt to save it a second time while it is still being asynchronously written the first time.
#rb matt.peters

[CL 26153970 by zousar shaker in ue5-main branch]
2023-06-21 13:06:10 -04:00
dan engelbrecht
20799c9f48 Store cooker computer remote ips in .projectstore
#rb zousar.shaker

[CL 26006400 by dan engelbrecht in ue5-main branch]
2023-06-15 04:51:13 -04:00
dan engelbrecht
790e6a8b46 Make sure we can gracefully destroy a FZenStoreWriter instance even if EndCook() is not called.
#jira UE-188177
#rb zousar.shaker

[CL 25888492 by dan engelbrecht in ue5-main branch]
2023-06-09 03:24:09 -04:00
Matt Peters
88d33c5f37 Cooker and SavePackage: Move calls to BeginCacheForCookedPlatformData and ClearCachedCookedPlatformData from SavePackage into the cooker. Moving the Clear calls allows us to clear only after repeated SavePackage attempts from the diffonly cooker have been completed, preventing a break in contract by having objects being saved after Clear is called. Moving the Begin calls is necessary to track which objects need Clear called on them.
#jira FORT-613920
#rn Minor, Cooking
#rb Zousar.Shaker

[CL 25851419 by Matt Peters in ue5-main branch]
2023-06-07 13:56:57 -04:00
zousar shaker
a3bf9945d7 Ensure that multi-process cook doesn't cause workers to delete the oplog created by the director when using ZenStoreWriter.
#rb dan.engelbrecht

[CL 25644245 by zousar shaker in ue5-main branch]
2023-05-26 12:01:58 -04:00
zousar shaker
b0d5721dde Initial implementation of DiffOnly mode for ZenServer as Cooked Output Store. Lacking detailed header diffing, but stubbed in for future addition. When using ZenServer as Cooked Output Store the packages use the ZenPackageSummary header format instead of the PackageFileSummary header format. This is how data is stored in ZenServer and the format it is when fetched. Data produced in process is transformed into this format ahead of diffing.
The code for parsing a view of the ZenPackageSummary has been extracted from AsyncLoading2.cpp into a private h/cpp and is possible to use from the header diffing code when printing out detailed diffs, but is not used yet.

#rb matt.peters
#rb pj.kack
#jira UE-175237

[CL 25549055 by zousar shaker in ue5-main branch]
2023-05-19 14:43:02 -04:00
dan engelbrecht
96500c25d1 Automatically detect a if a staged target is using zenstore without pak and configure zenserver host and project id.
#jira UE-144425
#rb zousar.shaker pj.kack
#preflight 64427d5b4944ef78923c85b6

[CL 25144415 by dan engelbrecht in ue5-main branch]
2023-04-21 08:56:22 -04:00
carlmagnus nordin
ecfc01c258 ZenLoader: Include preload dependencies in package headers instead of relying on predetermined export bundle dependencies. This means that each package can be treated in isolation when staging. Also removes the need to keep track of export and bundle counts in the package store which saves some memory.
#preflight 642e8838441e8b2f5506dbb4
#rb pk.kack,per.larsson,francis.hurteau

[CL 24944233 by carlmagnus nordin in ue5-main branch]
2023-04-06 10:50:30 -04:00
dan engelbrecht
db5e568754 Game client no longer needs -zenstoreproject or -zenstorehost arguments to correctly run with a COTF server which uses Zen.
Game client now accepts only -filehostip for connection to a CTOF server, -cookonthefly is no longer necessary and is ignored.
Game client will now receive ProjectName (ProjectId), Platform (OplogId), Zen server host name and port from COTF server if running with zenstore.
Fixed issues with generating the ProjectId from a path where drive letters where lower case.

#rb zousar.shaker pj.kack
#preflight

[CL 24889513 by dan engelbrecht in ue5-main branch]
2023-04-03 01:51:40 -04:00