Commit Graph

29 Commits

Author SHA1 Message Date
zousar shaker
e95a2b2c30 Change default port for autolaunched ZenServer local service to 8558. When connecting to an existing service (not autolaunched), the default port remains what it was before, but will change in the future.
[CL 28665636 by zousar shaker in ue5-main branch]
2023-10-11 11:18:41 -04:00
matt peters
a77af3f0e8 ZenStore structures: expose structures to interpret zenstore headers for use in -diffonly cooks.
PackageStoreOptimizer: Expose a static version of FindScriptObject and functions for calculating PublicExportHash.
FNameMap: Add a foreach iterator to provide a list of all data in the struct.
ZenPackageHeader: Move from private to internal, add a constructor that returns an error string rather than fatally asserting.
#rb Per.Larsson
#rnx

[CL 27855318 by matt peters in ue5-main branch]
2023-09-13 18:11:20 -04:00
carlmagnus nordin
ecfc01c258 ZenLoader: Include preload dependencies in package headers instead of relying on predetermined export bundle dependencies. This means that each package can be treated in isolation when staging. Also removes the need to keep track of export and bundle counts in the package store which saves some memory.
#preflight 642e8838441e8b2f5506dbb4
#rb pk.kack,per.larsson,francis.hurteau

[CL 24944233 by carlmagnus nordin in ue5-main branch]
2023-04-06 10:50:30 -04:00
carlmagnus nordin
462cc2a438 IoStore: Include imported package names in the header data. This removes the need to keep track of uncooked packages in the package store for editor builds.
#rnx
#rb pj.kack
#preflight 64256a01b01a25d6a044758d

[CL 24869323 by carlmagnus nordin in ue5-main branch]
2023-03-31 07:20:51 -04:00
carlmagnus nordin
f663db1c30 IoStore/ZenLoader: Some cleanup to facilitate future changes
#preflight 6423d8b2973e60967061727a
#rb pj.kack
#rnx

[CL 24832834 by carlmagnus nordin in ue5-main branch]
2023-03-29 04:40:20 -04:00
carlmagnus nordin
58b7c9404b IoStore: Refactor to prepare for staging without a global optimization pass
* Moved package redirection out of the package store optimizer
* Moved iocontainer header creation out of the package store optimizer
* Moved package load order sorting out of the package store optimizer

#preflight 641d8dab803cb466e83a05dc
#rb pj.kack
#rnx

[CL 24778887 by carlmagnus nordin in ue5-main branch]
2023-03-24 10:06:23 -04:00
carlmagnus nordin
be6dcbc558 IoStore:
Remove the requirement to resolve full export names when staging from Zen
#preflight 63d8efe9f626715201ca47d1
#rb pj.kack

[CL 23927635 by carlmagnus nordin in ue5-main branch]
2023-01-31 10:47:12 -05:00
dan engelbrecht
eccdbbbd02 Make attachment names written as temp-files in ZenStoreHttpClient uses a per-process unique salt
#rb zousar.shaker matt.peters
#preflight 63d7e0e7d21dbe1d29486735

[CL 23917462 by dan engelbrecht in ue5-main branch]
2023-01-30 17:38:18 -05:00
dan engelbrecht
e018c47697 Add marker file for project store oplog in Saved/Cooked/<platform> folder so Zen can check if an oplog should be garbage collected.
Companion Zen PR: https://github.com/EpicGames/zen/pull/209

#preflight 63971e55cb2317695e180344

[CL 23510788 by dan engelbrecht in ue5-main branch]
2022-12-14 07:18:38 -05:00
Per Larsson
2efa87995e Bulk Data - data resource table
This change aims to unify how BulkData and DerivedData references are stored and loaded from disk.
The structured part of Bulk Data i.e. Flags,Offset and Size has been moved out from the export data
and into a lookup table in the package header. This allows for bulk data to be rearranged w/o needing to recook but it will
also be used for derived data references to allow loading DDC data both from the runtime.
Currenlty this change ONLY effects cooked output.
Inline data can now be reloaded when loading from I/O store, all consistency checks
when discarding/reloading inline data has been removed.
When cooking, exports are serialized into it's own archive enabling the package header
to be modified after cooking the exports. This also makes it easer to rearrange the exports
in the export bundles for optimal loading performance, i.e. there is no longer a dependency
of the cooked serial offset from the begnning of the .uasset file.

#rb CarlMagnus.Nordi, PJ.Kack, Paul.Chipchase
#jira none
#preflight 637387db9e3bea807941a850
#rnx

[CL 23489396 by Per Larsson in ue5-main branch]
2022-12-13 03:51:37 -05:00
Zousar Shaker
b9548ddad4 Move Zen module public headers into an "Experimental" subfolder to have them reflect the feature status.
Fix locations that include the headers to use the "Experimental" subfolder and re-sort includes.  No functional change, purely a move, search+replace, and sort of includes.

#rb matt.peters
#preflight 63939eebc415e8dc78d35d63

[CL 23468934 by Zousar Shaker in ue5-main branch]
2022-12-09 16:11:41 -05:00
dan engelbrecht
2f8823f7dc Pass along path to project file when zen server is running locally so we can detect when it is removed and garbage-collect any projectstore entries on the zen side.
Allow Cook to remote Zen, if we detect that we are targetting a zenservice on a different machine we change the local file references to attachment based references.
#rb stefan.boberg
#preflight 63491c31ad0f7e2f20230b39

[CL 22525021 by dan engelbrecht in ue5-main branch]
2022-10-14 08:17:43 -04:00
carlmagnus nordin
bc0b9cf56d IoStore: Add support for staging and loading auto optional packages
#preflight 62a879e8d502a5543671efa4
#rb francis.hurteau

#ROBOMERGE-AUTHOR: carlmagnus.nordin
#ROBOMERGE-SOURCE: CL 20699731 via CL 20699906 via CL 20699913 via CL 20699917
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v968-20684695)

[CL 20700279 by carlmagnus nordin in ue5-main branch]
2022-06-17 04:25:23 -04:00
CarlMagnus Nordin
210d67b30e Promoted Zen project id generation to FApp
#rb pj.kack
#preflight 628769032c34da686ac82442

[CL 20316798 by CarlMagnus Nordin in ue5-main branch]
2022-05-23 01:49:25 -04:00
francis hurteau
b37b8af059 Fix crash in package optimizer when recursively resolving exports which an import outer in the chain
Removed default parameter to the function that obfuscated the omission

#rb CarlMagnus.Nordin
#preflight 6282cac0734d065770482e21

#ROBOMERGE-AUTHOR: francis.hurteau
#ROBOMERGE-SOURCE: CL 20243942 via CL 20260296 via CL 20260314 via CL 20260321
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v943-19904690)

[CL 20263026 by francis hurteau in ue5-main branch]
2022-05-18 12:56:12 -04:00
CarlMagnus Nordin
63b8cbc9dd AsyncLoading2: Implemented loading of optional package segments
#rnx
#preflight 62627548ac29fcf6a213fe33
#rb francis.hurteau

[CL 19900368 by CarlMagnus Nordin in ue5-main branch]
2022-04-25 07:37:07 -04:00
Johan Torp
ef8e0c4660 Optimize WITH_CASE_PRESERVING_NAME FName loading
LoadDisplayNames() 1.06s -> 0.24s when async loading the name batch in CachedAssetRegistry.bin during editor startup of an internal project

* Introduce FDisplayNameEntryId to avoid cache misses in FName::GetComparisonIdFromDisplayId() lookups
* Force name batch serialization to use display entry ids to help enforce cooking determinism
* Fix determinism problem when loading then saving via FPackageStoreNameMapBuilder
* Inline FNameEntryId::FromUnstableInt()
* Remove slow IsNumbered() check in CreateFromDisplayId when not using 4B FName
* Remove expensive heap sort by sorting up front instead

#rb rob.millar,matt.peters,pj.kack
#preflight 62664bd06968f18d5137d419

[CL 19899104 by Johan Torp in ue5-main branch]
2022-04-25 03:48:33 -04:00
Per Larsson
ddbf9eb5d2 Zen - cook to custom project/oplog ID with -ZenStoreProject=<ProjectId> and -ZenStorePlatform=<OplogId>
#rb none
#rnx
#preflight 19755222

[CL 19804798 by Per Larsson in ue5-main branch]
2022-04-19 02:37:06 -04:00
francis hurteau
eb9289d834 Added support to PackageStoreOptimizer to support optional package when generated from legacy cooked header
Support for Zen store will be added separately

#jira UE-129805
#rb CarlMagnus.Nordin, Matt.Peters
#preflight 61e6f8d73778a195debaffa6

#ROBOMERGE-AUTHOR: francis.hurteau
#ROBOMERGE-SOURCE: CL 18642382 in //UE5/Release-5.0/... via CL 18642393 via CL 18642414
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v900-18638592)

[CL 18642448 by francis hurteau in ue5-main branch]
2022-01-18 13:01:31 -05:00
pj kack
380bad2a75 Minor optimizations to PackageStoreOptimizer when cooking with ZenStore
#jira none
#rb carlmagnus.nordin
#rnx
#preflight 61bb19cc440d71c977b63c7b

#ROBOMERGE-AUTHOR: pj.kack
#ROBOMERGE-SOURCE: CL 18474617 in //UE5/Release-5.0/... via CL 18480638
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)

[CL 18480720 by pj kack in ue5-release-engine-test branch]
2021-12-16 18:22:15 -05:00
zousar shaker
0832797d28 Ensure that Zen ProjectIds are unique for projects with the same name in differing roots (or in the same root) while attempting to maintain some easily recognizable desriptors for the project.
Staged builds are explicitly assigned a project id via the persistent UECommandline.txt file.  This works on mobile/consoles, but isn't implemented for Windows/Linux/Mac, so staged builds for those platforms don't have a persistent assignment of project id and instead rely on the launch by the editor including the appropriate commandlines.
Unstaged builds on Windows/Mac/Linux now determine project id independently after they have determined the project file path.  This happens after the project file has been determined and has had case corrected.  This ensures it is different/unique for multiple blueprint projects that share the same executable.
Fix a bug where second cook in editor to zen would stall because the queue thinks all additions are finished due to a boolean that isn't reset.

#rb devin.doucette
#rb per.larsson
#preflight 61b39a35a2562c8b1c40f81e

#ROBOMERGE-AUTHOR: zousar.shaker
#ROBOMERGE-SOURCE: CL 18432317 in //UE5/Release-5.0/... via CL 18435404
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v897-18405271)

[CL 18435895 by zousar shaker in ue5-release-engine-test branch]
2021-12-10 18:06:39 -05:00
Marc Audy
0c3be2b6ad Merge Release-Engine-Staging to Test @ CL# 18240298
[CL 18241953 by Marc Audy in ue5-release-engine-test branch]
2021-11-18 14:37:34 -05:00
aurel cordonnier
a6e741e007 Merge from Release-Engine-Staging @ 17915896 to Release-Engine-Test
This represents UE4/Main @17911760, Release-5.0 @17915875 and Dev-PerfTest @17914035

[CL 17918595 by aurel cordonnier in ue5-release-engine-test branch]
2021-10-25 20:05:28 -04:00
aurel cordonnier
a12d56ff31 Merge from Release-Engine-Staging @ 17791557 to Release-Engine-Test
This represents UE4/Main @17774255, Release-5.0 @17791557 and Dev-PerfTest @17789485

[CL 17794212 by aurel cordonnier in ue5-release-engine-test branch]
2021-10-12 21:21:22 -04:00
matt peters
763b1ad106 ZenStoreHttpClient: Expose connection failure to the Oplog owner to handle.
#rb Per.Larsson
#rnx

#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 17661528 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v875-17642767)

[CL 17661553 by matt peters in ue5-release-engine-test branch]
2021-09-29 10:51:17 -04:00