41 Commits

Author SHA1 Message Date
steve robb
6d5b974842 Fixed up a lot of bool-taking container resize functions to take EAllowShrinking instead.
[CL 30735396 by steve robb in ue5-main branch]
2024-01-19 19:41:56 -05:00
matt peters
b6e8c4f2a9 Cooker: ZenStore -diffonly: Fix typo in AppendPathForPublicExportHash that caused an infinite loop.
#rnx
#rb None, trivial

[CL 28032424 by matt peters in ue5-main branch]
2023-09-20 08:41:40 -04:00
robert millar
9fbd6babc7 Save script serialization offsets into packages that use versioned property serialization.
Offset is saved relative to export serial data.
Add new object version entry SCRIPT_SERIALIZATION_OFFSET.

#rb PJ.Kack, Francis.Hurteau
#lockdown marc.audy

[CL 27898807 by robert millar in ue5-main branch]
2023-09-14 18:18:29 -04:00
matt peters
a77af3f0e8 ZenStore structures: expose structures to interpret zenstore headers for use in -diffonly cooks.
PackageStoreOptimizer: Expose a static version of FindScriptObject and functions for calculating PublicExportHash.
FNameMap: Add a foreach iterator to provide a list of all data in the struct.
ZenPackageHeader: Move from private to internal, add a constructor that returns an error string rather than fatally asserting.
#rb Per.Larsson
#rnx

[CL 27855318 by matt peters in ue5-main branch]
2023-09-13 18:11:20 -04:00
tomasz obrebski
9e09fa9069 Zen header alignment
#jira UE-179443
#rb pj.kack

[CL 27440335 by tomasz obrebski in ue5-main branch]
2023-08-29 07:38:16 -04:00
pj kack
3b92239e69 IoStore: Make sure the FZenPackageHeader array views starting with ImportedPublicExportHashes are 8-byte aligned
#jira UE-192747
#rb per.larsson

[CL 27135261 by pj kack in ue5-main branch]
2023-08-16 06:18:19 -04:00
carlmagnus nordin
ecfc01c258 ZenLoader: Include preload dependencies in package headers instead of relying on predetermined export bundle dependencies. This means that each package can be treated in isolation when staging. Also removes the need to keep track of export and bundle counts in the package store which saves some memory.
#preflight 642e8838441e8b2f5506dbb4
#rb pk.kack,per.larsson,francis.hurteau

[CL 24944233 by carlmagnus nordin in ue5-main branch]
2023-04-06 10:50:30 -04:00
carlmagnus nordin
462cc2a438 IoStore: Include imported package names in the header data. This removes the need to keep track of uncooked packages in the package store for editor builds.
#rnx
#rb pj.kack
#preflight 64256a01b01a25d6a044758d

[CL 24869323 by carlmagnus nordin in ue5-main branch]
2023-03-31 07:20:51 -04:00
carlmagnus nordin
f663db1c30 IoStore/ZenLoader: Some cleanup to facilitate future changes
#preflight 6423d8b2973e60967061727a
#rb pj.kack
#rnx

[CL 24832834 by carlmagnus nordin in ue5-main branch]
2023-03-29 04:40:20 -04:00
carlmagnus nordin
58b7c9404b IoStore: Refactor to prepare for staging without a global optimization pass
* Moved package redirection out of the package store optimizer
* Moved iocontainer header creation out of the package store optimizer
* Moved package load order sorting out of the package store optimizer

#preflight 641d8dab803cb466e83a05dc
#rb pj.kack
#rnx

[CL 24778887 by carlmagnus nordin in ue5-main branch]
2023-03-24 10:06:23 -04:00
dan thompson
c979e76663 Instead of rehashing the package name every time in sort fns, save it off.
#rb pj.kack
#preflight 63e2c796b923ed4bdd24cae0

[CL 24066995 by dan thompson in ue5-main branch]
2023-02-08 00:25:34 -05:00
pj kack
c1dfe83c61 UnrealPak: Add more fine grained trace cpu scopes for profiling on remote machines.
#rb trivial
#rnx
#preflight 63e10793b923ed4bddea469f

[CL 24031827 by pj kack in ue5-main branch]
2023-02-06 11:39:10 -05:00
carlmagnus nordin
be6dcbc558 IoStore:
Remove the requirement to resolve full export names when staging from Zen
#preflight 63d8efe9f626715201ca47d1
#rb pj.kack

[CL 23927635 by carlmagnus nordin in ue5-main branch]
2023-01-31 10:47:12 -05:00
Per Larsson
2efa87995e Bulk Data - data resource table
This change aims to unify how BulkData and DerivedData references are stored and loaded from disk.
The structured part of Bulk Data i.e. Flags,Offset and Size has been moved out from the export data
and into a lookup table in the package header. This allows for bulk data to be rearranged w/o needing to recook but it will
also be used for derived data references to allow loading DDC data both from the runtime.
Currenlty this change ONLY effects cooked output.
Inline data can now be reloaded when loading from I/O store, all consistency checks
when discarding/reloading inline data has been removed.
When cooking, exports are serialized into it's own archive enabling the package header
to be modified after cooking the exports. This also makes it easer to rearrange the exports
in the export bundles for optimal loading performance, i.e. there is no longer a dependency
of the cooked serial offset from the begnning of the .uasset file.

#rb CarlMagnus.Nordi, PJ.Kack, Paul.Chipchase
#jira none
#preflight 637387db9e3bea807941a850
#rnx

[CL 23489396 by Per Larsson in ue5-main branch]
2022-12-13 03:51:37 -05:00
carlmagnus nordin
bc0b9cf56d IoStore: Add support for staging and loading auto optional packages
#preflight 62a879e8d502a5543671efa4
#rb francis.hurteau

#ROBOMERGE-AUTHOR: carlmagnus.nordin
#ROBOMERGE-SOURCE: CL 20699731 via CL 20699906 via CL 20699913 via CL 20699917
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v968-20684695)

[CL 20700279 by carlmagnus nordin in ue5-main branch]
2022-06-17 04:25:23 -04:00
francis hurteau
b37b8af059 Fix crash in package optimizer when recursively resolving exports which an import outer in the chain
Removed default parameter to the function that obfuscated the omission

#rb CarlMagnus.Nordin
#preflight 6282cac0734d065770482e21

#ROBOMERGE-AUTHOR: francis.hurteau
#ROBOMERGE-SOURCE: CL 20243942 via CL 20260296 via CL 20260314 via CL 20260321
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v943-19904690)

[CL 20263026 by francis hurteau in ue5-main branch]
2022-05-18 12:56:12 -04:00
Dan Thompson
718ea8fe80 Several IoStore determinism fixes:
-- Shaders must be sorted prior to inlining, as they are inserted in to a sorted list.
-- Break ties sorting shaders on disk with chunkid (hash)
-- Sort the shader map hashes in the package store entry
-- Pick the lowest LoadOrderFactor instead of the first for shaders that are used in more than one package.
-- Fix -WriteFinalOrder to actually write the final order as well as all chunks.
#rb fabian.giesen, charles.bloom
#preflight 627441a7de15651eeb416fbf

[CL 20065025 by Dan Thompson in ue5-main branch]
2022-05-05 17:43:49 -04:00
CarlMagnus Nordin
63b8cbc9dd AsyncLoading2: Implemented loading of optional package segments
#rnx
#preflight 62627548ac29fcf6a213fe33
#rb francis.hurteau

[CL 19900368 by CarlMagnus Nordin in ue5-main branch]
2022-04-25 07:37:07 -04:00
Johan Torp
ef8e0c4660 Optimize WITH_CASE_PRESERVING_NAME FName loading
LoadDisplayNames() 1.06s -> 0.24s when async loading the name batch in CachedAssetRegistry.bin during editor startup of an internal project

* Introduce FDisplayNameEntryId to avoid cache misses in FName::GetComparisonIdFromDisplayId() lookups
* Force name batch serialization to use display entry ids to help enforce cooking determinism
* Fix determinism problem when loading then saving via FPackageStoreNameMapBuilder
* Inline FNameEntryId::FromUnstableInt()
* Remove slow IsNumbered() check in CreateFromDisplayId when not using 4B FName
* Remove expensive heap sort by sorting up front instead

#rb rob.millar,matt.peters,pj.kack
#preflight 62664bd06968f18d5137d419

[CL 19899104 by Johan Torp in ue5-main branch]
2022-04-25 03:48:33 -04:00
robert millar
5811293e2a Add TargetRules.bFNameOutlineNumber as an option to reduce the size of FName by 4 bytes by storing the number in the name table.
Adds a define UE_FNAME_OUTLINE_NUMBER.
Removes of FName, FMinimalName from memory image support. Adds of FMemoryImageName.
Removal of FMinimalName operator<<, all fields made private, size made variable.
All fields of FScriptName made private.
Added console commands for dumping numbered/unnumbered names and stats.

#rb johan.torp

#ROBOMERGE-OWNER: robert.millar
#ROBOMERGE-AUTHOR: robert.millar
#ROBOMERGE-SOURCE: CL 19058026 via CL 19058611 via CL 19058656 via CL 19061727 via CL 19061740 via CL 19064047
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v918-19018356)

[CL 19066701 by robert millar in ue5-main branch]
2022-02-21 02:35:48 -05:00
francis hurteau
eb9289d834 Added support to PackageStoreOptimizer to support optional package when generated from legacy cooked header
Support for Zen store will be added separately

#jira UE-129805
#rb CarlMagnus.Nordin, Matt.Peters
#preflight 61e6f8d73778a195debaffa6

#ROBOMERGE-AUTHOR: francis.hurteau
#ROBOMERGE-SOURCE: CL 18642382 in //UE5/Release-5.0/... via CL 18642393 via CL 18642414
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v900-18638592)

[CL 18642448 by francis hurteau in ue5-main branch]
2022-01-18 13:01:31 -05:00
pj kack
380bad2a75 Minor optimizations to PackageStoreOptimizer when cooking with ZenStore
#jira none
#rb carlmagnus.nordin
#rnx
#preflight 61bb19cc440d71c977b63c7b

#ROBOMERGE-AUTHOR: pj.kack
#ROBOMERGE-SOURCE: CL 18474617 in //UE5/Release-5.0/... via CL 18480638
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)

[CL 18480720 by pj kack in ue5-release-engine-test branch]
2021-12-16 18:22:15 -05:00
Marc Audy
0c3be2b6ad Merge Release-Engine-Staging to Test @ CL# 18240298
[CL 18241953 by Marc Audy in ue5-release-engine-test branch]
2021-11-18 14:37:34 -05:00
aurel cordonnier
a12d56ff31 Merge from Release-Engine-Staging @ 17791557 to Release-Engine-Test
This represents UE4/Main @17774255, Release-5.0 @17791557 and Dev-PerfTest @17789485

[CL 17794212 by aurel cordonnier in ue5-release-engine-test branch]
2021-10-12 21:21:22 -04:00
carlmagnus nordin
e6c7bce3ee Remove unused names from package headers, reducing the amount of data and number of names that have to be serialzed
* Strip names that are not used when cooking for UPS
* Strip names that are not used by the loader when staging for IoStore

#preflight 615424b7f4d2a4000107be68, 615450a875fae600010d4820
#rb pj.kack, francis.hurteau

#ROBOMERGE-AUTHOR: carlmagnus.nordin
#ROBOMERGE-SOURCE: CL 17674004 in //UE5/Release-5.0/... via CL 17674013
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v875-17642767)
#ROBOMERGE[STARSHIP]: UE5-Main

[CL 17674014 by carlmagnus nordin in ue5-release-engine-test branch]
2021-09-30 02:05:41 -04:00