Commit Graph

20 Commits

Author SHA1 Message Date
julien stjean
af4424e59c Introducing the relocation workflow.
When on, at load the packages will detect if they where moved and if this is the case they will modify their ImportMap and SoftObjectPathList. The package reference in those that were in the same MountPoint as the asset when it was saved will behave as if they were relative references.

#preflight 63f3e319500c05a624417605
#rb Matt.Peters

[CL 24332127 by julien stjean in ue5-main branch]
2023-02-20 20:57:13 -05:00
Matt Peters
1cd1f81cfc DirectoryTree: Give CA_ASSUME hint to suppress spurious static analysis warning.
#jira FORT-557639
#rnx
#preflight 63d2a41af626715201106296

[CL 23869656 by Matt Peters in ue5-main branch]
2023-01-26 11:55:37 -05:00
Matt Peters
c07fbb37c3 DirectoryTree: Remove COREUBOBJECT_API, because inline functions cannot be marked as dllexport/dllimport.
#rb None, trivial
#rnx
#preflight 63d083c6033e457ba87fcdd1

[CL 23842825 by Matt Peters in ue5-main branch]
2023-01-24 20:37:37 -05:00
Matt Peters
73bac25234 Containers: DirectoryTree. An O(log(n)) map from path to ValueType that also supports O(log(n)) queries for the closest parent directory in the tree of a given path.
#rb Devin.Doucette
#rnx
#preflight 63d0166cf2318350a289d7fc

[CL 23833605 by Matt Peters in ue5-main branch]
2023-01-24 12:48:49 -05:00
joe pribele
cffacc0be5 support for TObjectPtr handling renames and GetTypeHash(TObjectPtr) no long has to resolve
GetTypeHash will create a packaged object ref for hashing if the class is defined as lazy load otherwise raw pointer is hashed
a map was added for moved objects and ObjectPathId can be a WeakObjectPtr
the map allows UObjects to be mapped back to packed object refs

#rb zousar.shaker
https://p4-swarm.epicgames.net/reviews/23356491
#preflight 63b5de641c35d1cbdbccecf7
#preflight 63b70406e26e31879b8aa6d3

[CL 23589497 by joe pribele in ue5-main branch]
2023-01-05 12:27:16 -05:00
Per Larsson
b20b928cf7 Editor Domain - fix fallouts after bulk data refactor
- allow file region of type None
- don't early out when saving bulk data by reference

#rb trivial
#jira none
#preflight 639b6de30a67152550853ab9
#rnx

[CL 23529938 by Per Larsson in ue5-main branch]
2022-12-15 14:35:36 -05:00
Per Larsson
2efa87995e Bulk Data - data resource table
This change aims to unify how BulkData and DerivedData references are stored and loaded from disk.
The structured part of Bulk Data i.e. Flags,Offset and Size has been moved out from the export data
and into a lookup table in the package header. This allows for bulk data to be rearranged w/o needing to recook but it will
also be used for derived data references to allow loading DDC data both from the runtime.
Currenlty this change ONLY effects cooked output.
Inline data can now be reloaded when loading from I/O store, all consistency checks
when discarding/reloading inline data has been removed.
When cooking, exports are serialized into it's own archive enabling the package header
to be modified after cooking the exports. This also makes it easer to rearrange the exports
in the export bundles for optimal loading performance, i.e. there is no longer a dependency
of the cooked serial offset from the begnning of the .uasset file.

#rb CarlMagnus.Nordi, PJ.Kack, Paul.Chipchase
#jira none
#preflight 637387db9e3bea807941a850
#rnx

[CL 23489396 by Per Larsson in ue5-main branch]
2022-12-13 03:51:37 -05:00
joe pribele
0f45c0a9d5 Undo //UE5/Main/Engine/... changelist 23416765
#preflight 638fed855c5308d18c157af0

[CL 23423019 by joe pribele in ue5-main branch]
2022-12-06 20:58:56 -05:00
joe pribele
115c3e1a0e support for TObjectPtr handling renames and GetTypeHash(TObjectPtr) no long has to resolve
GetTypeHash will create a packaged object ref for hashing if the class is defined as lazy load otherwise raw pointer is hashed
a map was added for moved objects and ObjectPathId can be a WeakObjectPtr
the map allows UObjects to be mapped back to packed object refs
#preflight 638fad1b5624e6da5e166024

[CL 23416765 by joe pribele in ue5-main branch]
2022-12-06 16:27:20 -05:00
matt peters
daff9f8c5d EditorDomain: Implement support for PackageTrailers in the same package as BulkData in a regular BulkData section for PackageWriters, so that EditorDomain can have both references to BulkData in a WorkspaceDomain PackageTrailer and embedded BulkDatas that needed to be resaved.
Because the PackageFooter comes before the PackageTrailer, we need to change accounting for the PackageFooter in PackageWriters to include adjusting the VirtualExportsFileOffset that is passed into the PackageTrailer.
#rb Paul.Chipchase
#rnx
#preflight 638cbb1135192facc13f83e6

[CL 23386395 by matt peters in ue5-main branch]
2022-12-04 12:54:00 -05:00
joe pribele
9571d4eec5 [AssetRegistry] changed TryGetAssetByObjectPath to use FSoftObjectPath to match the rest of the API
#rb zousar.shaker
#preflight 63376f4d0d2b0d7bd8ce92c1

[CL 22283581 by joe pribele in ue5-main branch]
2022-09-30 19:00:07 -04:00
joe pribele
3db3104b1c [AssetRegistry] added two new methods TryGetAssetByObjectPath and TryGetAssetPackageData to allow lazy load to be able to accurately validate that objects and paths exist
otherwise an unresolve `TObjectPtr` could resolve to nullptr causing determinism issues

#rb matt.peters zousar.shaker
#rnx
#preflight 6335c8cb5c2225fe5f12a7d6

[CL 22263392 by joe pribele in ue5-main branch]
2022-09-30 00:54:16 -04:00
joe pribele
0b82102f66 [Core] changes to make lazyloadimports performant
changed lazy load back to looking at meta data to decide which types are lazyloaded
removed callback that handles lazyload can add back in the future if needed
only imports that marked as lazy are skipped. verifyimports still resolves other imports
skipping all imports causes performance problems because PackageLoadInternal is called many more times then necessary

#rb zousar.shaker
#p4v-preflight-copy 21567032
#preflight 6307d7753042e94f9f642a6d

[CL 21571620 by joe pribele in ue5-main branch]
2022-08-25 16:26:05 -04:00
robert millar
fb932b1cdb [KillPendingKill] Remove renamed packages and their exports from the async loading thread's caches.
#rb pj.kack

#ROBOMERGE-AUTHOR: robert.millar
#ROBOMERGE-SOURCE: CL 21060094 via CL 21060128 via CL 21060415 via CL 21060535
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v972-20964824)

[CL 21078246 by robert millar in ue5-main branch]
2022-07-13 15:37:02 -04:00
Matt Peters
02ba0b5b70 ZenStoreWriter: Fix assertion "Trying to add package that is already pending" if the cooker commits a package a second time while the first commit of the package is queued for asynchronous completion aand has not yet reached CommitPackageInternal.
This is necessary because as of CL 20151503, the cooker can commit the same package multiple times if the first save failed due to a timeout.

#fyi CarlMagnus.Nordin
#rnx
#rb Per.Larsson
#preflight 62853ba90f0d5076dee74ced

[CL 20265738 by Matt Peters in ue5-main branch]
2022-05-18 14:49:11 -04:00
Zousar Shaker
501acd133c Change how lazy load is configured to allow it to be driven by callback instead of by metadata. Added a callback from CookOnTheFlyServer to allow it to lazy load references to packages it doesn't intend to save (ie: iteratively skipped packages). This is only effective when cooking in a commandlet (not in editor).
#rb matt.peters
#preflight 625dabdbd412434587aa0b8d

[CL 19791002 by Zousar Shaker in ue5-main branch]
2022-04-18 14:35:51 -04:00
Dan Thompson
cd73707429 Update package writers to gather package hashes internally, then have CookByTheBookFinished copy them to asset package datas.
Add iostore chunk hashes to the asset registry during cook.
Remove old hashing code, which was extensive. This moves the async writes in to UE::Task from the old AsyncWorkSequence().
#rb Matt.Peters
#rb Jeff.Roberts
#preflight 624b24a5f73c316f68303946

[CL 19611812 by Dan Thompson in ue5-main branch]
2022-04-04 13:26:27 -04:00
francis hurteau
d262daac13 Added support to Loose File Package Writer to write multiple package output per input
Added similar support to the package store manifest
Zen Store support will be added separately
Added some extra data to linker object resources to be used by the iostore package optimizer to properly support optional package chunks.

Multiple cooking outputs is used to support "optional objects" which are editor data package exports that would be put in separate iostore containers than the normal cooked game content.

#jira UE-129801
#rb Matt.Peters, CarlMagnus.Nordin
#preflight 61e5d3bb873f2ea48f48166c

#ROBOMERGE-AUTHOR: francis.hurteau
#ROBOMERGE-SOURCE: CL 18639742 in //UE5/Release-5.0/... via CL 18639751 via CL 18639758
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v900-18638592)

[CL 18639762 by francis hurteau in ue5-main branch]
2022-01-18 09:56:23 -05:00
aurel cordonnier
fc542f6cfd Merge from Release-Engine-Staging @ 18081189 to Release-Engine-Test
This represents UE4/Main @18073326, Release-5.0 @18081140 and Dev-PerfTest @18045971

[CL 18081471 by aurel cordonnier in ue5-release-engine-test branch]
2021-11-07 23:43:01 -05:00
aurel cordonnier
a6e741e007 Merge from Release-Engine-Staging @ 17915896 to Release-Engine-Test
This represents UE4/Main @17911760, Release-5.0 @17915875 and Dev-PerfTest @17914035

[CL 17918595 by aurel cordonnier in ue5-release-engine-test branch]
2021-10-25 20:05:28 -04:00