Commit Graph

434 Commits

Author SHA1 Message Date
matt peters
2ac4cd0258 AssetRegistry::GetDerivedClassNames: Add handling of CoreRedirects so that subclasses registered with the oldname of a coreredirect for their parent class will still be found.
This fixes a problem that was exposed when  ANY_PACKAGE was removed; previously classes with a CoreRedirect to a different script package but the same class name would still be found even without searching CoreRedirects, but that is no longer the case and there is some content that was relying on it.

#rb Marcus.Wassmer
#rnx
#preflight 6296b5ad1946e0a8aa024b1d

#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 20445364 via CL 20445366 via CL 20445373 via CL 20445377
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v949-20362246)

[CL 20449673 by matt peters in ue5-main branch]
2022-06-01 04:28:03 -04:00
robert manuszewski
38f950042a Fixing editor startup crash related to parsing assed tag values that were stored as 'None'
#rb trivial
#preflight none

#ROBOMERGE-AUTHOR: robert.manuszewski
#ROBOMERGE-SOURCE: CL 20437944 via CL 20437964 via CL 20437969
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v949-20362246)

[CL 20448631 by robert manuszewski in ue5-main branch]
2022-06-01 03:53:11 -04:00
robert manuszewski
d1443992e1 Deprecating ANY_PACKAGE.
This change consists of multiple changes:

Core:
- Deprecation of ANY_PACKAGE macro. Added ANY_PACKAGE_DEPRECATED macro which can still be used for backwards compatibility purposes (only used in CoreUObject)
- Deprecation of StaticFindObjectFast* functions that take bAnyPackage parameter
- Added UStruct::GetStructPathName function that returns FTopLevelAssetPath representing the path name (package + object FName, super quick compared to UObject::GetPathName) + wrapper UClass::GetClassPathName to make it look better when used with UClasses
- Added (Static)FindFirstObject* functions that find a first object given its Name (no Outer). These functions are used in places I consider valid to do global UObject (UClass) lookups like parsing command line parameters / checking for unique object names
- Added static UClass::TryFindType function which serves a similar purpose as FindFirstObject however it's going to throw a warning (with a callstack / maybe ensure in the future?) if short class name is provided. This function is used  in places that used to use short class names but now should have been converted to use path names to catch any potential regressions and or edge cases I missed.
- Added static UClass::TryConvertShortNameToPathName utility function
- Added static UClass::TryFixShortClassNameExportPath utility function
- Object text export paths will now also include class path (Texture2D'/Game/Textures/Grass.Grass' -> /Script/Engine.Texture2D'/Game/Textures/Grass.Grass')
- All places that manually generated object export paths for objects will now use FObjectPropertyBase::GetExportPath
- Added a new startup test that checks for short type names in UClass/FProperty MetaData values

AssetRegistry:
- Deprecated any member variables (FAssetData / FARFilter) or functions that use FNames to represent class names and replaced them with FTopLevelAssetPath
- Added new member variables and new function overloads that use FTopLevelAssetPath to represent class names
- This also applies to a few other modules' APIs to match AssetRegistry changes

Everything else:
- Updated code that used ANY_PACKAGE (depending on the use case) to use FindObject(nullptr, PathToObject), UClass::TryFindType (used when path name is expected, warns if it's a short name) or FindFirstObject (usually for finding types based on user input but there's been a few legitimate use cases not related to user input)
- Updated code that used AssetRegistry API to use FTopLevelAssetPaths and USomeClass::StaticClass()->GetClassPathName() instead of GetFName()
- Updated meta data and hardcoded FindObject(ANY_PACKAGE, "EEnumNameOrClassName") calls to use path names

#jira UE-99463
#rb many.people
[FYI] Marcus.Wassmer
#preflight 629248ec2256738f75de9b32

#codereviewnumbers 20320742, 20320791, 20320799, 20320756, 20320809, 20320830, 20320840, 20320846, 20320851, 20320863, 20320780, 20320765, 20320876, 20320786

#ROBOMERGE-OWNER: robert.manuszewski
#ROBOMERGE-AUTHOR: robert.manuszewski
#ROBOMERGE-SOURCE: CL 20430220 via CL 20433854 via CL 20435474 via CL 20435484
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v949-20362246)

[CL 20448496 by robert manuszewski in ue5-main branch]
2022-06-01 03:46:59 -04:00
paul chipchase
f85fad88ff It is now possible to disable virtualization for all payloads owned by a specific asset type by adding the name of the asset to [Core.ContentVirtualization]DisabledAsset string array in the engine ini file.
#rb Per.Larsson
#rnx
#jira UE-151377
#preflight 628364050039ea57a52d6989

### Virtualization
- [Core.ContentVirtualization] in the engine ini file now supports an array called 'DisabledAsset' which can be used to name asset types that should not virtualize their payloads.
-- By default (in BaseEngine.ini) we have disabled the StaticMesh asset as we know it will crash if a payload is missing and the SoundWave asset as it still is pending testing.
- This new way to disable virtualization is data driven. The older hard coded method has not been removed but will likely be reworked in a future submit.
- Now when an editor bulkdata is adding it's payload to the package trailer builder during package save it will poll the virtualization system with a call to the new method ::IsDisabledForObject by passing in it's owner.
-- If the owner is valid and was present in the 'DisabledAsset' array then the method will return true and the EPayloadFlags::DisableVirtualization flag will be applied.

### Package Trailer
- The pre-existing functionality of enum EPayloadFilter has been moved to a new enum EPayloadStorageType as will only filter payloads based on their storage type.
- EPayloadFilter has been modified to filter payloads based on functionality although at the moment the only thing it can filter for is to return payloads that can be virtualized, it is left for future expansion.
- EPayloadFlags has been reduced to a uint16 with the remaining 2bytes being turned into a new member EPayloadFilterReason.
- This new member allows us to record the exact reason why a payload is excluded from virtualization. If it is zero then the payload can virtualize, otherwise it will contain one or more reasons as to why it is being excluded. For this reason the enum is a bitfield.
- Added overloads of ::GetPayloads and ::GetNumPayloads that take EPayloadFilter rather than a EPayloadStorageType
- Added wrappers around all AccessMode types for FLookupTableEntry.
- FPackageTrailerBuilder has been extended to take a EPayloadFilterReason so that the caller can already provide a reason why the payload cannot be virtualized.
-- As a future peace of work this will probably be changed and we will ask the caller to pass in the owner UObject pointer instead and then we will process the filtering when building the package trailer to a) keep all of the filtering code in one place b) keep the filtering consistent

### PackageSubmissionChecks
- The virtualization process in  will now request the payloads that can be virtualized from the package trailer, which respects the new payload flag, rather than requesting all locally stored payloads.

### UObjects
- There is no need for the SoundWave or MeshDescription classes to opt out of virtualization on construction. This will be done when the package is saved and is now data driven rather than being hardcoded.

### DumpPackagePayloadInfo
- The command has been updated to also display the filter reasons applied to each payload

[CL 20240971 by paul chipchase in ue5-main branch]
2022-05-17 07:54:28 -04:00
Matt Peters
19f2062bc0 AssetRegistryState: fix EnumerateAssets to handle duplicates in Filter.ObjectPaths, and optimize the handling of duplicates using a bit for each filters rather than counting the number of passed filter hits and requiring uniqueness in the filterresults.
Reported by PR #8466 submitted by LindyHopperGT.
#rb Johan.Torp
#rnx
#preflight 6282642ec57a894cf680ecac

[CL 20225262 by Matt Peters in ue5-main branch]
2022-05-16 11:31:06 -04:00
patrick enfedaque
c959bb103d Fix regression: infinite loop when timing out (from CL 20102122)
#rb matt.peters
#preflight skip
#rnx

#ROBOMERGE-AUTHOR: patrick.enfedaque
#ROBOMERGE-SOURCE: CL 20103832 via CL 20103851 via CL 20103858
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v943-19904690)

[CL 20109008 by patrick enfedaque in ue5-main branch]
2022-05-09 15:24:36 -04:00
patrick enfedaque
91d6a096be - AssetRegistry support for an added LoadCalculatedDependencies phase to the AssetDataGather to allow registered IAssetDependencyGatherers to add dependencies
- Implement a World Partition IAssetDependencyGatherer to add dependencies from the World to all its ExternalActor packages.

#rb matt.peters
#preflight 627905eb066ebb49109a0e5b

#ROBOMERGE-AUTHOR: patrick.enfedaque
#ROBOMERGE-SOURCE: CL 20102122 via CL 20102125 via CL 20102134
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v943-19904690)

[CL 20108813 by patrick enfedaque in ue5-main branch]
2022-05-09 15:19:07 -04:00
Matt Peters
07aedc80d3 AssetRegistry: Deprecate the include redirectors for "AssetData.h" -> "AssetRegistry/AssetData.h" and other AssetRegistry public includes.
#rb Johan.Torp
#rnx
#preflight 6273f90f4a2e93404863731d

[CL 20061252 by Matt Peters in ue5-main branch]
2022-05-05 14:11:33 -04:00
Jason Adcock
0b77016bcc Fix for Cooked Dedicated Server AssetRegistry Enabled Issue.
#jira UE-151045
#rb matt.peters
#preflight

[CL 20060836 by Jason Adcock in ue5-main branch]
2022-05-05 13:59:40 -04:00
Matt Peters
0a76650344 AssetRegistry fix for CrashReporterClient: disable the premade preloaded assetregistry in programs that require cookedcontent. It is only available in cooked game or in cooked editor.
#rb Patrick.Laflamme
#rnx
#preflight 627177349d6c2f8f5b267967

[CL 20029937 by Matt Peters in ue5-main branch]
2022-05-03 14:52:59 -04:00
Matt Peters
7ad238a806 AssetRegistry includes (Engine/Source): change #include "AssetData.h" -> #include "AssetRegistry/AssetData.h", and similar for the other moved AssetRegistry headers.
#rb Zousar.Shaker
#rnx
#preflight 6270509a220f89f0ad573030

[CL 20016982 by Matt Peters in ue5-main branch]
2022-05-02 18:06:48 -04:00
Johan Torp
f0c0f9610d Fix false error when asset registry loading detects a version change
#rb trivial
#preflight nope
#fyi matt.peters
#rnx

[CL 19899403 by Johan Torp in ue5-main branch]
2022-04-25 04:47:19 -04:00
Johan Torp
c0d0d99351 Optimize AssetDataGatherer cache load
* Blake3 -> xxhash3
* Enabled parallel name batch loading
* Add memory mapped file loading path to reduce memcpy
* Preload file in parallel to some CPU work

#rb matt.peters
#preflight 62664bfe0634d0904cb98c02

[CL 19899126 by Johan Torp in ue5-main branch]
2022-04-25 03:54:17 -04:00
Johan Torp
ef8e0c4660 Optimize WITH_CASE_PRESERVING_NAME FName loading
LoadDisplayNames() 1.06s -> 0.24s when async loading the name batch in CachedAssetRegistry.bin during editor startup of an internal project

* Introduce FDisplayNameEntryId to avoid cache misses in FName::GetComparisonIdFromDisplayId() lookups
* Force name batch serialization to use display entry ids to help enforce cooking determinism
* Fix determinism problem when loading then saving via FPackageStoreNameMapBuilder
* Inline FNameEntryId::FromUnstableInt()
* Remove slow IsNumbered() check in CreateFromDisplayId when not using 4B FName
* Remove expensive heap sort by sorting up front instead

#rb rob.millar,matt.peters,pj.kack
#preflight 62664bd06968f18d5137d419

[CL 19899104 by Johan Torp in ue5-main branch]
2022-04-25 03:48:33 -04:00
Dan Thompson
954c6b0297 Reset intead of Empty() - edit that should have been in last change but got lost in the shelf.
#rb trivial
#preflight 625ddeb548670f31a62d2685

[CL 19796135 by Dan Thompson in ue5-main branch]
2022-04-18 18:02:26 -04:00
Dan Thompson
2c625e2b6a Provide a common way for determining a single "representative" asset for a package in the case where there are multiple assets in a package.
#rb matt.peters
#preflight 625da440b21bb49791c4db81

[CL 19790494 by Dan Thompson in ue5-main branch]
2022-04-18 14:06:09 -04:00
Dan Thompson
b3b6ec9bdf Expose loading an asset registry from disk.
#rb matt.peters
#preflight 6256f8ed6520cc7123844ac1

[CL 19741379 by Dan Thompson in ue5-main branch]
2022-04-13 12:33:15 -04:00
Matt Peters
037786fc9f Preload the AssetDataGatherer's cache to remove a stall when waiting on the gatherer to load the data for packages requested early in engine startup.
#jira UE-146665
#rb Zousar.Shaker
#rnx
#preflight 6255932f9a0f82a3151b55a1

[CL 19721987 by Matt Peters in ue5-main branch]
2022-04-12 11:33:35 -04:00
Dan Thompson
cd73707429 Update package writers to gather package hashes internally, then have CookByTheBookFinished copy them to asset package datas.
Add iostore chunk hashes to the asset registry during cook.
Remove old hashing code, which was extensive. This moves the async writes in to UE::Task from the old AsyncWorkSequence().
#rb Matt.Peters
#rb Jeff.Roberts
#preflight 624b24a5f73c316f68303946

[CL 19611812 by Dan Thompson in ue5-main branch]
2022-04-04 13:26:27 -04:00
Matt Peters
fb70aad760 AssetDataGatherer: Fix typo in ReadAssetFile found by static analysis.
#rb None, trivial
#rnx
#preflight 6246eef63a5a4c1622dff481

[CL 19586857 by Matt Peters in ue5-main branch]
2022-04-01 08:38:46 -04:00
Matt Peters
46290e86ee AssetDataGatherer: Change serialization format of dependencies in the editor's AssetDataGatherer cache to improve load performance.
#rb Zousar.Shaker
#rnx
#preflight 6245a64e637925b5d3c6c729

[CL 19571741 by Matt Peters in ue5-main branch]
2022-03-31 09:16:55 -04:00
Matt Peters
63269cfdf7 AssetRegistry: Add EnumerateAllPackages.
#rb Zousar.Shaker
#rnx
#preflight 623c9eb19f11d3c73943818d

[CL 19498012 by Matt Peters in ue5-main branch]
2022-03-24 12:55:04 -04:00
Matt Peters
6bc3029111 PackageReader and EditorDomain: add all objects of type UScriptStruct to the list of ImportedClasses. UScriptStruct objects might be used for serialization even if they are not the class of an export in the package. One example of this is UDataTable's RowStruct.
#rb Zousar.Shaker
#rnx
#jira UE-146751
#preflight 623a2cb789625f0612b655ae

[CL 19471743 by Matt Peters in ue5-main branch]
2022-03-22 16:31:12 -04:00
Matt Peters
2cb495da5d AssetRegistry Redirector references fixup: Skip the fixup for package versions that do not need it. The fixup pessimizes the cook for any package that contains both redirectors and other assets with editor-only references.
#rb Zousar.Shaker
#rnx
#preflight 623231e4f41c515c3717e6e5

[CL 19409953 by Matt Peters in ue5-main branch]
2022-03-16 15:13:49 -04:00
Matt Peters
0e341a856d Add DoesPackageExistOnDisk to the AssetRegistry. Use it to check whether the AssetRegistry thinks a package exists without having to copy the entire FAssetPackageData.
In a future change we will add tracking of extension and of package name capitalization to FAssetPackageData, and will use DoesPackageExistOnDisk to replace the more expensive PackageResourceManager::TryMatchCaseOnDisk calls that occur when loading a package.

Change FPackageDatas to use DoesPackageExist, which is based on the AssetPackageDatas, rather than using GetAssetsByPackageName, because GetAssetsByPackageName can have false negatives from packages that are missing assets and can have false positives from in-memory packages that have added their Assets to the AssetRegistryState via ProcessLoadedAssetsToUpdateCache.

#jira UE-144429
#jira UE-143860
#rb Johan.Torp
#rnx
#preflight 622f75d26131e07703ac806b

[CL 19374576 by Matt Peters in ue5-main branch]
2022-03-14 13:20:30 -04:00