This change consists of multiple changes:
Core:
- Deprecation of ANY_PACKAGE macro. Added ANY_PACKAGE_DEPRECATED macro which can still be used for backwards compatibility purposes (only used in CoreUObject)
- Deprecation of StaticFindObjectFast* functions that take bAnyPackage parameter
- Added UStruct::GetStructPathName function that returns FTopLevelAssetPath representing the path name (package + object FName, super quick compared to UObject::GetPathName) + wrapper UClass::GetClassPathName to make it look better when used with UClasses
- Added (Static)FindFirstObject* functions that find a first object given its Name (no Outer). These functions are used in places I consider valid to do global UObject (UClass) lookups like parsing command line parameters / checking for unique object names
- Added static UClass::TryFindType function which serves a similar purpose as FindFirstObject however it's going to throw a warning (with a callstack / maybe ensure in the future?) if short class name is provided. This function is used in places that used to use short class names but now should have been converted to use path names to catch any potential regressions and or edge cases I missed.
- Added static UClass::TryConvertShortNameToPathName utility function
- Added static UClass::TryFixShortClassNameExportPath utility function
- Object text export paths will now also include class path (Texture2D'/Game/Textures/Grass.Grass' -> /Script/Engine.Texture2D'/Game/Textures/Grass.Grass')
- All places that manually generated object export paths for objects will now use FObjectPropertyBase::GetExportPath
- Added a new startup test that checks for short type names in UClass/FProperty MetaData values
AssetRegistry:
- Deprecated any member variables (FAssetData / FARFilter) or functions that use FNames to represent class names and replaced them with FTopLevelAssetPath
- Added new member variables and new function overloads that use FTopLevelAssetPath to represent class names
- This also applies to a few other modules' APIs to match AssetRegistry changes
Everything else:
- Updated code that used ANY_PACKAGE (depending on the use case) to use FindObject(nullptr, PathToObject), UClass::TryFindType (used when path name is expected, warns if it's a short name) or FindFirstObject (usually for finding types based on user input but there's been a few legitimate use cases not related to user input)
- Updated code that used AssetRegistry API to use FTopLevelAssetPaths and USomeClass::StaticClass()->GetClassPathName() instead of GetFName()
- Updated meta data and hardcoded FindObject(ANY_PACKAGE, "EEnumNameOrClassName") calls to use path names
#jira UE-99463
#rb many.people
[FYI] Marcus.Wassmer
#preflight 629248ec2256738f75de9b32
#codereviewnumbers 20320742, 20320791, 20320799, 20320756, 20320809, 20320830, 20320840, 20320846, 20320851, 20320863, 20320780, 20320765, 20320876, 20320786
#ROBOMERGE-OWNER: robert.manuszewski
#ROBOMERGE-AUTHOR: robert.manuszewski
#ROBOMERGE-SOURCE: CL 20430220 via CL 20433854 via CL 20435474 via CL 20435484
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v949-20362246)
[CL 20448496 by robert manuszewski in ue5-main branch]
#rb Paul.Chipchase, PJ.Kack, CarlMagnus.Nordin
#jira none
#preflight 629085d14f63120d8e170977
###
This change attempts to merge the editor and the runtime version of bulk data, i.e. bulk data one and two.
The major difference between the two is the fact that the editor version supported element vice
serialization via inheritance. This has been complelety removed in the runtime version. In order
to mitigate this divergance a non-virtual base class has been extracted that forms the base
for the untyped and typed versions of bulk data.
The untyped version is only used in editor and will be deprecated. However this is a breaking change,
previously it was possible to inherit from FByteBulkData in editor builds.
Other differences between the two includes:
* IsBulkDataLoaded - returns true for bulk data 1 if an allocation has been made even if the size is zero
* StartStreaming - was removed in bulk data 2 and has been removed
* FlushAsyncLoading - bulk data 2 was busy waiting, now an event is used
All I/O related functionality has been extracted into BulkDataStreaming.cpp and is using
a FBulkDataChunkId that is either a package path or a package ID.This could potentially
be extended to contain a cache key in development builds to support loading from DDC, i.e derived
data references.
[CL 20391050 by Per Larsson in ue5-main branch]
This is necessary because as of CL 20151503, the cooker can commit the same package multiple times if the first save failed due to a timeout.
#fyi CarlMagnus.Nordin
#rnx
#rb Per.Larsson
#preflight 62853ba90f0d5076dee74ced
[CL 20265738 by Matt Peters in ue5-main branch]
#rb Per.Larsson
#rnx
#jira UE-152823
#preflight 6284afb4f239239af66dd215
### Problem
- When saving an editor bulkdata with an invalid (empty) payload, it was being incorrectly identified has sharing a payload with another bulkdata instance that had already been virtualized and so was applying the virtualization flag to itself on post save.
- When saved a second time this incorrect flag would be saved to the bulkdata object itself.
- When loading we would detect that the bulkdata believed itself to be virtualized even though virtualizing an empty payload makes no sense and would emit the log warning.
### Fix
- When fixing up an editor bulkdata after saving we should not check to see if the payload is a virtualized duplicate if the payload is invalid.
- Additionally we should not record the package path if the payload is invalid as there would be nothing to load from.
[CL 20258991 by paul chipchase in ue5-main branch]
#rb Per.Larsson
#rnx
#jira UE-151377
#preflight 628364050039ea57a52d6989
### Virtualization
- [Core.ContentVirtualization] in the engine ini file now supports an array called 'DisabledAsset' which can be used to name asset types that should not virtualize their payloads.
-- By default (in BaseEngine.ini) we have disabled the StaticMesh asset as we know it will crash if a payload is missing and the SoundWave asset as it still is pending testing.
- This new way to disable virtualization is data driven. The older hard coded method has not been removed but will likely be reworked in a future submit.
- Now when an editor bulkdata is adding it's payload to the package trailer builder during package save it will poll the virtualization system with a call to the new method ::IsDisabledForObject by passing in it's owner.
-- If the owner is valid and was present in the 'DisabledAsset' array then the method will return true and the EPayloadFlags::DisableVirtualization flag will be applied.
### Package Trailer
- The pre-existing functionality of enum EPayloadFilter has been moved to a new enum EPayloadStorageType as will only filter payloads based on their storage type.
- EPayloadFilter has been modified to filter payloads based on functionality although at the moment the only thing it can filter for is to return payloads that can be virtualized, it is left for future expansion.
- EPayloadFlags has been reduced to a uint16 with the remaining 2bytes being turned into a new member EPayloadFilterReason.
- This new member allows us to record the exact reason why a payload is excluded from virtualization. If it is zero then the payload can virtualize, otherwise it will contain one or more reasons as to why it is being excluded. For this reason the enum is a bitfield.
- Added overloads of ::GetPayloads and ::GetNumPayloads that take EPayloadFilter rather than a EPayloadStorageType
- Added wrappers around all AccessMode types for FLookupTableEntry.
- FPackageTrailerBuilder has been extended to take a EPayloadFilterReason so that the caller can already provide a reason why the payload cannot be virtualized.
-- As a future peace of work this will probably be changed and we will ask the caller to pass in the owner UObject pointer instead and then we will process the filtering when building the package trailer to a) keep all of the filtering code in one place b) keep the filtering consistent
### PackageSubmissionChecks
- The virtualization process in will now request the payloads that can be virtualized from the package trailer, which respects the new payload flag, rather than requesting all locally stored payloads.
### UObjects
- There is no need for the SoundWave or MeshDescription classes to opt out of virtualization on construction. This will be done when the package is saved and is now data driven rather than being hardcoded.
### DumpPackagePayloadInfo
- The command has been updated to also display the filter reasons applied to each payload
[CL 20240971 by paul chipchase in ue5-main branch]
#rb Phillip.Kavan
Add missing generate public hash entries to ArchiveStackTrace
#fyi Francis.Hurteau
#preflight 626c692cb046e6ecc338080e
#lockdown JeanFrancois.Dube
[CL 19988606 by Marc Audy in ue5-main branch]
#rb trivial
#rnx
#preflight 626b9cbb53253f874b90efd3
- Although we don't really want to write out the virtualization flag, by removing it during ::BuildFlagsForSerialization we later check UpdatedFlags to see if the payload is virtualized and if we should write it out to disk or not.
- Removing the flag means we re-hydrate virtualized payloads when saved to disk, or in the case of an editordomain, try to store it as a local payload in the editor domain trailer which is not supported (if we did have a newly generated payload in an editor domain save it should go to the legacy path for now)
- The fix for this is probably quite simple, but the testing it would require means it's better to pull out the line of code and allow packages to serialize out the virtualization flag for now until a better fix is ready.
[CL 19976045 by paul chipchase in ue5-main branch]
#rb PJ.Kack
#rnx
#preflight 626a7ef2485a2fed3a2e04d7
- Note that these problems were not causing crashes, but they did cause errors to be logged about invalid formats and request the packages to be re-saved.
- Most of the problem comes from the code paths being too complicated, hopefully now things are locked down we should be able to start pruning this soon and make it easier to maintain in the future.
### Problem
- The problem comes from certain package setups that allow for the existence of duplicate payload entries where one entry is local or referenced and another version is virtualized. If a trailer is created from this data then the virtualized entry will be discarded (as the look up table is a map) and we revert to using the non virtualized payload instead. Ideally we'd want to discard the local/referenced payloads in favour of the virtualized one to save space.
-- By itself this bug would waste disk space but not cause other problems, however we were also writing out the virtualized flag so that when the bulkdata is next loaded it has the virtualized flag but then checks with the trailer and is told that it is not virtualized which triggers some asserts I added to find badly formed data.
- How the package gets duplicate entries like this is somewhat complicated involing duplicating objects from other (already virtualized) packages during load, then saving them (found in packages with Niagra assets at least)
### Fix
- When creating a trailer from a trailer build, we first find any local and/or referenced entries and check if there are duplicate entries in the virtualized payload entries. If so we remove the local/referenced version in favor of using the virtualized version.
-- Ideally we'd do this during the serialization of exports as it might let us skip the loading of some payloads for save entirely, however we don't know all the entries until after export serialization is completed so this is the only point we can search for duplicates safely.
- Added a way to get the full name of the owning asset, including package name + asset, when asking the user to re-save. This makes debugging the problems easier.
- Add a new check when loading to find payloads saved with the virtualization flag and to a trailer. If we do find one we warn the user that they should re-save the package and remove the flag so that no further problems occur.
- Demoted the old check for bad formats to a warning as we don't need to fail builds because of it.
-- Additionally we now only give the warning if we detect that we are loading from a package, as being virtualized and not in a package trailer would be valid when loading from other sources.
- Added comments to consider removing both checks asking the user to re-save the package after 5.2, as in theory we should only ever see these problems internally at Epic as it is unlikely that externals will have enabled the system.
- When saving a virtualized payload to the trailer, make sure to remove the virtualized flag, that should be picked up on load by polling the trailer for the payload status and should not be baked into the export data.
-- Hypothetically this will allow for us to re-hydrate the package without re-saving it through manipulating the trailer.
[CL 19963043 by paul chipchase in ue5-main branch]
#rb Matt.Peters
#rnx
#preflight 626a9f386461dd769fea25df
- If a payload was virtualized it was incorrectly being marked as requiring the legacy serialization path.
- This would technically work but was triggering a log warning about the bulkdata being stored in the wrong format (since it was in the legacy format) as we do not expect bulkdata with virtualized payloads to use the legacy path.
- We need to change the editor domain version to clear out any bad entries
[CL 19961539 by paul chipchase in ue5-main branch]
Remove bValid from FMetaData and FData return structs; it is redundant with the RawHash/Buffer data already present in those structs.
#preflight 62680b42430b9997ebe0a46d
#rb Devin.Doucette
#rnx
[CL 19923039 by Matt Peters in ue5-main branch]
