Commit Graph

878 Commits

Author SHA1 Message Date
PJ Kack
70f75f6648 AsyncLoading2: Make sure to call PostLoadInstances and CreateClusters before ClearConstructedObjects, because after ClearConstructedObjects unreferenced private objects in the package may be garbage collected.
Fix the underlying problem for the workaround in 19840451.

#jira FORT-456060
#rb carlmagnus.nordin
#robomerge EngineMerge
#rnx
#preflight 62a983aa3774a57fdc4f07f5

[CL 20666261 by PJ Kack in ue5-main branch]
2022-06-15 03:14:53 -04:00
Catalin Dragoiu
24f5bb2f6c Improve tracking of the LoadTime value in CookProfiler
#rb Matt.Peters Francis.Hurteau
#preflight 62a8b512136e62bef1cec185

[CL 20653168 by Catalin Dragoiu in ue5-main branch]
2022-06-14 13:45:02 -04:00
CarlMagnus Nordin
8188784a84 AsyncLoading2:
Merge loaded package store with global import store
Use global import store to keep track of pinned exports instead of relying on FindObjectWithOuter
#preflight 62a876325d0f5c5aab6d380c
#rb pj.kack

[CL 20645631 by CarlMagnus Nordin in ue5-main branch]
2022-06-14 08:09:17 -04:00
PJ Kack
b0bc254d82 AsyncLoading 1&2: Refactor initial load to remove zen loader dependency on script object blob in order to support editor loading.
NotifyRegistrationEvent:
Remove all NRP_Started events since they add no value.
Move the NRP_Finished event for type NRT_Class from InitializePrivateStaticClass (Class.cpp) to UObjectLoadAllCompiledInDefaultProperties (UObjectBase.cpp) since this is where the outer registration function for the class is actually executed.
Add a new NRT_NoExportObject event in ConstructUPackage for top level noexport objects like UScriptStruct, UDelegateFunction and USparseDelegateFunction that are just part of the package and has no registration info.

AsyncLoader:
Move EDLBootNotificationManager from AsyncPackageLoader to AsyncLoading.
Add FEarlyRegistrationEventsRecorder to replay early events that occur before the loader is created.

AsyncLoading:
Ignore the new NRT_NoExportObject event.
Allow for objects to skip the NRP_Added and NRP_Started phases.

AsyncLoading2:
Remove LazyInitializeFromLoadPackage and the reading and parsing of the ScriptObjects binary blob.
Remove FindScriptImportObjectFromIndex/GFindExistingScriptImport that was dependent on the ScriptObject blob, instead all script objects are added to the global import store directly from the notification system callbacks.
Change GetUnresolvedCDOs logic to check for imported classes with missing or not-yet-initialized CDOs.
Add FAsyncPackageDesc2::ReferencerRequestId to let each package track its most recent load package request referencer.
Change ProcessPendingCDOs logic to avoid recursive reentrancy and to process the most relevant CDOs first, based on the new ReferencerRequestId.

Tests:
Lyra: Editor, Runtime EDL and Zen loader on Win64 and Console
ShooterGame: Editor, Runtime EDL and Zen loader on Win64 and Console
AncientGame: Editor, Runtime EDL and Zen loader on Win64
Other: Editor, CookedEditor, Runtime EDL and Zen loader on Win64 (client and server)

#jira none
#rb carlmagnus.nordin
#robomerge EngineMerge
#rnx
#preflight 62a84ba7cf54a658eeffc200

[CL 20643384 by PJ Kack in ue5-main branch]
2022-06-14 05:37:32 -04:00
Per Larsson
9dd47b0f5c Bulk Data - fix bug when PollCompletion returns true before the callback is triggered
#rb Paul.Chipchase
#preflight 62a2fc792e1cc34f11d2c02e

[CL 20593479 by Per Larsson in ue5-main branch]
2022-06-10 07:36:56 -04:00
Per Larsson
bd3422ad50 Bulk Data - correctly convert async file I/O priority to I/O dispatcher priority
#rb CarlMagnus.Nordin
#jira UE-156135
#prefligh t62a32338d708f949d6486971

[CL 20593192 by Per Larsson in ue5-main branch]
2022-06-10 06:58:03 -04:00
carlmagnus nordin
bcc92ae155 AsyncLoading2: Also store public export objects without the RF_WasLoaded in the import store when loading in the editor (BP compilation can clear this flag)
#preflight 62a1ab00ba2f7b30780f5640
#rb pj.kack

#ROBOMERGE-AUTHOR: carlmagnus.nordin
#ROBOMERGE-SOURCE: CL 20572851 via CL 20573416 via CL 20573422 via CL 20573436
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v954-20466795)

[CL 20573897 by carlmagnus nordin in ue5-main branch]
2022-06-09 07:37:41 -04:00
paul chipchase
337ff65efa Remove the validation option when loading payloads from sidecar files as we now validate all loaded payloads at a higher level.
#rb trivial
#rnx
#preflight 62a1cb1fcf774bcd46cac2de

[CL 20573562 by paul chipchase in ue5-main branch]
2022-06-09 06:50:55 -04:00
paul chipchase
d802d4d4b2 EditorBulkdata now longer stores a EPackageSegment member since it is always set to EPackageSegment::Header
#rb Per.Larsson
#rnx
#preflight 629f530958032863438a0b30

- Storing the segment type as a member made the code would imply to the reader that it could be changed, even though in all paths it was being set to  EPackageSegment::Header. Removing the member and just hard coding the segment use should make the code easier to follow.
- Editor bulkdata always stores it's payload in the package trailer, which for now counts as the package header.
- The only deviation for this is the experimental feature that allows us to store the payloads in a sidecar file. This code path is not currently supported, but has been updated to use hardcoded EPackageSegment::PayloadSidecar.
- Given that PackageSegment was always set to EPackageSegment::Header we can easily remove any assert of if statement checking for that.

[CL 20553326 by paul chipchase in ue5-main branch]
2022-06-08 04:15:48 -04:00
Per Larsson
3da8d7d7ab Bulk Data - fix bug when calling IBulkDataRequest::GetReadResult() before calling IBulkDataRequest::GetSize()
#jira UE-155629
#rb Paul.Chipchase
#preflight 6299b9cb9e46af6513cd8aa0
#rnx

[CL 20484034 by Per Larsson in ue5-main branch]
2022-06-03 03:57:02 -04:00
Per Larsson
a97300da1f Bulk Data - remove checking the global GEventDrivenLoader and add support for loading from optional packages
#rb CarlMagnus.Nordin
#jira none
#preflight 62986a87a07cbdb4285b3699

[CL 20469669 by Per Larsson in ue5-main branch]
2022-06-02 10:53:10 -04:00
Per Larsson
1a72e611cf Bulk data - fix bug when loading bulk data referenced from the original .uasset file
- added additional check that the bulk data is not loading from a referenced .uasset file when using the attached archive
- reverted back to set default Offset value to INDEX_NONE
- removed setting SizeOnDisk when transacting

#rb Paul.Chipchase
#jira UE-155428
#preflight 629759c4e809e387be1a8fc3

[CL 20451384 by Per Larsson in ue5-main branch]
2022-06-01 08:52:47 -04:00
Matt Peters
3c2010d281 AppendToClassSchemaContext: Hide the use of FBlake3 behind an interface so we can easily change it.
#rb Steve.Robb
#rnx
#preflight 62975270a660a44a23b4ac94

[CL 20451128 by Matt Peters in ue5-main branch]
2022-06-01 08:05:42 -04:00
robert manuszewski
d1443992e1 Deprecating ANY_PACKAGE.
This change consists of multiple changes:

Core:
- Deprecation of ANY_PACKAGE macro. Added ANY_PACKAGE_DEPRECATED macro which can still be used for backwards compatibility purposes (only used in CoreUObject)
- Deprecation of StaticFindObjectFast* functions that take bAnyPackage parameter
- Added UStruct::GetStructPathName function that returns FTopLevelAssetPath representing the path name (package + object FName, super quick compared to UObject::GetPathName) + wrapper UClass::GetClassPathName to make it look better when used with UClasses
- Added (Static)FindFirstObject* functions that find a first object given its Name (no Outer). These functions are used in places I consider valid to do global UObject (UClass) lookups like parsing command line parameters / checking for unique object names
- Added static UClass::TryFindType function which serves a similar purpose as FindFirstObject however it's going to throw a warning (with a callstack / maybe ensure in the future?) if short class name is provided. This function is used  in places that used to use short class names but now should have been converted to use path names to catch any potential regressions and or edge cases I missed.
- Added static UClass::TryConvertShortNameToPathName utility function
- Added static UClass::TryFixShortClassNameExportPath utility function
- Object text export paths will now also include class path (Texture2D'/Game/Textures/Grass.Grass' -> /Script/Engine.Texture2D'/Game/Textures/Grass.Grass')
- All places that manually generated object export paths for objects will now use FObjectPropertyBase::GetExportPath
- Added a new startup test that checks for short type names in UClass/FProperty MetaData values

AssetRegistry:
- Deprecated any member variables (FAssetData / FARFilter) or functions that use FNames to represent class names and replaced them with FTopLevelAssetPath
- Added new member variables and new function overloads that use FTopLevelAssetPath to represent class names
- This also applies to a few other modules' APIs to match AssetRegistry changes

Everything else:
- Updated code that used ANY_PACKAGE (depending on the use case) to use FindObject(nullptr, PathToObject), UClass::TryFindType (used when path name is expected, warns if it's a short name) or FindFirstObject (usually for finding types based on user input but there's been a few legitimate use cases not related to user input)
- Updated code that used AssetRegistry API to use FTopLevelAssetPaths and USomeClass::StaticClass()->GetClassPathName() instead of GetFName()
- Updated meta data and hardcoded FindObject(ANY_PACKAGE, "EEnumNameOrClassName") calls to use path names

#jira UE-99463
#rb many.people
[FYI] Marcus.Wassmer
#preflight 629248ec2256738f75de9b32

#codereviewnumbers 20320742, 20320791, 20320799, 20320756, 20320809, 20320830, 20320840, 20320846, 20320851, 20320863, 20320780, 20320765, 20320876, 20320786

#ROBOMERGE-OWNER: robert.manuszewski
#ROBOMERGE-AUTHOR: robert.manuszewski
#ROBOMERGE-SOURCE: CL 20430220 via CL 20433854 via CL 20435474 via CL 20435484
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v949-20362246)

[CL 20448496 by robert manuszewski in ue5-main branch]
2022-06-01 03:46:59 -04:00
CarlMagnus Nordin
3d5a2052dc AsyncLoading2: Add support for having multiple PackageStore backends
#preflight 6296fc872a1851b4ccae7e39
#rb pj.kack

[CL 20447121 by CarlMagnus Nordin in ue5-main branch]
2022-06-01 02:12:33 -04:00
Devin Doucette
87340081ea DerivedData: Added a new API in place of the use of IoDispatcher
#preflight 62961234e62c873b96c939d5
#rb Zousar.Shaker

[CL 20445536 by Devin Doucette in ue5-main branch]
2022-05-31 22:01:40 -04:00
Per Larsson
b1fcb5183a BulkData - enable serialize bulk data size as 64bit integer by setting BULKDATA_Size64Bit
#rb Paull.Chipchase
#jira none
#preflight 62964d390126cadbd3cccf89

[CL 20440503 by Per Larsson in ue5-main branch]
2022-05-31 15:18:37 -04:00
Per Larsson
88f2de0b83 Removed disable optimization pragma and unused header
#rb none
#jira none
#preflight 629094a9f6668c14a3c758a1

[CL 20391200 by Per Larsson in ue5-main branch]
2022-05-27 05:24:20 -04:00
Per Larsson
c30e677d1b Merge BulkData 1&2
#rb Paul.Chipchase, PJ.Kack, CarlMagnus.Nordin
#jira none
#preflight 629085d14f63120d8e170977

###
This change attempts to merge the editor and the runtime version of bulk data, i.e. bulk data one and two.
The major difference between the two is the fact that the editor version supported element vice
serialization via inheritance. This has been complelety removed in the runtime version. In order
to mitigate this divergance a non-virtual base class has been extracted that forms the base
for the untyped and typed versions of bulk data.
The untyped version is only used in editor and will be deprecated. However this is a breaking change,
previously it was possible to inherit from FByteBulkData in editor builds.
Other differences between the two includes:
* IsBulkDataLoaded - returns true for bulk data 1 if an allocation has been made even if the size is zero
* StartStreaming - was removed in bulk data 2 and has been removed
* FlushAsyncLoading - bulk data 2 was busy waiting, now an event is used

All I/O related functionality has been extracted into BulkDataStreaming.cpp and is using
a  FBulkDataChunkId that is either a package path or a package ID.This could potentially
be extended to contain a cache key in development builds to support loading from DDC, i.e derived
data references.

[CL 20391050 by Per Larsson in ue5-main branch]
2022-05-27 04:31:58 -04:00
devin doucette
b2a07ea03e DDC: Merge from UE5/Main
#preflight 6288ff678828ea88c8af7034
#preflight 628ab5d93246d5019db76ed2
#rb none
#rnx

#ROBOMERGE-OWNER: devin.doucette
#ROBOMERGE-AUTHOR: Devin.Doucette
#ROBOMERGE-SOURCE: CL 20353148 via CL 20353832 via CL 20353839
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v948-20297126)

[CL 20355348 by devin doucette in ue5-main branch]
2022-05-24 16:40:25 -04:00
Matt Peters
0baa95d653 EditorBulkDatas need to create new guids to avoid duplication when they are loading from an instanced load, since instanced loads duplicate data from a source package into a new package of a different name.
#jira UE-152829
#rb Paul.Chipchase
#rnx
#preflight 628cd21cf057b981ca5edf3f

[CL 20347479 by Matt Peters in ue5-main branch]
2022-05-24 09:36:42 -04:00
CarlMagnus Nordin
a694071dbe AsyncLoading2: Keep the full package path for loaded packages instead of just the package name
#rb pj.kack
#preflight 628c99cff057b981ca4cdbc3

[CL 20345705 by CarlMagnus Nordin in ue5-main branch]
2022-05-24 05:33:01 -04:00
PJ Kack
c64e25117a AsyncLoading2: Fix nullptr crash inside check format statement.
#jira UE-153083
#rb trivial
#rnx
#preflight 62873f779016c6dd89bfedca

[CL 20291931 by PJ Kack in ue5-main branch]
2022-05-20 03:26:53 -04:00
Matt Peters
02ba0b5b70 ZenStoreWriter: Fix assertion "Trying to add package that is already pending" if the cooker commits a package a second time while the first commit of the package is queued for asynchronous completion aand has not yet reached CommitPackageInternal.
This is necessary because as of CL 20151503, the cooker can commit the same package multiple times if the first save failed due to a timeout.

#fyi CarlMagnus.Nordin
#rnx
#rb Per.Larsson
#preflight 62853ba90f0d5076dee74ced

[CL 20265738 by Matt Peters in ue5-main branch]
2022-05-18 14:49:11 -04:00
Matt Peters
820edc49b5 EditorDomain: Add UObject::AppendToClassSchema so that textures can declare EnableLegacyAlphaCoverageThresholdScaling and use different EditorDomain packages when that variable is changed. This will also add to the new hybrid iterative cook system to allow detection of when iteratively-cooked packages need to be invalidated due to changes in Serialize and PostLoad.
#rb Zousar.Shaker
#rnx
#preflight 6284e723614041edb7763cef

[CL 20259777 by Matt Peters in ue5-main branch]
2022-05-18 09:02:27 -04:00