As part of this change we also promote dynamic resolution and IO/PackageQueueDepth stats to Minimal since they're important for high level performance reporting.
Also fix up a few places that were redundantly using #if CSV_PROFILER around CSV macros.
#rb mickael.gilabert
[CL 34386798 by ben woodhouse in ue5-main branch]
- Fix OnAssetLoaded so it is called in -game to match the behavior of the previous loaders
#rnx
#rb kevin.macaulayvacher
[CL 34306179 by danny couture in ue5-main branch]
- Any code causing a linker mismatch needs to explicitly clear loading flags for their object to avoid asserting. This is to make the intention clear that rename will make loading correctly impossible when moving the object to a different linker unintentionally. So either clear the loading flags of the object (i.e. finish loading via load calls such as ConditionalPostLoad) or forcibly allow mismatching via the new flag.
- When marking an export as invalid in FLinkerLoad::InvalidateExport() we now also clear loading flags since it is wasteful to load invalid objects.
- Exports marked as invalid are no longer reset to be valid when the linker is reset for the object instance. This is to prevent reloading invalid objects when re-using a linker already resident in memory if a subsequent package load is requested.
- When UStruct upgrades UFields to FFields, the UStruct objects are marked as invalid to prevent reloading the objects unnecessarily
- Fixes an issue where since Rename can clear the linker for mismatched types, any calls to FLinkerLoad::InvalidateExport needs to be moved before the rename operation to be effective.
- CollectUnreachableObjects will no longer mark unreachable objects as invalid in the linkerloader. That code would mark objects as invalid until the linker is destroyed which incorrectly assumed would happen soon. If however a load for the same package occurs before the linker is destroyed, the in-memory linker would be re-used keeping the marked exports as invalid so those exports would not be loaded (even though they should have been).
#jira UE-212466 UE-214849
#rb Francis.Hurteau, Michael.Galetzka
[RN] UObject::Rename will always remove the renamed object's linker if the rename moves the object into a different package. As a result, any renames occuring while an object is loading will now assert instead of leading to hard to diagnose crashes long after the Rename call has completed. If renaming to a different package is intentional and the linker should not be cleared, the new flag REN_AllowPackageLinkerMismatch can be used to prevent clearing linkers on the object even though the linker is for a different package than the package has been renamed to be part of.
[CL 34304935 by kevin macaulayvacher in ue5-main branch]
Implementations have been effectively using the thread context and these changes make that the only available context.
#rb kevin.macaulayvacher
[CL 34011179 by devin doucette in ue5-main branch]
This fixes:
- The previous container relied on a fixed array of sorted indices. Each insertion would mean memmov'ing chunks indices and values
- Removing elements doesn't actually shrink the table (that's mostly fine) but instead adds a sentinel value. This means anytime a value is looked, for a sentinel value comparison is needed. When fetching all publicexports, we need to walk the whole table despite it not being uncommon in editor for exports to be invalidated and can be removed from iteration entirely.
When loading a UMap with 2M exports, we now take 25s compared to 44m. This usecase comes from a real customer umap.
#jira UE-201114
#rb danny.couture, Francis.Hurteau
[CL 33723462 by kevin macaulayvacher in ue5-main branch]
Backout until we can determine all potential knock-ons
[FYI] kevin.macaulayvacher
Original CL Desc
-----------------------------------------------------------------
Fixes two issues in zenloader around package export invalidation:
- Previously CollectUnreachableObjects would mark GC'd sub-objects as invalid. This could break future loads of the package while the package (but not all its sub-objects) are still in memory. (UE-213934)
- Previously importing a package which may convert/upgrade it's exports could result in the old objects lingering. These lingering objects might cause issues during cook/saving (UE-213910, note while this bug is specifically for -nozenloader, the changes in this CL fix the same issue for -zenloader)
To fix the above, when a package is finished loading we synchronize the FAsyncPackage2 exports with LFLinkerLoad's exports and ensure the FAsyncPackage2 contains at least the same invalid/bExportLoadFailed=true exports. These exports and any export that is a sub-object to an invalid export, are stored in the GlobalImportStore. When we next directly load or import a package, we will filter out any invalid exports. Note, we synchronize the invalid exports at the end of package loading because Preload and PostLoad codepaths have opportunities to alter exports, so we wait for those to complete before we do any recording. When a package is removed from memory, we remove all invalid exports from the GlobalImportStoreto prevent the InvalidExport map growing unbounded.
CollectUnreachableObjects no longer marks GC'd objects as invalid, and instead we rely on the logic described above. This prevents us from incorrectly marking valid objects, we should reload when later requested, as invalid.
Two tests were added to validate the two issues above, and in doing so the existing AsyncLoading test files that were scattered in the test directory have been moved to a test specific directory for clarity
#jira UE-213934, UE-213910
#rb danny.couture, Francis.Hurteau
#virtualized
[CL 33659845 by kevin macaulayvacher in ue5-main branch]
- Previously CollectUnreachableObjects would mark GC'd sub-objects as invalid. This could break future loads of the package while the package (but not all its sub-objects) are still in memory. (UE-213934)
- Previously importing a package which may convert/upgrade it's exports could result in the old objects lingering. These lingering objects might cause issues during cook/saving (UE-213910, note while this bug is specifically for -nozenloader, the changes in this CL fix the same issue for -zenloader)
To fix the above, when a package is finished loading we synchronize the FAsyncPackage2 exports with LFLinkerLoad's exports and ensure the FAsyncPackage2 contains at least the same invalid/bExportLoadFailed=true exports. These exports and any export that is a sub-object to an invalid export, are stored in the GlobalImportStore. When we next directly load or import a package, we will filter out any invalid exports. Note, we synchronize the invalid exports at the end of package loading because Preload and PostLoad codepaths have opportunities to alter exports, so we wait for those to complete before we do any recording. When a package is removed from memory, we remove all invalid exports from the GlobalImportStoreto prevent the InvalidExport map growing unbounded.
CollectUnreachableObjects no longer marks GC'd objects as invalid, and instead we rely on the logic described above. This prevents us from incorrectly marking valid objects, we should reload when later requested, as invalid.
Two tests were added to validate the two issues above, and in doing so the existing AsyncLoading test files that were scattered in the test directory have been moved to a test specific directory for clarity
#jira UE-213934, UE-213910
#rb danny.couture, Francis.Hurteau
#virtualized
[CL 33602309 by kevin macaulayvacher in ue5-main branch]
- Fix TSAN warning on FApp::IsGame() changing on GT while creating linkers by moving the check as the very last condition where it is highly unlikely to ever get reached in editor (the only place it can change).
- Since the value should never changes in -game and GIsEditor should never changes in editor we should never get into a state where that value dynamically affects the outcome of the loading but TSAN complained because we read it in all cases.
#jira UE-175442
#rb kevin.macaulayvacher
[CL 33536251 by danny couture in ue5-main branch]
#rb Per.Larsson
#rnx
- If a user tries to make a read request from FBulkDataBatchRequest that goes past the end of the bulkdata's payload we will eventually assert, but only at the point of reading which is often quite far away (context wise) from the code that initiated the request.
- Previously we were asserting if the user made a read request that exceeded the size of the bulkdata payload but this did not take into account reading from an offset.
- Fixed the assert to take offset into account so we now asset on invalid reads at the point that the read is first requested. In addition I have added more debug info to the assert making it easier to tell from the log file what was incorrect about the read request.
- Changed the assert to an ensure because although a too large size usually indicates a higher level bug, the lower level file system code will generally prevent us from reading past the end of the payload and return the data up until the end of the payload. So far in every case where we have seen this happening, reading to the end of the payload was acceptable and the higher level code worked. An ensure will allow the problem to remain visibile to programmers but not block content teams should the error make it to production.
-- This logic needs to be revisited as we should be mroe consistent with async and sync file reads and how we return errors, ideally allowing the higher level code to decide if a failure is important or not as we do not really have enough context at lower level to decide.
[CL 33479013 by paul chipchase in ue5-main branch]
- Detect when loader is hung and dump its state if it happens
- Add support for full flush from inside the last loaded package's callback if it ever happens
- Add unittest for that use case
#jira UE-207196, UE-207324
#rb kevin.macaulayvacher
[CL 33195963 by danny couture in ue5-main branch]
It does not actually require any package summary information since it only relies on TotalExportBundleCount which is known already in InitializeAsyncPackageFromPackageStore.
Fix nullptr crash in FAsyncLoadingThread2::FindOrInsertPackage for the rare special case path where the following dependency is added:
ImportedByPackage->GetExportBundleNode(ExportBundle_Process, DependentExportBundleIndex).DependsOn(&Package->GetExportBundleNode(ExportBundle_DeferredPostLoad, DependsOnExportBundleIndex));
#rb danny.couture, Per.Larsson
#tests Running Client, Server, Editor, CookedEditor
[CL 33194567 by pj kack in ue5-main branch]
