Create a new interface IPackageResourceManager that handles all attempts to load packages from storage. The default implementation passes through to loading from IFileManager. Future implementations will load from other services such as the EditorDomain and TargetDomain.
Add FPackagePath class that holds a LocalFilePath or LongPackageName, or a relative path to a package under a mounted content root (which can be converted into either the LocalFilePath or LongPackageName).
Modify LinkerLoad, BulkData, and AsyncLoader to use FPackagePath and IResourceManager.
Modify UPackage and FLinkerLoad to have a FPackagePath instead of a Filename.
#rb Paul.Chipchase, CarlMagnus.Nordin, Francis.Hurteau, Devin.Doucette
#rn Major Core
[CL 14814912 by Matt Peters in ue5-main branch]
- This stops code compiling fine in the editor then breaking when compiled for a different config.
#rb Martin.Ridgers
#fyi ben.ingram
#rnx
[CL 14812609 by paul chipchase in ue5-main branch]
- Add protection against async property accessed during async build and postload.
- Split PostLoad into 3 steps with the most part now being made async.
- Make Build/BatchBuild API async when the feature is activated.
- Add a new StaticMesh compilation manager. (This will be refactory later to reduce duplicated code with texture compiler).
- Skip RenderState creation for any static mesh component still being built so they only show up when ready.
- Fixed DistanceField and MeshCard computation that might trigger too soon if fixups are required to RenderData during PostLoad.
- Dynamic priorisation for building and updating static meshes nearest to the viewport.
- Implement different PIE modes, current default is to stall only for mesh that might affect nagivation/collision for players/bots.
- Add a new generic interface for assets being built (still a WIP).
- Add ability in FrontEnd filters to skip serialization of asset being async compiled and refresh when compilation finishes.
- Prevent auto-save while textures and static meshes are being built (same as shaders)
- Logic has been reordered in some Fortnite Building Component to avoid unnecessarily touching static mesh properties.
General Optimizations
- Use cached thumbnails until shaders/textures/static meshes are ready to improve performance and avoid rendering incomplete thumbnails.
DEBUGGING
- Can be forcibly enabled/disabled through command-line via -asyncstaticmeshcompilation=[off, on, paused]
- Can pause staticmesh compilation using Editor.AsyncStaticMeshCompilation = 2 or -asyncstaticmeshcompilation=paused
- Can manually resume a specified amount of paused compilation using Editor.AsyncStaticMeshCompilationResume [Num]
- Can forcibly wait on all compilation using Editor.AsyncStaticMeshCompilationFlushAll
BENCHMARKS
- 2m19s to 17.9s for Loading FortGPUTestbed's LumenTest with an empty local only DDC
- 2h45m to 5mxxs for Loading Reverb's P_World with an empty local only DDC
- 17m29 to 10m27s for Loading Apollo_Terrain_Edit with an empty local only DDC
TESTS
- Loading maps from Reverb, Fortnite, ShooterGame, QAGame, FortGPUTestbed
- Content browser interactions while still under heavy compilation
- Importing with Datasmith / Dataprep
- Working with the new Mesh Modeling Tool
- Shutting down the editor while loading
- Most of EngineTest Test Suite
REFERENCES
- Design https://docs.google.com/document/d/1O4GI1G9AtQN6l0SaGFfXw0DyNhlLunLoEqne5INvxIQ
- Documentation https://docs.google.com/document/d/1KCdFEMhhcsGwfgOisTVwlOwtsmEd7qbB0V6Tc39Gb10
#rb Francis.Hurteau
[CL 14688146 by danny couture in ue5-main branch]
Also add protection so meshdescription can be read from multiple threads in editor
- Loading Reverb's P_World with empty DDC
- 319GB to 242GB peak private byte
- 166GB to 80GB permanent memory usage after map load
#rb Richard.Talbot-Watkins, Matt.Peters, Paul.Chipchase
[CL 14544739 by danny couture in ue5-main branch]
-Moved processing to async thread
-Switched to much more efficient batched IO using the new IODispatch API
-Added fallback/legacy path for when IODispatch is disabled or not supported on the platform
-Changed page size back to 128KB (from 256KB) and doubled number of streaming pages
-256MB saving from smaller root pages. 32MB of those used for IO staging buffer
#rb graham.wihlidal
#fyi brian.karis
[CL 14110783 by Rune Stubbe in ue5-main branch]
Move RefCount tracking for imports from import object level to package level.
Only track currently loaded public exports, and their packages.
Replace import object weak pointers with public export raw pointers (uobject index).
Add new NotifyUnreachableObjects GC hook to get information about all objects to be destroyed.
When a public export is destroyed, clean up its raw pointer, and mark its package as not loaded.
When a package is destroyed, clean up its ref count tracking data.
#rb per.larsson
#ushell-cherrypick of 13114335 by PJ.Kack
#ROBOMERGE-SOURCE: CL 13147624 via CL 13147625 via CL 13147626 via CL 13147627
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v688-13145358)
[CL 13147628 by pj kack in Main branch]
#jira none
#rb none
#ushell-cherrypick of 12997588 by PJ.Kack
#ROBOMERGE-SOURCE: CL 12997796 in //UE4/Release-4.25Plus/... via CL 12997808
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v682-12900288)
[CL 12997823 by pj kack in Main branch]
#jira none
#rb devin.doucette, stefan.boberg
#lockdown stefan.boberg
#ushell-cherrypick of 12918250 by paul.chipchase
#ROBOMERGE-SOURCE: CL 12919209 in //UE4/Release-4.25Plus/... via CL 12919222
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v682-12900288)
[CL 12919263 by paul chipchase in Main branch]
Sets ArIsError and ArIsCriticalError to private
[at]Graeme.Thornton [at]Johan.Torp [at]Steve.Robb [at]Carlos.Cuello
#rb Steve.Robb
#ROBOMERGE-OWNER: Eric.Knapik
#ROBOMERGE-AUTHOR: eric.knapik
#ROBOMERGE-SOURCE: CL 11949030 via CL 11949058 via CL 11949116 via CL 11949174
#ROBOMERGE-BOT: (v656-11643781)
[CL 11953024 by Eric Knapik in Main branch]
- Attempting to memory map inline data will give a log error(as the original implementation did but will not attempt to actually memory map it as it is very unlikely that inline data would have the correct alignment requirements for the operation to succeed so to keep the code simple we just serialize the inline data which was the existing fallback behaviour anyway.
- Added the memory mapping implementation as it's own method (MemoryMapBulkData) since we only need to support the non-inline case.
- Added a flag (bShouldForceLoad) that when true will force the BulkData2 data to be streamed in immediately. Currently used as a fallback if the memory mapping fails (all callers assume 100% success) but I made it generic enough that we could use it for something else if the use case occurs.
- This is a fairly quick fix to unblock testing, future work includes:
- Reduce the added memory bloat (adding 2 additional pointers to BulkData2)
- Implement copy/assignment of memory mapped BulkData (based on the above work)
#jira UE-85974
#rnx
#rb Per.Larsson
[CL 11194361 by paul chipchase in Main branch]
