Updating FVirtualizedUntypedBulkData and textures to use the BulkDataRegistry.
BulkDataRegistry: Add get/put accessors for the cached BulkDataList of packages.
EditorDomain: Move ClassDigests into a global variable that can be shared with BulkDataRegistry.
EditorDomain: Improve performance of GetFileSize by fetching metadata only.
Tickable Cook Objects, for systems used by the cooker that need to be ticked.
Implementation of the the BulkDataRegistry that uses the DDC cache for persistent storage of the BulkDataList.
#rb Devin.Doucette, Paul.Chipchase, Zousar.Shaker
[CL 16768772 by Matt Peters in ue5-main branch]
#rb Danny.Couture
#rnx
#preflight 60d0386978c3b00001d06c06
* Macro removal
- UE_USE_VIRTUALBULKDATA: Asssumed to be 1.
- UE_VBD_TO_OLD_BULKDATA_PATH: Assumed to be 0 and code paths entirely deleted.
- UE_VBD_CONST: Assumed to be'const'
- Now that these macros are no longer in use some includes of VirtualizationManager.h can be removed (or moved to the cpp).
*VirtualizedBulkData
- No longer need need to include VirtualizationManager.h in the bulkdata header file so that is moved to the cpp.
- Removed ::GetBulkDataSize which was only added to keep compatibility to older code paths. This method has not yet been released publicly(it's not in EA) so we don't need to worry that anyone is actually using it.
- Add a new method ::HasPayloadData that allows the caller to poll if the payload exists and if it has a valid data length rather than calling ::GetPayloadSize (I'd like to move away from the need for calling code to know how large the payload is until the payload is actually pulled as it will make eventual thread safety code easier to do in the future).
- Added a method ::SetCompressionFormat that allows the compression format to be changed. Normally this should only be done when ::UpdatePayload is called but we need a way to fix old textures that were saved with the wrong compression flags for a time and to do that we need to be able to 'reset' the compression format in some cases. The method is documented to try and make it clear that this is not really intended for general use. At some point in the future if the engine mandates that packages be resaved to a specific version we can look into deprecating this call.
*StaticMeshSourceData
- MESHDATAKEY_STATICMESH_DERIVEDDATA_VER is currently 1 character shorter than it should be, but that doesn't seem worth changing the key to fix.
*Texture
- When accessing data from IImageWrapper we can take the data pointer directly from the TArray and give it's ownership to the FSharedBuffer being returned rather than cloning it.
- Deprecated FTextureSource::LoadBulkDataWithFileReader as with VBD it will always return true.
- FMipAllocation cannot yet be deleted as it is still required for Lock/Unlock but we can remove the support for old style FByteBulkData for now. There is further clean up work to be done here but we would have to make bigger changes to the FTextureSource api.
- FMipData can now store the data in FSharedBuffer format. In the future we can allow people to access mips via views rather than always taking copies to reduce unnecessary data transfer. This also eliminates the need to pass in a lock.
- FTextureSource::InitLayered still supports the parameter 'NewData' potentially being nullptr (in which case we set the texture data to uninitialized memory) this is another candidate for a future code clean up.
- Added a ::HasPayloadData to use instead of ::GetSizeOnDisk as in many cases the calling code is only interested if there is data at all and not what the exact size is.
* Testing
- Ran benchmarks before/after for cold cooks of an internal project showing overall cook times going from 1:54:28 down to 1:49:36 on average.
[CL 16727381 by paul chipchase in ue5-main branch]
This will cause the old bulkdata objects to be upgraded when loaded in the editor and saved out in the new format when the package is next saved.
#rb PJ.Kack
#rnx
#preflight 609a4df10ba7b600015e0da2
* UE5CookerObjectVersion
- Note that this work was done in CL 16266112 (it was easier to update this on the DevCooker branch)
- Moved UE_USE_VIRTUALBULKDATA to VirtualizationManager.
- Moved the virtualization versions to UE5MainStreamObjectVersion.
- Note that no package would have been saved with UE_USE_VIRTUALBULKDATA so it is safe to just move the versions.
* UE5MainStreamObjectVersion
- Added the virtialization versions here
- Fixed up some white space issues
* VirtualizationManager
- UE_USE_VIRTUALBULKDATA is now controlled from here.
- When disabled (set to 0) a new code path UE_VBD_TO_OLD_BULKDATA_PATH will be enabled that will attempt to allow packages saved with VBD to be reverted safely.
-- This requires that a new version be added to FUE5MainStreamObjectVersion, there is a static assert to warn people about this.
* VirtualizedBulkData
- Added a new method ::ConvertToOldBulkData which will load the VBD payload from disk and then pass it to the old bulkdata object provided. This is not as efficient as the OldBulkData->VBD path but is only provided as an emergancy way for us to back out if needed, we do not expect this code to actually be used.
- Fixed a bug where converting an empty bulkdata object to VBD would give a warning about invalid guids. If the provided bulkdata object is empty then it is likely that we will not have generated a unique guid for it anyway. A VBD object with no payload would expect to have an unset FPayloadId AND an invalid bulkdata identifier anyway so we can just do that and only warn in the case that a valid payload has no valid guid provided. This has a bonus of making the code easier to read as well.
* StaticMeshSourceData
- Updated MESHDATAKEY_STATICMESH_DERIVEDDATA_VER when UE_USE_VIRTUALBULKDATA is enabled again.
-- This probably isn't required but in theory some of our tests could've written mesh description data to the DDC with the old key, so we should update it to play it safe.
* Texture
- Converted to use FUE5MainStreamObjectVersion for it's version number
- Added a code path that will allow us to convert back to using old style bulkdata if UE_USE_VIRTUALBULKDATA is disabled and UE_VBD_TO_OLD_BULKDATA_PATH is enabled.
*MeshDescription
- Converted to use FUE5MainStreamObjectVersion for it's version number
-- Note that we don't have to tell the archive that we are using this because this was already happening.
- Added a code path that will allow us to convert back to using old style bulkdata if UE_USE_VIRTUALBULKDATA is disabled and UE_VBD_TO_OLD_BULKDATA_PATH is enabled.
[CL 16266408 by paul chipchase in ue5-main branch]
- The mirage specific code is disabled behind the define UE_USE_VIRTUALBULKDATA, this means that some code paths in Texture/Mesh are much more complex than they need to be as we support both old and new paths. Once the system has been turned on and confirmed to cause no issues then this will be stripped out.
- SavePackageUtilities.cpp, SavePackage.cpp and SavePackage2.cpp are editgrates rather than integrations as those files have changes in DevCooker that we don't want to bring over immediately.
- Also includes a prototype system for storing bulkdata in a sidecar file in the workspace domain rather than in the .uasset/.umap file which although has been discontinued as part of mirage, will have applications for future work for non-virtualized projects and/or text based assets.
#rb Patrick.Finegan (all changes have been reviewed when submitted to Dev-Cooker)
#tests Cooking and running ShooterGame/Frosty and other sample programs using megascan assets
#rnx
#preflight 608be50d870cf400013ff99d
[CL 16167285 by paul chipchase in ue5-main branch]
- Preserve guid during save if static mesh bulkdata content hasn't changed to avoid triggering unnecessary rebuilds
- Make sure the next HLOD generation is going to use the HashAsGuid feature to avoid recompiling staticmesh that are generated if they are exactly the same as before
- 31m10s to 14m15s to cook P_Construct_WP with a warmed-up DDC
#fyi Richard.TalbotWatkin
#rb Matt.Peters, Sebastien.Lussier, Yuriy.ODonnell
[CL 15755322 by danny couture in ue5-main branch]
* Added missing write lock in FMeshDescriptionBulkData::SaveMeshDescription( FMeshDescription& MeshDescription )
* Skip serialization of bulk data when archive is a ref collector
#rb danny.couture, richard.talbotwatkin
#ROBOMERGE-SOURCE: CL 15502374 in //UE5/Release-5.0-EarlyAccess/...
#ROBOMERGE-BOT: STARSHIP (Release-5.0-EarlyAccess -> Main) (v771-15082668)
[CL 15503758 by sebastien lussier in ue5-main branch]
- Moved source models related classes and implementation into separate files.
- Created a new UObject wrapper for holding mesh description bulk data; this is created as a transactable object inside a UStaticMesh.
- The cached mesh description is now an inner object of the bulk data wrapper; this allows transactions on the bulk data to affect the cached mesh.
- The static mesh SourceModels array can no longer be accessed directly: FStaticMeshSourceModel requires extra initialization before it should be used, so new LODs should be added through the static mesh API.
- Undo/redo is now fixed within the geometry tools
- Fixed regression where mesh description was being unpacked unnecessarily during transactions
- MeshDescription bulk data and its cached MeshDescriptions are now emptied prior to removing the UObject reference, in order to immediately free memory instead of needing to wait for garbage collection
- No more deadlocks when clearing or committing mesh descriptions during garbage collection
- Bulk data UObjects are constructed, where possible, at startup, in order to prevent it from happening during critical moments, e.g. during garbage collection or package saving.
#rb Danny.Couture, Alexis.Matte, Ryan.Schmidt
[CL 15420827 by Richard TalbotWatkin in ue5-main branch]
- The MeshDescription for each source model LOD is now wrapped in a UStaticMeshDescription, meaning that it can be transacted separately to the parent static mesh. This improves memory and time performance in static mesh transactions, e.g. when assigning materials.
- FStaticMeshSourceModel now has its own API for managing the mesh description and its bulk data.
- A new optional Hi-res SourceModel member has been added to StaticMesh; this allows the original source data to be stored here, while a reduced LOD0 can be stored as SourceModel LOD0, to allow geometry modelling tools to work efficiently.
- MeshDescription is now serialized using a proxy archive object which ensures that FNames can always be archived as strings.
- UStaticMeshDescription is now serialized.
- UStaticMeshDescription has been reverted to only holding its own mesh description, instead of being able to reference any external mesh description. This previous change was never released.
- RawMesh is now immediately converted to MeshDescription in PostLoad, so that the MeshDescription[BulkData] pointers are always valid, and can be tag serialised automatically during transactions. RawMesh should now not be used in the editor at all, and will soon be deprecated.
This change results in far quicker and memory-efficient transactions in StaticMeshes. e.g. assign material to a static mesh section for a 20000 vertex mesh now takes less than a second (discounting time to rebuild the render proxy) instead of 7+ seconds.
#rb Alexis.Matte, Danny.Couture
[CL 15248935 by Richard TalbotWatkin in ue5-main branch]
Removed attribute views.
Refactored MeshAttributeArray implementation.
Added a new GetArrayView() method for all types of TMeshAttributeRefs which can be used generically.
#rb Alexis.Matte
[CL 14651594 by Richard TalbotWatkin in ue5-main branch]
Also add protection so meshdescription can be read from multiple threads in editor
- Loading Reverb's P_World with empty DDC
- 319GB to 242GB peak private byte
- 166GB to 80GB permanent memory usage after map load
#rb Richard.Talbot-Watkins, Matt.Peters, Paul.Chipchase
[CL 14544739 by danny couture in ue5-main branch]
This is the correct versioning for new MeshDescription objects.
The original versioning (in ReleaseObjectVersion) clashes with changes being brought over from 4.26. MeshDescription objects already saved in UE5 will only be valid for precisely FReleaseObjectVersion::MeshDescriptionNewFormat; subsequent version numbers belong to assets from 4.26 (and derived streams) which have been resaved and merged into UE5, and do *not* have the new MeshDescription format.
Going forward, UE5MainStreamObjectVersion holds the latest version for MeshDescription objects.
#rb Steve.Robb, Marc.Audy
[CL 14377658 by Richard TalbotWatkin in ue5-main branch]
Prepared direct attribute access for deprecation, preferring use of APIs to access static mesh attributes.
Fixed recently merged Enterprise code to comply with new APIs.
Changed all tools to use triangle-centric iteration where possible.
Added new MeshAttributeArray APIs for querying attribute flags, and added a new Mandatory flag.
Various bug fixes.
#rb Alexis.Matte
[CL 13873755 by Richard TalbotWatkin in ue5-main branch]
- Removed hardcoded element type arrays (Vertices, Edges, Triangles etc.). Mesh element types can now be arbitrarily added, with any number of channels.
- Mesh element containers have a much leaner format; instead of sparse arrays, they are now represented by a simple bitarray, determining whether an index is used or not. Consequently, mesh topology is now entirely described with the attribute system, e.g. edge start and end vertices, triangle vertices, etc.
- Support added for attributes of arbitrary dimensions, e.g. float[4] or int[2].
- Support added for attributes which index into another mesh element container.
- Added FMeshElementIndexer: this is an efficient container for maintaining backward references from one element type to another; for example, edges have an attribute specifying which vertices are at each end (an attribute of type FVertexID[2]). With an indexer, it is possible to look up which edges contain a given vertex, even though this is not explicitly stored. Indexers are designed to do minimal allocations and update lazily and in batch when necessary.
- Added support for preserving UV topology in static meshes. UVs are now a first-class element type which may be indexed directly from triangles.
- Added the facility to access the underlying array in an attribute array directly.
- Triangles now directly reference their vertex, edge and UV IDs. Vertex instances are to be deprecated.
- Changed various systems to be triangle-centric rather than polygon-centric, as this is faster. Triangles are presumed to be the elementary face type in a MeshDescription, even if polygons are still supported. The concept of polygons will be somewhat shifted to mean a group of triangles which should be treated collectively for editing purposes.
- Optimised normal/tangent generation and FBX import.
- Deprecated EditableMesh, MeshEditor and StaticMeshEditorExtension plugins - these are to be removed, but they still have certain hooks in place which need removing.
#rb
[CL 13568702 by Richard TalbotWatkin in ue5-main branch]
CL 10373564 by danny.couture
Optimize Material Baking (Phase 1)
- Introduce a mecanism to override the vertex/index buffer allocator used for dynamic meshes
- Avoid GDynamicMesh non-ticked pools build-up by using our own vertex/index buffer pool during baking
- Reduce reallocation and incurred soft page faults by reusing a single set of vertex/index buffers big enough for the biggest mesh
- Preemptively detect if smearing would result in monochrome texture to avoid useless work
- Shrink smeared monochrome textures during the baking process for huge memory savings
- Move UV smearing in worker threads to avoid blocking the game thread
- Required shaders are now built asynchronously
- Add progress bar for material baking
- 28m23 [at] 150 GB RAM -> 2m14s [at] 45 GB RAM for 6 channels [at] 512x512 when baking materials on ProxyLOD for DATASET-0008a with DDC empty
#rb Jurre.deBaare, Sebastien.Lussier
CL 10516258 by danny.couture
Optimize Material Baking (Phase 2)
- Implement pipelining with staging buffers to avoid GPU stalls when reading from render targets
- Reuse the same prepared FMeshBatch instead of rebuilding it for each draw pass
- Prepare the RenderItem in advance on other threads to reduce work on the game thread
- Move the staging surface copy out of the render thread
- Small vertex and index buffers are not reused to avoid dependency locks when mapping them
- Fix bug in Canvas Flush_RenderThread found while running HLOD rebuild commandlet on Fortnite
- Delete old and unused MaterialBakingModule.h from public files
- 4m44s -> 59s for baking 6 channel [at] 1024x1024 when baking materials on ProxyLOD for DATASET-0008a with shaders already compiled
- Time spent in Material Baking when rebuilding all HLOD on Apollo_POI_Large_HLOD (Phase 1 + 2 combined)
- 10m18s -> 2m36s for a first rebuild all in editor with no shaders in DDC (cold)
- 1m23s -> 20s for a second rebuild all in editor (warm)
#rb Jeremy.Moore, Sebastien.Lussier
CL 11135986 by sebastien.lussier
Optimized mesh merging
* Added DeletePolygons() & DeleteTriangles methods to FMeshDescription which rely on TSets<> instead of performing costly TArray::AddUnique() calls()
* Parallelized UV generation and avoided duplicate processing of the same mesh+lod pairs
* Optimized FMeshDescriptionOperations::GenerateUniqueUVsForStaticMesh()
* Goes from 100s to 10s in my test case
#rb danny.couture, jeanfrancois.dube, richard.talbotwatkin
#ROBOMERGE-OWNER: sebastien.lussier
#ROBOMERGE-AUTHOR: sebastien.lussier
#ROBOMERGE-SOURCE: CL 11206337 via CL 11206341 via CL 11206346
#ROBOMERGE-BOT: (v643-11205221)
[CL 11206493 by sebastien lussier in Main branch]
#rnx
#rb none
#ROBOMERGE-OWNER: ryan.durand
#ROBOMERGE-AUTHOR: ryan.durand
#ROBOMERGE-SOURCE: CL 10869210 via CL 10869511 via CL 10869900
#ROBOMERGE-BOT: (v613-10869866)
[CL 10870549 by ryan durand in Main branch]
