Changed some init function of the stuct FTextureSource to take the new struct FSharedBufferWithID.
Changed the interchange texture factory to precompute the PayloadId outside of the game thread.
On a very large udim texture (7.6 GB of source data), this can help us avoid a 2 seconds stall on the gamethread.
#jira UETOOL-3874
#rb Paul.Chipchase
#ROBOMERGE-SOURCE: CL 17069191 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v853-17066230)
[CL 17069241 by julien stjean in ue5-release-engine-test branch]
This drastically improves the performance when doing the test to determine if the newly imported texture is a normal map or not.
(Before 0.288150 seconds, after 0.000161 seconds) These are from RGBA 16 bits 8k tif texture.
The improvement is made by:
1- avoiding copying the source mip.
2- avoiding a compression and decompression of the payload of a VirtualBulkData when getting the payload, if the payload is already loaded.
#jira UETOOL-3856
#rb Paul.Chipchase, JeanMichel.Dignard
#preflight 61094e454cd7930001783127
#ROBOMERGE-SOURCE: CL 17030758 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v846-17029325)
[CL 17030763 by julien stjean in ue5-release-engine-test branch]
Also add the writing of AppendAdditionalData to a callback on the PackageStoreWriter, so that FVirtualizedUntypedBulkDatas can be separated out into separate archives in the same way FUntypedBaseData are.
#rnx
#rb Paul.Chipchase
#ROBOMERGE-SOURCE: CL 17015164 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v839-17012307)
[CL 17015166 by matt peters in ue5-release-engine-test branch]
#rb PJ.Kack
#rnx
#preflight 610110305938f900010846ae
- Previously we would print the profiling data to the log if we have entries in either PushStats or PullStats but and entry in one of those maps only proves that a backend has been mounted, not if a Push or Pull has actually been carried out.
- Now we iterate over the maps and check that we actually have at least one entry before logging anything.
- This will help keep the logs clean in cases where Mirage is enabled but irrelevant, such as a commandlet that doesn't actually load a package etc.
- Added some improvements to the formatting so that it looks more like the DDC profiling output.
-- Line break before the profiling info
-- Use of == to mark different sections
-- Use of --- to clearly mark the headers vs data
#ROBOMERGE-SOURCE: CL 16981019 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v838-16927207)
[CL 16981036 by paul chipchase in ue5-release-engine-test branch]
#rb Per.Larsson
#rnx
#preflight 60f93e100fc49a0001c95bea
- When a package is saved we now push it's payloads to local storage backends (in memory/local file cache etc)
- In a future piece of work, when a package is submitted we will push it's payloads to the persistent storage backends (jupiter/source control etc)
- When pulling a payload we will first search the local storage backends for it and then go to the slower persistent storage backends if it was not found.
-- Once a payload has been found when being pulled we will immediately cache it to the local storage backends so that subsequent pulls will be faster.
-- At some point a ZenStorage backend will be added which will most likely be the primary local storage backend.
#ROBOMERGE-SOURCE: CL 16922588 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v836-16769935)
[CL 16922803 by paul chipchase in ue5-release-engine-test branch]
#rb Per.Larsson
#rnx
#preflight 60f7b910a6959a000164fb78
* SVirtualizationStatusIndicator
- Added the total time taken for each activity to the widget's tooltip and reformatted the output to be clearer.
- The initial tool tip should just give a brief highlevel overview of the activity taking place so groups all backends together across all threads.
* VirtualizationManager
- Profiling data for Mirage is now stored as a struct per activity type, making it easier to edit in the future.
- We now store the profiling data in the base format and let the different display methods format it instead.
#ROBOMERGE-SOURCE: CL 16906627 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v836-16769935)
[CL 16906635 by paul chipchase in ue5-release-engine-test branch]
Change texture and meshdescription BulkData guids to be unique.
#rb Paul.Chipchase
#rnx
#ROBOMERGE-SOURCE: CL 16903425 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v836-16769935)
[CL 16903431 by matt peters in ue5-release-engine-test branch]
Updating FVirtualizedUntypedBulkData and textures to use the BulkDataRegistry.
BulkDataRegistry: Add get/put accessors for the cached BulkDataList of packages.
EditorDomain: Move ClassDigests into a global variable that can be shared with BulkDataRegistry.
EditorDomain: Improve performance of GetFileSize by fetching metadata only.
Tickable Cook Objects, for systems used by the cooker that need to be ticked.
Implementation of the the BulkDataRegistry that uses the DDC cache for persistent storage of the BulkDataList.
#rb Devin.Doucette, Paul.Chipchase, Zousar.Shaker
#ROBOMERGE-SOURCE: CL 16768772 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v835-16672529)
[CL 16768778 by matt peters in ue5-release-engine-test branch]
#rb Danny.Couture
#rnx
#preflight 60d0386978c3b00001d06c06
* Macro removal
- UE_USE_VIRTUALBULKDATA: Asssumed to be 1.
- UE_VBD_TO_OLD_BULKDATA_PATH: Assumed to be 0 and code paths entirely deleted.
- UE_VBD_CONST: Assumed to be'const'
- Now that these macros are no longer in use some includes of VirtualizationManager.h can be removed (or moved to the cpp).
*VirtualizedBulkData
- No longer need need to include VirtualizationManager.h in the bulkdata header file so that is moved to the cpp.
- Removed ::GetBulkDataSize which was only added to keep compatibility to older code paths. This method has not yet been released publicly(it's not in EA) so we don't need to worry that anyone is actually using it.
- Add a new method ::HasPayloadData that allows the caller to poll if the payload exists and if it has a valid data length rather than calling ::GetPayloadSize (I'd like to move away from the need for calling code to know how large the payload is until the payload is actually pulled as it will make eventual thread safety code easier to do in the future).
- Added a method ::SetCompressionFormat that allows the compression format to be changed. Normally this should only be done when ::UpdatePayload is called but we need a way to fix old textures that were saved with the wrong compression flags for a time and to do that we need to be able to 'reset' the compression format in some cases. The method is documented to try and make it clear that this is not really intended for general use. At some point in the future if the engine mandates that packages be resaved to a specific version we can look into deprecating this call.
*StaticMeshSourceData
- MESHDATAKEY_STATICMESH_DERIVEDDATA_VER is currently 1 character shorter than it should be, but that doesn't seem worth changing the key to fix.
*Texture
- When accessing data from IImageWrapper we can take the data pointer directly from the TArray and give it's ownership to the FSharedBuffer being returned rather than cloning it.
- Deprecated FTextureSource::LoadBulkDataWithFileReader as with VBD it will always return true.
- FMipAllocation cannot yet be deleted as it is still required for Lock/Unlock but we can remove the support for old style FByteBulkData for now. There is further clean up work to be done here but we would have to make bigger changes to the FTextureSource api.
- FMipData can now store the data in FSharedBuffer format. In the future we can allow people to access mips via views rather than always taking copies to reduce unnecessary data transfer. This also eliminates the need to pass in a lock.
- FTextureSource::InitLayered still supports the parameter 'NewData' potentially being nullptr (in which case we set the texture data to uninitialized memory) this is another candidate for a future code clean up.
- Added a ::HasPayloadData to use instead of ::GetSizeOnDisk as in many cases the calling code is only interested if there is data at all and not what the exact size is.
* Testing
- Ran benchmarks before/after for cold cooks of an internal project showing overall cook times going from 1:54:28 down to 1:49:36 on average.
#ROBOMERGE-SOURCE: CL 16727381 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v835-16672529)
[CL 16727410 by paul chipchase in ue5-release-engine-test branch]
- If the time for the selection is 0.0 then just use 0.0 as the result, don't divide.
#rb trivial
#preflight 60cb73229f123700012ec005
#ROBOMERGE-SOURCE: CL 16706893 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v835-16672529)
[CL 16706916 by paul chipchase in ue5-release-engine-test branch]
#rb Per.Larsson
#rnx
* VirtualizationManager
- Add method ::IsEnabled which allows the caller to poll if content virtualization is enabled or not.
- Add method ::GetPayloadActivityInfo to return profiling data
- Added a new Profiling namespace containing all of the profiling code.
-- The idea is to try and keep all profiling data contained in the virtualization manager and not require the backends to be aware of it.
- The actual pull/push operations have been moved to new private methods ::TryPushDataToBackend and ::PullDataFromBackend, by limiting the scolpe of where the actual operation occur makes it easier to add the profiling code.
- We use the cooking stats system for recording our profiling data. Currently this adds no real value and we could've implemented our own but longer term this code might get hooked into FCookStatsManager:: CookStatsCallbacks to add it to our telemetry systems and this will be easier to do if it is already in the cooking stats formats.
* SVirtualizationStaticIndicator
- The widget is based on SDDCStatusIndicator for the DDC and is placed just before it in the UI.
- The widget will only be shown if the content virtualization system is enabled, so functions as an easy way to check that (maybe in the future it can be on all the time in which case we'd need the widget to reflect when the systems are disabled)
- Currently shows a green down arrow when a payload has been pulled and a green up arrow when a payload has been pushed.
- The tool tip shown on mouse over will show the total data sizes pulled and pushed from the backends.
#preflight 60b87c34ae46a100017d5334
[CL 16544645 by paul chipchase in ue5-main branch]
This will cause the old bulkdata objects to be upgraded when loaded in the editor and saved out in the new format when the package is next saved.
#rb PJ.Kack
#rnx
#preflight 609a4df10ba7b600015e0da2
* UE5CookerObjectVersion
- Note that this work was done in CL 16266112 (it was easier to update this on the DevCooker branch)
- Moved UE_USE_VIRTUALBULKDATA to VirtualizationManager.
- Moved the virtualization versions to UE5MainStreamObjectVersion.
- Note that no package would have been saved with UE_USE_VIRTUALBULKDATA so it is safe to just move the versions.
* UE5MainStreamObjectVersion
- Added the virtialization versions here
- Fixed up some white space issues
* VirtualizationManager
- UE_USE_VIRTUALBULKDATA is now controlled from here.
- When disabled (set to 0) a new code path UE_VBD_TO_OLD_BULKDATA_PATH will be enabled that will attempt to allow packages saved with VBD to be reverted safely.
-- This requires that a new version be added to FUE5MainStreamObjectVersion, there is a static assert to warn people about this.
* VirtualizedBulkData
- Added a new method ::ConvertToOldBulkData which will load the VBD payload from disk and then pass it to the old bulkdata object provided. This is not as efficient as the OldBulkData->VBD path but is only provided as an emergancy way for us to back out if needed, we do not expect this code to actually be used.
- Fixed a bug where converting an empty bulkdata object to VBD would give a warning about invalid guids. If the provided bulkdata object is empty then it is likely that we will not have generated a unique guid for it anyway. A VBD object with no payload would expect to have an unset FPayloadId AND an invalid bulkdata identifier anyway so we can just do that and only warn in the case that a valid payload has no valid guid provided. This has a bonus of making the code easier to read as well.
* StaticMeshSourceData
- Updated MESHDATAKEY_STATICMESH_DERIVEDDATA_VER when UE_USE_VIRTUALBULKDATA is enabled again.
-- This probably isn't required but in theory some of our tests could've written mesh description data to the DDC with the old key, so we should update it to play it safe.
* Texture
- Converted to use FUE5MainStreamObjectVersion for it's version number
- Added a code path that will allow us to convert back to using old style bulkdata if UE_USE_VIRTUALBULKDATA is disabled and UE_VBD_TO_OLD_BULKDATA_PATH is enabled.
*MeshDescription
- Converted to use FUE5MainStreamObjectVersion for it's version number
-- Note that we don't have to tell the archive that we are using this because this was already happening.
- Added a code path that will allow us to convert back to using old style bulkdata if UE_USE_VIRTUALBULKDATA is disabled and UE_VBD_TO_OLD_BULKDATA_PATH is enabled.
[CL 16266408 by paul chipchase in ue5-main branch]
- The mirage specific code is disabled behind the define UE_USE_VIRTUALBULKDATA, this means that some code paths in Texture/Mesh are much more complex than they need to be as we support both old and new paths. Once the system has been turned on and confirmed to cause no issues then this will be stripped out.
- SavePackageUtilities.cpp, SavePackage.cpp and SavePackage2.cpp are editgrates rather than integrations as those files have changes in DevCooker that we don't want to bring over immediately.
- Also includes a prototype system for storing bulkdata in a sidecar file in the workspace domain rather than in the .uasset/.umap file which although has been discontinued as part of mirage, will have applications for future work for non-virtualized projects and/or text based assets.
#rb Patrick.Finegan (all changes have been reviewed when submitted to Dev-Cooker)
#tests Cooking and running ShooterGame/Frosty and other sample programs using megascan assets
#rnx
#preflight 608be50d870cf400013ff99d
[CL 16167285 by paul chipchase in ue5-main branch]
- FUniqueBuffer has been added as a single-ownership mutable buffer.
- FSharedBuffer remains a shared-ownership buffer and is now always immutable. In addition to the factory functions that were available previously, it can be also be created by moving FUniqueBuffer.
- FSharedBuffer is now used directly rather than requiring several pointer and reference types.
- FWeakSharedBuffer has been added as a weak reference to a shared buffer, in place of the previous weak pointer types.
- FBufferOwner has been promoted out of FSharedBuffer and can be used for advanced buffer management, including lazy buffer materialization where allocating and/or populating the buffer may be deferred until the buffer is accessed.
#jira none
#rb Zousar.Shaker
#fyi Paul.Chipchase
#rnx
[CL 14939736 by Devin Doucette in ue5-main branch]
Create a new interface IPackageResourceManager that handles all attempts to load packages from storage. The default implementation passes through to loading from IFileManager. Future implementations will load from other services such as the EditorDomain and TargetDomain.
Add FPackagePath class that holds a LocalFilePath or LongPackageName, or a relative path to a package under a mounted content root (which can be converted into either the LocalFilePath or LongPackageName).
Modify LinkerLoad, BulkData, and AsyncLoader to use FPackagePath and IResourceManager.
Modify UPackage and FLinkerLoad to have a FPackagePath instead of a Filename.
#rb Paul.Chipchase, CarlMagnus.Nordin, Francis.Hurteau, Devin.Doucette
#rn Major Core
[CL 14814912 by Matt Peters in ue5-main branch]
