Updating FVirtualizedUntypedBulkData and textures to use the BulkDataRegistry.
BulkDataRegistry: Add get/put accessors for the cached BulkDataList of packages.
EditorDomain: Move ClassDigests into a global variable that can be shared with BulkDataRegistry.
EditorDomain: Improve performance of GetFileSize by fetching metadata only.
Tickable Cook Objects, for systems used by the cooker that need to be ticked.
Implementation of the the BulkDataRegistry that uses the DDC cache for persistent storage of the BulkDataList.
#rb Devin.Doucette, Paul.Chipchase, Zousar.Shaker
[CL 16768772 by Matt Peters in ue5-main branch]
- Avoid redundant iterations and processing of UPackages by implementing a listener instead
- Process many packages asynchronously (not just one) by keeping them in memory longer and finishing them later
- Make the ddc fill job wait for async tasks properly
- Fix texture always being single threaded by allowing async at all time for ddc fill jobs
- Avoid calling IsCachedCookedPlatformDataLoaded too often on objects still being compiled
- Use a more sensible max memory usage in WorldPartitionHelpers to avoid calling the GC too often
- 30m16s to 7m44s when running -run=DerivedDataCache -fill -map=P_Construct_WP -mapsonly -projectonly with hot cache
- 2h30m to 1h11s when running -run=DerivedDataCache -fill -map=P_Construct_WP -mapsonly -projectonly with cold cache
#rb Zousar.Shaker, Devin.Doucette
#preflight 60d1fd5571002f0001b8880e
[CL 16745009 by danny couture in ue5-main branch]
#rb Danny.Couture
#rnx
#preflight 60d0386978c3b00001d06c06
* Macro removal
- UE_USE_VIRTUALBULKDATA: Asssumed to be 1.
- UE_VBD_TO_OLD_BULKDATA_PATH: Assumed to be 0 and code paths entirely deleted.
- UE_VBD_CONST: Assumed to be'const'
- Now that these macros are no longer in use some includes of VirtualizationManager.h can be removed (or moved to the cpp).
*VirtualizedBulkData
- No longer need need to include VirtualizationManager.h in the bulkdata header file so that is moved to the cpp.
- Removed ::GetBulkDataSize which was only added to keep compatibility to older code paths. This method has not yet been released publicly(it's not in EA) so we don't need to worry that anyone is actually using it.
- Add a new method ::HasPayloadData that allows the caller to poll if the payload exists and if it has a valid data length rather than calling ::GetPayloadSize (I'd like to move away from the need for calling code to know how large the payload is until the payload is actually pulled as it will make eventual thread safety code easier to do in the future).
- Added a method ::SetCompressionFormat that allows the compression format to be changed. Normally this should only be done when ::UpdatePayload is called but we need a way to fix old textures that were saved with the wrong compression flags for a time and to do that we need to be able to 'reset' the compression format in some cases. The method is documented to try and make it clear that this is not really intended for general use. At some point in the future if the engine mandates that packages be resaved to a specific version we can look into deprecating this call.
*StaticMeshSourceData
- MESHDATAKEY_STATICMESH_DERIVEDDATA_VER is currently 1 character shorter than it should be, but that doesn't seem worth changing the key to fix.
*Texture
- When accessing data from IImageWrapper we can take the data pointer directly from the TArray and give it's ownership to the FSharedBuffer being returned rather than cloning it.
- Deprecated FTextureSource::LoadBulkDataWithFileReader as with VBD it will always return true.
- FMipAllocation cannot yet be deleted as it is still required for Lock/Unlock but we can remove the support for old style FByteBulkData for now. There is further clean up work to be done here but we would have to make bigger changes to the FTextureSource api.
- FMipData can now store the data in FSharedBuffer format. In the future we can allow people to access mips via views rather than always taking copies to reduce unnecessary data transfer. This also eliminates the need to pass in a lock.
- FTextureSource::InitLayered still supports the parameter 'NewData' potentially being nullptr (in which case we set the texture data to uninitialized memory) this is another candidate for a future code clean up.
- Added a ::HasPayloadData to use instead of ::GetSizeOnDisk as in many cases the calling code is only interested if there is data at all and not what the exact size is.
* Testing
- Ran benchmarks before/after for cold cooks of an internal project showing overall cook times going from 1:54:28 down to 1:49:36 on average.
[CL 16727381 by paul chipchase in ue5-main branch]
- Fixed tracing of UObject names and function names (UObject::CallFunction, FScopeCycleCounterUObject) when "stat named events" toggle is enabled (-statnamedevents, "stat NamedEvents" console command).
- Added FCpuProfilerTrace::OutputBeginDynamicEvent(const FName), in addition to existing API that receives const ANSICHAR* or const TCHAR*. It uses a separate DynamicFNameScopeNamesMap.
#rb Johan.Berg
#fyi Jeff.Farris
[CL 16657504 by ionut matasaru in ue5-main branch]
Fixed issue where changing a function signature in an interface wouldn't update nodes in a blueprint.
Changed the API of the deferred registry to be more inline with UE coding standards.
#rb ben.marsh phillip.kavan
#rnx
#jira UE-113662
#preflight 60c0c0fdc61264000190e16a
[CL 16606206 by Tim Smith in ue5-main branch]
Add an enum value for transient package flags and clear them all on load from PackageFileSummary in addition to the current clearing on save, so that newly-marked transient flags that were previously saved out can still avoid being set when loading.
#rnx
#rb Francis.Hurteau
[CL 16547525 by Matt Peters in ue5-main branch]
#rb Per.Larsson
#rnx
* VirtualizationManager
- Add method ::IsEnabled which allows the caller to poll if content virtualization is enabled or not.
- Add method ::GetPayloadActivityInfo to return profiling data
- Added a new Profiling namespace containing all of the profiling code.
-- The idea is to try and keep all profiling data contained in the virtualization manager and not require the backends to be aware of it.
- The actual pull/push operations have been moved to new private methods ::TryPushDataToBackend and ::PullDataFromBackend, by limiting the scolpe of where the actual operation occur makes it easier to add the profiling code.
- We use the cooking stats system for recording our profiling data. Currently this adds no real value and we could've implemented our own but longer term this code might get hooked into FCookStatsManager:: CookStatsCallbacks to add it to our telemetry systems and this will be easier to do if it is already in the cooking stats formats.
* SVirtualizationStaticIndicator
- The widget is based on SDDCStatusIndicator for the DDC and is placed just before it in the UI.
- The widget will only be shown if the content virtualization system is enabled, so functions as an easy way to check that (maybe in the future it can be on all the time in which case we'd need the widget to reflect when the systems are disabled)
- Currently shows a green down arrow when a payload has been pulled and a green up arrow when a payload has been pushed.
- The tool tip shown on mouse over will show the total data sizes pulled and pushed from the backends.
#preflight 60b87c34ae46a100017d5334
[CL 16544645 by paul chipchase in ue5-main branch]
* Added a function to RedirectCollector to ingest soft package references
* Push the harevest soft package references from save to the redirect collector on a succesful save
* Moved the processing of soft package reference associated with a package during cook to after that package has been saved
#rb Matt.Peters
#jira UE-115137
[CL 16534223 by Francis Hurteau in ue5-main branch]
Add missing CoreMinimal.h when compiling core
- 251s -> 245s for QAGame Win64 Development with -NoXGE on AMD TR 3970X
#rb Ben.Marsh
[CL 16527504 by danny couture in ue5-main branch]
