#rb Per.Larsson
#rnx
#preflight 60f93e100fc49a0001c95bea
- When a package is saved we now push it's payloads to local storage backends (in memory/local file cache etc)
- In a future piece of work, when a package is submitted we will push it's payloads to the persistent storage backends (jupiter/source control etc)
- When pulling a payload we will first search the local storage backends for it and then go to the slower persistent storage backends if it was not found.
-- Once a payload has been found when being pulled we will immediately cache it to the local storage backends so that subsequent pulls will be faster.
-- At some point a ZenStorage backend will be added which will most likely be the primary local storage backend.
[CL 16922588 by paul chipchase in ue5-main branch]
The following functions would gracefully handle a count of zero:
- EmptyAndAddValues
- EmptyAndAddUninitializedValues
The following functions would assert if given a count of zero:
- AddUninitializedValues
- AddValues
- InsertValues
- RemoveValues
- ClearValues
This change makes all of the above functions gracefully handle a count of zero, which addresses some editor crashes that could easily occur in user scripts when using unreal.Array in Python.
#fyi Malachi.Duncan
#rb Matt.Peters
#preflight 60f74556e885f20001b56003
[CL 16909007 by Jamie Dale in ue5-main branch]
#rb Per.Larsson
#rnx
#preflight 60f7b910a6959a000164fb78
* SVirtualizationStatusIndicator
- Added the total time taken for each activity to the widget's tooltip and reformatted the output to be clearer.
- The initial tool tip should just give a brief highlevel overview of the activity taking place so groups all backends together across all threads.
* VirtualizationManager
- Profiling data for Mirage is now stored as a struct per activity type, making it easier to edit in the future.
- We now store the profiling data in the base format and let the different display methods format it instead.
[CL 16906627 by paul chipchase in ue5-main branch]
2) Added a quick method to query the deferred CppStructOps until we decide on a long term solution
3) Added method to test to see if a source file is the NoExports include file.
#rb none
#rnx
#preflight 60f6c42f25c166000152cbdc
[CL 16895867 by Tim Smith in ue5-main branch]
useful for RVT on mobile platforms, to save the cost of runtime compression
#rb Dmitriy.Dyomin mihnea.balta
[CL 16872220 by yangke li in ue5-main branch]
Add OnEndLoad calls for Async loaded packages, since the editor sometimes uses async loads.
#rb Francis.Hurteau
#rnx
[CL 16861817 by Matt Peters in ue5-main branch]
Updating FVirtualizedUntypedBulkData and textures to use the BulkDataRegistry.
BulkDataRegistry: Add get/put accessors for the cached BulkDataList of packages.
EditorDomain: Move ClassDigests into a global variable that can be shared with BulkDataRegistry.
EditorDomain: Improve performance of GetFileSize by fetching metadata only.
Tickable Cook Objects, for systems used by the cooker that need to be ticked.
Implementation of the the BulkDataRegistry that uses the DDC cache for persistent storage of the BulkDataList.
#rb Devin.Doucette, Paul.Chipchase, Zousar.Shaker
[CL 16768772 by Matt Peters in ue5-main branch]
- Avoid redundant iterations and processing of UPackages by implementing a listener instead
- Process many packages asynchronously (not just one) by keeping them in memory longer and finishing them later
- Make the ddc fill job wait for async tasks properly
- Fix texture always being single threaded by allowing async at all time for ddc fill jobs
- Avoid calling IsCachedCookedPlatformDataLoaded too often on objects still being compiled
- Use a more sensible max memory usage in WorldPartitionHelpers to avoid calling the GC too often
- 30m16s to 7m44s when running -run=DerivedDataCache -fill -map=P_Construct_WP -mapsonly -projectonly with hot cache
- 2h30m to 1h11s when running -run=DerivedDataCache -fill -map=P_Construct_WP -mapsonly -projectonly with cold cache
#rb Zousar.Shaker, Devin.Doucette
#preflight 60d1fd5571002f0001b8880e
[CL 16745009 by danny couture in ue5-main branch]
#rb Danny.Couture
#rnx
#preflight 60d0386978c3b00001d06c06
* Macro removal
- UE_USE_VIRTUALBULKDATA: Asssumed to be 1.
- UE_VBD_TO_OLD_BULKDATA_PATH: Assumed to be 0 and code paths entirely deleted.
- UE_VBD_CONST: Assumed to be'const'
- Now that these macros are no longer in use some includes of VirtualizationManager.h can be removed (or moved to the cpp).
*VirtualizedBulkData
- No longer need need to include VirtualizationManager.h in the bulkdata header file so that is moved to the cpp.
- Removed ::GetBulkDataSize which was only added to keep compatibility to older code paths. This method has not yet been released publicly(it's not in EA) so we don't need to worry that anyone is actually using it.
- Add a new method ::HasPayloadData that allows the caller to poll if the payload exists and if it has a valid data length rather than calling ::GetPayloadSize (I'd like to move away from the need for calling code to know how large the payload is until the payload is actually pulled as it will make eventual thread safety code easier to do in the future).
- Added a method ::SetCompressionFormat that allows the compression format to be changed. Normally this should only be done when ::UpdatePayload is called but we need a way to fix old textures that were saved with the wrong compression flags for a time and to do that we need to be able to 'reset' the compression format in some cases. The method is documented to try and make it clear that this is not really intended for general use. At some point in the future if the engine mandates that packages be resaved to a specific version we can look into deprecating this call.
*StaticMeshSourceData
- MESHDATAKEY_STATICMESH_DERIVEDDATA_VER is currently 1 character shorter than it should be, but that doesn't seem worth changing the key to fix.
*Texture
- When accessing data from IImageWrapper we can take the data pointer directly from the TArray and give it's ownership to the FSharedBuffer being returned rather than cloning it.
- Deprecated FTextureSource::LoadBulkDataWithFileReader as with VBD it will always return true.
- FMipAllocation cannot yet be deleted as it is still required for Lock/Unlock but we can remove the support for old style FByteBulkData for now. There is further clean up work to be done here but we would have to make bigger changes to the FTextureSource api.
- FMipData can now store the data in FSharedBuffer format. In the future we can allow people to access mips via views rather than always taking copies to reduce unnecessary data transfer. This also eliminates the need to pass in a lock.
- FTextureSource::InitLayered still supports the parameter 'NewData' potentially being nullptr (in which case we set the texture data to uninitialized memory) this is another candidate for a future code clean up.
- Added a ::HasPayloadData to use instead of ::GetSizeOnDisk as in many cases the calling code is only interested if there is data at all and not what the exact size is.
* Testing
- Ran benchmarks before/after for cold cooks of an internal project showing overall cook times going from 1:54:28 down to 1:49:36 on average.
[CL 16727381 by paul chipchase in ue5-main branch]
- Fixed tracing of UObject names and function names (UObject::CallFunction, FScopeCycleCounterUObject) when "stat named events" toggle is enabled (-statnamedevents, "stat NamedEvents" console command).
- Added FCpuProfilerTrace::OutputBeginDynamicEvent(const FName), in addition to existing API that receives const ANSICHAR* or const TCHAR*. It uses a separate DynamicFNameScopeNamesMap.
#rb Johan.Berg
#fyi Jeff.Farris
[CL 16657504 by ionut matasaru in ue5-main branch]
