Always try to find the likely culprit in case of illegal references
#rb Matt.Peters
#jira UE-115971
#ROBOMERGE-SOURCE: CL 16774967
#ROBOMERGE-BOT: (v836-16769935)
[CL 16774971 by francis hurteau in ue5-main branch]
Updating FVirtualizedUntypedBulkData and textures to use the BulkDataRegistry.
BulkDataRegistry: Add get/put accessors for the cached BulkDataList of packages.
EditorDomain: Move ClassDigests into a global variable that can be shared with BulkDataRegistry.
EditorDomain: Improve performance of GetFileSize by fetching metadata only.
Tickable Cook Objects, for systems used by the cooker that need to be ticked.
Implementation of the the BulkDataRegistry that uses the DDC cache for persistent storage of the BulkDataList.
#rb Devin.Doucette, Paul.Chipchase, Zousar.Shaker
[CL 16768772 by Matt Peters in ue5-main branch]
- Avoid redundant iterations and processing of UPackages by implementing a listener instead
- Process many packages asynchronously (not just one) by keeping them in memory longer and finishing them later
- Make the ddc fill job wait for async tasks properly
- Fix texture always being single threaded by allowing async at all time for ddc fill jobs
- Avoid calling IsCachedCookedPlatformDataLoaded too often on objects still being compiled
- Use a more sensible max memory usage in WorldPartitionHelpers to avoid calling the GC too often
- 30m16s to 7m44s when running -run=DerivedDataCache -fill -map=P_Construct_WP -mapsonly -projectonly with hot cache
- 2h30m to 1h11s when running -run=DerivedDataCache -fill -map=P_Construct_WP -mapsonly -projectonly with cold cache
#rb Zousar.Shaker, Devin.Doucette
#preflight 60d1fd5571002f0001b8880e
[CL 16745009 by danny couture in ue5-main branch]
Implement fast path for FCoreRedirects::GetMatchingRedirects when looking for a single redirect type.
Saves ~18% of the time spent in FLinkerLoad::FixupImportMap, which is time sliced in a 5ms window in PIE.
#rb francis.hurteau
[CL 16728888 by JeanFrancois Dube in ue5-main branch]
#rb Danny.Couture
#rnx
#preflight 60d0386978c3b00001d06c06
* Macro removal
- UE_USE_VIRTUALBULKDATA: Asssumed to be 1.
- UE_VBD_TO_OLD_BULKDATA_PATH: Assumed to be 0 and code paths entirely deleted.
- UE_VBD_CONST: Assumed to be'const'
- Now that these macros are no longer in use some includes of VirtualizationManager.h can be removed (or moved to the cpp).
*VirtualizedBulkData
- No longer need need to include VirtualizationManager.h in the bulkdata header file so that is moved to the cpp.
- Removed ::GetBulkDataSize which was only added to keep compatibility to older code paths. This method has not yet been released publicly(it's not in EA) so we don't need to worry that anyone is actually using it.
- Add a new method ::HasPayloadData that allows the caller to poll if the payload exists and if it has a valid data length rather than calling ::GetPayloadSize (I'd like to move away from the need for calling code to know how large the payload is until the payload is actually pulled as it will make eventual thread safety code easier to do in the future).
- Added a method ::SetCompressionFormat that allows the compression format to be changed. Normally this should only be done when ::UpdatePayload is called but we need a way to fix old textures that were saved with the wrong compression flags for a time and to do that we need to be able to 'reset' the compression format in some cases. The method is documented to try and make it clear that this is not really intended for general use. At some point in the future if the engine mandates that packages be resaved to a specific version we can look into deprecating this call.
*StaticMeshSourceData
- MESHDATAKEY_STATICMESH_DERIVEDDATA_VER is currently 1 character shorter than it should be, but that doesn't seem worth changing the key to fix.
*Texture
- When accessing data from IImageWrapper we can take the data pointer directly from the TArray and give it's ownership to the FSharedBuffer being returned rather than cloning it.
- Deprecated FTextureSource::LoadBulkDataWithFileReader as with VBD it will always return true.
- FMipAllocation cannot yet be deleted as it is still required for Lock/Unlock but we can remove the support for old style FByteBulkData for now. There is further clean up work to be done here but we would have to make bigger changes to the FTextureSource api.
- FMipData can now store the data in FSharedBuffer format. In the future we can allow people to access mips via views rather than always taking copies to reduce unnecessary data transfer. This also eliminates the need to pass in a lock.
- FTextureSource::InitLayered still supports the parameter 'NewData' potentially being nullptr (in which case we set the texture data to uninitialized memory) this is another candidate for a future code clean up.
- Added a ::HasPayloadData to use instead of ::GetSizeOnDisk as in many cases the calling code is only interested if there is data at all and not what the exact size is.
* Testing
- Ran benchmarks before/after for cold cooks of an internal project showing overall cook times going from 1:54:28 down to 1:49:36 on average.
[CL 16727381 by paul chipchase in ue5-main branch]
- If the time for the selection is 0.0 then just use 0.0 as the result, don't divide.
#rb trivial
#preflight 60cb73229f123700012ec005
[CL 16706893 by paul chipchase in ue5-main branch]
#fix actually allow for 32-byte aligned UObject allocations, allocate by size and aligment rather than aligned size and added development/debug build verification of resulting mem adress
#jira UE-117778
#rb Johan.Torp
#fyi Andrew.Davidson, Zak.Middleton
#preflight 60cb32e79ecd3e000199a396
[CL 16703994 by Jurre deBaare in ue5-main branch]
- This has gone so long without being noticed as it is not expected that bulkdata will be compressed during a cook, as the pakfiles themselves will be compressed.
- InternalLoadFromIoStore now only loads via the IoDispatcher directly to the destination buffer if the data is in uncompressed format.
- If the data is going to be compressed we let the IoDispatcher allocate and return the compressed buffer and bulkdata will decompress that to the destination buffer instead.
#rb Per.Larsson, CM.Nordin
#fyi Charles.deRousiers
#preflight 60c83e2143ec380001e2ad1b
[CL 16670240 by paul chipchase in ue5-main branch]
- Fixed tracing of UObject names and function names (UObject::CallFunction, FScopeCycleCounterUObject) when "stat named events" toggle is enabled (-statnamedevents, "stat NamedEvents" console command).
- Added FCpuProfilerTrace::OutputBeginDynamicEvent(const FName), in addition to existing API that receives const ANSICHAR* or const TCHAR*. It uses a separate DynamicFNameScopeNamesMap.
#rb Johan.Berg
#fyi Jeff.Farris
[CL 16657504 by ionut matasaru in ue5-main branch]
~40x speedup for set property loading, 281ms -> 6.5ms, initial load time for Win64 Test client in an internal project
Slight refactor in preparation of more advanced container allocators and allocating all UObject property memory linearly after UObject itself
#rb steve.robb
[CL 16623583 by Johan Torp in ue5-main branch]
