#rb Per.Larsson
#rnx
#jira UE-151377
#preflight 628364050039ea57a52d6989
### Virtualization
- [Core.ContentVirtualization] in the engine ini file now supports an array called 'DisabledAsset' which can be used to name asset types that should not virtualize their payloads.
-- By default (in BaseEngine.ini) we have disabled the StaticMesh asset as we know it will crash if a payload is missing and the SoundWave asset as it still is pending testing.
- This new way to disable virtualization is data driven. The older hard coded method has not been removed but will likely be reworked in a future submit.
- Now when an editor bulkdata is adding it's payload to the package trailer builder during package save it will poll the virtualization system with a call to the new method ::IsDisabledForObject by passing in it's owner.
-- If the owner is valid and was present in the 'DisabledAsset' array then the method will return true and the EPayloadFlags::DisableVirtualization flag will be applied.
### Package Trailer
- The pre-existing functionality of enum EPayloadFilter has been moved to a new enum EPayloadStorageType as will only filter payloads based on their storage type.
- EPayloadFilter has been modified to filter payloads based on functionality although at the moment the only thing it can filter for is to return payloads that can be virtualized, it is left for future expansion.
- EPayloadFlags has been reduced to a uint16 with the remaining 2bytes being turned into a new member EPayloadFilterReason.
- This new member allows us to record the exact reason why a payload is excluded from virtualization. If it is zero then the payload can virtualize, otherwise it will contain one or more reasons as to why it is being excluded. For this reason the enum is a bitfield.
- Added overloads of ::GetPayloads and ::GetNumPayloads that take EPayloadFilter rather than a EPayloadStorageType
- Added wrappers around all AccessMode types for FLookupTableEntry.
- FPackageTrailerBuilder has been extended to take a EPayloadFilterReason so that the caller can already provide a reason why the payload cannot be virtualized.
-- As a future peace of work this will probably be changed and we will ask the caller to pass in the owner UObject pointer instead and then we will process the filtering when building the package trailer to a) keep all of the filtering code in one place b) keep the filtering consistent
### PackageSubmissionChecks
- The virtualization process in will now request the payloads that can be virtualized from the package trailer, which respects the new payload flag, rather than requesting all locally stored payloads.
### UObjects
- There is no need for the SoundWave or MeshDescription classes to opt out of virtualization on construction. This will be done when the package is saved and is now data driven rather than being hardcoded.
### DumpPackagePayloadInfo
- The command has been updated to also display the filter reasons applied to each payload
[CL 20240971 by paul chipchase in ue5-main branch]
Deprecation of FPlatformMath::IsNegative*() functions.
New FPlatformMath::IsNegativeOrNegativeZero().
Fix up of existing usage to either use < 0 or IsNegativeOrNegativeZero where appropriate.
Fixes for aliasing problems in various FMath functions, including IsNegative*().
Resubmission of CL# 19833778 with fixes for problematic Mac and Android toolchains, causing spurious errors while building PCHs.
#rb devin.doucette, charles.bloom, will.damon, chris.babcock
#jira UE-148435
#preflight 6260764d91376845adf9893f
[CL 19840896 by Steve Robb in ue5-main branch]
Deprecation of FPlatformMath::IsNegative*() functions.
New FPlatformMath::IsNegativeOrNegativeZero().
Fix up of existing usage to either use < 0 or IsNegativeOrNegativeZero where appropriate.
Fixes for aliasing problems in various FMath functions, including IsNegative*().
#rb devin.doucette, charles.bloom
#jira UE-148435
#preflight 625ece48f16e0d2accab15d9
[CL 19833778 by Steve Robb in ue5-main branch]
#jira UE-122078
#rb Andrew.Davidson, Colin.McGinley
#preflight standard build
#ROBOMERGE-AUTHOR: fred.kimberley
#ROBOMERGE-SOURCE: CL 18817999 in //UE5/Release-5.0/... via CL 18818012 via CL 18822871
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v910-18824042)
[CL 18824721 by fred kimberley in ue5-main branch]
#rb Per.Larsson, Devin.Doucette
#jira UE-133497
#rnx
#preflight 61f835491c5ac5523462810a
- A lot of files have been touched but most of this is changing FPayload::IsValid to !FIoHash::IsZero, the main changes are in EditorBulkData/EditorBulkDataTests
- The only difference between FPayloadId and FIoHash is that the former considered the hash of an invalid or empty buffer to be 'invalid' too where as FIoHash would return a valid hash
-- To keep behaviour the same, we only hash payloads in EditorBulkData using a utility method ::HashBuffer which will not hash empty or invalid payloads and return a default FIoHash instead.
-- Unit tests have been extended to prove that the behaviour has not changed.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18806362 in //UE5/Release-5.0/... via CL 18808527 via CL 18821790
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v908-18788545)
[CL 18822154 by paul chipchase in ue5-main branch]
Fix MeshOrientation
A triangle is defined by its 3 vertices: v0 v1 v2 and 3 edges: e0(v0v1), e1 (v1v2), e2(v2v0)
Swapping triangle orientation is done be swapping:
- 2 vertex v0 and v1 => new triangle is (v1 v0 v2)
- 2 edge but thy are e1 and e2 that need to be swapped and not e0 and e1, so (e0(v0v1), e2(v2v0), e1 (v1v2)
[FYI] jeanluc.corenthin
#preflight 61f890b8f657e25a590729f2
#ROBOMERGE-AUTHOR: david.lesage
#ROBOMERGE-SOURCE: CL 18803755 in //UE5/Release-5.0/... via CL 18803773 via CL 18821682
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v908-18788545)
[CL 18822070 by david lesage in ue5-main branch]
#rb PJ.Kack
#rnx
#jira UE-139708
#preflight 61eaaedfc92021e535b6d1e0
- The goal is to show a clear separation between the bulkdata object and the virtualization system.
- NOTE that all of these classes were added during 5.0 development so we are just renaming and moving them, no deprecation paths.
- Renamed FVirtualizedUntypedBulkData to FEditorBulkData
-- Also removed the derived templated types, they were never actually used
- Renamed FVirtualizedBulkDataReader to FEditorBulkDataReader
- Renamed FVirtualizedBulkDataWriter to FEditorBulkDataWriter
- Moved the renamed classes to the UE::Serialization namespace from UE::Virtualization
- Renamed the files of the renamed classes where required.
- Replaced use of LogVirtualization with LogSerialization.
- Renamed defines prefixed with VBD_* to UE_ and make sure they are undefed by the end of the cpp
- Edited unit tests to show up under "System.CoreUObject.Serialization.EditorBulkData.*"
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18688778 in //UE5/Release-5.0/... via CL 18688787 via CL 18688810
#ROBOMERGE-BOT: UE5 (Release-Engine-Test -> Main) (v903-18687472)
[CL 18688823 by paul chipchase in ue5-main branch]
MeshDescription: Eliminate unnecessary copy of index variables in for loop that triggers Clang's -Wrange-loop-analysis compile error.
#preflight 61d45bcd6c000c869b04004a
#rb Richard.TalbotWatkin
#rnx
#ROBOMERGE-AUTHOR: matt.peters
#ROBOMERGE-SOURCE: CL 18511960 in //UE5/Release-5.0/... via CL 18511990
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)
[CL 18512013 by matt peters in ue5-release-engine-test branch]
#rb PJ.Kack
#rnx
#preflight 61a8d6d8a7179bfa550218af
- This is an experimental feature being used during testing and development, it is not intended to be used by a shippable project.
-- The code is all wrapped by a single define, UE_ENABLE_VIRTUALIZATION_TOGGLE, making it easy to disable and remove
-- The actual method called on bulkdata, ::SetVirtualizationOptOut is being submitted already deprecated to prevent it's accidental use.
- The use case is to allow projects to opt out of virtualizing static meshes as they are more fragile (i.e. hard crashes) when their payloads cannot be accessed.
- Even when ::SetVirtualizationOptOut is called the feature will not work unless [Core.System.Experimental]AllowVirtualizationOptOut=True is set in the config file. This allows projects to opt into this.
- When a bulkdata is opting out, we simply send the payload to be stored at the end of the package like we used to, rather than send it to the package trailer. This way we do not need to store any additional state in the trailer to track if a payload is allowed to be virtualized or not. Since this feature is throw away I did not want to make any data format changes.
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 18352158 in //UE5/Release-5.0/... via CL 18352163
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)
[CL 18352180 by paul chipchase in ue5-release-engine-test branch]
This represents UE4/Main @17911760, Release-5.0 @17915875 and Dev-PerfTest @17914035
[CL 17918595 by aurel cordonnier in ue5-release-engine-test branch]
This represents UE4/Main @17774255, Release-5.0 @17791557 and Dev-PerfTest @17789485
[CL 17794212 by aurel cordonnier in ue5-release-engine-test branch]
Change texture and meshdescription BulkData guids to be unique.
#rb Paul.Chipchase
#rnx
#ROBOMERGE-SOURCE: CL 16903425 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v836-16769935)
[CL 16903431 by matt peters in ue5-release-engine-test branch]
Updating FVirtualizedUntypedBulkData and textures to use the BulkDataRegistry.
BulkDataRegistry: Add get/put accessors for the cached BulkDataList of packages.
EditorDomain: Move ClassDigests into a global variable that can be shared with BulkDataRegistry.
EditorDomain: Improve performance of GetFileSize by fetching metadata only.
Tickable Cook Objects, for systems used by the cooker that need to be ticked.
Implementation of the the BulkDataRegistry that uses the DDC cache for persistent storage of the BulkDataList.
#rb Devin.Doucette, Paul.Chipchase, Zousar.Shaker
#ROBOMERGE-SOURCE: CL 16768772 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v835-16672529)
[CL 16768778 by matt peters in ue5-release-engine-test branch]
#rb Danny.Couture
#rnx
#preflight 60d0386978c3b00001d06c06
* Macro removal
- UE_USE_VIRTUALBULKDATA: Asssumed to be 1.
- UE_VBD_TO_OLD_BULKDATA_PATH: Assumed to be 0 and code paths entirely deleted.
- UE_VBD_CONST: Assumed to be'const'
- Now that these macros are no longer in use some includes of VirtualizationManager.h can be removed (or moved to the cpp).
*VirtualizedBulkData
- No longer need need to include VirtualizationManager.h in the bulkdata header file so that is moved to the cpp.
- Removed ::GetBulkDataSize which was only added to keep compatibility to older code paths. This method has not yet been released publicly(it's not in EA) so we don't need to worry that anyone is actually using it.
- Add a new method ::HasPayloadData that allows the caller to poll if the payload exists and if it has a valid data length rather than calling ::GetPayloadSize (I'd like to move away from the need for calling code to know how large the payload is until the payload is actually pulled as it will make eventual thread safety code easier to do in the future).
- Added a method ::SetCompressionFormat that allows the compression format to be changed. Normally this should only be done when ::UpdatePayload is called but we need a way to fix old textures that were saved with the wrong compression flags for a time and to do that we need to be able to 'reset' the compression format in some cases. The method is documented to try and make it clear that this is not really intended for general use. At some point in the future if the engine mandates that packages be resaved to a specific version we can look into deprecating this call.
*StaticMeshSourceData
- MESHDATAKEY_STATICMESH_DERIVEDDATA_VER is currently 1 character shorter than it should be, but that doesn't seem worth changing the key to fix.
*Texture
- When accessing data from IImageWrapper we can take the data pointer directly from the TArray and give it's ownership to the FSharedBuffer being returned rather than cloning it.
- Deprecated FTextureSource::LoadBulkDataWithFileReader as with VBD it will always return true.
- FMipAllocation cannot yet be deleted as it is still required for Lock/Unlock but we can remove the support for old style FByteBulkData for now. There is further clean up work to be done here but we would have to make bigger changes to the FTextureSource api.
- FMipData can now store the data in FSharedBuffer format. In the future we can allow people to access mips via views rather than always taking copies to reduce unnecessary data transfer. This also eliminates the need to pass in a lock.
- FTextureSource::InitLayered still supports the parameter 'NewData' potentially being nullptr (in which case we set the texture data to uninitialized memory) this is another candidate for a future code clean up.
- Added a ::HasPayloadData to use instead of ::GetSizeOnDisk as in many cases the calling code is only interested if there is data at all and not what the exact size is.
* Testing
- Ran benchmarks before/after for cold cooks of an internal project showing overall cook times going from 1:54:28 down to 1:49:36 on average.
#ROBOMERGE-SOURCE: CL 16727381 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v835-16672529)
[CL 16727410 by paul chipchase in ue5-release-engine-test branch]
This will cause the old bulkdata objects to be upgraded when loaded in the editor and saved out in the new format when the package is next saved.
#rb PJ.Kack
#rnx
#preflight 609a4df10ba7b600015e0da2
* UE5CookerObjectVersion
- Note that this work was done in CL 16266112 (it was easier to update this on the DevCooker branch)
- Moved UE_USE_VIRTUALBULKDATA to VirtualizationManager.
- Moved the virtualization versions to UE5MainStreamObjectVersion.
- Note that no package would have been saved with UE_USE_VIRTUALBULKDATA so it is safe to just move the versions.
* UE5MainStreamObjectVersion
- Added the virtialization versions here
- Fixed up some white space issues
* VirtualizationManager
- UE_USE_VIRTUALBULKDATA is now controlled from here.
- When disabled (set to 0) a new code path UE_VBD_TO_OLD_BULKDATA_PATH will be enabled that will attempt to allow packages saved with VBD to be reverted safely.
-- This requires that a new version be added to FUE5MainStreamObjectVersion, there is a static assert to warn people about this.
* VirtualizedBulkData
- Added a new method ::ConvertToOldBulkData which will load the VBD payload from disk and then pass it to the old bulkdata object provided. This is not as efficient as the OldBulkData->VBD path but is only provided as an emergancy way for us to back out if needed, we do not expect this code to actually be used.
- Fixed a bug where converting an empty bulkdata object to VBD would give a warning about invalid guids. If the provided bulkdata object is empty then it is likely that we will not have generated a unique guid for it anyway. A VBD object with no payload would expect to have an unset FPayloadId AND an invalid bulkdata identifier anyway so we can just do that and only warn in the case that a valid payload has no valid guid provided. This has a bonus of making the code easier to read as well.
* StaticMeshSourceData
- Updated MESHDATAKEY_STATICMESH_DERIVEDDATA_VER when UE_USE_VIRTUALBULKDATA is enabled again.
-- This probably isn't required but in theory some of our tests could've written mesh description data to the DDC with the old key, so we should update it to play it safe.
* Texture
- Converted to use FUE5MainStreamObjectVersion for it's version number
- Added a code path that will allow us to convert back to using old style bulkdata if UE_USE_VIRTUALBULKDATA is disabled and UE_VBD_TO_OLD_BULKDATA_PATH is enabled.
*MeshDescription
- Converted to use FUE5MainStreamObjectVersion for it's version number
-- Note that we don't have to tell the archive that we are using this because this was already happening.
- Added a code path that will allow us to convert back to using old style bulkdata if UE_USE_VIRTUALBULKDATA is disabled and UE_VBD_TO_OLD_BULKDATA_PATH is enabled.
[CL 16266408 by paul chipchase in ue5-main branch]
