# Results
- In my small tests this resulted in compiling 8.9x less shaders (1558 vs. 174)
- Spent 107 seconds less compiling.
# Changes
- Adds CacheGivenTypesForCooking to UMaterialInterface and implemented for UMaterial and UMaterialInstance. This is responsible for compiling just the given types of shaders which is fully qualified by VF Type, Shader Type (or Shader Pipeline Type).
- Fully qualify each shader compile request from the client by FeatureLevel and QualityLevel. This way we don't compile multiple quality levels that aren't actively rendered with.
- CacheGivenTypesForCooking for UMaterial and UMaterialInterface explicity tries to re-use and find existing resources so we can progressively build up the shadermap while running.
- FMaterial::CacheGivenTypes now can compile shader pipelines.
- Adds CompileODSCMaterialsForRemoteRecompile which compiles just the requested shaders. This was because I wanted to leave the functionality in CompileMaterialsForRemoteRecompile alone.
- FShaderRecompileData now has a serialize function so the Load/Save serialize code can be the same on host and client.
- Made CompileMaterialsForRemoteRecompile compiled only for the Editor.
- Don't assert on needing complete shader maps when using ODSC.
- Mass conversion from NULL to nullptr in ShaderCompiler.cpp
- Refactoring some common code from RecompileShadersForRemote into separate functions.
#rb Ben.Ingram
#jira UE-136438
#preflight 6261bb9afe26b98ad8479633
[CL 19863961 by Jason Nadro in ue5-main branch]
For example: recompileshaders /Engine/Private/Lumen/LumenVisualizeHardwareRayTracing.usf
1. Game sends the shader file string across the network to the cooking server.
2. Cooking server finds all shaders of that type.
3. Cooking server flushes those specific types from the global shader map.
4. Cooking server recompiles just those shaders and sends them back to the game.
# Todo
* Refactor duplicate code in RecompileShadersForRemote
* Implement the ability for material shader maps to compile just specific shader types requested (this doesn't exist).
#rb none
#jira UE-148085, UE-114418
#preflight 625d722648670f31a61eb3de
[CL 19785256 by Jason Nadro in ue5-main branch]
Fix serialization bug that cause old file based COTF to ignore what shader platform we were requesting (fixed for Zen COTF in 19595445).
Extend the -noshaderddc command line option with -noglobalshaderddc and -nomaterialshaderddc options (to improve COTF debugging iteration times).
#jira none
#rb jason.nadro
#rnx
#preflight 624c497bbd5b36eec30c314f
[CL 19643402 by PJ Kack in ue5-main branch]
Fix ShaderPlatform crash on platforms with a common folder that contains ShaderPlatform config data.
#jira none
#rb carlmagnus.nordin
#rnx
#preflight 6239ccbb9304d833f4f26c19
[CL 19465715 by PJ Kack in ue5-main branch]
This change allows the cooking server to communicate with clients using platform-specific communication protocols instead of TCP (the default).
This new mode is activated by adding '-platformprotocol' command-line parameter when starting the cooking server (in addition to the usual '-cookonthefly').
The new functionality is included in FNetworkFileServerPlatformProtocol, which is a variant of the preexisting FNetworkFileServer but using ITargetDevice/ITargetDeviceSocket interfaces instead of traditional sockets.
Direct communication with targets is only supported on some platforms, which can be checked by calling ITargetPlatform::SupportsFeature(ETargetPlatformFeatures::DirectDataExchange).
#rb jeff.newquist
#preflight 62166a4237178b0175af6269
[CL 19094838 by Wojciech Krywult in ue5-main branch]
[FYI] Jason.Nadro
Original CL Desc
-----------------------------------------------------------------
[ODSC] - Break out recompile shaders functionality into its own file.
- Break out common functionality for both ShaderCompiler.cpp and RecompileShaders.cpp into it's own shared file.
#rb Ben.Ingram
#jira none
#preflight 619cffb0801b361978aa77d3
#ROBOMERGE-AUTHOR: jason.nadro
#ROBOMERGE-SOURCE: CL 18273892 in //UE5/Release-5.0/... via CL 18273919
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)
[CL 18273945 by jason nadro in ue5-release-engine-test branch]
- Break out common functionality for both ShaderCompiler.cpp and RecompileShaders.cpp into it's own shared file.
#rb Ben.Ingram
#jira none
#preflight 619cffb0801b361978aa77d3
#ROBOMERGE-AUTHOR: jason.nadro
#ROBOMERGE-SOURCE: CL 18273499 in //UE5/Release-5.0/... via CL 18273518
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)
[CL 18273542 by jason nadro in ue5-release-engine-test branch]
## Refactoring
- FShaderRecompileData is the de-factor data structure for holding recompile inputs/outputs (these should be separated).
- FShaderRecompileData has moved from being a CookingServer only data structure to living in ShaderCompiler (Engine runtime).
- UE::Cook::FRecompileShaderRequest was basically a duplicate struct of FShaderRecompileData. Now it just holds a FShaderRecompileData struct and a callback.
- RecompileShadersForRemote(), which previously accepted a grab bag of arguments, now just takes the FShaderRecompileData struct.
- ShaderPlatform is no longer a int32 but a EShaderPlatform to be more explicit.
- The downside to this approach is all these systems are more tightly coupled, but the sharing of code makes this easier to modify in the future.
## Bug Fixes
- recompileshaders changed would only sometimes work depending on what materials the cooking server had loaded. Now the command issued by the client will explicitly say which materials to load and compile for.
- In RecompileShadersForRemote remove the call to GetOutdatedShaderTypes because in the cooking server it doesn't work. That function only works when the shader maps exist, but in the cooking server when loading materials we do not create the maps. This creates a chicken/egg issue. Where we need to maps to determine what's out of date and to recompile...but we don't have the maps because we haven't done a compile yet.
## Features
- Display how long the shader request took on the cooking server.
- Print shader stats out at the end of recompile request.
#rb none
#jira UE-121436, UE-113113
#preflight 6196a3f0f020c4e616acff15
#ROBOMERGE-AUTHOR: jason.nadro
#ROBOMERGE-SOURCE: CL 18242119 in //UE5/Release-5.0/... via CL 18242138
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v895-18170469)
[CL 18242996 by jason nadro in ue5-release-engine-test branch]
#rb CarlMagnus.Nordin
#rnx
#tests Ran overnight preflights on several platforms, opened/cooked/staged/ran the oldest version of InfiltratorDemo that can be downloaded (4.11)
### ObjectVersion
- Add a new version enum EUnrealEngineObjectUE5Version.
-- This version number starts at 1000 which leaves more than enough for for EUnrealEngineObjectUEVersion to be expanded
- Even though very few changes (if any at all) to EUnrealEngineObjectUE4Version are expected there is a static assert to make sure that EUnrealEngineObjectUEVersion::AUTOMATIC_VERSION never overtakes EUnrealEngineObjectUE5Version::INITIAL_VERSION.
- Add a struct FPackageFileVersion that wraps around the version numbers and is used to store them instead of raw int32 values which was done before. This should make it easier to add new version numbers in the future if we desire (although this will cause problems in places that serialize the struct directly)
### FPackageFileSummary
- Adding a new entry to CurrentLegacyFileVersion at value -8 which shows the UE5 version being added. This lets us make the changes without needing to submit anything to UE4 Main.
- When loading a package that does not have a UE5 version, it will remain at 0.
- Added ::IsFileVersionTooOld and ::IsFileVersionTooNew to replace hardcoded tests in the code base for version validity. This will make it easier to make changes in the future.
- A few months ago most of the accessors of the version number were deprecated in favour of a version that did not contain the Engine number (ie UE4Ver -> UEVer in Archive) but to work with these changes the renamed methods now will return or accept the version as FPackageFileVersion rather than int32. The old UE4 methods will remain deprecated and direct licensees to use the new methods.
### Archive
- Now stores the version as a FPackageFileVersion rather than int32
### LinkerLoad
- Reports the larger version number if we detect a higher version number than we support. Note that this could cause an issue if the UE4 version is ever raised but helps keep the code simple.
### AssetData
- Need to add a new version here to manage existing data that only has the UE4 version
### EditorDomain
- We do not need to version the format, we can just invalidate existing editor domain entries via EditorDomainVersion
### EditorServer
- When reporting that a package is too old we report the UE4 version as that is the only version that can be older than VER_UE4_OLDEST_LOADABLE_PACKAGE
- When reporting that a package is too new it can be either the UE4 or the UE5 version so we print them together "UE4Ver|UE5Ver"
### ContentCommandlets
- The min and max resave versions have been kept as a single value, you will not be able to resave against different UE4 and UE5 versions at the same time. It doesn't seem like a useful feature and would greatly increase the complexity of the code.
- We will also only report the file version as a single value.
### ManifestUObject
- This class was setting an older obsolete version on purpose to try and maintain compatibility with older clients so we need to provide a way to create an older UE4 only version that will leave the UE5 version as unset.
### NetworkPlatformFile
- I was unable to test the code path in FNetworkPlatformFile::ProcessServerCachedFilesResponse as I am unsure how to run the game in a mode that will actually use it.
- When reading an older "CookedVersion.txt" that was saved with a single version, the reads will fail and this will count as a version change in the code so that all of the existing files will be deleted. The existing code would not give the user a log message when this happens and given the very small time window where this might happen caused by this change I have opted to leave this alone and not add any additional logging.
- If we do detect a version mismatch we will still only log the version number as a single version.
### CookOnTheFlyServer
- We now add each version number to the IniVersionMap rather than merge the version and license version as a key/value pair. This allows us to a) use both the UE4 and UE5 version numbers b) we now log a warning that the version values don't match when it is changed, previously since it was a key value we would log a warning about an additional setting instead.
-- I also added "vs" to the log message when values are mismatched to make the space between the two values being printed clearer.
#ROBOMERGE-OWNER: paul.chipchase
#ROBOMERGE-AUTHOR: paul.chipchase
#ROBOMERGE-SOURCE: CL 17549459 via CL 17550236 via CL 17550238 via CL 17550582
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v870-17433530)
#ROBOMERGE[STARSHIP]: UE5-Main
[CL 17550583 by paul chipchase in ue5-release-engine-test branch]
Updated references to UE4 to Unreal in NetworkFileConnection.cpp
This required references to GPackageFileUE4Version and GPackageFileLicenseeUE4Version to also be updated to GPackageFileUnrealVersion and GPackageFileLicenseeUnrealVersion accordingly
#ROBOMERGE-SOURCE: CL 15850452 in //UE5/Release-5.0-EarlyAccess/...
#ROBOMERGE-BOT: STARSHIP (Release-5.0-EarlyAccess -> Main) (v786-15839533)
[CL 15851651 by mark lintott in ue5-main branch]
- target platform code can manage the file transfer from the cook server.
- client ip address and platform-specific metadata is sent from the client.
#rb josh.adams
#rnx
#lockdown marcus.wassmer
#jira MTHRL-18
#ROBOMERGE-SOURCE: CL 12653783 in //UE4/Release-4.25Plus/... via CL 12653792
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v675-12543919)
[CL 12653798 by david harvey in Main branch]
Fix recompileshaders running on target with COTF server
#rb none
[FYI] Yuriy.ODonnell
#ROBOMERGE-SOURCE: CL 11892983 in //UE4/Release-4.25/... via CL 11892985
#ROBOMERGE-BOT: RELEASE (Release-4.25Plus -> Main) (v656-11643781)
[CL 11892989 by ben ingram in Main branch]
#rnx
#rb none
#ROBOMERGE-OWNER: ryan.durand
#ROBOMERGE-AUTHOR: ryan.durand
#ROBOMERGE-SOURCE: CL 10869210 via CL 10869511 via CL 10869900
#ROBOMERGE-BOT: (v613-10869866)
[CL 10870549 by ryan durand in Main branch]
