[FYI] Laura.Hermanns
Original CL Desc
-----------------------------------------------------------------
Add host machine name to OOM report in DXC backend and refactor global SCW error-code handling.
#rb Jason.Nadro, Dan.Elksnitis, Yuriy.Odonnell, Arciel.Rekman
#preflight 638e610c255f07df8ea212f4
#rnx
[CL 23421737 by kate ellis in ue5-main branch]
- allow 0 TimeToLive to be specified explicitly on commandline (useful when debugging single-job debug worker input files, without this it keeps running the same job repeatedly) and modify single job worker input commandline txt file to use this instead of 0.5f
- change config-conditional behaviour serializing to memory instead of file on disk to be conditional on an optional cmdline argument instead, and move file copy into the block that does the file writing (otherwise it repeatedly tries and fails to copy the nonexistent file)
#preflight 6377fed033774509008c42c6
#rb Jason.Nadro
[CL 23234934 by dan elksnitis in ue5-main branch]
- fix collisions in debug dumps for material shaders - name the folder based on the material instance name (if building a material instance) instead of just the base material name, also hash the asset path and static parameters to differentiate materials with the same name and different path and different static permutations
- add a debug dump option for writing single-job ShaderCompileWorker input files to allow debugging SCW in normal operation mode (including a separate txt file containing commandline args for SCW to do so)
- use path concatenation operator instead of + in shadercompileworker when constructing input/output file paths
#preflight 6362a45d6000216d2e15cd5b
#rb Christopher.Waters
#rb Rolando.Caloca
#rb Yuriy.ODonnell
[CL 22920594 by dan elksnitis in ue5-main branch]
- Focused around moving GlobalBeginCompileShader and friends.
- ModifyCompilationEnvironment and ValidateCompiledResult now only compiled in Editor builds.
- Measured 0.5MB to 1.0MB elf size reduction depending on platform.
#jira none
#rb jason.nadro, arciel.rekman, florin.pascu
#preflight 63613f992b5338aceb442902
[CL 22890964 by christopher waters in ue5-main branch]
- each platform compilation request is now wrapped in __try/__except rather than a single exception handling block at the top level of the worker; this allows us to log an exception (with callstack) as a compilation error and continue the batch
- remove SEH code from ShaderConductorContext, the above makes this redundant (and it didn't provide any actionable information)
- strip down SEH code in D3DShaderCompiler; now only used for the purposes of pre-compiling with DXC in the case of an FXC crash. dumping preprocessed source will be handled in a different manner in a forthcoming CL
- minor change in the DXC precompile path to not log an unnecessary warning when performing an explicitly-requested DXC precompile
#rb Jason.Nadro
#rb Laura.Hermanns
#rb Yuriy.ODonnell
#preflight 63514c798176062ea73acb41
#jira FORT-524383
[CL 22654436 by dan elksnitis in ue5-main branch]
This can be enabled by modifying `UnrealEngine\Engine\Saved\UnrealBuildTool\BuildConfiguration.xml` like so:
<?xml version="1.0" encoding="utf-8" ?>
<Configuration xmlns="https://www.unrealengine.com/BuildConfiguration">
<BuildConfiguration>
<bShaderCompilerWorkerTrace>true</bShaderCompilerWorkerTrace>
</BuildConfiguration>
</Configuration>
- Added a build configuration xml value, `bShaderCompilerWorkerTrace`.
- Turning this on will set USE_SHADER_COMPILER_WORKER_TRACE=1
- Move the parameter -nothreading to be set when we launch the process instead of internally as an extra cmd line arg.
- Unreal Insights uses a separate thread to send events so threading support is needed for the program. When we have USE_SHADER_COMPILER_WORKER_TRACE enabled we need to turn off `-nothreading`.
- When USE_SHADER_COMPILER_WORKER_TRACE is enabled we pass in `-trace=default` to get CPU event markers.
- The SCW program needs to turn on the following defines to be able to perform CPU and memory traces:
ENABLE_LOW_LEVEL_MEM_TRACKER=1
UE_MEMORY_TAGS_TRACE_ENABLED=1
UE_TRACE_ENABLED=1
- Instrument Shader Compiler Worker with TRACE_CPUPROFILER_EVENT_SCOPE. This are no-ops when this is turned off.
todo: Make the shader compiler worker inherit the trace args from the main process it was launched from.
#rb Yuriy.ODonnell
#jira none
#preflight 634ef80269246074db9637c2
[CL 22625183 by Jason Nadro in ue5-main branch]
Changed ShaderArchive, GlobalShaderCache, ShaderDebugInfo and Autogen to use ShaderPlatformName and not ShaderFormat when naming their output files.
#rb Jack.Porter, Chris.Waters, Mihnea.Balta, Jason.Nadro
#jira UE-120561
#preflight 62c31f6fc9410537282296c6
[CL 20937870 by Florin Pascu in ue5-main branch]
- Non-engine modules/targets will have to specify the "version" of includes via IncludeOrderVersion in TargetRules or ModuleRules.
- This setting will control the value of UE_ENABLE_INCLUDE_ORDER_DEPRECATED_IN_XXX where XXX is the version of the engine.
- When moving types out of a header, users will need to include the new location of the type in the header it was removed from but only if UE_ENABLE_INCLUDE_ORDER_DEPRECATED_IN_XXX is set.
- If a target does not change its IncludeOrderVersion to the latest version, UBT will print out a message telling users how to upgrade.
- This change introduces a new set of SharedPCH permutations to make sure modules with older versions get the PCH with UE_ENABLE_INCLUDE_ORDER_DEPRECATED_IN_XXX set correctly.
#jira none
#rb jonathan.adamczewski, joe.kirchoff
#preflight 623e1d3d196f3ae80b4c37ee
[CL 19518359 by christopher waters in ue5-main branch]
- Provides about 8% runtime memory savings (in local tests).
- Also, adds more compression types for shaders.
- Impact on the shader compilation (in SCWs) seems to be negligible in local tests.
#rb Devin.Doucette, Charles.Bloom
[REVIEW] [at]Devin.Doucette, [at]Charles.Bloom, [at]Jason.Nadro
#jira UE-136845
#ROBOMERGE-AUTHOR: arciel.rekman
#ROBOMERGE-SOURCE: CL 18502862 via CL 18503105 via CL 18503112 via CL 18505939 via CL 18505950
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)
[CL 18505961 by arciel rekman in ue5-release-engine-test branch]
This represents UE4/Main @18073326, Release-5.0 @18081140 and Dev-PerfTest @18045971
[CL 18081471 by aurel cordonnier in ue5-release-engine-test branch]
Local shader compiler:
- Not making 1 job batches (for High prio jobs) on startup
- Randomizing pending job selection to reduce chance that multiple 60+ sec jobs get into a single batch
(the above have the most effect on speed up, below is optional/misc)
- Pushing the completed jobs back to manager before, and not after, pulling new ones to reduce idle time
- Processing jobs in FIFO rather than LIFO order (change to LIFO seems like an ODSC regression? Hard to say definitively if it's a regression from the numbers, but seems odd to have the earliest jobs processed last)
- Parallel processing of input and output files (starts sequential by default to reduce the CPU overhead, but is enabled if we ever see too long write/read)
- More insights scopes
Distributed:
- Not avoiding local machine for XGE on startup
Both:
- Input file compression (disabled by default, need to better figure out when it's beneficial in a general case. Mostly for people with really slow I/O and XGE over VPN).
- More stats, also more dense stat output. Removed an unimportant one, added stats about the batches.
#rb Jason.Nadro, Ben.Ingram
#[review] [at]Jason.Nadro, [at]Ben.Ingram
#preflight 6132ec79bf137d0001ae91ee
#jira UE-125101
#ushell-cherrypick of 17448576 by Arciel.Rekman
#ROBOMERGE-AUTHOR: arciel.rekman
#ROBOMERGE-SOURCE: CL 17448989 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v870-17433530)
[CL 17448996 by arciel rekman in ue5-release-engine-test branch]
