- Shaders are searched by TypeName and PermutationId, so move those fields out of FShader and into separate arrays inside ShaderMap, to reduce cache misses while finding shaders
#rb none
#ROBOMERGE-OWNER: ben.ingram
#ROBOMERGE-AUTHOR: ben.ingram
#ROBOMERGE-SOURCE: CL 12646854 via CL 12646966 via CL 12646981
#ROBOMERGE-BOT: RELEASE (Release-Engine-Staging -> Main) (v675-12543919)
[CL 12646995 by ben ingram in Main branch]
- Each shadermap is now associated with one or more assets, and shaders are laid out according to the
open order of those assets, if provided. If not provided, the original ordering is kept. The original ordering
will not be always deterministic, particularly if shader compilation needs to happen, because jobs can get
reordered during the compilation.
Other fixes:
- Fixed sorting within the shadermap not being deterministic (FCompareShaderPrimaryKey)
- Fixed Niagara shadermap duplicating Platform property and not initializing it properly.
- Removed (unused) support for Child shader libraries (would not be deterministic).
#rb Ben.Ingram, Rob.Krajcarski
#ROBOMERGE-OWNER: Arciel.Rekman
#ROBOMERGE-AUTHOR: arciel.rekman
#ROBOMERGE-SOURCE: CL 12249260 via CL 12249262 via CL 12250173
#ROBOMERGE-BOT: (v668-12245121)
[CL 12250313 by Arciel Rekman in Main branch]
Including StringFwd.h in place of forward declarations offers more flexibility to change how these types are declared.
#jira none
#rb Steve.Robb
[RN] minor Core
#ROBOMERGE-SOURCE: CL 12235779 in //UE4/Release-4.25/... via CL 12235792
#ROBOMERGE-BOT: RELEASE (Release-4.25Plus -> Main) (v664-12234175)
[CL 12237495 by devin doucette in Main branch]
#rb Josh.Adams, Ben.Woodhouse
#rnx
#ROBOMERGE-OWNER: Arciel.Rekman
#ROBOMERGE-AUTHOR: arciel.rekman
#ROBOMERGE-SOURCE: CL 11450810 via CL 11452365 via CL 11452390 via CL 11452432
#ROBOMERGE-BOT: (v654-11333218)
[CL 11456136 by Arciel Rekman in Main branch]
- Optimize parsing of FVertexElement, FDepthStencilStateInitializerRHI, FBlendStateInitializerRHI, FPipelineFileCacheRasterizerState, FPipelineCacheFileFormatPSO using FStringView and TArray<FStringView, TInlineAllocator<N>> to avoid temporary allocations.
- Optimize parsing and string conversion of FCompactFullName, FStableShaderKeyAndValue using FStringView and T[Ansi]StringBuilder to avoid temporary allocations.
- Use TInlineAllocator<16> for FCompactFullName to eliminate dynamic allocations. The size was selected by profiling a title with many shaders.
- Serialize FEditorShaderStableInfo directly to UTF-8 instead of converting to it later to avoid converting every name from ASCII to UTF-16 to UTF-8 during the serialization process.
- Load input files in parallel now that millions of allocations have been eliminated.
- Make the PSOSC serialization self-test only run when verbose logging is enabled.
- Optimize ParseQuoteComma by using FStringView and TArray<FStringView, TInlineAllocator<N>> to avoid temporary allocations.
These changes reduced the execution time of the build command by 90% on a large title. The time savings affect full, iterative, and single-package cooks.
#rb Arciel Rekman, Zousar.Shaker
#ushell-cherrypick of 10860101 by devin.doucette
#ROBOMERGE-SOURCE: CL 11163162 via CL 11163163 via CL 11163164
#ROBOMERGE-BOT: (v640-11091645)
[CL 11166858 by devin doucette in Main branch]
#rnx
#rb none
#ROBOMERGE-OWNER: ryan.durand
#ROBOMERGE-AUTHOR: ryan.durand
#ROBOMERGE-SOURCE: CL 10869210 via CL 10869511 via CL 10869900
#ROBOMERGE-BOT: (v613-10869866)
[CL 10870549 by ryan durand in Main branch]
- Added shader pipeline hash as a key attribute.
- Added shaders from shader pipelines (they can have different code if RHI removes unused interpolators between the stages, like D3D does).
#rb Dmitriy.Dyomin, Richard.Wallis
#ROBOMERGE-SOURCE: CL 9959336 via CL 9959339 via CL 9959342
#ROBOMERGE-BOT: (v558-9892490)
[CL 9959347 by arciel rekman in Main branch]
Shader code library was always emitting "<shadermodel>.scl.csv" files while cooking. This output data is only needed for users (or automated jobs) that update the PSO cache. With this change the configuration value "NeedsShaderStableKeys" needs to be set to true for platforms that use PSO (e.g. AndroidEngine.ini and IOSEngine.ini).
This saves diskspace and around 90 seconds on a iterative cook on a large project.
#rb Dmitriy.Dyomin
#ROBOMERGE-SOURCE: CL 6604300 via CL 6604799
#ROBOMERGE-BOT: (v351-6581450)
[CL 6604833 by johan berg in Main branch]
New AssetRegistryState::InitializeFromExistingAndPrune temporarily disabled until some bugs have been fixed.
Test Scenario:
1) BuildCookStageAndRun with these arguments: -platform=Win64 -configuration=Development
2) CookIterate with these arguments: -run=Cook -CookCultures=en -TargetPlatform=WindowsClient -unversioned -stdout -unattended -iterate
Wall Time Results (as an average of running step 2) two times):
Before: ~09:40 (580 seconds) cook commandlet time
After: ~02:30 (150 seconds) cook commandlet time
=> 07:10 (430 seconds ) faster, i.e. a ~ 3.9x speedup
Win32 FileSystem Results:
Before: 1.5 million GetFileAttribute calls and 1.2 million FindNextFile calls
After: 35 0000 GetFileAttribute calls and 1.6 million FindNextFile calls
=> ~400 000 calls to FindNextFile replaces ~1.5 million calls to GetFileAttribute
#rb none
(peafour-cherrypick of //UE4/Dev-Core/[at]5645695 by PJ.Kack)
#ROBOMERGE-OWNER: robert.manuszewski
#ROBOMERGE-AUTHOR: pj.kack
#ROBOMERGE-SOURCE: CL 5533504 via CL 5533655 via CL 5536177 via CL 5772728 via CL 5772753
[CL 5772793 by pj kack in Main branch]
Integrate all cook -iterate optimizations from //UE4/Dev-core/...
Test Scenario:
1) BuildCookStageAndRun with these arguments: -platform=Win64 -configuration=Development
2) CookIterate with these arguments: -run=Cook -CookCultures=en -TargetPlatform=WindowsClient -unversioned -stdout -unattended -iterate
Wall Time Results (as an average of running step 2) two times):
Before: ~09:40 (580 seconds) cook commandlet time
After: ~02:30 (150 seconds) cook commandlet time
=> 07:10 (430 seconds ) faster, i.e. a ~ 3.9x speedup
Win32 FileSystem Results:
Before: 1.5 million GetFileAttribute calls and 1.2 million FindNextFile calls
After: 35 0000 GetFileAttribute calls and 1.6 million FindNextFile calls
=> ~400 000 calls to FindNextFile replaces ~1.5 million calls to GetFileAttribute
#rb none
#ROBOMERGE-SOURCE: CL 5533504 via CL 5533655
[CL 5536177 by pj kack in Main branch]
