* Major define optimization involves converting map of defines to use FName keys and variant values rather than strings -- this eliminates most of the cost of string hashing, allocation, and conversion.
* Lower overhead FHashTable used instead of TMap.
* An initial map of defines can optionally be provided globally. Anything using an initial define can have its map index cached for optimized lookup when reading or writing.
Other micro-optimizations:
* Added Reserve calls for uniform buffer related maps, to eliminate map resizing / rehashing. Saved around 15% perf (after define optimizations).
* Added a map for UB lookup, instead of iterating through the linked list. Saved around 10% perf.
Non-optimization: Sort the order in which uniform buffer variable names are searched in BuildShaderFileToUniformBufferMap, to create determinism in ShaderDebug data for A/B testing (previously order was dependent on global constructor order for UB definitions, which could vary arbitrarily with unrelated changes).
#jira UE-187334
#rnx
#rb dan.elksnitis jason.nadro yuriy.odonnell
[CL 26142884 by jason hoerner in ue5-main branch]
- When checking if we should output debug info, the output won't always be successful
- When preprocessing the shader source, make sure the output buffer is cleared before we do anything to prevent doubling up the shader source on retries.
#rb dan.elksnitis
[CL 25857810 by christopher waters in ue5-main branch]
- keep a preprocessor-specific shared cache of loaded shader files in ANSI format similar to the one in ShaderCore but skipping the unnecessary additional load to widechar, along with the associated extra allocations and conversions, and stripping comments directly as part of the load
- for in-memory source contained in the environment, which can't use the above due to potential different contents of same-named includes across jobs, perform the widechar->ansichar conversion and comment strip in a single step rather than converting then stripping, to save an additional allocation of the full source
#rb Jason.Nadro
#rb Yuriy.ODonnell
#preflight 64775dca2e6c1a0737f6aaef
[CL 25702732 by dan elksnitis in ue5-main branch]
#fyi dan.elksnitis
Original CL Desc
-----------------------------------------------------------------
[shader preprocessor] optimizations
- keep a preprocessor-specific shared cache of loaded shader files in ANSI format similar to the one in ShaderCore but skipping the unnecessary additional load to widechar, along with the associated extra allocations and conversions, and stripping comments directly as part of the load
- for in-memory source contained in the environment, which can't use the above due to potential different contents of same-named includes across jobs, perform the widechar->ansichar conversion and comment strip in a single step rather than converting then stripping, to save an additional allocation of the full source
#rb Jason.Nadro
#rb Yuriy.ODonnell
#preflight 646f74ac407983b99801c870
[CL 25626032 by dan elksnitis in ue5-main branch]
- keep a preprocessor-specific shared cache of loaded shader files in ANSI format similar to the one in ShaderCore but skipping the unnecessary additional load to widechar, along with the associated extra allocations and conversions, and stripping comments directly as part of the load
- for in-memory source contained in the environment, which can't use the above due to potential different contents of same-named includes across jobs, perform the widechar->ansichar conversion and comment strip in a single step rather than converting then stripping, to save an additional allocation of the full source
#rb Jason.Nadro
#rb Yuriy.ODonnell
#preflight 646f74ac407983b99801c870
[CL 25621499 by dan elksnitis in ue5-main branch]
- add new IShaderFormat API for separate preprocessing and compilation; backends can implement one or the other depending on the return value of SupportsIndependentPreprocessing
- add support for executing preprocessing in the cook process prior to job submission and constructing job input hashes based on preprocessed source (and a subset of the environment used as compile inputs). controlled by a cvar for now and disabled by default
- add a BaseShaderFormat class in ShaderCompilerCommon which implements common behaviour for output of debug data - note this function is only called for formats which support independent preprocessing, so is expected to be used only by formats which have been converted to use this API
- add new cvars for output of some additional shader debug data - 1. a txt file containing the input hash a.k.a. job cache key 2. a text file containing all diagnostic messages (errors and warnings) for the job
- minor change to how input hashes are constructed for pipeline jobs - sum hashes as 256-bit ints instead of adding to a buffer and re-hashing. faster and simpler, and also more collision resistant (sum of two well distributed hashes equally well distributed)
#rb Jason.Nadro
#rb Yuriy.ODonnell
#preflight 64512c88c86798f650b953d3
[CL 25317218 by dan elksnitis in ue5-main branch]
- Remove shader platform specific info from uniform buffers, we can generate code with macros that translates to what each platform needs.
- With the per-platform differences removed, uniform buffer declarations (read: generated text) can be created once per shader parameter type that needs them instead of once per shader type's shader parameter type usage.
- FShaderType and FVertexFactoryType now hold a TSet of used uniform buffer names instead of a TMap that held potentially duplicated FStrings of uniform buffer defintions.
- Lists of uniform buffers are *potentially* different per platform, so the annoying CachedUniformBufferPlatform variables need to stick around for now.
- GenerateReferencedUniformBuffers renamed to GenerateReferencedUniformBufferNames and now populates a TSet instead of a TMap.
- ShaderType::AddReferencedUniformBufferIncludes renamed to ShaderType::AddUniformBufferIncludesToEnvironment which calls the shared UE::ShaderParameters::AddUniformBufferIncludesToEnvironment.
- CreateUniformBufferShaderDeclaration moved to UE::ShaderParameters::CreateUniformBufferShaderDeclaration and now returns its FString.
- CacheUniformBufferIncludes was removed since it was only populated values from CreateUniformBufferShaderDeclaration.
- Deprecated FShaderCompilerInput::SourceFilePrefix since nothing has populated it in 9 years.
#jira none
#rb dan.elksnitis, yuriy.odonnell
#preflight 63643e5ff233f06a127c11c1
[CL 22988611 by christopher waters in ue5-main branch]
Changed ShaderArchive, GlobalShaderCache, ShaderDebugInfo and Autogen to use ShaderPlatformName and not ShaderFormat when naming their output files.
#rb Jack.Porter, Chris.Waters, Mihnea.Balta, Jason.Nadro
#jira UE-120561
#preflight 62c31f6fc9410537282296c6
[CL 20937870 by Florin Pascu in ue5-main branch]
