- add FShaderSource class which wraps source as populated by preprocessing and subsequently accessed by compilation and other debug features; this class automatically inserts zeroed padding such that 16-byte-wide SIMD string comparison operations do not require a non-SIMD tail to process any overhang.
- add typedefs for the string/view/character types and update preprocessing code to use these typedefs instead of the explicit types
- add explicit if constexprs in minifier code around char width to disable simd optimizations for char width != 2 (and subsequently skip the non-simd tail if char width == 2 since FShaderSource automatically adds the required padding)
#rb Jason.Nadro, Yuriy.ODonnell
[CL 30358137 by dan elksnitis in ue5-main branch]
- move sequence of preprocessing steps out of ShaderPreprocessor module and into UE::ShaderCompilerCommon::ExecuteShaderPreprocessingSteps; the former is now explicitly just the low-level preprocessor lib
- add an implementation of PreprocessShader in FBaseShaderFormat so backends which have no custom code to execute as part of preprocessing can just automatically inherit this implementation, and fix up such backends to eliminate now-unnecessary overrides
#rb christopher.waters, Laura.Hermanns
[CL 30178136 by dan elksnitis in ue5-main branch]
- never append the environment defines as commented code to the source used for further preprocessing/compilation; instead only append it to the debug USF
- strip comments after loading the debug usf in direct compile mode as some backends expect comments to have already been removed and the extra ones we add to the debug dump cause them to barf
- change all #if 0s in the debug usf to block comments instead so the above can strip them (said backends also don't like preprocessor directives left in the file)
#rb Jason.Nadro, rob.krajcarski
[CL 30161438 by dan elksnitis in ue5-main branch]
- move uniform buffer cleanup and dead stripping into ShaderPreprocessor module's PreprocessShader function
- add "required symbols" to compiler input struct to specify additional symbols to keep during minification aside from those specified by the entrypoint; modify API such that both an entry point string and additional symbols can be specified (to avoid each backend needing to manually parse the compound RT entry point string)
- make use of ModifyShaderCompilerInput in all backends to set additional defines and required symbols on input struct up front; only use the AdditionalDefines map in cases where it's actually necessary
- remove the various per-platform defines for enabling minifier, no longer required now that this has been rolled out for all backends
- fix SCW directcompile mode; this had rotted due to pieces of the FShaderCompilerEnvironment having been added that weren't explicitly serialized to either cmdline or in the shader source. this now serializes as a base64 string written inside the USF containing all portions of the environment required for compilation (using the same serialization function as is used to write/read the SCW input file)
- use a debug flag for indicating we're in "direct compile" mode and should load the debug USF off disk, rather than the poorly named "bSkipPreprocessedCache" (this name is both inaccurate and also confusing due to the addition of the preprocessed job cache)
- modify platform "force wave32" mechanism to use a pragma directive to set a compiler define, instead of doing string replacement in the preprocessed source
- add a view version of the RT entrypoint parsing to use in preprocessing, note that other paths still need to construct fstrings due to further manipulation so keeping the FString path around too
- clean up backends manually checking the "directcompile" cmdline arg
#rb christopher.waters, Yuriy.ODonnell
#rb Chris.Waters
#rb Laura.Hermanns
[CL 30023082 by dan elksnitis in ue5-main branch]
Flattened include dependencies are generated during include scanning at startup, basically for free (perf difference was well below noise). Bulk dependencies reduce round trips to the shader cache (which require mutex locks), and are indexed by the ANSI text exactly as it appears in the include directive in the source files, allowing a faster case sensitive hash, and avoiding the need for expensive path string operations. Anything found in a bulk dependency is stored in an array that parallels the dependency array, rather than a map. Includes stored in IncludeVirtualPathToSharedContentsMap also use an array.
Noting that our string classes (FString) are already case insensitive by default, some unnecessary case conversions were removed. The separate map of "seen" shaders was also removed, as we can just use the LoadedIncludesCache map for the same purpose. Where possible, existing ANSI strings are referenced, avoiding dynamic allocation.
#jira UE-197213
#rnx
#rb yuriy.odonnell jason.nadro
[CL 29095249 by jason hoerner in ue5-main branch]
* Moved identifier copy and macro bloom filter from maybe_expand_macro to test into copy_to_action_point / copy_to_action_point_macro_expansion. 13.1% of improvement.
* SSE implementation of scan_to_directive, 10x faster, 5.2%
* SSE implementation of identifier copy, 3x faster, 4.5%
* SSE ShaderConvertAndStripComments, 4x faster, 3.6%
* Fast inline string equality comparison, 4x faster, 1.5%
To make SSE implementations "safe" without needing special cases near the end of a buffer, it's necessary to ensure padding is present in the relevant buffers, anything that goes through a preprocess_string call. This includes the string arena allocator, temporary stbds arrays that hold strings, and file buffers passed in. The latter all pass through ShaderConvertAndStripComments, where we can add padding. (ShaderConvertAndStripComments itself has special cases for end of buffer). Code related to original 1 and 2 character macro filter removed, since I can't see a reason to enable it over the bloom filter.
I also attempted SSE optimization of copy_to_action_point and copy_line_without_comments, but improvement wasn't big enough to be worth the complexity (around 2% for the former, but massive code complexity, 0.5% for the latter). That's pretty much everything SSE friendly that's over 1% on a profile, although I think copy_argument can be made a lot faster, not primarily through SSE.
#jira UE-197212
#rnx
#rb yuriy.odonnell jason.nadro
[CL 28834324 by jason hoerner in ue5-main branch]
* Low overhead bloom filter for identifier hash lookups in maybe_expand_macro (3.1%)
* Optimized inline version of library calls (isspace, isalnum), and avoid strtoul for single digit numbers in evaluate_if (2.6%)
* Array reserve function inlining (1.7%)
* Fetch output size from preprocessor library instead of calling strlen (1.2%)
* Avoid FName -> string -> FName round trip conversion of key in FShaderCompilerEnvironment::Merge (0.3%)
* Use string concatentation instead of printf and inline storage in AddStbDefines (0.3%)
* Other overhead reduction (remove unused fast_dest feature, smaller stbds_array_header struct, savings from inlining) (1.5%)
#rnx
#rb dan.elksnitis jason.nadro
[CL 28537574 by jason hoerner in ue5-main branch]
* Inline array memory allocation added to low level preprocessor for output and various temporary buffers to reduce dynamic memory allocation and reallocation overhead. Saved 4.6%.
* FShaderPreprocessOutput::StripCode optimized to write to FString as TCHAR array, rather than using AppendChar (over 4x speedup). Saved 2.9%
* Shader source file cache now also stores stripped and ANSI converted source, to avoid need to convert and strip the source, plus allocating a copy is avoided. Saved 4.3%
* Uniform buffer structure declarations stored as ANSI converted source, avoiding convert and copy. Saved 4.9%
#rnx
#rb dan.elksnitis jason.nadro
[CL 28219741 by jason hoerner in ue5-main branch]
* Major define optimization involves converting map of defines to use FName keys and variant values rather than strings -- this eliminates most of the cost of string hashing, allocation, and conversion.
* Lower overhead FHashTable used instead of TMap.
* An initial map of defines can optionally be provided globally. Anything using an initial define can have its map index cached for optimized lookup when reading or writing.
Other micro-optimizations:
* Added Reserve calls for uniform buffer related maps, to eliminate map resizing / rehashing. Saved around 15% perf (after define optimizations).
* Added a map for UB lookup, instead of iterating through the linked list. Saved around 10% perf.
Non-optimization: Sort the order in which uniform buffer variable names are searched in BuildShaderFileToUniformBufferMap, to create determinism in ShaderDebug data for A/B testing (previously order was dependent on global constructor order for UB definitions, which could vary arbitrarily with unrelated changes).
#jira UE-187334
#rnx
#rb dan.elksnitis jason.nadro yuriy.odonnell
[CL 26142884 by jason hoerner in ue5-main branch]
- When checking if we should output debug info, the output won't always be successful
- When preprocessing the shader source, make sure the output buffer is cleared before we do anything to prevent doubling up the shader source on retries.
#rb dan.elksnitis
[CL 25857810 by christopher waters in ue5-main branch]
- keep a preprocessor-specific shared cache of loaded shader files in ANSI format similar to the one in ShaderCore but skipping the unnecessary additional load to widechar, along with the associated extra allocations and conversions, and stripping comments directly as part of the load
- for in-memory source contained in the environment, which can't use the above due to potential different contents of same-named includes across jobs, perform the widechar->ansichar conversion and comment strip in a single step rather than converting then stripping, to save an additional allocation of the full source
#rb Jason.Nadro
#rb Yuriy.ODonnell
#preflight 64775dca2e6c1a0737f6aaef
[CL 25702732 by dan elksnitis in ue5-main branch]
#fyi dan.elksnitis
Original CL Desc
-----------------------------------------------------------------
[shader preprocessor] optimizations
- keep a preprocessor-specific shared cache of loaded shader files in ANSI format similar to the one in ShaderCore but skipping the unnecessary additional load to widechar, along with the associated extra allocations and conversions, and stripping comments directly as part of the load
- for in-memory source contained in the environment, which can't use the above due to potential different contents of same-named includes across jobs, perform the widechar->ansichar conversion and comment strip in a single step rather than converting then stripping, to save an additional allocation of the full source
#rb Jason.Nadro
#rb Yuriy.ODonnell
#preflight 646f74ac407983b99801c870
[CL 25626032 by dan elksnitis in ue5-main branch]
- keep a preprocessor-specific shared cache of loaded shader files in ANSI format similar to the one in ShaderCore but skipping the unnecessary additional load to widechar, along with the associated extra allocations and conversions, and stripping comments directly as part of the load
- for in-memory source contained in the environment, which can't use the above due to potential different contents of same-named includes across jobs, perform the widechar->ansichar conversion and comment strip in a single step rather than converting then stripping, to save an additional allocation of the full source
#rb Jason.Nadro
#rb Yuriy.ODonnell
#preflight 646f74ac407983b99801c870
[CL 25621499 by dan elksnitis in ue5-main branch]
- add new IShaderFormat API for separate preprocessing and compilation; backends can implement one or the other depending on the return value of SupportsIndependentPreprocessing
- add support for executing preprocessing in the cook process prior to job submission and constructing job input hashes based on preprocessed source (and a subset of the environment used as compile inputs). controlled by a cvar for now and disabled by default
- add a BaseShaderFormat class in ShaderCompilerCommon which implements common behaviour for output of debug data - note this function is only called for formats which support independent preprocessing, so is expected to be used only by formats which have been converted to use this API
- add new cvars for output of some additional shader debug data - 1. a txt file containing the input hash a.k.a. job cache key 2. a text file containing all diagnostic messages (errors and warnings) for the job
- minor change to how input hashes are constructed for pipeline jobs - sum hashes as 256-bit ints instead of adding to a buffer and re-hashing. faster and simpler, and also more collision resistant (sum of two well distributed hashes equally well distributed)
#rb Jason.Nadro
#rb Yuriy.ODonnell
#preflight 64512c88c86798f650b953d3
[CL 25317218 by dan elksnitis in ue5-main branch]
