Flattened include dependencies are generated during include scanning at startup, basically for free (perf difference was well below noise). Bulk dependencies reduce round trips to the shader cache (which require mutex locks), and are indexed by the ANSI text exactly as it appears in the include directive in the source files, allowing a faster case sensitive hash, and avoiding the need for expensive path string operations. Anything found in a bulk dependency is stored in an array that parallels the dependency array, rather than a map. Includes stored in IncludeVirtualPathToSharedContentsMap also use an array.
Noting that our string classes (FString) are already case insensitive by default, some unnecessary case conversions were removed. The separate map of "seen" shaders was also removed, as we can just use the LoadedIncludesCache map for the same purpose. Where possible, existing ANSI strings are referenced, avoiding dynamic allocation.
#jira UE-197213
#rnx
#rb yuriy.odonnell jason.nadro
[CL 29095249 by jason hoerner in ue5-main branch]
* Moved identifier copy and macro bloom filter from maybe_expand_macro to test into copy_to_action_point / copy_to_action_point_macro_expansion. 13.1% of improvement.
* SSE implementation of scan_to_directive, 10x faster, 5.2%
* SSE implementation of identifier copy, 3x faster, 4.5%
* SSE ShaderConvertAndStripComments, 4x faster, 3.6%
* Fast inline string equality comparison, 4x faster, 1.5%
To make SSE implementations "safe" without needing special cases near the end of a buffer, it's necessary to ensure padding is present in the relevant buffers, anything that goes through a preprocess_string call. This includes the string arena allocator, temporary stbds arrays that hold strings, and file buffers passed in. The latter all pass through ShaderConvertAndStripComments, where we can add padding. (ShaderConvertAndStripComments itself has special cases for end of buffer). Code related to original 1 and 2 character macro filter removed, since I can't see a reason to enable it over the bloom filter.
I also attempted SSE optimization of copy_to_action_point and copy_line_without_comments, but improvement wasn't big enough to be worth the complexity (around 2% for the former, but massive code complexity, 0.5% for the latter). That's pretty much everything SSE friendly that's over 1% on a profile, although I think copy_argument can be made a lot faster, not primarily through SSE.
#jira UE-197212
#rnx
#rb yuriy.odonnell jason.nadro
[CL 28834324 by jason hoerner in ue5-main branch]
* Low overhead bloom filter for identifier hash lookups in maybe_expand_macro (3.1%)
* Optimized inline version of library calls (isspace, isalnum), and avoid strtoul for single digit numbers in evaluate_if (2.6%)
* Array reserve function inlining (1.7%)
* Fetch output size from preprocessor library instead of calling strlen (1.2%)
* Avoid FName -> string -> FName round trip conversion of key in FShaderCompilerEnvironment::Merge (0.3%)
* Use string concatentation instead of printf and inline storage in AddStbDefines (0.3%)
* Other overhead reduction (remove unused fast_dest feature, smaller stbds_array_header struct, savings from inlining) (1.5%)
#rnx
#rb dan.elksnitis jason.nadro
[CL 28537574 by jason hoerner in ue5-main branch]
* Inline array memory allocation added to low level preprocessor for output and various temporary buffers to reduce dynamic memory allocation and reallocation overhead. Saved 4.6%.
* FShaderPreprocessOutput::StripCode optimized to write to FString as TCHAR array, rather than using AppendChar (over 4x speedup). Saved 2.9%
* Shader source file cache now also stores stripped and ANSI converted source, to avoid need to convert and strip the source, plus allocating a copy is avoided. Saved 4.3%
* Uniform buffer structure declarations stored as ANSI converted source, avoiding convert and copy. Saved 4.9%
#rnx
#rb dan.elksnitis jason.nadro
[CL 28219741 by jason hoerner in ue5-main branch]
- modify versioning mechanisms to use UEMETADATA pragmas
- update commands which generate version.ush files to generate such pragmas instead of comments
- include the above pragma-driven version(s) in the input hash when preprocessed job cache is enabled (not needed for the disabled case since as before the entire contents of the version files are hashed)
- fix bug in stb_preprocessor that was causing it to stop expanding macros if any diagnostic was encountered (and rename the array field storing diagnostic messages to diagnostics, since it contains more than just errors)
#rb Yuriy.ODonnell
[CL 27515060 by dan elksnitis in ue5-main branch]
* Major define optimization involves converting map of defines to use FName keys and variant values rather than strings -- this eliminates most of the cost of string hashing, allocation, and conversion.
* Lower overhead FHashTable used instead of TMap.
* An initial map of defines can optionally be provided globally. Anything using an initial define can have its map index cached for optimized lookup when reading or writing.
Other micro-optimizations:
* Added Reserve calls for uniform buffer related maps, to eliminate map resizing / rehashing. Saved around 15% perf (after define optimizations).
* Added a map for UB lookup, instead of iterating through the linked list. Saved around 10% perf.
Non-optimization: Sort the order in which uniform buffer variable names are searched in BuildShaderFileToUniformBufferMap, to create determinism in ShaderDebug data for A/B testing (previously order was dependent on global constructor order for UB definitions, which could vary arbitrarily with unrelated changes).
#jira UE-187334
#rnx
#rb dan.elksnitis jason.nadro yuriy.odonnell
[CL 26142884 by jason hoerner in ue5-main branch]
- When checking if we should output debug info, the output won't always be successful
- When preprocessing the shader source, make sure the output buffer is cleared before we do anything to prevent doubling up the shader source on retries.
#rb dan.elksnitis
[CL 25857810 by christopher waters in ue5-main branch]
- keep a preprocessor-specific shared cache of loaded shader files in ANSI format similar to the one in ShaderCore but skipping the unnecessary additional load to widechar, along with the associated extra allocations and conversions, and stripping comments directly as part of the load
- for in-memory source contained in the environment, which can't use the above due to potential different contents of same-named includes across jobs, perform the widechar->ansichar conversion and comment strip in a single step rather than converting then stripping, to save an additional allocation of the full source
#rb Jason.Nadro
#rb Yuriy.ODonnell
#preflight 64775dca2e6c1a0737f6aaef
[CL 25702732 by dan elksnitis in ue5-main branch]
#fyi dan.elksnitis
Original CL Desc
-----------------------------------------------------------------
[shader preprocessor] optimizations
- keep a preprocessor-specific shared cache of loaded shader files in ANSI format similar to the one in ShaderCore but skipping the unnecessary additional load to widechar, along with the associated extra allocations and conversions, and stripping comments directly as part of the load
- for in-memory source contained in the environment, which can't use the above due to potential different contents of same-named includes across jobs, perform the widechar->ansichar conversion and comment strip in a single step rather than converting then stripping, to save an additional allocation of the full source
#rb Jason.Nadro
#rb Yuriy.ODonnell
#preflight 646f74ac407983b99801c870
[CL 25626032 by dan elksnitis in ue5-main branch]
- keep a preprocessor-specific shared cache of loaded shader files in ANSI format similar to the one in ShaderCore but skipping the unnecessary additional load to widechar, along with the associated extra allocations and conversions, and stripping comments directly as part of the load
- for in-memory source contained in the environment, which can't use the above due to potential different contents of same-named includes across jobs, perform the widechar->ansichar conversion and comment strip in a single step rather than converting then stripping, to save an additional allocation of the full source
#rb Jason.Nadro
#rb Yuriy.ODonnell
#preflight 646f74ac407983b99801c870
[CL 25621499 by dan elksnitis in ue5-main branch]
