- When checking if we should output debug info, the output won't always be successful
- When preprocessing the shader source, make sure the output buffer is cleared before we do anything to prevent doubling up the shader source on retries.
#rb dan.elksnitis
[CL 25857810 by christopher waters in ue5-main branch]
- keep a preprocessor-specific shared cache of loaded shader files in ANSI format similar to the one in ShaderCore but skipping the unnecessary additional load to widechar, along with the associated extra allocations and conversions, and stripping comments directly as part of the load
- for in-memory source contained in the environment, which can't use the above due to potential different contents of same-named includes across jobs, perform the widechar->ansichar conversion and comment strip in a single step rather than converting then stripping, to save an additional allocation of the full source
#rb Jason.Nadro
#rb Yuriy.ODonnell
#preflight 64775dca2e6c1a0737f6aaef
[CL 25702732 by dan elksnitis in ue5-main branch]
#fyi dan.elksnitis
Original CL Desc
-----------------------------------------------------------------
[shader preprocessor] optimizations
- keep a preprocessor-specific shared cache of loaded shader files in ANSI format similar to the one in ShaderCore but skipping the unnecessary additional load to widechar, along with the associated extra allocations and conversions, and stripping comments directly as part of the load
- for in-memory source contained in the environment, which can't use the above due to potential different contents of same-named includes across jobs, perform the widechar->ansichar conversion and comment strip in a single step rather than converting then stripping, to save an additional allocation of the full source
#rb Jason.Nadro
#rb Yuriy.ODonnell
#preflight 646f74ac407983b99801c870
[CL 25626032 by dan elksnitis in ue5-main branch]
- keep a preprocessor-specific shared cache of loaded shader files in ANSI format similar to the one in ShaderCore but skipping the unnecessary additional load to widechar, along with the associated extra allocations and conversions, and stripping comments directly as part of the load
- for in-memory source contained in the environment, which can't use the above due to potential different contents of same-named includes across jobs, perform the widechar->ansichar conversion and comment strip in a single step rather than converting then stripping, to save an additional allocation of the full source
#rb Jason.Nadro
#rb Yuriy.ODonnell
#preflight 646f74ac407983b99801c870
[CL 25621499 by dan elksnitis in ue5-main branch]
- add new IShaderFormat API for separate preprocessing and compilation; backends can implement one or the other depending on the return value of SupportsIndependentPreprocessing
- add support for executing preprocessing in the cook process prior to job submission and constructing job input hashes based on preprocessed source (and a subset of the environment used as compile inputs). controlled by a cvar for now and disabled by default
- add a BaseShaderFormat class in ShaderCompilerCommon which implements common behaviour for output of debug data - note this function is only called for formats which support independent preprocessing, so is expected to be used only by formats which have been converted to use this API
- add new cvars for output of some additional shader debug data - 1. a txt file containing the input hash a.k.a. job cache key 2. a text file containing all diagnostic messages (errors and warnings) for the job
- minor change to how input hashes are constructed for pipeline jobs - sum hashes as 256-bit ints instead of adding to a buffer and re-hashing. faster and simpler, and also more collision resistant (sum of two well distributed hashes equally well distributed)
#rb Jason.Nadro
#rb Yuriy.ODonnell
#preflight 64512c88c86798f650b953d3
[CL 25317218 by dan elksnitis in ue5-main branch]
- add handling of #pragma directives in macro definitions
- add handling of #pragma message to create diagnostic outputs
- add and update comments in preprocessor.h
- unifying memory allocation mechanisms between stb_ds, stb_alloc and preprocessor - allow configuring all functions performing memory allocation/freeing via macros which should be specified via a header whose include path is specified via the STB_CONFIG macro (this will be included in stb_common.h if defined)
- don't pass an stb_arena to the include resolution callback; this is unnecessary complexity (implementation can handle its own memory allocation as long as the resulting strings have the appropriate lifetime) and indeed the arena used was global (and so not threadsafe)
- expose function allowing external modification of error modes so we can disable the error case for preprocessor directives not being at the start of a line (and so handle the case where we #define a macro whose value is a #pragma, or any other string possibly containing a # that we want to pass unmodified to the output i.e. the case of #define COMPILER_DEFINE #define which we use to keep #defines in the preprocessed code for handling by the platform compilers)
- fix leak of pp diagnostic messages
- remove internal caches for include resolution; they are problematic since (a) they aren't threadsafe, and (b) the caching of resolved filenames based on the include path found in source is inherently problematic (can potentially cause issues with same-named files in different folders included via relative paths)
- strip all code related to explicit file loading and lists of include paths; this has no purpose for our use case. note this makes the callbacks for file load/unload and include resolve mandatory, comments in preprocessor.h updated to reflect this.
- fix a few other memory leaks
#preflight 638611a9fa053c489a54b2ec
#rb Yuriy.ODonnell
[CL 23305324 by dan elksnitis in ue5-main branch]
- add explicit extern "C" on preprocessor methods to allow linking from C++ code
- add custom callbacks for include path resolution and file loading/unloading
- add padding space following macro expansion ending with a > to work around DXC bug (not handling >> correctly if it occurs at the end of a templated type)
- adding explicit handling for HLSL infinity constant (1.#INF); the # character outside of a directive was previously putting the preprocessor into an error state
- change line directive output to use #line format rather than # (DXC is ok with the latter but minifier doesn't currently handle it), and also always include the filename (so when minifying don't have to be careful not to strip the line directive with the filename in it); this also matches mcpp behaviour
- adding const-correctness and casts where needed to avoid const casts/compile errors in calling C++ code and satisfy Mac/Intellisense compilers
- bugfix: parsing of '.' was immediately putting the scanning state machine into "number" mode resulting in skipped macro expansion in code of the form "var.MACRO.var2"/numbers; added extra state transitions and changing the default transition when encountering '.' to an explicit mode that can allow either transitioning to parsing identifiers or numbers depending on following characters.
- bugfix: macros with an empty parameter list were incorrectly terminating said list on expansion (adding extra rparens)
- bugfix: set file load mode as "callback" when callback used for loading, and check the mode in the free callbacks
- various minor changes to satisfy Clang code analysis
#preflight 637e3391e30d438849ddf902
#rb Yuriy.ODonnell
[CL 23249332 by dan elksnitis in ue5-main branch]
- Remove shader platform specific info from uniform buffers, we can generate code with macros that translates to what each platform needs.
- With the per-platform differences removed, uniform buffer declarations (read: generated text) can be created once per shader parameter type that needs them instead of once per shader type's shader parameter type usage.
- FShaderType and FVertexFactoryType now hold a TSet of used uniform buffer names instead of a TMap that held potentially duplicated FStrings of uniform buffer defintions.
- Lists of uniform buffers are *potentially* different per platform, so the annoying CachedUniformBufferPlatform variables need to stick around for now.
- GenerateReferencedUniformBuffers renamed to GenerateReferencedUniformBufferNames and now populates a TSet instead of a TMap.
- ShaderType::AddReferencedUniformBufferIncludes renamed to ShaderType::AddUniformBufferIncludesToEnvironment which calls the shared UE::ShaderParameters::AddUniformBufferIncludesToEnvironment.
- CreateUniformBufferShaderDeclaration moved to UE::ShaderParameters::CreateUniformBufferShaderDeclaration and now returns its FString.
- CacheUniformBufferIncludes was removed since it was only populated values from CreateUniformBufferShaderDeclaration.
- Deprecated FShaderCompilerInput::SourceFilePrefix since nothing has populated it in 9 years.
#jira none
#rb dan.elksnitis, yuriy.odonnell
#preflight 63643e5ff233f06a127c11c1
[CL 22988611 by christopher waters in ue5-main branch]
