Split Work Graph shaders into multiple frequencies.
This is somewhat in anticipation of graphic nodes.
But also it is a replacement for using CFLAG_WorkgraphLocalNodes to differentiate nodes with local or global root signature.
#rb Yuriy.ODonnell
[CL 33146442 by jeremy moore in ue5-main branch]
- strip old job cache path (constructing an input hash based on all inputs); the cache key based on preprocessed source is now the One True Job Cache
- strip parallelfor-based job submission path (gamethread-blocking); the task path has been enabled for a long while now without issues
- remove the "compile job inputs" debug dump; this is no longer relevant since it's based on inputs to the now-stripped compile job path (similar functionality will be provided by the new form of debug usf once completed)
#rb Laura.Hermanns
[CL 32825320 by dan elksnitis in ue5-main branch]
This will provide the detail needed to connect failed compile jobs (due terminated parent processes) to their UBA job:
Previously, such error looked like this:
> LogUbaController: Display: [CookWorker 4]: LogWindows: Error: appError called: Assertion failed: IFileManager::Get().FileSize(*FilePath) == INDEX_NONE [File:[...]\ShaderCompileWorker.cpp] [Line: 877]
> LogUbaController: Display: [CookWorker 4]: Exiting due to OpenProcess(ParentProcessId) failing and the input file is present!
And now provides the file path:
> LogUbaController: Display: [CookWorker 4]: LogWindows: Error: appError called: Assertion failed: IFileManager::Get().FileSize(*FilePath) == INDEX_NONE [File:[...]\ShaderCompileWorker.cpp] [Line: 877]
> LogUbaController: Display: [CookWorker 4]: Exiting due to OpenProcess(ParentProcessId) failing and the input file is present! FilePath=D:\Example\42.uba.out
#rb henrik.karlsson, Jason.Nadro
#rnx
[CL 31774825 by laura hermanns in ue5-main branch]
It's about 1/4 of CoreMinimal.h but rarely needed (Compression.h pulls on CriticalSection.h and Map.h that are costly).
#rb Yoan.StAmant
[CL 30683417 by aris theophanidis in ue5-main branch]
* Fixed so CrashOutputFile is updated when process is reused (the crash output file should be the same as the output file)
[CL 30643496 by henrik karlsson in ue5-main branch]
Replaced PLATFORM_COMPILER_HAS_DECLTYPE_AUTO blocks.
Replaced PLATFORM_COMPILER_HAS_FOLD_EXPRESSIONS blocks.
[CL 30600164 by steve robb in ue5-main branch]
* Fixed function pointer format for getting next process in uba
* Fixed so output files are written even when return code is non-zero (error)
[CL 30585779 by henrik karlsson in ue5-main branch]
* Removed code checking if worker is running remotely. local and remote should be treated the same way
[CL 30525050 by henrik karlsson in ue5-main branch]
* Changed so when building through uba we don't delete the .in file. Since uba supports rescheduling when remote helpers disconnect we can't delete this file until out is fully sent back
[CL 30468743 by henrik karlsson in ue5-main branch]
- move uniform buffer cleanup and dead stripping into ShaderPreprocessor module's PreprocessShader function
- add "required symbols" to compiler input struct to specify additional symbols to keep during minification aside from those specified by the entrypoint; modify API such that both an entry point string and additional symbols can be specified (to avoid each backend needing to manually parse the compound RT entry point string)
- make use of ModifyShaderCompilerInput in all backends to set additional defines and required symbols on input struct up front; only use the AdditionalDefines map in cases where it's actually necessary
- remove the various per-platform defines for enabling minifier, no longer required now that this has been rolled out for all backends
- fix SCW directcompile mode; this had rotted due to pieces of the FShaderCompilerEnvironment having been added that weren't explicitly serialized to either cmdline or in the shader source. this now serializes as a base64 string written inside the USF containing all portions of the environment required for compilation (using the same serialization function as is used to write/read the SCW input file)
- use a debug flag for indicating we're in "direct compile" mode and should load the debug USF off disk, rather than the poorly named "bSkipPreprocessedCache" (this name is both inaccurate and also confusing due to the addition of the preprocessed job cache)
- modify platform "force wave32" mechanism to use a pragma directive to set a compiler define, instead of doing string replacement in the preprocessed source
- add a view version of the RT entrypoint parsing to use in preprocessing, note that other paths still need to construct fstrings due to further manipulation so keeping the FString path around too
- clean up backends manually checking the "directcompile" cmdline arg
#rb christopher.waters, Yuriy.ODonnell
#rb Chris.Waters
#rb Laura.Hermanns
[CL 30023082 by dan elksnitis in ue5-main branch]
* Inline array memory allocation added to low level preprocessor for output and various temporary buffers to reduce dynamic memory allocation and reallocation overhead. Saved 4.6%.
* FShaderPreprocessOutput::StripCode optimized to write to FString as TCHAR array, rather than using AppendChar (over 4x speedup). Saved 2.9%
* Shader source file cache now also stores stripped and ANSI converted source, to avoid need to convert and strip the source, plus allocating a copy is avoided. Saved 4.3%
* Uniform buffer structure declarations stored as ANSI converted source, avoiding convert and copy. Saved 4.9%
#rnx
#rb dan.elksnitis jason.nadro
[CL 28219741 by jason hoerner in ue5-main branch]
- Remove the condition causing SCW to automatically exit after a single job if HLSLcc was used. This doesn't appear to be beneficial; HLSLcc uses a lot of memory but does not appear to leak it, so we're wasting time spinning up new processes unnecessarily here. This allows us to deprecated the associated field in the compiler output and function in IShaderFormat.
- Strip out compiler invocations stats; this was not accurate in some cases and it was decided after discussion that at this point the effort to fix the problems was not worth the utility we get from it.
#rb Jason.Nadro
#rb Laura.Hermanns
[CL 26256933 by dan elksnitis in ue5-main branch]
* Early out when adding uniform buffer resource entries, by checking if uniform buffer had been processed, before processing individual entries.
* Changed resource entry list from a map to an array, saving the cost of map operations adding items to it.
* Resource entries for global uniform buffers cached once at startup.
* Resource entry member names stored in an external buffer, to avoid per-entry string memory allocation overhead.
* Miscellaneous smaller optimizations -- for example, using ByHash to avoid redundant hashing and FString construction.
#jira none
#rnx
#rb jason.nadro dan.elksnitis
#preflight 646cfdbc1134ffac7034af60
[CL 25596308 by jason hoerner in ue5-main branch]
- add new IShaderFormat API for separate preprocessing and compilation; backends can implement one or the other depending on the return value of SupportsIndependentPreprocessing
- add support for executing preprocessing in the cook process prior to job submission and constructing job input hashes based on preprocessed source (and a subset of the environment used as compile inputs). controlled by a cvar for now and disabled by default
- add a BaseShaderFormat class in ShaderCompilerCommon which implements common behaviour for output of debug data - note this function is only called for formats which support independent preprocessing, so is expected to be used only by formats which have been converted to use this API
- add new cvars for output of some additional shader debug data - 1. a txt file containing the input hash a.k.a. job cache key 2. a text file containing all diagnostic messages (errors and warnings) for the job
- minor change to how input hashes are constructed for pipeline jobs - sum hashes as 256-bit ints instead of adding to a buffer and re-hashing. faster and simpler, and also more collision resistant (sum of two well distributed hashes equally well distributed)
#rb Jason.Nadro
#rb Yuriy.ODonnell
#preflight 64512c88c86798f650b953d3
[CL 25317218 by dan elksnitis in ue5-main branch]
