GIsCriticalError was not set to true inside exception handling block, preventing logs in SCW to be dumped from memory to disk despite -LogToMemory argument.
#rnx
#rb sebastian.schoner
[CL 35461655 by laura hermanns in ue5-main branch]
- Enable logging to memory in SCW target script to decide at runtime whether we need to dump the log to disk.
- Add "-DumpLogOnExitCrashOnly" command line argument.
This is used in combination with DUMP_LOG_ON_EXIT to only dump the memory stored log to file if a critical error was reported (i.e. GIsCriticalError).
- Add CVar "r.ShaderCompiler.DebugDumpWorkerCrashLog" to dump SCW logs to disk when they crash.
For example, a SCW with worker index 2 will store its log at "Engine/Programs/ShaderCompileWorker/Saved/Logs/ShaderCompileWorker-2.log" when run locally.
On build machines, the respective path for artifacts is used.
- Make WITH_LOGGING_TO_MEMORY the only condition to allocate a memory backed output device for logging:
CL 2940738 introduced the preprocessor guard for FOutputDeviceMemory '!IS_PROGRAM && !WITH_EDITORONLY_DATA', while the specific request via WITH_LOGGING_TO_MEMORY was added later.
For the SCW, this is contradictory if we enable it explicitly but IS_PROGRAM is true for the worker processes.
We therefore remove the original condition and allow logs to be stored in memory solely on the condition WITH_LOGGING_TO_MEMORY.
- Add "-ShaderCompileWorkerCrashLogsDir=<DIR>" argument to shader compiling manager to allow dumping SCW logs to customized base directory.
#rnx
#rb Matt.Peters, sebastian.schoner
[CL 35144547 by laura hermanns in ue5-main branch]
Split Work Graph shaders into multiple frequencies.
This is somewhat in anticipation of graphic nodes.
But also it is a replacement for using CFLAG_WorkgraphLocalNodes to differentiate nodes with local or global root signature.
#rb Yuriy.ODonnell
[CL 33146442 by jeremy moore in ue5-main branch]
- strip old job cache path (constructing an input hash based on all inputs); the cache key based on preprocessed source is now the One True Job Cache
- strip parallelfor-based job submission path (gamethread-blocking); the task path has been enabled for a long while now without issues
- remove the "compile job inputs" debug dump; this is no longer relevant since it's based on inputs to the now-stripped compile job path (similar functionality will be provided by the new form of debug usf once completed)
#rb Laura.Hermanns
[CL 32825320 by dan elksnitis in ue5-main branch]
This will provide the detail needed to connect failed compile jobs (due terminated parent processes) to their UBA job:
Previously, such error looked like this:
> LogUbaController: Display: [CookWorker 4]: LogWindows: Error: appError called: Assertion failed: IFileManager::Get().FileSize(*FilePath) == INDEX_NONE [File:[...]\ShaderCompileWorker.cpp] [Line: 877]
> LogUbaController: Display: [CookWorker 4]: Exiting due to OpenProcess(ParentProcessId) failing and the input file is present!
And now provides the file path:
> LogUbaController: Display: [CookWorker 4]: LogWindows: Error: appError called: Assertion failed: IFileManager::Get().FileSize(*FilePath) == INDEX_NONE [File:[...]\ShaderCompileWorker.cpp] [Line: 877]
> LogUbaController: Display: [CookWorker 4]: Exiting due to OpenProcess(ParentProcessId) failing and the input file is present! FilePath=D:\Example\42.uba.out
#rb henrik.karlsson, Jason.Nadro
#rnx
[CL 31774825 by laura hermanns in ue5-main branch]
- Added ability to optionally skip SDK validation when making a Target instance even if we don't want to skip Target validation (used when we make the base UnrealEditor target to compare to project's Editor target - we don't need to manage SDK for that temporary UnrealEditor target).
- ShaderCompileWorker is now BuildEnvironment.UniqueIfNeeded, which will build into the project binaries dir if needed (based on per-project SDK override)
- UnrealPak is marked as not caring about an SDK versions so it can be built along with SCW that does care
- Added $(BinaryDir) variable to UEBuildTarget, etc, for the location of the first Binary in the output binaries
#rb David.Harvey, Joe.Kirchoff
[CL 31661939 by josh adams in ue5-main branch]
It's about 1/4 of CoreMinimal.h but rarely needed (Compression.h pulls on CriticalSection.h and Map.h that are costly).
#rb Yoan.StAmant
[CL 30683417 by aris theophanidis in ue5-main branch]
* Fixed so CrashOutputFile is updated when process is reused (the crash output file should be the same as the output file)
[CL 30643496 by henrik karlsson in ue5-main branch]
Replaced PLATFORM_COMPILER_HAS_DECLTYPE_AUTO blocks.
Replaced PLATFORM_COMPILER_HAS_FOLD_EXPRESSIONS blocks.
[CL 30600164 by steve robb in ue5-main branch]
* Fixed function pointer format for getting next process in uba
* Fixed so output files are written even when return code is non-zero (error)
[CL 30585779 by henrik karlsson in ue5-main branch]
* Removed code checking if worker is running remotely. local and remote should be treated the same way
[CL 30525050 by henrik karlsson in ue5-main branch]
* Changed so when building through uba we don't delete the .in file. Since uba supports rescheduling when remote helpers disconnect we can't delete this file until out is fully sent back
[CL 30468743 by henrik karlsson in ue5-main branch]
- move uniform buffer cleanup and dead stripping into ShaderPreprocessor module's PreprocessShader function
- add "required symbols" to compiler input struct to specify additional symbols to keep during minification aside from those specified by the entrypoint; modify API such that both an entry point string and additional symbols can be specified (to avoid each backend needing to manually parse the compound RT entry point string)
- make use of ModifyShaderCompilerInput in all backends to set additional defines and required symbols on input struct up front; only use the AdditionalDefines map in cases where it's actually necessary
- remove the various per-platform defines for enabling minifier, no longer required now that this has been rolled out for all backends
- fix SCW directcompile mode; this had rotted due to pieces of the FShaderCompilerEnvironment having been added that weren't explicitly serialized to either cmdline or in the shader source. this now serializes as a base64 string written inside the USF containing all portions of the environment required for compilation (using the same serialization function as is used to write/read the SCW input file)
- use a debug flag for indicating we're in "direct compile" mode and should load the debug USF off disk, rather than the poorly named "bSkipPreprocessedCache" (this name is both inaccurate and also confusing due to the addition of the preprocessed job cache)
- modify platform "force wave32" mechanism to use a pragma directive to set a compiler define, instead of doing string replacement in the preprocessed source
- add a view version of the RT entrypoint parsing to use in preprocessing, note that other paths still need to construct fstrings due to further manipulation so keeping the FString path around too
- clean up backends manually checking the "directcompile" cmdline arg
#rb christopher.waters, Yuriy.ODonnell
#rb Chris.Waters
#rb Laura.Hermanns
[CL 30023082 by dan elksnitis in ue5-main branch]
* Inline array memory allocation added to low level preprocessor for output and various temporary buffers to reduce dynamic memory allocation and reallocation overhead. Saved 4.6%.
* FShaderPreprocessOutput::StripCode optimized to write to FString as TCHAR array, rather than using AppendChar (over 4x speedup). Saved 2.9%
* Shader source file cache now also stores stripped and ANSI converted source, to avoid need to convert and strip the source, plus allocating a copy is avoided. Saved 4.3%
* Uniform buffer structure declarations stored as ANSI converted source, avoiding convert and copy. Saved 4.9%
#rnx
#rb dan.elksnitis jason.nadro
[CL 28219741 by jason hoerner in ue5-main branch]
