Commit Graph

143 Commits

Author SHA1 Message Date
joe kirchoff
652353f2cc ShaderCompileWorker: Fix postbuild copy step if the target name is renamed for any reason
#rnx

[CL 25904169 by joe kirchoff in ue5-main branch]
2023-06-09 16:50:37 -04:00
jason hoerner
88d847a0ad GlobalBeginCompileShader optimizations. 3.2x improvement in performance of the function on Lyra cook:
* Early out when adding uniform buffer resource entries, by checking if uniform buffer had been processed, before processing individual entries.
* Changed resource entry list from a map to an array, saving the cost of map operations adding items to it.
* Resource entries for global uniform buffers cached once at startup.
* Resource entry member names stored in an external buffer, to avoid per-entry string memory allocation overhead.
* Miscellaneous smaller optimizations -- for example, using ByHash to avoid redundant hashing and FString construction.

#jira none
#rnx
#rb jason.nadro dan.elksnitis
#preflight 646cfdbc1134ffac7034af60

[CL 25596308 by jason hoerner in ue5-main branch]
2023-05-23 20:16:11 -04:00
dan elksnitis
f645f93588 [shaders] remove legacy preprocessor option & mcpp library
#rb Jason.Nadro
#preflight 6467bf0c2c0a5da0dcd7aaf2
#fyi Yuriy.ODonnell

[CL 25549427 by dan elksnitis in ue5-main branch]
2023-05-19 14:50:25 -04:00
bryan sefcik
91c57d395e Removed redundant module includes.
#preflight 645d4bf3aa3c584c0b5b3a67

[CL 25435653 by bryan sefcik in ue5-main branch]
2023-05-11 16:48:21 -04:00
Steve Robb
0f44db07d5 Replaced some container operator new usage.
#rb james.hopkin
#jira none
#preflight 6459670f28155a0f41565a34

[CL 25384456 by Steve Robb in ue5-main branch]
2023-05-09 05:17:49 -04:00
dan elksnitis
b52faed5d5 [shaders]
- add new IShaderFormat API for separate preprocessing and compilation; backends can implement one or the other depending on the return value of SupportsIndependentPreprocessing
- add support for executing preprocessing in the cook process prior to job submission and constructing job input hashes based on preprocessed source (and a subset of the environment used as compile inputs). controlled by a cvar for now and disabled by default
- add a BaseShaderFormat class in ShaderCompilerCommon which implements common behaviour for output of debug data - note this function is only called for formats which support independent preprocessing, so is expected to be used only by formats which have been converted to use this API
- add new cvars for output of some additional shader debug data - 1. a txt file containing the input hash a.k.a. job cache key 2. a text file containing all diagnostic messages (errors and warnings) for the job
- minor change to how input hashes are constructed for pipeline jobs - sum hashes as 256-bit ints instead of adding to a buffer and re-hashing. faster and simpler, and also more collision resistant (sum of two well distributed hashes equally well distributed)

#rb Jason.Nadro
#rb Yuriy.ODonnell
#preflight 64512c88c86798f650b953d3

[CL 25317218 by dan elksnitis in ue5-main branch]
2023-05-03 10:17:48 -04:00
ionut matasaru
3d7c0e8440 UBT: Removed deprecated TargetInfo::bUseMallocProfiler toggle in Source/Programs.
#jira UE-141779
#rb Joe.Kirchoff
#rb Johan.Berg
#preflight 64380aea7a00790bfd116430

[CL 25101875 by ionut matasaru in ue5-main branch]
2023-04-19 04:15:59 -04:00
dan elksnitis
2971b65bc4 [shaders] deprecated FShaderPipelineCompileJob::bFailedRemovingUnused field; this is only used to append some dubiously useful information to a log message (the underlying error causing the failure is always reported as any other shader preprocess/compile error). opting to remove rather than dragging this along with forthcoming preprocessing refactoring.
#preflight 6435679c1d61ecec3b56c78d
#rb Jason.Nadro

[CL 24991496 by dan elksnitis in ue5-main branch]
2023-04-11 11:05:47 -04:00
christopher waters
6a4206d490 Removing bad Launch include paths from programs.
[CL 24328631 by christopher waters in ue5-main branch]
2023-02-20 17:39:13 -05:00
dan elksnitis
b5acd2894d [shaders] unify handling of single job bSucceeded flag; wasn't getting set properly when not using workers (resubmit with pipeline serialization fix)
#rb Yuriy.ODonnell
#rb Laura.Hermanns
#rb Jason.Nadro
#preflight 63d2e011d21dbe1d29be43c1

[CL 23893259 by dan elksnitis in ue5-main branch]
2023-01-27 17:02:08 -05:00
dan elksnitis
017b41a8b3 [Backout] - CL23885168
#fyi dan.elksnitis
Original CL Desc
-----------------------------------------------------------------
[shaders] unify handling of single job bSucceeded flag; wasn't getting set properly when not using workers

#rb Yuriy.ODonnell
#rb Laura.Hermanns
#rb Jason.Nadro
#preflight 63d2e011d21dbe1d29be43c1

[CL 23889694 by dan elksnitis in ue5-main branch]
2023-01-27 14:57:37 -05:00
dan elksnitis
5788613c9d [shaders] unify handling of single job bSucceeded flag; wasn't getting set properly when not using workers
#rb Yuriy.ODonnell
#rb Laura.Hermanns
#rb Jason.Nadro
#preflight 63d2e011d21dbe1d29be43c1

[CL 23885168 by dan elksnitis in ue5-main branch]
2023-01-27 12:04:51 -05:00
dan elksnitis
da802ed74c [shaders]
- move compile job types into RenderCore and use them in SCW
- using the above factor out a helper function for pipeline job compilation to further reduce code duplication
- make flags indicating job success/failure conditions part of the SCW output data rather than evaluating them when reading the output (so they can be evaluated once in the helper function mentioned above)
- modify debug worker input dumps to write a single file for combined-pipeline compilation rather than one file per stage

#rb Yuriy.ODonnell
#preflight 63d1725d574ab9cae4c93d76

[CL 23868225 by dan elksnitis in ue5-main branch]
2023-01-26 10:35:39 -05:00
bryan sefcik
377f3c2775 * Updated AssemblyUtils to add an event callback to AssemblyResolve when calling AddFileToAssemblyCache. You had to before explicitly call InstallRecursiveAssemblyResolver to get the assembly cache resolver to work.
* Updated RulesAssembly to add the parent assembles to the assembly cache so the types in the other assemblies can be used when calling the constructor.
#preflight 63cf2823b84de45a0c12a6a4

[CL 23825371 by bryan sefcik in ue5-main branch]
2023-01-23 19:47:44 -05:00
dan elksnitis
be95382d48 [shaders] factor out single-input compilation and exception handling into shared utility functions
#preflight 63caa36ad45afa2a8f236855
#rb Jason.Nadro
#rb Laura.Hermanns
#rb Yuriy.ODonnell

[CL 23789494 by dan elksnitis in ue5-main branch]
2023-01-20 09:39:05 -05:00
dan elksnitis
cfcde3fcbf [shaders] remove functionality which allows setting cvars via the shader input environment, along with the only existing usage of it (which is obsolete)
this has much potential for misuse, and is generally incompatible with the goal of in-process multithreaded compilation

#rb Jason.Nadro
#rb Yuriy.ODonnell
#preflight 63c94ba902024f93d83dfbbc

[CL 23772691 by dan elksnitis in ue5-main branch]
2023-01-19 09:12:23 -05:00
laura hermanns
5e725d3838 Reissue shader compile jobs on OOM error.
#rb Jason.Nadro, Yuriy.Odonnell
[FYI] Dan.Elksnitis
#jira UE-171694, FORT-540901, FORT-534644
#preflight 639273aa67018b14b5b53ddf
#rnx

[CL 23499949 by laura hermanns in ue5-main branch]
2022-12-13 18:21:32 -05:00
laura hermanns
daffbf30c9 Bump GUID in ShaderVersion.ush to invalidate shader DDC and fix SCW output file protocol on the non-error path. Appendix to CL 23427669.
#rb Aaron.Eady
[FYI] Brandon.Schaefer, Chihyu.Hsieh, Fredrik.Seehuusen, Jordan.Triomphe
#preflight none
#rnx

[CL 23429492 by laura hermanns in ue5-main branch]
2022-12-07 11:27:09 -05:00
laura hermanns
bcf95545bd Add host machine name to OOM report in DXC backend and refactor global SCW error-code handling. Bumped ShaderCompileWorkerOutputVersion to 9.
#rb Jason.Nadro, Dan.Elksnitis, Yuriy.Odonnell, Arciel.Rekman
#preflight 638e610c255f07df8ea212f4
#rnx

[CL 23428544 by laura hermanns in ue5-main branch]
2022-12-07 10:29:08 -05:00
kate ellis
839f9f1889 [Backout] - CL23401851
[FYI] Laura.Hermanns
Original CL Desc
-----------------------------------------------------------------
Add host machine name to OOM report in DXC backend and refactor global SCW error-code handling.

#rb Jason.Nadro, Dan.Elksnitis, Yuriy.Odonnell, Arciel.Rekman
#preflight 638e610c255f07df8ea212f4
#rnx

[CL 23421737 by kate ellis in ue5-main branch]
2022-12-06 19:47:06 -05:00
laura hermanns
c17b2976f9 Add host machine name to OOM report in DXC backend and refactor global SCW error-code handling.
#rb Jason.Nadro, Dan.Elksnitis, Yuriy.Odonnell, Arciel.Rekman
#preflight 638e610c255f07df8ea212f4
#rnx

[CL 23421619 by laura hermanns in ue5-main branch]
2022-12-06 19:45:07 -05:00
dan elksnitis
8801f83bb2 [shaders] add support to run STB preprocessor instead of MCPP; add a cvar to disable this and instead use the legacy preprocessor (currently this is set to true - i.e. still using MCPP - by default)
#rb Yuriy.ODonnell
#rb Jason.Nadro
#preflight 6387663c3377450900c5522b

[CL 23350844 by dan elksnitis in ue5-main branch]
2022-12-01 09:27:13 -05:00
dan elksnitis
b8959132c3 [shaders] add option to SCW to keep input files rather than deleting them; modify the cmdline args generated for the "single job" debug worker input files to include this option so the debug input file won't be deleted after executing to completion.
#rb Jeremy.Moore
#preflight 6386190af213a116eb214130

[CL 23306380 by dan elksnitis in ue5-main branch]
2022-11-29 10:00:37 -05:00
dan elksnitis
92c56f938f [ShaderCompileWorker] minor fixes:
- allow 0 TimeToLive to be specified explicitly on commandline (useful when debugging single-job debug worker input files, without this it keeps running the same job repeatedly) and modify single job worker input commandline txt file to use this instead of 0.5f
- change config-conditional behaviour serializing to memory instead of file on disk to be conditional on an optional cmdline argument instead, and move file copy into the block that does the file writing (otherwise it repeatedly tries and fails to copy the nonexistent file)

#preflight 6377fed033774509008c42c6
#rb Jason.Nadro

[CL 23234934 by dan elksnitis in ue5-main branch]
2022-11-22 09:42:02 -05:00
Matt Peters
22cb6cec34 LLM: Change ENABLE_LOW_LEVEL_MEM_TRACKER to not be separately definable, and make LLM_ENABLED_IN_CONFIG separately definable instead. This is necessary because ENABLE_LOW_LEVEL_MEM_TRACKER can only be true if LLM_ENABLED_IN_CONFIG is also true, because c-language files include the definition of LLM_ENABLED_IN_CONFIG but cannot include the platform-specific definition of ENABLE_LOW_LEVEL_MEM_TRACKER.
#rb Devin.Doucette
#rnx
#preflight 636abd377c2b505190b898a8

[CL 23038845 by Matt Peters in ue5-main branch]
2022-11-08 15:57:03 -05:00