Commit Graph

259 Commits

Author SHA1 Message Date
dan elksnitis
cfa3219e31 [shaders] use ImageGetDigestStream for hashing DXC DLLs (to exclude signature/certificate data) instead of hashing the entire binary. This allows us to get the same version # for both signed and unsigned DLLs, and so fixes recompilation of shaders on initial startup of packaged release builds. Hashing was changed to xxhash which provides an iterative hashing API as required by this change (cityhash does not have such an API).
#jira UE-175638
#preflight 63e3e34dc11233cdcad2694c
#rb Yuriy.ODonnell
#rb Jason.Nadro
#lockdown Juan.Canada

[CL 24082354 by dan elksnitis in ue5-main branch]
2023-02-08 16:13:26 -05:00
Laura Hermanns
5a8a3b3063 Unify shader debug name in shader code output for DXC backends to improve cache hits.
The shader code output contains a little bit of plaintext such as I/O identifiers and the shader name, which was generated for the DXC backend in a way that prevents caching.
This unifies the shader name in the output code and improves the cache hit ratio for a Lyra cook from 0% to roughly 50% cache hits (for certain platforms).

#rb Jason.Nadro, Carl.Lloyd
#fyi Brian.White, Arciel.Rekman, Brandon.Schaefer
#jira FORT-548603
#preflight 63e26c6c07207b2570ba2d5b
#rnx

[CL 24050212 by Laura Hermanns in ue5-main branch]
2023-02-07 11:15:47 -05:00
dan elksnitis
34c034f2b2 [shaders] add a mutex acquired around all calls into hlslcc as a temporary fix for parallel in-process compilation. also some minor cleanup (include sorting, remove an unused/unimplemented function prototype)
#rb Jason.Nadro
#rb Yuriy.ODonnell
#rb Laura.Hermanns
#preflight 63e272b08374ef68dfd142ea

[CL 24049810 by dan elksnitis in ue5-main branch]
2023-02-07 11:04:07 -05:00
dan elksnitis
2834fb7c85 [shaders] strip out some 5.2 deprecations
#rb Jason.Nadro
#preflight 63d9227ecf52968117a45f9a

[CL 23926994 by dan elksnitis in ue5-main branch]
2023-01-31 10:14:30 -05:00
Laura Hermanns
325df88cbd Include SPIR-V header from SPIRV-Headers folder instead of SPIRV-Reflect.
#rb Jason.Nadro
#jira none
#rnx
#preflight 63c968fd0274c9f5fb6ed1e4

[CL 23789338 by Laura Hermanns in ue5-main branch]
2023-01-20 09:25:39 -05:00
jeannoe morissette
a4b446428f VulkanRHI: Override ShaderParameterParser for Vulkan bindless resource declarations. Use a typedef for each resource's type and declare its variable array with that type, at the end (in SPIRV) collapse all the arrays of a same descriptor type on the same binding. Move 'per-stage' uniform buffer bindings into descriptor sets AFTER the descrciptor sets for the resource arrays (since the resource array bindings will never change, but the ubo will change at a very high frequency). Also add the possibility for platform common headers to overload UB_RESOURCE_MEMBER_RESOURCE to match the altered declaration in ShaderParameterParser (temporary solution). The result gets around the lack of ResourceDescriptorHeap in Vulkan. (Vulkan Bindless WIP)
#rb Yuriy.ODonnell
#fyi Christopher.Waters
#jira UE-162018
#preflight 63c6e1933f587d9e14f6f27c
#rnx

[CL 23741527 by jeannoe morissette in ue5-main branch]
2023-01-17 13:23:40 -05:00
Yuriy ODonnell
fc92061821 Automatic bindless resource declaration conversion step will now explicitly expand AUTO_BINDLESS_SAMPLER/RESOURCE_INDEX/VARIABLE macros
This is required because subsequent shader compilation steps assume that the code has been fully preprocessed. In particular, the dead code elimination system does not handle macros such as ones in BindlessResources.ush.

#preflight 63bef4ba6e6e8d4662ffd485
#rb christopher.waters

[CL 23649118 by Yuriy ODonnell in ue5-main branch]
2023-01-11 13:03:47 -05:00
henrik karlsson
ce587eb612 Added markups for headers that can't be compiled standalone.. this is needed for both Header units and IWYU
#preflight 63789b0c170bc34a93a869e5
#rb none

[CL 23552160 by henrik karlsson in ue5-main branch]
2022-12-21 02:46:50 -05:00
Yuriy ODonnell
e5d4d76bed ShaderMinifier - Add basic namespace support
* Track namespaces for symbols
* Re-emit namespace scopes
* Conservatively include symbols from all namespaces in minified code (as if they were all global)
* Additional parsing work will be required to more accurately track used symbols based on active namespaces (i.e. handle `using` keyword, etc.)
* Added RemoveDeadCode() overload that takes an explicit list of required symbols instead of just an entry point name

#rb dan.elksnitis
#preflight 6398be5e35203bc7aa7f9b47

[CL 23494729 by Yuriy ODonnell in ue5-main branch]
2022-12-13 13:42:42 -05:00
laura hermanns
bcf95545bd Add host machine name to OOM report in DXC backend and refactor global SCW error-code handling. Bumped ShaderCompileWorkerOutputVersion to 9.
#rb Jason.Nadro, Dan.Elksnitis, Yuriy.Odonnell, Arciel.Rekman
#preflight 638e610c255f07df8ea212f4
#rnx

[CL 23428544 by laura hermanns in ue5-main branch]
2022-12-07 10:29:08 -05:00
kate ellis
839f9f1889 [Backout] - CL23401851
[FYI] Laura.Hermanns
Original CL Desc
-----------------------------------------------------------------
Add host machine name to OOM report in DXC backend and refactor global SCW error-code handling.

#rb Jason.Nadro, Dan.Elksnitis, Yuriy.Odonnell, Arciel.Rekman
#preflight 638e610c255f07df8ea212f4
#rnx

[CL 23421737 by kate ellis in ue5-main branch]
2022-12-06 19:47:06 -05:00
laura hermanns
c17b2976f9 Add host machine name to OOM report in DXC backend and refactor global SCW error-code handling.
#rb Jason.Nadro, Dan.Elksnitis, Yuriy.Odonnell, Arciel.Rekman
#preflight 638e610c255f07df8ea212f4
#rnx

[CL 23421619 by laura hermanns in ue5-main branch]
2022-12-06 19:45:07 -05:00
yuriy odonnell
a9591ee1e1 ShaderMinifier - Disable some of the debug features by default
#rb none (trivial)
#preflight skip

[CL 23385508 by yuriy odonnell in ue5-main branch]
2022-12-04 02:56:30 -05:00
Yuriy ODonnell
6911020bba ShaderMinifier - Fix handling of anonymous struct shader parameters
Declarations like `struct { uint Param; } Foo;` were previously incorrectly removed from code.

Also report explicit error when encountering 'namespace' keyword (which is currently unsupported)

#preflight 6387c5b1f213a116ebc18c5f
#rb dan.elksnitis

[CL 23343976 by Yuriy ODonnell in ue5-main branch]
2022-11-30 18:57:20 -05:00
Matt Peters
c9f018dced Use ZipArchiveReader to merge shader symbol zip files created during MPCook.
#rb Christopher.Waters
#rnx
#preflight 638643c47b4bd3f057981494

[CL 23313433 by Matt Peters in ue5-main branch]
2022-11-29 13:13:26 -05:00
axel riffard
105479f091 Fix Metal 3.0 shader compilation on iOS
#rb jack.porter
#jira UE-166909
#preflight 6385cd67d76995cb142c0695

[CL 23303343 by axel riffard in ue5-main branch]
2022-11-29 05:55:31 -05:00
Matt Peters
2b78f8441a ShaderSymbolExport: Rename temporary class to avoid name collision with the real class that has just been implemented.
#rb None, trivial
#rnx
#preflight None, Nottestedbypreflight

[CL 23300398 by Matt Peters in ue5-main branch]
2022-11-28 19:20:39 -05:00
matt peters
d2bb8963dd ShaderSymbolExport: Fix missing header for non-unity compile.
#rnx
#rb None, trivial
#preflight None, fixing build break

[CL 23263099 by matt peters in ue5-main branch]
2022-11-24 19:15:00 -05:00
christopher waters
85bd9bb007 Moving ShaderSymbolExport implementation into its own compilation unit.
#jira none
#rb matt.peters
#preflight 637d24918b12eb83a7ddea74

[CL 23241530 by christopher waters in ue5-main branch]
2022-11-22 16:14:14 -05:00
dan elksnitis
a3253bae46 [shaders] tiny change to allow skipping output of directcompile.txt file ignoring bool in compile input (used in some forthcoming debug code which dumps extra copies of preprocessed shader code and doesn't need these)
#preflight 6377fcab8b12eb83a7e6e3d1
#rb Jason.Nadro

[CL 23235560 by dan elksnitis in ue5-main branch]
2022-11-22 10:25:53 -05:00
matt peters
df72d2d4e9 ShaderCompilerCommon: Fix missing symbols in monolithic editor build due to spurious inline keyword in on function definition.
#rb None, trivial
#rnx
#preflight None, fixing build break

[CL 23212668 by matt peters in ue5-main branch]
2022-11-18 20:36:49 -05:00
Matt Peters
3fc09ce67b Multiprocess editor support: Add the parameter -MultiprocessId=<N> to allow instances of the engine to have a unique id in their group of multiprocess instances.
MPCook and Shaders: Change ShaderSymbolExport to export to different .zip files when in a -MultiprocessId environment.
Add part of the implementation for merging the different zip files created by CookWorkers together. The rest of the implementation requires reading from .zip files and will be coming in a future change.

#rb Christopher.Waters, Zousar.Shaker
#rnx
#preflight 6377fcedfa348e8480e25a2b

[CL 23209604 by Matt Peters in ue5-main branch]
2022-11-18 18:47:21 -05:00
Yuriy ODonnell
398f6e6d79 ShaderMinifier - Add an option to preserve comment lines
#rb dan.elksnitis
#preflight 63630eac522c8f7ab36344e0

[CL 22943808 by Yuriy ODonnell in ue5-main branch]
2022-11-03 02:13:18 -04:00
Yuriy ODonnell
f9824703c2 ShaderMinifier - Add basic test for MinifyShader, change SkipUntilNextLine to exclude new line character
#rb none (trivial)
#preflight 636068cde271e75dea59c739

[CL 22876523 by Yuriy ODonnell in ue5-main branch]
2022-10-31 21:34:29 -04:00
yuriy odonnell
9de862ab0b Add UE::ShaderCompilerCommon::RemoveDeadCode() high-level wrapper for shader minifier, to be consistently used in all shader compiler back-ends
#preflight 635af3aa1b7de936a5295629
#rb jason.nadro

[CL 22850483 by yuriy odonnell in ue5-main branch]
2022-10-29 01:51:27 -04:00