This coalescing has a few benefits. First, it allows async LPI and mesh command creation jobs to handle level streaming events. Second, batches more work together which opens more doors to parallelization in follow-up CL's.
Finally, this change splits the mesh caching jobs for nanite and ray tracing off into their own tasks, which improves parallelism.
#preflight 63ea8a675d72661e36f1a9b9
[CL 24195188 by zach bethel in ue5-main branch]
Due to the fact that UnrealPak doesn't load INIs, the cvar was always its default value. This, combined with the fact that shaders don't store what codec they are compressed with means that setting this cvar to anything other than default (oodle) causes a crash in UnrealPak as DecompressShader will try to decompress with Oodle independent of the actual codec. Rather than plumb this information down through ProjectParams, I felt it prudent to just always use Oodle.
Additionally, in order to facilitate faster iteration times for development build farm jobs, the shader compression level used during UnrealPak will inherit from the overall package compression level set via Project Settings (PackageCompressionLevel_*).
This tacitly avoids patch issues as the default shader compression level happens to be the default package compression level, except now it inherits any faster levels for iteration time.
Note that we still use Mermaid as the codec as it's faster for decompressing than Kraken, which we care about for shaders.
#rb Fabian.Giesen
#rb Arciel.Rekman
#preflight 63e2f94c786751d1e0ffb290
[CL 24092953 by dan thompson in ue5-main branch]
[FYI] christopher.waters
Original CL Desc
-----------------------------------------------------------------
RHI header dependency cleanup
- Reducing other header dependencies from RHIDefinitions.h
- Moved a few functions to the new RHIStrings.h
- Uniform buffer layout initializer moved to its own file to reduce RHI dependency on MemoryLayout.
#preflight 63bca752577437afe604b097
[CL 23631957 by bob tellez in ue5-main branch]
- Reducing other header dependencies from RHIDefinitions.h
- Moved a few functions to the new RHIStrings.h
- Uniform buffer layout initializer moved to its own file to reduce RHI dependency on MemoryLayout.
#preflight 63bca752577437afe604b097
[CL 23631946 by christopher waters in ue5-main branch]
- required to support RTPSO with hitgroups with different payloads.
#preflight 63b4858f2540a78d27bef4de
#rb chris.kulla
#jira UE-173204
[CL 23569623 by tiago costa in ue5-main branch]
- Allows construction of StatePrecachePSOHash in FGraphicsPipelineStateInitializers that are not associated with MDCs.
- Also better reflects the actual PSO compatibility.
[REVIEW] [at]Graham.Wihlidal, [at]Kenzo.TerElst
#preflight
[CL 23300306 by arciel rekman in ue5-main branch]
- Avoids oversubscription deadlock when multiple background threads are creating shaders during the PSO precaching.
#rb CM Nordin, Yuriy O'Donnell
#jira UE-166109
#review
#preflight 6365bb2b4b0e01486a9753be
[CL 23010571 by Arciel Rekman in ue5-main branch]
Every shader now has the ability to declare which payload it uses. HitGroup, Miss and Callable shaders may only specify a single type, whereas Raygen shaders may provide more than one.
The global raytracing shader libraries have been categorized according to their payload type, which helps facilitate the creation of minimal ray-tracing pipelines.
Avoid adding callable shaders to the RTPSO if they are not being used.
All RHI shader compilers and shader loaders that support ray tracing have been modified to track an extra uint representing the payload type flagged in the source (at time of compilation).
Add the RayTracingPayloadType to the shadermap DDC key so that we can properly detect when a shader's payload type is invalidated. This is required because we serialize the payload type into the shaders from the C++ side.
#rb Yuriy.ODonnell
#jira none
#preflight 635715cae6096564af4dd28e
[CL 22742893 by chris kulla in ue5-main branch]
- modified serialization to instead add the warnings along with debug info to an "editor only data" struct, and store one of these per deduplicated bytecode entry (note the previous separate deduplication step for debug data was determined to be redundant)
- append additional context information to each warning via a DebugName argument passed to AddShaderCompilerOutput and store all warnings associated with a single bytecode (including those encountered for different permutations with potentially different source code resulting in the same bytecode) in sorted order
- add a simple ToString helper to FShaderCompileJobKey for easy construction of the DebugName argument for the above
- modify all engine usage of AddShaderCompilerOutput to construct this debug name from the job key
#rb Yuriy.ODonnell
#rb Jason.Nadro
#preflight 633ae2263629a1a4f913a041
#jira UE-150069
[CL 22301631 by dan elksnitis in ue5-main branch]
Add PSOPrecache file which wraps engine level PSO structs and functions
- FPSOPrecacheParams used to wrap certain parameters which drive the PSO collection (mostly set from components)
- IPSOCollector interface used as engine entry to the mesh pass processors in renderer project
- FPSOCollectorCreateManager to wrap all statically registered IPSOCollector CreateFuncions
- PSOCollectorStats structs and helper functions
Add PSOPrecache support to PipelineStateCache:
- functions to precache compute and graphics PSOs
- functions to check current precache state
- FPrecacheGraphicsPipelineCache which wraps all currently precached PSOs
IMaterial entry function to PrecachePSOs with given vertex factories and params (implemented by UMaterial & UMaterialInstance) which forwards call to FMaterial and finally the FMaterialShaderMap
- FMaterialShaderMap will iterate all registered IPSOCollector to collect all possible PSOs used and forward them to the PipelineStateCache for actual (async) precaching
RHI Changes:
- Add RHI specific functions to retrieve & compare the FRHIVertexDeclaration and FGraphicsPipelineStateInitializer hash for each RHI
- Add helper functions to match RHI*State data by retrieving initializer
Minimal vertex factory changes for PSO precaching as prep - actual changes in upcoming CLs
MeshPassProcessor base functions to collect PSOs which can be used by all mesh pass processor implemention (actual changes in upcoming CLs)
- Cache complete precache data hash of stored minimal PSO so it can be used for fast actual PSO initializer hash computation (full PSO initializer is build at runtime)
- Cache if minimal PSO was successfully precache and option to skip draw when PSO is still precaching (disabled by default and doesn't work yet via GPU scene based rendering)
- Add helper functions to setup the per mesh pass render target data used during PSO collection
Refactor SceneTextureConfig so it can be initialized from an init struct and move all function to setup the internal members from SceneTexture to SceneTextureConfig using the data from the init struct (so it's available in Engine project)
Fix GBufferInfo format for velocity target (was always floating point while it should be unorm when not android
Make certain helper functions definitions to h file so they can be used everywhere (SupportsNaniteRendering, FTextureRenderTarget2DResource::GetFormat & CreateFlags)
#preflight 63189a5a967ffc68fb9044a5
#jira UE-139584
#rb Mihnea.Balta
[CL 21907508 by kenzo terelst in ue5-main branch]
- GetRayTracingCallableLibrary() similar to how we handle hitgroups.
FindRayTracingCallableIndex(...) to query callable shader index in RTPSO.
#rb yuriy.odonnell
#preflight 628620c99016c6dd8974b510
[CL 20278150 by tiago costa in ue5-main branch]
Changed macro used to resolve overloads for these 2 functions to the one requiring one actual float/double type in the arguments. Removed old macro that allowed mixing signed/unsigned int types and replaced it with one that requires all the types to be signed to match any overloads (tested but currently unused).
Resolved some compile errors that resulted from newly ambiguous calls that mixed signed/unsigned integer types.
#jira UE-143473
#rb Andrew.Davidson
#preflight 6215382b30639b44d2f52355
#ROBOMERGE-AUTHOR: zak.middleton
#ROBOMERGE-SOURCE: CL 19098314 via CL 19099699 via CL 19100188 via CL 19100709 via CL 19109894
#ROBOMERGE-BOT: UE5 (Release-Engine-Staging -> Main) (v921-19075845)
[CL 19147214 by zak middleton in ue5-main branch]
- Provides about 8% runtime memory savings (in local tests).
- Also, adds more compression types for shaders.
- Impact on the shader compilation (in SCWs) seems to be negligible in local tests.
#rb Devin.Doucette, Charles.Bloom
[REVIEW] [at]Devin.Doucette, [at]Charles.Bloom, [at]Jason.Nadro
#jira UE-136845
#ROBOMERGE-AUTHOR: arciel.rekman
#ROBOMERGE-SOURCE: CL 18502862 via CL 18503105 via CL 18503112 via CL 18505939 via CL 18505950
#ROBOMERGE-BOT: STARSHIP (Release-Engine-Staging -> Release-Engine-Test) (v899-18417669)
[CL 18505961 by arciel rekman in ue5-release-engine-test branch]
- Splitting symbol generation and shader "debug info" generation into two different cvars. This should now allow us to generate symbols post-packaging without generating different shader variants. This should also enable us to always generate symbols and still have the final deduplicated shader sizes.
- r.Shaders.Symbols is now the primary way to enable shader symbols for debugging.
- r.Shaders.GenerateSymbols can be enabled to always generate symbols without writing them to disk.
- r.Shaders.WriteSymbols can be used to force write symbols if they were previously generated.
- r.Shaders.ExtraData can be used to generate shader names for engine runtime systems to use.
- NotifyShaderCooked replaced with NotifyShaderCompiled, prep for upcoming non-cook symbol writing.
- PC DXC shader compiles should only generate debug info when requested, not all the time.
- PC FXC shader compiles should generate debug info when requested, not just when r.Shaders.Optimize is 0.
#jira none
#rb arciel.rekman, lukas.hermanns
#preflight 611ad9035e737200015992db
#ROBOMERGE-SOURCE: CL 17186155 in //UE5/Main/...
#ROBOMERGE-BOT: STARSHIP (Main -> Release-Engine-Test) (v855-17104924)
[CL 17186162 by christopher waters in ue5-release-engine-test branch]
