- Only in places where it is trivially proven the call is only made on the render thread, due to an existing check(IsInRenderingThread()) assert somewhere in the function.
- FRHICommandListImmediate::Get() itself contains a check(IsInRenderingThread()), so this enforces correct threading, and removes the need for extra checks at the call sites.
- Remaining uses of FRHICommandListExecutor::GetImmediateCommandList() need investigation. Some may be bugs.
- Also some changes to make use of the passed-in RHICmdList where possible (e.g. render commands that are given the immediate command list, but call the global getter rather than using the argument they were given).
#rb zach.bethel
[CL 31699633 by luke thatcher in ue5-main branch]
Significant refactor of RHI command list management and submission, and RHI breadcrumbs / RenderGraph (RDG) scopes, to allow for parallel translation of most RHI command lists.
See individual changelists in //UE5/Dev-ParallelRendering for details. A summary of the changes is as follows:
This work's primary goal was to allow as many RHI command lists as possible to be parallel translated, to make more efficient use of many-core systems. To achieve this:
- The submission code paths for the immediate and parallel RHI command lists have been merged into a single function: FRHICommandListExecutor::Submit().
- A "dispatch thread" (which is simply a series of chained task graph tasks) is used to decide which command lists are batched together in a single parallel translate job.
- Individual command lists can disable parallel translate, which forces them to be executed on the RHI thread. This happens automatically if an RHI command list performs an operation that is not thread safe (e.g. buffer lock, or low-level resource transition).
One of the primary blockers for parallel translation was the RHI breadcrumb system, and the way RDG builds scopes. This was also refactored to remove these limitations:
- RDG could only push/pop events on the immediate command list, which resulted in parallel and immediate work being interleaved, breaking any opportunity for parallelism.
- Platform RHI implementations of breadcrumbs (e.g. in D3D12 RHI) was not correct across multiple RHI contexts. Push/pop operations aren't necessarily balanced within any one RHI context given that RDG builds "parallel pass sets" containing arbitrary ranges of renderer passes.
A summary of the new RHI breadcrumb system is as follows:
- A tree of breadcrumb nodes is built by the render thread and RDG. Each node contains the node name, and pointers to the parent and next nodes. When fully built, the nodes form a depth-first linked list which is used for traversing the tree for GPU crash debugging.
- The memory for breadcrumb nodes is provided by ref-counted allocator objects. These allocators are pipelined through the RHI, allowing the platform RHI implementation to extend their lifetime for GPU crash debugging purposes.
- RHIPushEvent / RHIPopEvent have been removed, replaced with RHIBeginBreadcrumbGPU / RHIEndBreadcrumbGPU. Platform RHIs implement these functions to perform GPU immediate writes using the unique ID of each node, for tracking GPU progress.
- Format string arguments are captured by-value to remove the cost of string formatting while building the breadcrumb tree. String formatting only occurs when the actual formatted string is required (e.g. during GPU crash breadcrumb stack traversal, or when calling platform GPU profiling APIs).
RenderGraph scopes have been simplified:
- The separate scope trees / arrays of ops have been combined. There is now a single tree of RDG scopes containing all types.
- Each RDG pass holds a pointer to the scope it was created under.
- BeginCPU / EndCPU is called on each RDG scope as the various RDG threads enter / exit them. This allows us to mark-up each worker thread with the relevant Unreal Insights scopes.
Other changes include:
- Fixes for bugs uncovered when parallel translate was enabled.
- Adjusted platform affinities necessary due to the new layout of thread tasks in the renderer.
- Refactored RHI draw call stats to better fit the new pipeline design.
#rb jeannoe.morissette, zach.bethel
#jira UE-139543
[CL 30973133 by Luke Thatcher in ue5-main branch]
The views made by DumpGPU must be created with lifetime extension disabled, so they release the underlying resources immediately after each dump pass.
#rnx
#jira UE-157708
#rb Guillaume.Abadie, Zach.Bethel
[CL 26769211 by mihnea balta in ue5-main branch]
- RHICreate{Vertex, Index, Structured}Buffer
- RHICreate{ShaderResource, UnorderedAccess}View
- RHIUpdateUniformBuffer
- Various initialization / locking methods for helper buffer types in RHIUtilities.h
The goal is to continue to force resource creation through command lists to avoid surprises with moving things off the render thread.
#rb christopher.waters
[CL 26183242 by zach bethel in ue5-main branch]
Fixed missing AtomicCompatible flags in NaniteCullRaster
Fixed incorrect image transition when using DumpGPU
#rb Luke.Thatcher
[CL 26125943 by carl lloyd in ue5-main branch]
r.DumpGPU.Stream=1 instead allocate and reuse staging resources from its own pool, only issuing copy to staging and gpu fence rhi commands.
Then every frame it polls whether the resource staging is lockable with the GPU fence on render thread.
When ready, the render thread kicks of a background task that cakes care of resource CPU post processing and disk writing.
Once disk write is complete, the render thread polls everyframe a FEvent to know when complete and unlock the staging resource.
The staging resource is then release to staging resource pool ready to be reused for dumping another resource of the current frame.
#rb none
#jira UE-179496
#preflight 64713ebcb310540a8d8e7da3
[CL 25657285 by Guillaume Abadie in ue5-main branch]
This notiably make the difference between a PreprocessedPixelFormat and PostprocessedPixelFormat in
FTextureSubresourceDumpDesc to handle the special logic for GL not able to write uav with less than 4byte
directly in TranslateSubresourceDumpDesc().
Motivation of this change is to have a very modular PostProcessTexture() easily reusable from asynchronous
stream GPU resource capture to disk.
#rb dmitriy.dyomin
#jira none
#preflight 645bc04d8f4d53ff22a289fa
[CL 25447955 by Guillaume Abadie in ue5-main branch]
Since FRHICommandListImmediate::ImmediateFlush() was calling FlushPendingDeletes() after executing the current list of commands, the lambda enqueued by that function was actually running as part of the next command list. Besides unnecessarily extending resource lifetimes, this also meant that the new command list started by flushing the deferred deletion queue, so anything added directly in there (instead of via another enqueued command) would be deleted before the commands were executed on the GPU. This was very easy to reproduce with -norhithread on DX12, because texture unlock operations add the staging buffer to the deferred deletion queue immediately, instead of enqueuing a command to do it, so the staging buffers were gone by the time the GPU tried to copy from them.
This changelist adds a boolean to RHISubmitCommandLists() which is true when we're flushing the immediate command list with the FlushRHIThreadFlushResources mode, so that the RHI can process the deletion queue internally after submission, instead of doing it in RHIPerFrameRHIFlushComplete(). I didn't want to move RHIPerFrameRHIFlushComplete() itself to another point in the timeline, because old RHIs (D3D11 and OpenGL) use that for other purposes, and it seems unwise to alter their behavior (e.g. D3D11 wants to resolve timing queries in there with a blocking wait, and running that at the end of the current command list would introduce a CPU/GPU sync point).
Also:
* deprecated FRHIResource::FlushPendingDeletes(), since all it does is call FlushPendingDeletes() on the command list being passed in, and code outside of the RHI really shouldn't be doing that.
* made FlushPendingDeleteRHIResources_RenderThread() flush the immediate command list instead of calling FlushPendingDeletes() directly
#jira UE-184426
#rnx
#preflight https://horde.devtools.epicgames.com/job/6455194d023fe5d3ad8faa64
#rb Luke.Thatcher
[CL 25423894 by mihnea balta in ue5-main branch]
