Count FIFO dispatches, not packets, and stop paying for two hot loads

Three separate fixes to the same hot path.

The profiler's FIFO figure counted the wrong thing. g_fifo_commands is
incremented once per run_FIFO entry, and one of those drains a whole packet, so
dividing by it priced a packet rather than a method. Sonic '06 averages about 17
methods per packet, so the reported 612 ns per command was really 612 ns per
packet and the per-method cost was closer to 36 ns. Cross-checks: 413.6 FIFO
refills a frame at 4096 bytes is 1.69 MB, about 423000 words, which 24248
packets can only consume at roughly 17 words each. Dispatches are now counted
where they are dispatched, both figures are reported, and the per-method
histogram divides by the right one. The line is labelled packets, and notes that
fifo_decode is a catch-all holding every handler body too, since no handler
carries its own scope.

rsx_state::decode was a cross-TU call per dispatched method. The body is one
exchange, but the definition lived in rsx_methods.cpp and LTO is disabled
project-wide, so it never inlined, and being opaque it also forced the caller to
reload its context pointer afterwards. Moved to the header.

set_transform_constant and set_transform_program read ctrl->put through a
seq_cst load. That is an ldar on ARM64, on a cache line shared with ctrl->get
which the guest PPU writes from another cluster, in the two hottest handlers in
this title. Relaxed: a stale value only shrinks the batch, and the remainder is
picked up on the next call.

rpcs3/Emu/RSX/RSXFIFO.cpp

@@ -912,7 +912,14 @@ namespace rsx
 
 			m_ctx->register_state->decode(reg, value);
 
-			if (rsx::prof::enabled()) [[unlikely]] rsx::prof::g_method_counts[reg & (rsx::prof::method_slot_count - 1)]++;
+			if (rsx::prof::enabled()) [[unlikely]]
+			{
+				// The sequential counter is what the per-method figure should divide by.
+				// g_fifo_commands is incremented once per run_FIFO entry, which is a whole
+				// packet, so it prices packets.
+				rsx::prof::g_fifo_dispatches++;
+				rsx::prof::g_method_counts[reg & (rsx::prof::method_slot_count - 1)]++;
+			}
 
 			if (auto method = methods[reg])
 			{

rpcs3/Emu/RSX/rsx_methods.cpp

@@ -1198,11 +1198,6 @@ namespace rsx
 		registers[NV406E_SEMAPHORE_OFFSET] = 0x10;
 	}
 
-	void rsx_state::decode(u32 reg, u32 value)
-	{
-		// Store new value and save previous
-		latch = std::exchange(registers[reg], value);
-	}
 
 	bool rsx_state::test(u32 reg, u32 value) const
 	{

rpcs3/Emu/RSX/rsx_methods.h

@@ -173,7 +173,15 @@ namespace rsx
 
 		~rsx_state() = default;
 
-		void decode(u32 reg, u32 value);
+		// Inlined deliberately. run_FIFO calls this once per dispatched method and the body
+		// is a single exchange, but with the definition in rsx_methods.cpp and LTO disabled
+		// project-wide it compiled to a real cross-TU call, which also forced the caller to
+		// reload its context pointer afterwards because the call is opaque.
+		void decode(u32 reg, u32 value)
+		{
+			// Store new value and save previous
+			latch = std::exchange(registers[reg], value);
+		}
 
 		bool test(u32 reg, u32 value) const;
 

rpcs3/Emu/RSX/rsx_profiler.cpp

@@ -19,6 +19,7 @@ namespace rsx::prof
 	const void* g_owner_thread = nullptr;
 	u64 g_fifo_refills = 0;
 	u64 g_fifo_commands = 0;
+	u64 g_fifo_dispatches = 0;
 	u32 g_method_counts[method_slot_count] = {};
 	u64 g_fifo_refill_bytes = 0;
 	u64 g_fifo_refill_stalls = 0;
@@ -255,9 +256,20 @@ namespace rsx::prof
 			// bucket with no owner left in it.
 			const u64 decode_ticks = g_acc.ticks[static_cast<usz>(bucket::fifo_decode)];
 
-			fmt::append(report, "\n\tFIFO commands   %.0f/frame, %.0f ns each in FIFO decode",
+			// Labelled packets, because that is what it counts. fifo_decode is also a
+			// catch-all holding every method handler body, since no handler carries its
+			// own scope, so neither figure is dispatch overhead alone.
+			fmt::append(report, "\n\tFIFO packets    %.0f/frame, %.0f ns each in FIFO decode",
 				static_cast<double>(g_fifo_commands) / frames,
 				static_cast<double>(decode_ticks) * to_ms * 1'000'000.0 / static_cast<double>(g_fifo_commands));
+
+			if (g_fifo_dispatches)
+			{
+				fmt::append(report, "\n\tFIFO dispatches %.0f/frame, %.1f/packet, %.0f ns each",
+					static_cast<double>(g_fifo_dispatches) / frames,
+					static_cast<double>(g_fifo_dispatches) / static_cast<double>(g_fifo_commands),
+					static_cast<double>(decode_ticks) * to_ms * 1'000'000.0 / static_cast<double>(g_fifo_dispatches));
+			}
 		}
 
 		{
@@ -284,7 +296,7 @@ namespace rsx::prof
 					fmt::append(list, "\n\t  %-46s %8.0f/frame  %4.1f%%",
 						name.empty() ? fmt::format("0x%05x", slot << 2) : std::string(name),
 						static_cast<double>(count) / frames,
-						static_cast<double>(count) * 100.0 / static_cast<double>(g_fifo_commands));
+						static_cast<double>(count) * 100.0 / static_cast<double>(g_fifo_dispatches ? g_fifo_dispatches : g_fifo_commands));
 				}
 				fmt::append(report, "\n\ttop methods%s", list);
 			}
@@ -293,6 +305,7 @@ namespace rsx::prof
 		prof_log.success("%s", report);
 
 		g_fifo_commands = 0;
+		g_fifo_dispatches = 0;
 		std::fill(std::begin(g_method_counts), std::end(g_method_counts), 0u);
 		g_fifo_refills = 0;
 		g_fifo_refill_bytes = 0;

rpcs3/Emu/RSX/rsx_profiler.h

@@ -189,6 +189,15 @@ namespace rsx::prof
 	 */
 	extern u64 g_fifo_commands;
 
+	/**
+	 * Methods actually dispatched, as opposed to [g_fifo_commands], which counts entries
+	 * into run_FIFO. One of those drains a WHOLE packet, so dividing by it prices a packet
+	 * and not a method. Sonic '06 averages roughly 17 methods per packet, so the two differ
+	 * by more than an order of magnitude and the per-packet figure was being read as a
+	 * per-command one.
+	 */
+	extern u64 g_fifo_dispatches;
+
 	/**
 	 * Commands seen per RSX method register, indexed by (id >> 2).
 	 *