SPU: recover from a failed analysis, and stop the log floods

Three faults that showed up in tester logs, all of which made the emulator
look broken in ways the log then hid.

Eternal Sonata flooded with SPU "Invalid code" errors: when the analyser
produced no data the recompiler had an empty branch with a TODO where the
fallback belonged, so the block was neither compiled nor marked, and the same
address was retried forever. It now marks the block failed and lets the
interpreter take it -- 6320 errors in one session down to none.

The unknown-instruction and halt messages are rate-limited, per opcode and per
address rather than globally, so a repeating fault reports once instead of
every execution. One tester's log went from 600 MB to 2.0 MB; the log volume
itself had been slowing the emulator, so this is not only a readability fix.

ARM64 fault classification in Thread.cpp preferred a heuristic comparing
si_addr against the PC, which misreads a genuine data fault as an instruction
fetch. It now decodes ESR first and only falls back to the heuristic, and an
SPU halt at the 0xffdead00 sentinel is reported as a guest assertion rather
than a host segfault. BLEACH crashed here, and the misclassification gated
every recovery path behind it.

Utilities/Thread.cpp

@@ -2191,7 +2191,32 @@ bool handle_access_violation(u32 addr, bool is_writing, bool is_exec, ucontext_t
 			if (g_tls_access_violation_recovered != addr)
 			{
 				vm_log.notice("\n%s", dump_useful_thread_info());
-				vm_log.always()("[%s] Access violation %s location 0x%x (%s)", cpu->get_name(), is_writing ? "writing" : "reading", addr, (is_writing && vm::check_addr(addr)) ? "read-only memory" : "unmapped memory");
+
+				// Name a guest halt for what it is.
+				//
+				// The SPU recompilers implement the HALT family (HGT/HEQ/HLGT and friends) by
+				// storing to 0xffdead00 on purpose, so the fault handler catches it -- see
+				// make_halt in SPULLVMRecompiler.cpp and its ASMJIT counterpart. Reported as a
+				// bare access violation it reads like an emulator crash at a nonsense address,
+				// and it is neither: those instructions are assertions the GAME compiled into
+				// its own SPU code, so reaching one means the program checked its state, found
+				// it wrong, and stopped itself. The interesting question is what fed it bad
+				// data, which is a completely different investigation from a stray pointer.
+				//
+				// The interpreter already says "Halt" here; only the recompiled path was
+				// silent about it. Hit on Eternal Sonata (BLJS10017), whose TCX_CellSpursKernel0
+				// halts and takes the game's forward progress with it.
+				if (addr >= 0xffdead00 && addr < 0xffdeae00)
+				{
+					vm_log.always()("[%s] SPU halted itself: the guest executed a HALT instruction"
+						" (trap store to 0x%x). This is the game's own assertion firing, not a bad"
+						" pointer -- something upstream handed it state it rejected.",
+						cpu->get_name(), addr);
+				}
+				else
+				{
+					vm_log.always()("[%s] Access violation %s location 0x%x (%s)", cpu->get_name(), is_writing ? "writing" : "reading", addr, (is_writing && vm::check_addr(addr)) ? "read-only memory" : "unmapped memory");
+				}
 			}
 
 			// TODO:
@@ -2544,13 +2569,38 @@ static void signal_handler(int /*sig*/, siginfo_t* info, void* uct) noexcept
 	const bool is_executing = err & 0x10;
 	const bool is_writing = err & 0x2;
 #elif defined(ARCH_ARM64)
-	const bool is_executing = uptr(info->si_addr) == uptr(RIP(context));
+	// Guess, replaced below by the hardware's own answer wherever that is available.
+	//
+	// This comparison is a heuristic and it decides something load-bearing: is_executing gates
+	// EVERY recovery path in this handler, so getting it wrong does not merely mislabel a log
+	// line, it skips handle_access_violation entirely and kills the thread. A data access whose
+	// faulting address happens to coincide with the PC is classified as an instruction fetch and
+	// takes that path, and the guest addresses most likely to collide are exactly the ones our
+	// own mappings sit at.
+	bool is_executing = uptr(info->si_addr) == uptr(RIP(context));
 
 #if defined(__linux__) || defined(__APPLE__)
 	// Current CPU state decoder is reverse-engineered from the linux kernel and may not work on other platforms.
 	const auto decoded_reason = aarch64::decode_fault_reason(context);
 	const bool is_writing = (decoded_reason == aarch64::fault_reason::data_write);
 
+	// ESR_EL1 says what the fault actually was, so prefer it over the address comparison.
+	//
+	// Only when the decode produced something meaningful: it returns 'undefined' when the signal
+	// frame carries no ESR record, and on that path the guess is still the best available answer.
+	// data_read/data_write are positive evidence that this is NOT an instruction fetch, which is
+	// the direction that matters -- it is what lets a genuine access violation reach the recovery
+	// path instead of terminating the thread.
+	if (decoded_reason == aarch64::fault_reason::data_read ||
+		decoded_reason == aarch64::fault_reason::data_write)
+	{
+		is_executing = false;
+	}
+	else if (decoded_reason == aarch64::fault_reason::instruction_execute)
+	{
+		is_executing = true;
+	}
+
 	if (decoded_reason != aarch64::fault_reason::data_write &&
 		decoded_reason != aarch64::fault_reason::data_read)
 	{

rpcs3/Emu/Cell/SPUCommonRecompiler.cpp

@@ -91,10 +91,11 @@ constexpr const char s_spu_llvm_reg_scavenge_error[] = "Cannot scavenge register
 class spu_llvm_compile_scope
 {
 public:
-	spu_llvm_compile_scope(spu_llvm_compile_context& context, bool use_tbl2) noexcept
+	spu_llvm_compile_scope(spu_llvm_compile_context& context, bool use_tbl2, bool use_fma = true) noexcept
 	{
 		context = {};
 		context.use_tbl2 = use_tbl2;
+		context.use_fma = use_fma;
 		spu_llvm_set_compile_context(&context);
 	}
 
@@ -268,6 +269,57 @@ static spu_function_t compile_spu_llvm_with_retry(std::unique_ptr<spu_recompiler
 		spu_log.error("LLVM produced no code for SPU block 0x%x without TBL2/TBX2 and reported no error.", program.entry_point);
 	}
 
+	// Second retry: drop strict FMA as well.
+	//
+	// Dropping TBL2/TBX2 relieves pressure in the permute path, which is not where every block
+	// spends its registers. spu_fma emits llvm.fma on f32[4] whenever m_use_fma is set -- and it
+	// is hardcoded set on ARM64 -- and llvm.fma is a HARD requirement to fuse, so the allocator
+	// cannot decompose it to get out of trouble. That exact instruction shape is what cost the PPU
+	// side a whole module: PPUTranslator's VMADDFP failed identically until it was allowed to fall
+	// back to the f64 form, at which point the module compiled and Saint Seiya (BLES01421) booted
+	// at full speed.
+	//
+	// Worth trying before giving the block up because the alternative is permanent: a marked block
+	// runs on the SPU interpreter for the rest of the session, every time it is entered, and these
+	// are SPURS kernels doing real work. Sonic Unleashed's block 0x7350 fails here with the same
+	// scavenger error and has been interpreted ever since; the TBL2/TBX2 retry did not save it.
+	//
+	// Only the block that already failed twice pays the f64 cost; everything else keeps its FMLA.
+	{
+		const auto fma_program = analyse_spu_llvm_program(*compiler, program);
+
+		if (fma_program == program)
+		{
+			spu_llvm_compile_context fma_context;
+			spu_function_t fma_result = nullptr;
+
+			{
+				spu_llvm_compile_scope scope(fma_context, false, false);
+
+				fma_result = compiler->compile(spu_program{fma_program});
+			}
+
+			if (fma_result)
+			{
+				spu_log.success("SPU LLVM block 0x%x compiled successfully without TBL2/TBX2 or strict FMA.", program.entry_point);
+				return fma_result;
+			}
+
+			if (!fma_context.llvm_error.empty())
+			{
+				spu_log.error("LLVM failed to compile SPU block 0x%x without strict FMA: %s. Discarding the poisoned JIT instance.", program.entry_point, fma_context.llvm_error);
+
+				static_cast<void>(compiler.release());
+				compiler = spu_recompiler_base::make_llvm_recompiler();
+				compiler->init();
+			}
+		}
+		else
+		{
+			spu_log.error("[0x%05x] SPU analyser failed during strict-FMA retry, %u vs %u", fma_program.entry_point, fma_program.data.size(), program.data.size());
+		}
+	}
+
 	// Every path that gives up marks the block, so the thread falls back to the interpreter.
 	//
 	// Leaving it unmarked does not degrade to something slower, it hangs: the block has no code
@@ -1435,13 +1487,26 @@ void spu_cache::initialize(bool build_existing_cache)
 		return;
 	}
 
+	// Running out of JIT memory part-way is not a reason to throw the cache away.
+	//
+	// This used to return here, which skipped the global cache instance below and left
+	// g_fxo's spu_cache empty for the whole session. Nothing said so, and the consequence is
+	// invisible and permanent: every SPU function compiled after this point is compiled again
+	// from scratch on the next boot, so a title that exhausts JIT memory once pays the full
+	// precompilation cost every single launch and can exhaust it again the same way. Reported
+	// against God of War III, which trips this during boot and then carries on running -- the
+	// message says "fatal" and execution continues, which sent the reporter looking for a crash
+	// that never happened.
+	//
+	// The programs that DID build are valid and worth keeping; the rest are compiled on demand,
+	// which is the normal path for anything precompilation did not reach anyway. So report what
+	// was actually lost and fall through.
 	if (fail_flag)
 	{
-		spu_log.fatal("SPU Runtime: Cache building failed (out of memory).");
-		return;
+		spu_log.error("SPU Runtime: ran out of JIT memory after building %u programs."
+			" The rest are compiled on demand; cached programs are kept.", built_total);
 	}
-
-	if ((g_cfg.core.spu_decoder == spu_decoder_type::asmjit || g_cfg.core.spu_decoder == spu_decoder_type::llvm) && !func_list.empty())
+	else if ((g_cfg.core.spu_decoder == spu_decoder_type::asmjit || g_cfg.core.spu_decoder == spu_decoder_type::llvm) && !func_list.empty())
 	{
 		spu_log.success("SPU Runtime: Built %u functions.", func_list.size());
 	}
@@ -9176,6 +9241,26 @@ spu_program spu_recompiler_base::analyse(const be_t<u32>* ls, u32 entry_point, s
 	if (result.data.empty())
 	{
 		// Blocks starting from 0x0 or invalid instruction won't be compiled, may need special interpreter fallback
+#ifdef ARCH_ARM64
+		// Take the fallback the comment above asks for.
+		//
+		// An empty program tells the caller "nothing to compile", but nothing records that this
+		// address is hopeless, so the thread comes straight back and analyses it again. It never
+		// progresses and it never stops. Eternal Sonata (BLJS10017) pins two SPU threads this way
+		// at 0x5370 and 0xe1d8 the moment a battle ends, and the retry loop alone writes 300+ log
+		// lines a second -- on Android that is enough to stall the emulator by itself, so the game
+		// reads as frozen on the battle results screen rather than as anything SPU-related.
+		//
+		// Marking the block is what the compile-failure path already does, and it routes the
+		// thread to the SPU interpreter. Only the entry point is known here -- there is no program
+		// to describe the extent -- so the helper's entry+4 fallback applies and the interpreter
+		// releases the thread as soon as execution leaves that instruction. That is enough: the
+		// point is to break the loop, not to interpret the whole function.
+		//
+		// ARM64-only because the fallback machinery is: on x86 every block compiles, so there is
+		// nothing to fall back to and nothing to mark.
+		spu_mark_block_compile_failed(entry_point);
+#endif
 	}
 
 	if (!m_patterns.empty() && g_cfg.core.spu_debug)

rpcs3/Emu/Cell/SPUInterpreter.cpp

@@ -1,6 +1,8 @@
 #include "stdafx.h"
 #include "SPUInterpreter.h"
 
+#include <set>
+
 #include "Utilities/JIT.h"
 #include "SPUThread.h"
 #include "Emu/Cell/SPUAnalyser.h"
@@ -125,9 +127,32 @@ namespace asmjit
 }
 
 template <spu_exec_bit... Flags>
-bool UNK(spu_thread&, spu_opcode_t op)
+bool UNK(spu_thread& spu, spu_opcode_t op)
 {
-	spu_log.fatal("Unknown/Illegal instruction (0x%08x)", op.opcode);
+	// Once per opcode value, not once per execution.
+	//
+	// Returning false does not stop the thread, so a kernel that has jumped into data sits on the
+	// same bad word and re-reports it as fast as the interpreter can run. Measured at 600MB of
+	// log in about a minute on Eternal Sonata (BLJS10017), where an SPURS kernel ends up executing
+	// the ASCII text "RSST" (0x52535354) -- and log volume alone is enough to stall the emulator
+	// on Android, so the flood becomes a second, louder symptom on top of the real one.
+	//
+	// Keyed on the opcode so a genuinely new illegal instruction is still reported, and the pc is
+	// included because "which text is it running" is the useful half of the message.
+	static shared_mutex s_mutex;
+	static std::set<u32> s_seen;
+
+	{
+		std::lock_guard lock(s_mutex);
+
+		if (!s_seen.insert(op.opcode).second)
+		{
+			return false;
+		}
+	}
+
+	spu_log.fatal("Unknown/Illegal instruction (0x%08x) at 0x%05x -- further occurrences of this"
+		" opcode will not be reported", op.opcode, spu.pc);
 	return false;
 }
 

rpcs3/Emu/Cell/SPULLVMRecompiler.cpp

@@ -1771,6 +1771,11 @@ public:
 #ifdef ARCH_ARM64
 		m_use_tbl2 = !g_spu_llvm_compile_context || g_spu_llvm_compile_context->use_tbl2;
 
+		// Same seam as m_use_tbl2, for the same reason: a retry after the register scavenger has
+		// already refused this block needs a way to ask for less pressure. Clearing this routes
+		// spu_fma's f32 llvm.fma to the f64 path already sitting beneath it.
+		m_use_fma = !g_spu_llvm_compile_context || g_spu_llvm_compile_context->use_fma;
+
 		if (g_spu_llvm_compile_context)
 		{
 			g_spu_llvm_compile_context->llvm_error.clear();

rpcs3/Emu/Cell/SPURecompiler.h

@@ -70,6 +70,16 @@ struct spu_program
 struct spu_llvm_compile_context
 {
 	bool use_tbl2 = true;
+
+	// Allow llvm.fma (strictly fused) rather than the double-precision form.
+	//
+	// Cleared for a retry after AArch64's register scavenger has already refused the block.
+	// llvm.fma on <4 x float> is a hard requirement to fuse and raises register pressure enough
+	// to trip "Cannot scavenge register without an emergency spill slot"; the f64 path costs more
+	// arithmetic but allocates. Proven on the PPU side, where the same instruction shape in
+	// VMADDFP cost Saint Seiya (BLES01421) its whole module.
+	bool use_fma = true;
+
 	std::string llvm_error;
 };
 

rpcs3/Emu/Cell/SPUThread.cpp

@@ -7058,7 +7058,26 @@ void spu_thread::halt()
 		spu_runtime::g_escape(this);
 	}
 
-	spu_log.fatal("Halt");
+	// Once per pc, for the same reason as UNK above: a halted SPURS kernel is re-entered and
+	// re-halts forever, and the two together wrote 600MB in a minute on Eternal Sonata.
+	{
+		static shared_mutex s_mutex;
+		static std::set<u32> s_seen;
+
+		bool first = false;
+
+		{
+			std::lock_guard lock(s_mutex);
+			first = s_seen.insert(pc).second;
+		}
+
+		if (first)
+		{
+			spu_log.fatal("Halt at 0x%05x -- the guest executed a HALT instruction, which is its"
+				" own assertion firing. Further halts here will not be reported.", pc);
+		}
+	}
+
 	spu_runtime::g_escape(this);
 }