brmis: appendix — how FEX-Emu handles guest returns and indirects

Read FEX's call-ret shadow stack + per-site inline L1 lookup (BranchOps.cpp,
Dispatcher.cpp, ThreadManager.cpp; clone at /home/bmd/pcsx2/FEX, untracked).
Production MAMBO-X64 shape: adr+stp{guestRA,landing}+bl at calls, ldp+cbz+ret
at returns with EVERY exit path ending in ret to keep the hardware RAS
balanced (corrects the miss-path B in our dismissed Step-1 sketch), balanced
xzr pushes for unknown continuations, guard-page+SIGSEGV recentering instead
of bounds checks, DONTNEED wipe on invalidation, and no shared dispatcher
mega-Br at all -- per-site flat-L1 probe + per-site br/blr. Blueprint if
P2-2 reopens; gate arithmetic unchanged (call-sparse EE workload).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

tools/perf/brmis-2026-07.md

@@ -161,6 +161,67 @@ the dispatch rate is high, (2) per-site inline LUT+`Br` A/B (de-aliases the
 shared `Br`; the r22 delta measures the funnel by construction, and it IS
 the candidate fix).
 
+## Appendix: how FEX does it (source read 2026-07-10)
+
+FEX-Emu (x86→AArch64 usermode DBT, MIT; cloned at `/home/bmd/pcsx2/FEX`,
+shallow) ships a production MAMBO-X64-style **call-ret shadow stack**,
+unconditional, plus per-site inline lookup for all other indirects. Their
+workload (call-dense x86 userspace) is MAMBO's world, so for them the prize
+is real. Key files: `FEXCore/Source/Interface/Core/JIT/BranchOps.cpp`
+(~line 100 has the full pattern catalog as a comment),
+`Dispatcher/Dispatcher.cpp`, `LookupCache.h`,
+`Source/Tools/LinuxEmulation/LinuxSyscalls/ThreadManager.cpp:185` (alloc),
+`SyscallsSMCTracking.cpp:48` (fault recovery). ⚠️ Their CLAUDE.md forbids
+AI-generated contributions **to FEX** — read-as-reference only, never send
+them a PR from here.
+
+Design elements:
+
+1. **True shadow stack, not a direct-mapped cache.** Per-thread 4MB
+   (`CALLRET_STACK_SIZE`), dedicated pinned reg **x25 = REG_CALLRET_SP**,
+   spilled/reloaded to `State.callret_sp` at JIT-loop boundaries.
+2. **Call site:** `adr TMP1, <landing>; stp {guestRA, landing},
+   [x25, -0x10]!; bl <target>` — the `bl` pushes the hardware RAS. With
+   Multiblock on, the frontend translates ACROSS the call so the return
+   continuation is usually the fall-through block right after the `bl`
+   (zero-cost landing); when block layout doesn't cooperate the landing is a
+   direct `b <return-block>`. A call whose continuation isn't translated
+   pushes **{xzr, xzr}** — a never-matching frame that keeps the stack
+   depth aligned with guest call depth so grandparent returns still hit.
+3. **Return site:** `ldp {ra, landing}, [x25], 0x10; sub; cbz -> hit` —
+   on hit `ret landing` pairs with the caller's `bl` and the RAS predicts
+   it. On mismatch, fall into the inline L1 probe, and **every exit path
+   still ends in `ret`** (even the dispatcher fallback): "follow the normal
+   path (but ending in a ret)". That keeps the hardware RAS balanced —
+   a miss-path `b` would leave the paired RAS entry unpopped and desync
+   every subsequent return. (Our Step-1 sketch had `B DispatcherReg` on
+   miss — FEX's detail is correct, ours was wrong.)
+4. **Non-return indirects: NO shared mega-`Br`.** Every indirect site
+   emits its own inline L1 probe — flat direct-mapped `{HostCode, GuestRIP}`
+   16B-entry array (8k–1M entries, auto-resizing, base+pre-shifted mask
+   loaded from STATE in one `ldp`), ~6 insns, one dependent-load level —
+   ending in a per-site `br` (or `blr` for indirect calls, which also
+   pushes the RAS). The central dispatcher is only the L1-miss slow path.
+   This is exactly the P2-3 flat-LUT + per-site-Br combination.
+5. **No bounds checks in the hot path.** Guard pages both sides; the
+   SIGSEGV handler recenters the SP to Base+SIZE/4 on any fault in the
+   region (1MB overflow runway, 3MB underflow runway — underflow is the
+   common imbalance). Thread switches / longjmp-style guest behavior
+   degrade to compare-misses, never to wrong execution.
+6. **Invalidation = `madvise(DONTNEED)` the whole stack** on any SMC /
+   code-cache invalidation: pages lazily refill as zeros, and a zero frame
+   is a never-matching sentinel. (yaps2's Remove()-redirect-stub property
+   is stronger — our landings stay VALID across recClear without any wipe;
+   only recResetRaw would need one.)
+
+Verdict interplay: FEX is the blueprint to copy if this campaign ever
+reopens (it validates the Step-1 design and corrects the miss-path detail),
+but it doesn't change the gate arithmetic — they built it for call-dense
+x86; our measured guest-indirect pool stays ≤0.8–1.3% of EE-thread cycles.
+The transferable-today idea is #4: per-site probe + per-site `br` needs no
+shadow stack, no reserved register, and doubles as the decisive
+funnel-measurement A/B.
+
 ## Repro
 
 ```