Exponent-0xFF operands through VMADDw and VMSUB at every dest mask, one
case per half of MADDw's set. MADDx/y/z runs the same shape with its
broadcast lanes in range as a scope control.
DIV.S, SQRT.S and RSQRT.S ran on host doubles on every recompiler rung,
which makes them correctly rounded; the EE's divide/square-root unit is
a digit recurrence with no rounding step and is not. The interpreter has
run the recurrence since the model landed, so mode 4 now calls the same
two functions out of line rather than reproducing them. eeDivide and
eeSqrtBits stop being static for it.
RSQRT.S composes the two with an ordinary single in between, as silicon
does, and its intermediate crosses the square root's call through the
island's scratch. Only the value moves: the negative-operand and
zero-divisor arms, and every flag they set, are the code they were.
Modes 1 to 3 keep the host instruction and the FPUDivFPCR swap.
The rsqrt sign table gains a mode-4 leg, which owes every row; the
filter for the twenty rows mode 3 misses by one ULP stays where it is,
now as that mode's own pin. The divide unit's console table gains a
mode-4 leg beside its interpreter one.
A standalone probe of the exponent band the FP matrix corpus does not
reach: a large operand at exponent 255 against an addend walked across
the guard mask's erasure boundary and the narrowing boundary just above
it, through all eight add/sub-family forms. Both engines and both rungs
of the DOUBLE path are scored separately, and the table is checked for
rows of each polarity so an engine that never raises O cannot pass it.
The mask keys on the exponent difference and clears low mantissa bits,
and a widened slot carries both: the exponent field moves from bits
23..30 to 52..62 and mantissa bit k to bit k+29, so FPU_ADD_SUB_D --
already the form the accumulates use -- is the same law with the shift
29 further left. That leaves nothing wanting the architectural single,
so both operands go straight through SlotToDouble, and the narrowing
pair and the widening's exponent-0xff test go with the single-domain
form. ADD.S and SUB.S drop from 127 host instructions to 93, ADDA.S and
SUBA.S from 132 to 98.
recFPUOp was the only caller reaching the arm that masks ft, which the
console corpus never sampled, so the new witness table covers it against
the interpreter and the console rows now also run through the DOUBLE
path.
ToPS2FPU_Full's saturating and flushing arms are for results that leave
the PS2's range, and a root does not, so both sat dead behind their
compares. The body goes from 54 host instructions to 23, under the 26
the single-precision path spends.
Both capture-backed tables read the operands the fpmul3 sweep produced,
which is exponent 127 with both operands positive. Every model of the
deficit here reads the significands and nothing else, and nothing said
so.
Sixteen rows from a probe that moves the other two fields: four sign
combinations across five exponent placements for eight separating
operand pairs, of which the extremes are kept. All 160 rows the probe
ran came back one ULP low and no operand pair changed its verdict.
The two rungs differ on two classes of operand and the 8137-case
hardware corpus reaches neither: its zero-tail multiplies all carry ft
mantissas the boundary term cannot read, and its rows inside the array's
band either saturate or come back exact. It scores mode 3 and mode 4 the
same, so nothing in the tree said which of them silicon agrees with.
The fpmul3 capture does. It swept eight fs significands against every
one of the 2^23 ft significands on an SCPH-90000, recording for each
whether the console returned the correctly-rounded product or that
product one ULP low, and those two are the only outcomes it ever saw.
Twenty rows where the boundary term decides and twelve inside the band
are transcribed here with both values, so each asserts mode 4 against
silicon and mode 3 against the rounding it is supposed to keep.
Half of each table has the two values equal, which is the polarity that
stops an emitter that decremented every product from passing.
It patched one product, 0.25 * pi, from the correctly-rounded 0x3f490fdb
to the 0x3f490fda the EE's multiplier returns, so Tales of Destiny stops
hanging. That product is one sample of the multiplier's one-ULP deficit,
and eeClampMode 3 models the deficit for every operand pair -- including
the asymmetry the gamefix reproduced by comparing fs and ft against
their own constants, which falls out of a predicate that reads ft alone.
The seven serials carry the clamp mode instead, and iFPU-arm64.cpp's
multiply is a bare Fmul again.
The toggle goes from the Qt, FullscreenUI, Android and iOS front ends,
along with the GameDB schema enum and the harness helper that set it.
Fix_FpuMultiply and the config bit stay: vu_capture's on-disk gamefix
mask is bit-indexed by GamefixId, so dropping ordinal 0 would
reinterpret every capture already recorded. tbl_GamefixNames keeps the
name, so a GameDB that still lists it parses and does nothing.
Both iFPUd modes emitted the multiplier deficit in full: the Booth term,
the boundary predicate over it, and an out-of-line call to the multiply
array for what neither decides. eeClampMode 3 now emits the Booth term
alone, three instructions off ft's mantissa; a new eeClampMode 4 keeps
the other two. Nothing else differs between them.
Mode 4 reaches the config through the GameDB and the INI and has no
picker entry, but the front ends still write the bit: ApplySanityCheck
rejects a config whose bits are not a whole mode, and a rejected config
falls back to the default rather than to the mode that was picked.
The harness's clamp-mode helpers set whole modes for the same reason.
A product whose tail is non-zero but smaller than the array's 2^15
borrow still loses an ULP, and ft alone does not say which of those it
is; only reconstructing the truncated low columns does. Mode 3 guards
for that band and calls eeMulOneUlpLow, which the interpreter already
reaches inline.
The call is plain AAPCS from inside a block, and the only one this file
emits, so it carries an island: the allocator's live caller-saved homes
are spilled around it and the EE pin mirrors go through their
flush/reload pair.
A slot does not say which format it holds, and the global that does
moves with eeClampMode, so a snapshot decoded under a mode other than
the one it was taken in returns a different number for every non-zero
word. Two harnesses alive at once is enough to reach it: an interpreter
leg read after a mode-3 leg has run.
The format now travels with the bytes, the FPRs are read through the
snapshot rather than through FPRreg::Word(), and a restore re-encodes
them for the file it is going into.
The predicate was a Cmtst against a mask parked in d10, which can
express the Booth term and nothing else, so iFPUd came back one ULP high
wherever the boundary term at the truncation column was the deciding one
-- a class the interpreter models. Building the predicate in GPRs
instead admits both terms, and turns the old shape's implicit tail test,
which worked only because FPUFPCR holds round-toward-zero, into an
explicit one.
Nothing reads d10 after that, so it leaves the reserved list and the
allocator has its callee-saved slot back.
iFPUd widened every operand with ToDoubleFrom: eleven instructions
branching on exponent field 0xff, since a PS2 single with that exponent
is an ordinary large number IEEE reads as Inf or NaN. Against a
relocated slot the widening is one Fmul, and a compare goes from 34.75
host instructions to 8.75. CVT.W loses its NaN fix-up the same way,
leaving Fcvtzs.
Which format the file is in follows the clamp mode, so modes 0-2 keep
the architectural word in the low half. FPRreg's accessors read the
format and the emitters read the mode; the two engines meet at the
boundaries the word is still observable at -- MFC1, MTC1, MOV.S, the two
conversions, LWC1, SWC1 and the allocator's fill and spill.
The unscale factor takes q11 and d10's mask moves to the relocated
layout, so the allocator's callee-saved range is q12-q15.
The header called the range's size the minimum because a 3-operand MMI
op needs three homes at once. The floor is three; the slots above it
keep an FPU multiply-accumulate's four FPR homes inside the
call-surviving range.
EeFuzz.CalleeSavedNeonBudget reads high-water marks of the per-op needed
set from a test-build probe in _clearNeededNEONregs.
The slot is 64 bits and the architectural register is 32, so a word view
of the union is only right while the two coincide. Word() and SetWord()
replace f/UL/SL and are the identity, so nothing that runs changes; what
it buys is that the compiler names every place that reads a slot as a
word, which is the set the next commit has to relocate.
The x86 tier is not built here and still reads the members directly.
EeFpuFormat.h defines the format and AsmHelpers.h emits it. Widening a
slot already held in it is one Fmul against a parked 2^896, where mode
3's ToDoubleFrom branches on exponent field 0xFF. Nothing computes in it
yet.
Arm64JitBuffer.h is arm64_emit_test.cpp's MAP_JIT buffer, unchanged,
moved out so the relocation's emit tests can call the code they emit
too.
The word stays in the slot's low half. Nothing but the stride moves.
fpuRegisters was itself the savestate's wire form; fpuRegistersWire is
now, and the freeze boundary converts.
* EE: the D-cache store-tag lookup dropped the top three bits of the tag
DXSTG takes a guest physical page from TagLo and has to turn it into the
host pointer our tags carry. It did that by routing the page through its
KSEG0 alias, which meant masking the tag to 29 bits first -- and KSEG0 is
only 512 MB wide, so the mask was not a formality. Every physical page at
or above 0x20000000 folded into the low half of the map and resolved to
whatever happened to live at the folded address.
The consequence that matters is that a page past the end of the physical
map folded onto real memory: 0x60129000 resolved to 0x00129000, and the
eviction wrote 64 bytes of cache line into guest RAM the tag never named.
Use vtlb_GetPhyPtr instead, which is what the debugger and PSM already use
to ask this question. It covers the whole 1 GB physical map and answers
null both for a handler page and for an address off the end of the map, so
the unbacked case is now decided by the same lookup that produces the
pointer rather than by a truncation.
Where a tag naming one of our main-RAM mirrors resolves changes as a side
effect of that, and is deliberately left unpinned. Those mirrors are our
physical map's, not a console's: an SCPH-30001 has no RAM at those physical
addresses, and an eviction steered at one reached nothing at all. There is
no hardware answer to hold us to, so nothing asserts one.
* Tests: point the DXSTG unresolvable-page check at a page that is unresolvable
The check named 0x1FFFF000, described as "BIOS/unmapped territory at the
top of the physical map". That page is the last one of the 4 MB BIOS ROM
mapped at 0x1FC00000, so it is real backing memory: the test took the
backed branch every time, wrote 64 bytes into the loaded BIOS image, and
asserted only that nothing faulted. The branch it was named for -- the one
carrying the safety property -- had no coverage at all.
Name 0x60129000 instead. It is past the end of the physical map, and it is
the page with teeth, because the old 29-bit fold sent it to 0x00129000 in
main RAM. A witness there turns "we did not fault" into "we did not write
somewhere the guest never named", which is the property worth holding.
An SCPH-30001 agrees with that much: an eviction steered above the end of
RAM puts nothing into RAM. Nothing beyond it is asserted -- where a tag
naming one of our main-RAM mirrors resolves is emulator-specific, so it
stays unpinned, with a comment saying so and why.
* Tests: stop the DXSTG write-back check skipping on 16K-page hosts
MapAt's candidate addresses are 4K-aligned and none is 16K-aligned, so on
a 16K-page kernel -- Asahi, Apple Silicon, some Android, and one of our own
CI jobs -- the kernel rejects every one of them and the mapping fails. The
write-back check treated that as a precondition and skipped outright, which
took its guest-side assertions with it: the ones that actually pin where a
DXSTG-steered eviction lands, none of which need anything from the host.
The mapping is only the negative control, there to show the write-back did
not ALSO reach the host page carrying the same number. Make it optional.
The guest-side half now runs everywhere and only the control drops out.
DxstgDirtyStaysInsideGuestMemory still skips, and should: it is entirely
about the host page. That leaves one skip here on a 16K-page host instead
of two, and none at all on a 4K one.
ipu_dither has had an SSE2 path and a scalar reference since forever, and
arm64 took the reference. The compiler closes half of that gap on its own —
with dithering off the loop is simple enough that clang vectorises it, and
measured here the scalar and NEON versions come out cycle-identical. With
dithering on it closes none of it: the clamp is written as std::max/std::min
around a table lookup, the destination is a 5/5/5/1 bitfield, and between
them the vectoriser gives up entirely. That arm ran at about 36 instructions
per pixel.
The NEON version is not a transliteration of the SSE2 one. x86 needs six
unpacks to split a row into channels because it has no deinterleaving load;
NEON has VLD4, so a whole 16-pixel row arrives already split one register per
channel and the shuffle chain simply does not exist. The dither tables are
the reference's coefficients with the sign folded into the choice of
operation, which lets saturating byte arithmetic supply the clamp for free —
the same trick the SSE2 path uses, and the reason both agree with the
reference bit for bit.
Measured on an M2 Max P-core, 2M macroblocks, two runs each:
dither on reference 18.76G instructions / 3.372G cycles
NEON 1.29G instructions / 0.293G cycles (11.5x)
dither off reference 1.08G instructions / 0.247G cycles
NEON 1.13G instructions / 0.247G cycles (even)
Function size drops from 476 to 208 bytes.
The tests are the point of the commit as much as the code is. Three
implementations of one function existed and nothing had ever compared them,
which is a bad shape here: a wrong result does not crash, it tints an FMV,
and nobody reports that. The transform depends on nothing but a pixel's four
bytes and its position modulo four in each axis, so the suite sweeps every
byte value through every one of the sixteen dither cells rather than
sampling. It holds whichever path the host selected to the reference, so it
gates the SSE2 arm on x86 exactly as it gates NEON here.
Proven to discriminate by mutation: transposing the r and b channels fails
three of four cases (correctly not the sweep that holds the channels equal),
perturbing one dither cell by one fails two, and dropping saturation fails
all four.
ipu_dither_reference loses its __ri so that a symbol survives into Release
for the tests to call.
The per-lane colour offsets were packed with the signed saturating pack while
both code generators used the unsigned one. The mask above the pack has already
put every lane in 0..65535, which makes the unsigned pack the identity and makes
the signed pack flatten everything from 32768 up to 32767. A descending gouraud
gradient is how a lane gets there: its offset is negative, the mask turns it
into a large positive, and the pack saturates it. Every pixel of the group then
carries that instead of its own colour, for the whole scanline.
The mask and the unsigned pack were introduced together to fix exactly this, in
"GS/SW: Mask color gradients to prevent incorrect clamping"; a later refactor
that rewrote the same lines to change how the shift table is loaded retyped the
tail back to the signed pack. The generators were not part of that refactor,
which is why only the C++ path regressed and why nothing noticed.
Where the path is reachable, measured rather than argued: with the rasteriser
JIT on, a probe at the top of the C++ setup never fires across corpus replays
that generate tens of kilobytes of scanline code apiece. It is entered only when
there is no code memory to compile into at all, and that same condition turns off
the EE, IOP and VU recompilers, so it is not a configuration anyone plays in.
What it is, is the path a measurement runs under -- the only way to ask what the
renderer computes without a JIT in the way, and so the arbiter of a
generated-code question. It was about to arbitrate one, and would have lied: the
gs-shade console capture re-run under it differed from the generated arm in
42,240 bytes, concentrated in exactly the gouraud colour it was to be asked
about. It is now byte-identical, and the generated arm is byte-identical to
before the change, so nothing a shipping build renders moves.
The new suite runs both paths over the same spans and compares the setup state
and the stored pixels, so the next divergence anywhere in the scanline fails
loudly instead of waiting for a capture to find it.
The recompiler already has a policy for arriving at address zero. A fetch
at PC=0 raises an Address Error and the BIOS handler takes over (AX-11),
because PS1 mode drives the IOP there through a register jump often
enough to be worth modelling rather than asserting on.
The immediate form of the same event never got there. Emitting a jump
whose target is zero asserted instead, so the two ways of reaching the
same address behaved differently: through a register it is emulated and
the guest carries on, through `j 0` it aborts a Devel build one
instruction before the handler would have seen it. Dropping the assert
routes the immediate form into the existing path — the tail stores pc,
links the block at zero, and the dispatcher hands it to psxRecompile,
which raises the Address Error.
Unlike the EE, nothing here compiles a jump the guest does not take: the
IOP scanner ends every block at the first branch, so an unresolved weak
symbol's guarded `jal 0` is never emitted. Reaching this needs the guest
to genuinely jump to zero — an unguarded weak call, a branch target that
computes to zero in low RAM, or a corrupted code word.
The test runs the JIT arm alone, which is what the new harness mode is
for: the interpreter has no PC=0 model at all, so the arms are meant to
disagree here and the differential harness has nothing to say.
A call to an unresolved weak symbol links as `jal 0`, guarded by a null
test on the symbol's address that always skips it. PS2SDK's libc glue
ships four such sites, so every homebrew ELF built against it carries the
shape, and the recompiler asserted the moment it met one.
It meets one because SL-03 continuation compiles the skipped path: the
guard branch becomes a continuation site, the scan runs on through the
dead call, and the emitter is handed a zero target for code that never
executes. The assert (inherited from the x86 recompiler, which aborts on
the same ELF) then takes down any Devel build before the program starts.
Nothing needs to happen at that target. If something did jump there,
address zero resolves like every other address — a block in RAM page 0,
or the unmapped-page handler — so the three tails just emit it.
The shape only reaches the emitter when the guard cannot be resolved at
compile time; a constant address folds the branch and the dead call is
never emitted, which is why an ELF carrying it can run clean until one
block boundary lands between the address materialization and the test.
The tests pin the reachable half.
Delete gl-arm-r44p1-attachment-self-read from the driver-bug database, so
r44p1 Mali takes GL_ARM_shader_framebuffer_fetch again on GLES and -- because
GSUtil::AndroidAutoPrefersVulkan asks the same table -- Auto resolves back to
OpenGL on those devices.
The rule was correct about the defect and wrong about the trade. Through
2.6.6.4 the gate it formalised was inert: the Mali profile block re-enabled
the ARM backend moments after the gate disabled it, so every r44p1 device
shipped on GL + fetch. 2.6.6.5 made the gate actually engage, and on GLES --
where fetch and the texture barrier are one capability -- every
self-referential draw became an RT copy plus a tile flush. Shadow of the
Colossus fell 30 -> 7 fps on the Anbernic RG 477V and users mass-downgraded
to 2.6.6.4. Offline replay of that scene under the device's feature shape
shows why no smaller fix could win the speed back: 890 render-target copies
and 938 render-pass breaks a frame against 1664 draws -- and a 2.6.6.4
replay under the same shape produces the same ledger (901/948/1664), so the
old build's speed WAS the in-tile read, not better GS decisions.
The known cost is unchanged from 2.6.6.4: r44p1's fetch corrupts some
content (MGS3 observed; most likely the driver grants the tile-read slot per
attachment format and silently degrades denied reads to memory fetches
inside a live feedback loop). Vulkan stays available as the
correct-rendering choice for those games, and its own r44p1 rule is
untouched -- there the in-tile read is a device loss, and the RT copy is an
ordinary image copy rather than a tile flush.
Unlike 2.6.6.4, the restored path is ordering-correct: db41082150 taught the
barrier-drop logic that ARM's fetch orders overlapping primitives by spec.
gs_vertex_tests 64/64, with the driver-profile pins flipped to assert the
restoration on GL and the copy path on Vulkan.
d8e2741234 stopped dropping the full barrier under framebuffer fetch when a
draw's primitives overlap, because fetch replaces the destination read without
necessarily ordering fragments. That is true of the EXT extension and it is
where the defect was measured -- Mesa 25.3.6 / Apple M2, 18% of an MGS3 frame
changing between identical replays. The conclusion was then written as
framebuffer_fetch_orders_overlap = false for the whole GL backend.
ARM_shader_framebuffer_fetch guarantees the opposite. Its spec: "when an
individual sample is covered by multiple primitives, rendering for that sample
is performed sequentially in the order in which the primitives were submitted",
and a read of gl_LastFragColorARM "must wait for the processing of all previous
fragments destined for the current pixel to complete". That is the same contract
Vulkan's rasterization-order attachment access and Metal's programmable blending
provide, and both of those keep the barrier-free path.
Every Mali device on Android takes the ARM path -- it is the only one that
works there, which is why the Mali profile selects it even when EXT is also
advertised. So the blanket answer put the entire Mali install base on a split
draw, one draw call per primitive group, for every overlapping blended draw.
That is a large part of what made 2.6.6.5 slower than 2.6.6.4 on Mali, which is
the population that reported it.
So make the question per-extension, where it belongs: ARM orders, EXT does not.
A driver that violates the ARM guarantee is a driver bug and goes in the
driver-bug database as a fetch blocklist entry, which is the mechanism r44p1
already uses -- not a blanket rule that also penalises every healthy Mali.
Vulkan, Metal and the EXT path are all unchanged. 21/21 policy tests.
Brian Degenhardt
pstef