DIV.S, SQRT.S and RSQRT.S ran on host doubles on every recompiler rung,
which makes them correctly rounded; the EE's divide/square-root unit is
a digit recurrence with no rounding step and is not. The interpreter has
run the recurrence since the model landed, so mode 4 now calls the same
two functions out of line rather than reproducing them. eeDivide and
eeSqrtBits stop being static for it.
RSQRT.S composes the two with an ordinary single in between, as silicon
does, and its intermediate crosses the square root's call through the
island's scratch. Only the value moves: the negative-operand and
zero-divisor arms, and every flag they set, are the code they were.
Modes 1 to 3 keep the host instruction and the FPUDivFPCR swap.
The rsqrt sign table gains a mode-4 leg, which owes every row; the
filter for the twenty rows mode 3 misses by one ULP stays where it is,
now as that mode's own pin. The divide unit's console table gains a
mode-4 leg beside its interpreter one.
raiseOrClearOU() was handed a sum recomputed from the unmasked operands
while the destination got the masked one, so an operand the guard mask
had erased could still move O. The add/sub family now rounds and flags a
single value, eeGuardedSum()'s, and the accumulates hand their second
raise the sum eeMulAccumulate() actually formed instead of rebuilding it
from the ACC and the unrounded product.
The overflow test moves with it, into eeRoundsOutOfRange(), so the value
path and the flag path agree about the band where a result narrows back
onto kEeFpuMax.
madFlushedProduct() goes with the rebuild it existed for: the product
reaches the adder through eeMulRound(), which already returns a signed
zero for an underflowing one.
The mask keys on the exponent difference and clears low mantissa bits,
and a widened slot carries both: the exponent field moves from bits
23..30 to 52..62 and mantissa bit k to bit k+29, so FPU_ADD_SUB_D --
already the form the accumulates use -- is the same law with the shift
29 further left. That leaves nothing wanting the architectural single,
so both operands go straight through SlotToDouble, and the narrowing
pair and the widening's exponent-0xff test go with the single-domain
form. ADD.S and SUB.S drop from 127 host instructions to 93, ADDA.S and
SUBA.S from 132 to 98.
recFPUOp was the only caller reaching the arm that masks ft, which the
console corpus never sampled, so the new witness table covers it against
the interpreter and the console rows now also run through the DOUBLE
path.
A product whose tail is non-zero but smaller than the array's 2^15
borrow still loses an ULP, and ft alone does not say which of those it
is; only reconstructing the truncated low columns does. Mode 3 guards
for that band and calls eeMulOneUlpLow, which the interpreter already
reaches inline.
The call is plain AAPCS from inside a block, and the only one this file
emits, so it carries an island: the allocator's live caller-saved homes
are spilled around it and the EE pin mirrors go through their
flush/reload pair.
iFPUd widened every operand with ToDoubleFrom: eleven instructions
branching on exponent field 0xff, since a PS2 single with that exponent
is an ordinary large number IEEE reads as Inf or NaN. Against a
relocated slot the widening is one Fmul, and a compare goes from 34.75
host instructions to 8.75. CVT.W loses its NaN fix-up the same way,
leaving Fcvtzs.
Which format the file is in follows the clamp mode, so modes 0-2 keep
the architectural word in the low half. FPRreg's accessors read the
format and the emitters read the mode; the two engines meet at the
boundaries the word is still observable at -- MFC1, MTC1, MOV.S, the two
conversions, LWC1, SWC1 and the allocator's fill and spill.
The unscale factor takes q11 and d10's mask moves to the relocated
layout, so the allocator's callee-saved range is q12-q15.
The slot is 64 bits and the architectural register is 32, so a word view
of the union is only right while the two coincide. Word() and SetWord()
replace f/UL/SL and are the identity, so nothing that runs changes; what
it buys is that the compiler names every place that reads a slot as a
word, which is the set the next commit has to relocate.
The x86 tier is not built here and still reads the members directly.
The divisor's exponent field decided the whole instruction: the zero arm
returned before the sign test, so -0 and the negative denormals came
back carrying the division's cause alone. Hoisted, on the interpreter
and on the fast path. fpuFullMode already tested the sign first and is
untouched.
DenormalDivisorTreatedAsZero asserted the shape the other two shared.
It never looked at the dividend, which is what decides the cause.
Both single-precision tiers move together: the differential fuzzer in
ee_rec_fpu_rsqrt_tests.cpp diffs the interpreter against the fast path
on exactly this operand class, so neither passes alone.
The recurrence returns the truncated quotient or one above it, and on a
large share of operands the remainder alone already says which.
eeDivideSignificand() and eeSqrtSignificand() now recognize those with
one integer division or one integer square root and return, so they no
longer run 24 digit steps. The frame, the caps, the share this covers
and which way the implication runs are at eeDivideCap().
The caps are the ones that used to ship in eeDivideTruncates(), which
the digit recurrence replaced.
eeISqrt48() comes back for the square root's half, its body unchanged
from the version the recurrence retired.
Both loops branched on the digit twice per step to choose the addend.
The selector's two comparisons are already the masks those selects want,
so it returns them and the digit is never rebuilt as -1/0/+1 inside the
step; why that form is at EeSrtDigitMask.
Square root takes the same shape, with both candidate addends and both
candidate roots built off the root before the digit arrives. That also
drops its integer digit.
No result changes.
DIV.S, SQRT.S and RSQRT.S applied a partial truncation law and fell
through to a correctly rounded host divide, which left every operand the
law did not reach one ULP away from the console. The unit is not a
rounding rule at all: it is a radix-2 SRT digit recurrence, so
eeDivide() and eeSqrtBits() now run the digits. What that recurrence
is, and what it was measured against, is at eeSrtDigit().
The recurrence is not ours: it is PS2Float.cpp's Div() and Sqrt() from
GitHubProUser67's proposed PCSX2 soft-float series, carried and since
revised in the pcsx2-reliquary fork, whose only documentation is a DOI.
The three ops are integer now, so ScopedDivRoundMode and eeISqrt48 go
with them and FPUDivFPCR's rounding mode no longer reaches the
interpreter.
The divide-unit tests move with it. They were written around a model
with a residue: the two tripwires are enabled, the console tables assert
the console on every cell, and the engine differentials now pin the
shape of the one-ULP divergence that is left rather than the region the
old law settled.
The interpreter modelled the one-ULP deficit with a closed form over
ft's mantissa. That form only holds where the exact product is
representable: once there is a tail below the ULP the decision needs fs
as well. Those rows came back IEEE.
Replace it with the array - Booth recode, carry-save tree, truncated low
columns - and read the decision off the column the truncation lands on.
eeMulDefectiveFt goes away, so the interpreter now models a superset of
what the arm64 emitters do. The emitters keep their cut of the closed
form; the comment in iFPUd-arm64.cpp says why. The randomized
differential classifies rows by the tail below the ULP and licenses a
second divergence, and a new console-row test pins the class the closed
form could not reach.
A COP1 result strictly below 2^-126 and not zero is not always flushed:
the add/sub family keeps the mantissa bits where normalisation left them
and forces the exponent field to 0, while the multiplies and the divide
clear them. The 72 console rows, how they were sampled and how they rule
out flushing and the true denormal value, are in
ee_fpu_underflow_console_tests.cpp.
Only the mode-3 recompiler had it, so on every one of these inputs the
interpreter was the less accurate engine. eeRoundToSingle takes an
addsub flag and eeGuardedAddSub is what passes it, which covers all
eight add/sub-family ops. The underflow fold also moves ahead of the
(float) cast.
The single-precision fast path computes in single and cannot hold these
values at all, so it is pinned as a divergence.
EeFpuGuardedAddSubConsole's randomized test caught the interaction on
its own. Its model computes in host floats, which FZ has already
flushed, so it cannot express the new rule; underflowing pairs are
skipped there and counted.
The previous commit moved eeSqrtBits() to integer arithmetic, which
turned FPUDiv.Roundmode from something the host FPCR delivered for free
into something the function reads and acts on itself. Nothing in the
suite covered that.
Four operands, one per case the truncation law and the rounding mode can
land in, plus an assertion that two of the four rows move with the mode
at all.
The comment at eeSqrtBits() now covers the one mode where the law and
the mode disagree, toward-positive-infinity, and which of the two wins
there.
Eleven rounds of console captures went looking for the whole rounding
rule of the EE's divide/square-root unit and did not find it. One
fragment of it is settled: past a per-branch bound on how far the exact
result sits below the upper candidate, the unit truncates. eeDivide()
and eeSqrtBits() now apply that fragment and keep the correctly rounded
answer everywhere else. A fragment can ship because the implication runs
one way. The frame, the caps and the captures behind them are in the
block comment at eeDivideTruncates().
Both recompilers keep the host's correctly-rounded fdiv/fsqrt, so the
interpreter now leaves them behind on those rows. The three fuzz
differentials stop asserting that the engines agree and assert instead
the shape they may differ in.
Two rsqrt.s rows of the console capture regress, and are named in
ee_fpu_divunit_console_tests.cpp. Both were right by cancellation:
a root one ULP high, then a division that rounded up to the word silicon
reached by truncating a smaller divisor. Modelling the square root
removed one half of the pair; the other half is a division with u =
6,884,762 against a cap of 10,043,841, inside the region the law does
not settle. Both rows now have a sqrt.s column that matches silicon
- the console's sqrt.s 4938608B is 445941C1, and the tree produces
exactly that.
SQRT_S now calls eeSqrtBits(). It carried its own copy of the zero case
and of the exponent-255 prescale, which is the drift that helper existed
to prevent. The prescale is retired with it: |Ft|/4 and the doubled
result were only ever a way to keep an ordinary EE binade inside a host
single, and in integers exponent 255 is the k = 23 path.
ScopedDivRoundMode goes too, since the integer path reads FPUDivFPCR
directly.
ee_fpu_divunit_exhaustive_tests.cpp is new. It pins what the exhaustive
captures rule out - the unit is not a rounding rule, and the decision
is not a function of (branch, divisor, u, nu2(T+1)) - next to the
silicon witnesses for the part that shipped.
rsqrt(EEMAX, EEMAX) is not an RSQRT question. Two captures on the
SCPH-90000, 2231 operand pairs run through sqrt.s, rsqrt.s and div.s,
say two things about the unit underneath it.
RSQRT.S is sqrt-then-divide and nothing cleverer: rsqrt.s Fs, Ft came
back bit-identical to div.s Fs, S with S the sqrt.s Ft silicon had just
produced, on every row of both captures, with a plain 24-bit single in
between. That is what RSQRT_S already computes, and it is now pinned.
Neither step is correctly rounded. sqrt.s lands one ULP low on a large
minority of arbitrary operands and never high, div.s misses in both
directions, and rsqrt.s compounds the two into two ULPs. The error is
deterministic, but it is not a rounding mode and not a function of
either operand alone, so there is no cheap model to adopt.
ee_fpu_divunit_console_tests.cpp carries 87 console rows and the tallies
behind all of this. It asserts the sqrt-then-divide decomposition in the
engine, the 193 op-rows where the unit is exact, the 68 where it is not
and by how many ULPs, and that the fast path does not drift off the top
binade. DISABLED_InterpMatchesConsoleOnEveryRow is the tripwire a real
model of the unit would have to pass; run today it fails on exactly
those 68 assertions.
The RSQRT_S comment blamed the row on the two-step rounding above it; it
is corrected to what was measured.
The exponent-255 clamp came off the EE arithmetic in previous commits
and was left on the compares, on the grounds that clamped and unclamped
compares can only differ when both operands are in the top binade and
that no captured row is like that. That reason went into FPU.cpp and it
is wrong: one clamped operand is enough, because it only has to collide
with an operand that already is 0x7F7FFFFF.
C_cond_S reads eeToDouble() now, and that was fpuDouble()'s last caller
with anything to do. Its only other site was SQRT_S's else branch, where
the exponent is 1..254 and both of its rewrites are the identity, so
fpuDouble() and fpuOperandBits() are deleted and eeToDouble() is the
single operand entry point in the file.
The fast path keeps its clamp: it compares in host singles, where every
top-binade word is a NaN and an unclamped Fcmp goes unordered. What
making it exact would cost is at ee_fpu_compare_console_tests.cpp.
That file transcribes all 86 compare rows of the capture: every c.f.s,
c.eq.s, c.lt.s and c.le.s case, with the expected FCR31 put through the
same mask CFC1 applies, so each row is a flag test as well as a
condition test. It asserts the interpreter, fpuFullMode
(DOUBLE::recC_*_xmm, which does not clamp) and the fast path separately,
the last including the four rows it is expected to miss.
EeRecFpu.CEqSInfinityClampsToMax asserted CC set for c.eq.s(2^128,
FLT_MAX), the clamp's answer taken for the PS2's. It is rewritten as
CEqSTopBinadeSplitsTheInterpFromTheFastPath, which asserts both tiers
and says which one is the console's. Same inverted-premise shape as the
four MAX/MIN tests and MulSFpuMulHackOffGivesNativeProduct.
CEqSPositiveNaNBothClampToMax becomes CEqSPositiveNaNIsEqualToItself: it
passes because a bit pattern equals itself, not because of any clamp.
These two were the last arithmetic reading their operands through
fpuDouble() and saturating at posFmax, so the EE's top binade could not
reach them. eeDivide() normalizes both operands and reassembles the
exponent, and eeSqrtBits() gives RSQRT the exponent-255 prescale SQRT.S
already uses. Why neither can take the eeToDouble() route the rest of
the family took is at eeDivide.
Two results move with them. Saturation goes to the EE's own maximum
0x7FFFFFFF, a binade above the fast path's FLT_MAX, so the tiers now
differ by design and the tests that used to diff them run their legs
separately. And RSQRT's zero-divisor sign comes from Fs rather than Ft,
matching upstream x86 recRSQRThelper1 and the console; DIV's xor of the
two operands is untouched.
The interpreter column of
EeRecFpuFull.RsqrtAboveEeMaxSaturatesInsteadOfWrappingToNegativeZero
moves to 0x7FFFFFFF with it, and the zero-divisor and RSQRT tripwires
that named this commit as their condition graduate.
Exponent 255 is an ordinary exponent on this FPU: 0x7F800000 is 2^128,
0x7FFFFFFF is the largest number the machine has, and there is no Inf
and no NaN. That is one binade above IEEE single, so fpuDouble() folded
every such operand down to +/-0x7F7FFFFF on the way in and
clampToEeRange() folded a host infinity back to the same word on the way
out.
ADD.S, SUB.S, MUL.S, their A-forms and the four multiply-accumulates now
read their operands through eeToDouble() and round once through a new
eeRoundToSingle(). Why the double arithmetic in between is exact, and
how the top binade is rounded with no host single to round it in, is at
eeGuardedAddSub, MUL_S and eeRoundToSingle.
The multiplier's one-ULP deficit had to move across with it or it would
have been lost here: it sat on top of fpuDouble() and read the operands
as clamped, which changed ft's mantissa and so changed its own
predicate. eeMulRound applies it to what eeRoundToSingle produces
instead, and the six multiply forms reach it through eeMulProduct's
replacement, eeMulAccumulate.
One behavior changed beyond the range: an overflowing product now ends a
MADD/MSUB, value as well as flag, where the value path used to carry on
accumulating and land a binade away.
The fast path is untouched. It computes in host singles and cannot hold
these values at all; the warning against changing posFmax to suit it is
at the top of ee_fpu_overflow_console_conformance_tests.cpp.
DIV and RSQRT still read operands through fpuDouble and still saturate
through checkDivideByZero's posFmax. A quotient of two singles is not
exact in a double, so the eeToDouble route would double-round under the
divide unit's round-to-nearest; they need the normalise-and-reassemble
treatment the next commit gives them.
The EE adder carries no guard bits to the right of the mantissa, so
a cancelling add or sub comes back one ULP toward zero from the IEEE
answer. x86 FPU_ADD_SUB and both arm64 emitters have masked for that for
years; the interpreter did not, which made it the less accurate engine
on this class.
fpuAddSubGuarded() ports the recompilers' mask into ADD.S, SUB.S,
ADDA.S, SUBA.S, MADD.S, MSUB.S, MADDA.S and MSUBA.S, unconditionally: it
does not read CHECK_FPU_GUARDED, so with fpuGuardedAddSub=false the two
engines disagree here by design, and
EeRecFpuGuardBit.GuardOffDivergesFromInterpreterByDesign asserts both
sides so that is not closed by accident.
fpuDouble() now reads the shared fpuOperandBits(), and MADDA.S/MSUBA.S
name their product instead of accumulating in one expression; the
comment at MADD_S has the rest.
ee_fpu_guarded_addsub_console_tests.cpp is the new console pin for the
class: the capture rows the port moves, the eight family members,
aliasing, a randomised sweep against the x86 model on both engines, and
the FCR31 axis it must not disturb.
Not fixed here: fpuOperandBits() still clamps an exponent-255 operand to
+-0x7F7FFFFF before the adder sees it, so a row with such an operand
gets one step closer to the console without reaching it. That is the
operand-clamp family SQRT.S already left (1a09344ba6).
checkOverflow() asked whether the result had come back as a host
infinity and checkUnderflow() whether it was a host denormal. Neither
ever appears under the FP environment a game runs in, so O and U were
raised only under a rounding mode no game selects: 18 of the 57 rows in
the first-party capture read back with the flags clear where the console
raised them.
Decide from the magnitude of the exact result instead, computed in
double by a new eeToDouble(). The multiply-accumulates round twice and
so raise twice; the block comments at raiseOrClearOU() and
madAccumulandOverflowed() carry the silicon rows the rule was read off.
The value path keeps the old helpers' folding, now in clampToEeRange(),
so this is a flag change end to end.
The family table in ee_fpu_fcr_console_conformance_tests.cpp called all
ten of its "overflow" rows overflows. Eight are +/-FLT_MAX sums landing
exactly on 0x7FFFFFFF, which is representable and which silicon does not
flag: the four adds move to the clear class, and the four
multiply-accumulates get operands that overflow the product instead. The
raise class is six multiplies and two underflows, and the clear class
grows to one row per emitter that owes a clear.
Because the answer no longer depends on the rounding mode, the two
DISABLED tests that said opposite things about that table in the two FP
environments are gone; ExceptionFlagsMatchConsoleExceptTheFastPathRaise
replaces them and runs both. EnginesAgreeOnTheOverflowFlagClear,
NanMathOverflowIsNotAnOperandClampModeDifference and
OverflowFlagsComposeAcrossOneBlock graduate from DISABLED, and
InterpreterRaisesOverflowAndUnderflowLikeTheConsole and
FullModeMatchesConsoleOnEveryRow are new.
DISABLED_UnderflowFlagsNeedFzOff stays disabled: what is left in it is
the denormal-result value, which no capture can settle, since the
console's FPU has no denormal results to capture.
The fast path is not touched. It clears O and U and raises neither, so
the tests pin its FCR31 as the interpreter's minus exactly O|U|SO|SU
rather than as a row list. A raise there needs the double arithmetic
fpuFullMode already pays for; the reverted fast-path emitter took the
cheap route and derived O from a host-Inf predicate, so it was
a function of eeRoundMode and fired on rows that are not overflows, for
+8 host instructions on every arithmetic op.
Exponent 255 is an ordinary binade on the EE - no Inf or NaN, and the
representable max is 0x7FFFFFFF rather than FLT_MAX - so an exponent-255
operand never needed saturating. The interpreter clamped it to
+/-FLT_MAX inside fpuDouble anyway, and landed short of the console:
sqrt.s 7F800000 -> 5F7FFFFF, silicon 5F800000
sqrt.s 7FFFFFFF -> 5F7FFFFF, silicon 5FB504F3
sqrt.s 7FC00000 -> 5F7FFFFF, silicon 5F9CC471
It now computes sqrt(|Ft|/4)*2, which recSQRT_S_xmm already emits. Why the
factor is 4 is at SQRT_S in FPU.cpp.
The two conformance tests that pinned the clamp pin the console value
instead, over every exponent-255 shape rather than the three the capture
holds.
Both this branch's engines gated SQRT.S's I|SI on `exp != 0 && sign`.
The console gates on the sign bit alone, so -0 and the negative
denormals, which flush to -0 and produce an ordinary +0, raise
invalid-operation there too. Those were the only two operand classes
that lost the flag.
From the first-party capture that records FCR31 alongside the result:
sqrt 80000000 : console 00000000/01020041 both engines 00000000/01000001
sqrt 80000001 : console 00000000/01020041 both engines 00000000/01000001
The rule holds across all 38 SQRT.S rows. A positive qNaN raises
nothing.
The fix is a deletion on both sides: the Tst(0x7F800000)/B.eq pair in
the arm64 fast path, and the flag set hoisted out of FPU.cpp's
negative-normal arm. x86's recSQRT_S_xmm has always tested MOVMSKPS's
sign bit alone (iFPU.cpp:1767), which is why upstream-x86-jit is the one
column in the capture that answers both rows correctly; the arm64
FULL-mode DOUBLE::recSQRT_S_xmm tested the sign alone too and was
unaffected.
A previous commit fixed the value half of a case where the interpreter
returned _FtValUl_ & 0x80000000 and so answered -0 where the console
answers +0.
EeRecFpu.SqrtSInvalidFlagFollowsTheSignBitAlone is the ten-row sign x
exponent matrix from the capture, each engine scored on the full FCR31
word. The flag half of EeRecFpu.SqrtSOfNegativeZeroIsPositiveZero moves
there; it had asserted the opposite on nothing but the two engines
agreeing.
Two leads from the same sweep, recorded in the capture's handoff and not
addressed here. The arm64 JIT reports 01000001 on every RSQRT.S row in
the capture, including ordinary-negative operands, where interp, x86-jit
and hardware all say 01020041; recRSQRT_S_xmm does contain the I|SI set,
so that looks like a lost flag write rather than a missing one. And
hardware raises I|SI, not D|SD, on RSQRT's 0/0 rows.
The interpreter ran DIV.S, SQRT.S and RSQRT.S under the ambient FPUFPCR
rather than the divide unit's FPUDivFPCR, so it chopped where both
recompilers and the console round: one ULP low on every inexact result
in the default mode a game runs in. It now takes the same swap, gated
the way the emitters gate it; ScopedDivRoundMode in FPU.cpp has the
rest.
The witness this was first reported against, 1.0 rsqrt 1.5 landing on
0x3F5105EC where hardware gives 0x3F5105EB, has two independent causes.
749392120f fixed one. The value came back when the harness moved to the
production FP environment and was read as that fix half-landing, "the
divide is still in double". It is not: fpuDouble returns float and the
quotient was always single.
Four EeFpuOverflowConsole cases go DISABLED here: their SQRT.S rows were
written against the interpreter's truncated values. "Tests: EE FPU
overflow against hardware" and "Fix: SQRT.S raises invalid on -0 and
negative denormals" later in the series bring them back.
pstef