cpu_on_stop() is a teardown hook and nothing enforces that it runs a single time.
A PPU thread that re-enters the stop path without exiting reports again on every
pass: one thread ("SPU Interrupt Thread2") was producing "PPU thread perf stats
are not available." roughly every 10 microseconds, near 100,000 lines a second.
That is survivable on a desktop. On Android the log goes to external storage, so
it pins the log writer and drags down the shutdown it is describing -- the
emulator logged "Stopping emulator..." and never reached "All threads have been
stopped", leaving the next boot stuck on the last frame of the previous game.
Guard the reporting with a flag so it happens once per thread. The flag is
deliberately left out of serialization: it describes this run's reporting, not
guest state.
This does not address why the thread re-enters the stop path, which is a separate
question -- it stops that from being an emulator-wide stall while it is open.
Continuing with the modules that did compile is correct -- the dispatcher
entry for an uncompiled function interprets, nothing runs garbage -- but it is
per-instruction dispatch and it is slower than the interpreter outright. Saint
Seiya measured 6fps against 23 with most of its modules missing.
A game at that speed looks broken, and it gets reported as broken, when the
real answer is one restart away. An honest stop is better than a degraded run
that invites the wrong bug report.
So this ends the boot the way running out of memory always did. What is
different from before is the reason: the message names it and says what to do,
and it stays on screen because the overlay is drawn by the RSX thread rather
than the one this stops.
The fallback itself stays in place for the ordinary case of a single module
failing for some other reason, which is upstream's design and is worth keeping
-- one bad module costing its own functions is a fair trade. Running out of
memory is not that case: it takes most of the executable with it.
Arkham City never finished compiling, and neither did LEGO Batman 2. The log
said "LLVM crash recovery invoked" 240 times and then killed main_thread,
which looks like a codegen bug and is not one.
What actually happened: utils::memory_commit failed with ENOMEM inside the
disposable LLVM worker. That thread dying is how run_recoverable_llvm reports
any failure, so an out-of-memory device was indistinguishable from bad
codegen. It cost a long detour through max_map_count, disk space and
overcommit before the errno in the fatal gave it away, so the JIT's allocator
now uses a checked commit and throws a plain "Out of memory" instead.
The fatal part was the symbol resolvers. ppu_initialize ensure()d that every
group's __resolve_symbols was present, but that function lives in the compiled
output: when every module in a group fails, it is simply absent. The ensure
turned a partial compile into a dead main_thread, which discarded the 170
modules that HAD compiled and surfaced as a boot that never ends.
That contradicts the design either side of it -- a module that fails to load
is deliberately not fatal, because a guest function with no compiled code
keeps its dispatcher entry and is interpreted. A missing resolver is now the
same: report it, skip it, let that group interpret. Losing one group's speed
beats losing the boot.
Also tell the user. Out of memory is the only compile failure they can act on,
and the useful action is not obvious: compiled modules are already in the
cache, so starting the game again resumes rather than restarting the work. One
message after the workers join, not one per module -- once memory is short
every remaining module fails identically, and two hundred popups would be
worse than none.
Lowering Max LLVM Compile Threads also avoids it, and is deliberately not
suggested in the message: compile time is already the common complaint, and
halving the workers to dodge a case that is now survivable is a bad trade.
It did its job -- it is what identified res - rtime == 128 as the constant behind
the Assassin's Creed hang -- and what it costs now is an atomic increment on a
shared static every time any conditional store fails, on every PPU thread. That is
cross-core traffic on the hot path of every guest atomic, for a question that has
been answered.
The measurement it produced is kept in the comment above the fix, which is the part
worth having.
ldarx has a fast path: re-reserving the same 128-byte line the thread's last
successful stdcx wrote skips reloading ppu.rtime, on the theory that the hardware
caches the reservation across a chain of stores. The branch is empty, so rtime
keeps the value it had before that store -- and the store left the line's counter
128 higher. Every conditional store after the first one on a line therefore fails
its rtime != (res & -128) test by exactly 128, and keeps failing, because a guest
that retries re-enters the same fast path with the same stale value.
Assassin's Creed never gets past its loading screen because of it. libsre's
cellSpursAddUrgentCommand walks four urgent-command slots inside a single
reservation loop: it stores slot 0 back unchanged to release it, then can never
claim slot 1, and its failure path restarts the whole scan. Measured at 490
million failures on 0x102ed6b8, res-rtime == 128 on every one, data unchanged.
Downstream of that, everything else looked like the bug and was not. The job
chain is valid (workloadId 4), three of its four urgent slots stay empty, no
workload is ever marked ready, all six SPURS SPU kernels sleep correctly on a
control block nothing writes, and the main thread burns 2.2 million syscalls a
second in a yield loop waiting for a load that cannot finish. One missing
increment, five symptoms.
047f71b43 added this fast path for MGS4 along with a "rtime -= 128" here and a
"rtime += 128" in the store; the two cancel, and both were later dropped rather
than corrected, which left the fast path reading a stale value. Advancing rtime
at the store is what makes the empty branch correct: it is already current.
Saint Seiya: Sanctuary Battle (BLES01421) stalled partway through PPU
compilation and booted to a black screen. The failure was in LLVM, not here:
on AArch64 the register scavenger ran out of registers under the GHC calling
convention, which pins most of the GPRs to guest state, and
AArch64FrameLowering::determineCalleeSaves returns early for GHC before it can
create the emergency spill slot the scavenger falls back on. The scavenger then
aborts, and because that takes down the whole MODULE rather than one function,
every function in it drops to the interpreter -- the boot never finishes, or
the game runs at interpreter speed with nothing in the log to explain it.
Fix creates the spill slot for GHC frames that actually need stack. 231/231
modules compile for Saint Seiya, and Sonic Unleashed's FMVs work for the same
reason. Because it is a codegen fix rather than a per-game workaround, any
title that hit this benefits.
The change itself lives in the LLVM submodule, whose remote is upstream
llvm/llvm-project, so it cannot travel in this repository. It is preserved
here as 3rdparty/llvm/armsx3-aarch64-ghc-emergency-spill.patch, applied
against the pinned submodule commit; a build without it applied will exhibit
the original stall.
Also bumps the ARM64 codegen cache version so caches produced before the fix
are not reused, and carries the PPUTranslator changes the same work needed.
FCTIW, FCTIWZ, FCTID and FCTIDZ carried a saturation correction that only
makes sense on x86. cvtsd2si returns 0x80000000 for any value it cannot
represent, so the result is XORed back into 0x7fffffff on overflow.
FCVTNS and FCVTZS already saturate on their own, so the same XOR turned a
correct result into its opposite: every overflowing conversion produced
INT_MIN where it should have produced INT_MAX. The mask is a no-op when
there is no overflow, so it never did anything except break that case.
Armored Core: For Answer put the player under the floor in the tutorial
because a coordinate that should have clamped high arrived clamped low.
The cache needed a build identity as well. Its key is the executable's
SHA-1 plus a settings bitset and nothing more, so the first attempt at this
fix silently reused objects compiled by the previous build and looked like
it had done nothing. Every earlier PPU codegen change had the same problem
for anyone with a warm cache.
Module loading also no longer abandons the remaining modules after one
object fails to load.
PPU: a module that fails codegen no longer takes the boot with it. ppu_initialize2
called the fatal jit.add(); run_recoverable_llvm and the try_* pair already existed
in this tree but were used only by the SPU recompiler, so LLVM's fatal handler threw
on a thread with no recovery context and killed the worker. That is not one lost
module: g_progr_pdone is incremented in the compile loop's INCREMENT, so the module
the dead worker held was never accounted for, g_progr_ptotal could never reach zero,
and the boot waited on it forever. Saint Seiya: The Sanctuary (BLES01421, issue #25)
stopped at 133 of 134 on 'Cannot scavenge register without an emergency spill slot'.
Now routed through try_add on ARCH_ARM64, mirroring the SPU branch, with
ppu_initialize2 returning bool so the caller stops logging a dead module as compiled.
Losing the worker also halved the rate for everything left.
VK: a data_heap block no longer frees through an allocator that is not the current
one. Borrowed pointer, cached at construction with nothing tying it to the
allocator's lifetime; declining the free costs nothing the device teardown does not
already release.
Android: g_strings held 180 of localized_string_id's 323 entries and the callbacks
ignored their args entirely, so every string carrying a name, date, size or error
code lost it -- including CELL_SAVEDATA_LOAD, which is why the save prompt was Yes
and No over an empty message (open_msg_dialog logged msgString=""). All 322 the Qt
switch provides are present, in enum order, with QString::arg's %0 substitution
reproduced and utf8_to_u32string on the u32 path so trophy names survive. A
static_assert on the table size fails the build when upstream adds an id.
probeDiscInfo: set g_fxo up before mounting. vfs::mount lazily constructs vfs_manager
through manual_typemap::init<T>(), which writes *m_order++, and clear() nulls that
when a game stops -- so scanning a new disc image after playing anything wrote
through null. Emu.IsStopped() cannot guard it, because stopped is the cleared state.
The low-memory serialisation added three days ago holds a std::mutex across the
LLVM compile itself. A worker that hits LLVM's fatal handler leaves through
pthread_exit, and bionic unwinds nothing on that path, so the mutex stays locked
by a thread that no longer exists and every remaining worker waits on it for the
rest of the session. Memory only falls as modules accumulate, so the tight-memory
branch is likeliest late in a run -- which is why it reads as the PPU cache
getting stuck at the very end, and why dropping to the interpreter avoids it.
Reported as Saint Seiya: The Sanctuary never finishing its module cache. A claim
taken by compare-and-swap and waited on with a timeout costs a stranded claim a
wait rather than the session; the memory back-pressure either side of it is
unchanged. Third time this fork has been bitten by an unbounded wait around a
thread bionic can kill without unwinding.
Demon's Souls precompiles for 27 minutes and was killed partway through. The
trajectory says why serialising alone could not save it: the process sat between
4.3GB and 5.8GB for the whole run on a 7GB device, so this was sustained
footprint rather than a transient overlap the existing mutex could flatten. It
died when the system wanted memory back, and Zygote logged signal 9 for four
processes at once, so the pressure was not ours alone.
Adds a second tier. Below 2GB free, workers already compile one at a time; below
1GB free, the worker holding that lock now waits for the system to recover before
starting the next module, up to ten seconds, rechecking every 100ms.
The wait happens AFTER taking the serialisation lock deliberately. Waiting first
would have the other worker still allocating, so the wait would be watching
memory it is not allowed to influence.
Safe to wait here because precompilation writes each object to disk and links
none of them: the modules are not mapped into the VM, so is_being_used_in_emulation
is false and no JIT instance is created. Pausing costs time and nothing else.
Bounded rather than indefinite, because failing to compile is worse than
compiling under pressure, and it gives up cleanly if the emulator is stopping.
The worker count is decided once, from a reading taken before the emulator has
mapped the PS3 address space or the game has loaded anything. On a cold cache
that reading goes stale almost immediately, and by the time it matters the
count is fixed and cannot respond.
Measured on Arkham City: a first boot peaked at 5228MB where the same session
with the modules already cached sits at 2636MB, and sampling RssAnon against
RssFile and RssShmem put the growth in anon, so it is the compilers holding
LLVM contexts rather than the GPU caches. The process was killed partway
through; on a warm cache the identical settings run fine.
So the number of workers is not really the problem, their overlap is. Below
2GB free, a worker now takes a mutex around a single module's compilation,
which makes it one at a time exactly when that is the difference between
finishing and being killed. Checked per module, at the point of use, because
rechecking is the entire point. With headroom the check fails and nothing is
serialised, so there is no cost in the common case.
The lock is scoped to the compile alone, so a worker waiting on it is never
holding a context while it waits.
Android only.
Follow-up to the previous commit, which lowered modules-per-JIT to a flat 25
on Android. That fixed the kill but paid for it everywhere, including on
devices with memory to spare and on titles that were never at risk. The two
things the constant governs are not the same concern: branch reachability is
about translated game code, while the symbol resolver is a one-shot boot-time
initialiser that fills the jumptable and never runs again. Only the second is
a memory problem, so only the second should be allowed to force a split.
The group is now sized from MemAvailable, the same way the compile worker
count already is. A quarter of what is free, with the ceiling set at what a
full group of 100 costs, since budgeting past the point where nothing would
be split buys nothing.
~2GB free -> 26 per JIT, resolver peak ~500MB
~4.4GB -> 57 per JIT, resolver peak ~1.1GB (this device)
8GB+ -> 100 per JIT, i.e. upstream, untouched
Two separate reasons this stays free in the common case. A device with room
lands on 100 and is not split at all. And a title whose parts fit in one group
gets a single JIT instance at any limit at or above its part count, so every
title below the threshold is unaffected regardless: same instances, same
codegen. At roughly 4000 functions per part that covers everything under about
100k analysed functions, which is most of the library. Arkham City, at ~457k,
is not, and that is the point.
The 5KB per function and 4000 functions per part are measured off the run that
died: 2.3GB across ~457k functions, with parts holding 2800 to 4900 each.
Arkham City was killed partway through compiling its PPU modules. Memory sat
level around 4GB for four minutes of ordinary compile-and-free, then went
4010MB -> 6323MB in seventeen seconds and the log stops mid-compile, no
tombstone: the kernel OOM killer on a 7GB device.
The module part that carries jit_bounds also builds the symbol resolver, and
GetSymbolResolver spans every function across its entire JIT instance rather
than its own part: an LLVM Function declaration and a constant-array entry per
function, then a relocation each through MCJIT. Arkham City analyses to about
457k functions, and at 100 modules per instance the first resolver covered all
of them at once. The log records it plainly, 457209 functions generated in one
module where every other module in the run reports between 2800 and 4900.
No new mechanism was needed. ppu_initialize already splits modules across JIT
instances and jit_mod.symbol_resolvers is already a vector with one entry per
instance, executed in a loop. The group size was simply tuned for a desktop.
Upstream's own comment on the constant names this exact trade: lowering it
lowers continuous memory requirements, at the cost of more branches unable to
reach with a direct B. The resolver's cost is linear in the functions it spans,
so a quarter of the group size is a quarter of the peak.
Android only. Desktop keeps 100.
Skate 3 still aborted in llvm::report_bad_alloc_error during PPU compilation with
4.6GB reported available, where the previous figures allowed three workers.
Both numbers were too optimistic. A single large PPU module can take well over a
gigabyte through MCJIT and relocation processing, so 1GB per worker does not
cover a big title, and the reading is taken before the emulator maps the PS3
address space, so some of what it counts is already spoken for.
Reserves 2GB for the emulator and budgets 1.5GB per worker. On a 7GB phone with
4.6GB free that is a single worker: slower to compile, but it finishes. Three was
faster right up to the point it killed the process, and a game that aborts during
compilation cannot be played at all.
Minecraft exposed this class of bug after a cache clear and Skate 3 exposed that
the first fix was not enough; neither is new, the original cap sized against
installed rather than available memory.
Clearing the shader cache forces every PPU module to recompile at once, and the
process aborted partway through: scudo internal map failure, NO MEMORY, in
RuntimeDyldELF relocation processing on a PPU worker thread.
The existing cap allowed one worker per 1.5GB of total RAM, so four on this
device. Total RAM is the wrong number. The same device reported 7.3GB installed
while sitting at 76MB actually free, because it is also holding everything else
the user is running. Four LLVM workers on top of the emulator's own couple of
gigabytes had nowhere to go.
Adds utils::get_avail_memory, reading MemAvailable from /proc/meminfo, which is
the kernel's own estimate of what can be handed out without swapping. Workers are
then budgeted against that with a 1.5GB floor reserved for the emulator, falling
back to a more conservative slice of total where it cannot be read.
At 4.9GB available that allows three workers instead of four, and near zero it
correctly allows one.
Boot audio aborted the process. init_audio does ensure() on
Emu.GetCallbacks().make_video_source(), and the Android callbacks return nullptr
because there is no media backend, so ensure() killed the app outright. It fired
for any game whose folder holds a SND0.AT3, which is every folder format game:
for an .iso the fs::is_file check in rsx::thread::thread looks inside the mounted
virtual device and never finds one, so every .iso booted so far dodged it by
accident. A null source is now handled and logged. Boot music does not play,
which it could not have anyway. PKG installed games were exposed to this too,
since they also sit on the real filesystem.
PPU compilation ran out of memory. jit_core_allocator::limit() sizes the LLVM
compile workers on core count alone, which is right on a desktop and fatal on a
handheld: eight workers on a 7 GB device aborted inside
llvm::report_bad_alloc_error partway through a large title, with Max LLVM
Compile Threads left at 0 for "use every core". The limit is now bounded by
physical memory as well, roughly one worker per 1.5 GB and never below one. It
only caps the automatic default; an explicit setting is still honoured.
Fix accidentally clearing a binding during remap if the same button is assigned to rightclick in the pad navigation.
Only allow clearing a binding on release if the same button was also pressed while no remapping occured.
jpolo1224
Elad
Megamouse
yahfz