armsx/ARMSX3
0
0
Fork 0
mirror of https://github.com/ARMSX2/ARMSX3.git synced 2026-08-24 16:58:52 -07:00
Code Issues Packages Projects Releases Wiki Activity

RSX: vectorize the primitive-restart index upload on ARM64

Browse Source 
The primitive-restart variant of upload_untouched had no SIMD path on
ARM64: the asmjit builder is x86-only, and clang cannot auto-vectorize
the scalar loop (-Rpass-analysis: "value that could not be identified
as reduction is used outside the loop") because the min/max updates are
conditional on the restart compare -- while the non-restart loop next to
it does auto-vectorize. Net effect: 16 scalar instructions per index on
a path some titles saturate. Measured on a Snapdragon 8 Elite (Odin 3),
Virtua Tennis 4 routes its entire indexed-draw traffic through this
loop: 2.81 billion indices in a 9-minute match session, median 159k
indices per frame.

Port the x86 lane algebra to NEON, 8x u16 / 4x u32 per iteration: the
restart-equal mask ORs the lane to all-ones for the min accumulator and
the store (all-ones is index_limit, exactly what the scalar loop
writes) and BICs it to zero for the max accumulator, so restart lanes
can never win either reduction; UMINV/UMAXV reduce once at the end and
the tail stays scalar. Baseline v8.0 AdvSIMD only.

Supporting results, all on the Odin 3 with the system driver:

- Correctness: 216-case differential (scalar vs NEON vs the dispatched
  path; every tail residue mod 8 and mod 4; restart index absent,
  present, 0, index_limit, all-restart, and index_limit present while
  not the restart value; u16 and u32) ran on device at RSX init in all
  four A/B runs: 0 mismatches. An independent 65,000-case host-side
  model of the same lane algebra also matched the scalar loop, and a
  blind review of the diff could not construct a diverging input.

- Performance A/B (cntvct_el0 around the dispatch, null-region
  calibration subtracted, per-window medians over 120-flip windows with
  >10k restart indices/flip, runs interleaved scalar/NEON/NEON/scalar):
    scalar: 0.02504 and 0.02464 ticks/index  (1.30 ns/index)
    NEON:   0.00346 and 0.00381 ticks/index  (0.19 ns/index)
  ~6.8x faster per index; scalar-scalar repeatability 1.6%. Worst
  single-frame cost in this loop fell from 3.64 ms to 0.99 ms. FPS
  stayed 60/60 in all runs on this device; the win is RSX-thread
  occupancy and worst-frame cost, and would be frame time where the
  RSX thread is the bottleneck.

Titles that never enable primitive restart are unaffected: they route
through the untouched path, which clang already vectorizes.
This commit is contained in:
Zulux91
2026-08-12 05:53:44 -05:00
parent 94b12dc216
commit 2df75aa604
1 changed files with 100 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
rpcs3/Emu/RSX/Common/BufferUtils.cpp
+100
View File
@@ -253,6 +253,7 @@ DECLARE(copy_data_swap_u32) = copy_data_swap_u32_naive<false>;
DECLARE(copy_data_swap_u32_cmp) = copy_data_swap_u32_naive<true>;
#endif
namespace
{
template <typename T>
@@ -392,6 +393,102 @@ namespace
return (u64{max_index} << 32) | u64{min_index};
}
#if defined(ARCH_ARM64)
// Eight u16 (four u32) per iteration instead of one.
//
// The SIMD build of this loop is assembled by asmjit under ARCH_X64 only, so ARM64
// fell through to the scalar loop above -- and unlike the non-restart loop, clang
// cannot auto-vectorize this one ("value that could not be identified as reduction
// is used outside the loop"): the min/max updates are themselves conditional on the
// restart compare. Measured on a Snapdragon 8 Elite, Virtua Tennis 4 routes its
// entire indexed-draw traffic through here -- a median of ~159k indices per frame
// in a match -- at 16 scalar instructions per index.
//
// Same lane algebra as the x86 asmjit builder below: the restart-equal mask ORs
// the lane to all-ones for the min accumulator and the store (all-ones is
// index_limit, exactly what the scalar loop writes), and BICs it to zero for the
// max accumulator, so restart lanes can never win either reduction. Horizontal
// UMINV/UMAXV once at the end. Tail stays scalar.
//
// CONTRACT: src and dst must not overlap. The vector body reads a full lane
// group before writing it back, so a partial overlap diverges from the scalar
// loop's element-wise order. Both callers (VK/GL vertex upload) hand a guest
// memory span as src and a freshly mapped ring-buffer span as dst.
static inline u64 upload_untouched_neon(const be_t<u16>* src, u16* dst, u32 count, u16 restart_index)
{
u32 i = 0;
u16 min_index = index_limit<u16>();
u16 max_index = 0;
if (count >= 8)
{
const uint16x8_t vrestart = vdupq_n_u16(restart_index);
uint16x8_t vmin = vdupq_n_u16(0xffff);
uint16x8_t vmax = vdupq_n_u16(0);
for (; i + 8 <= count; i += 8)
{
const uint16x8_t v = vreinterpretq_u16_u8(vrev16q_u8(vreinterpretq_u8_u16(
vld1q_u16(reinterpret_cast<const u16*>(src) + i))));
const uint16x8_t eq = vceqq_u16(v, vrestart);
const uint16x8_t v_or_ones = vorrq_u16(v, eq);
vmin = vminq_u16(vmin, v_or_ones);
vmax = vmaxq_u16(vmax, vbicq_u16(v, eq));
vst1q_u16(dst + i, v_or_ones);
}
min_index = vminvq_u16(vmin);
max_index = vmaxvq_u16(vmax);
}
for (; i < count; ++i)
{
const u16 index = src[i].value();
dst[i] = index == restart_index ? index_limit<u16>() : min_max(min_index, max_index, index);
}
return (u64{max_index} << 32) | u64{min_index};
}
static inline u64 upload_untouched_neon(const be_t<u32>* src, u32* dst, u32 count, u32 restart_index)
{
u32 i = 0;
u32 min_index = index_limit<u32>();
u32 max_index = 0;
if (count >= 4)
{
const uint32x4_t vrestart = vdupq_n_u32(restart_index);
uint32x4_t vmin = vdupq_n_u32(0xffffffffu);
uint32x4_t vmax = vdupq_n_u32(0);
for (; i + 4 <= count; i += 4)
{
const uint32x4_t v = vreinterpretq_u32_u8(vrev32q_u8(vreinterpretq_u8_u32(
vld1q_u32(reinterpret_cast<const u32*>(src) + i))));
const uint32x4_t eq = vceqq_u32(v, vrestart);
const uint32x4_t v_or_ones = vorrq_u32(v, eq);
vmin = vminq_u32(vmin, v_or_ones);
vmax = vmaxq_u32(vmax, vbicq_u32(v, eq));
vst1q_u32(dst + i, v_or_ones);
}
min_index = vminvq_u32(vmin);
max_index = vmaxvq_u32(vmax);
}
for (; i < count; ++i)
{
const u32 index = src[i].value();
dst[i] = index == restart_index ? index_limit<u32>() : min_max(min_index, max_index, index);
}
return (u64{max_index} << 32) | u64{min_index};
}
#endif
#ifdef ARCH_X64
template <typename T>
static void build_upload_untouched(asmjit::simd_builder& c, native_args& args)
@@ -470,6 +567,8 @@ namespace
r = upload_xi16(src.data(), dst.data(), count, restart_index);
else
r = upload_xi32(src.data(), dst.data(), count, restart_index);
#elif defined(ARCH_ARM64)
r = upload_untouched_neon(src.data(), dst.data(), count, restart_index);
#else
r = upload_untouched_naive(src.data(), dst.data(), count, restart_index);
#endif
@@ -884,3 +983,4 @@ std::tuple<u32, u32, u32> write_index_array_data_to_buffer(std::span<std::byte>
fmt::throw_exception("Unreachable");
}
}