fix: rewrite GNU testsuite harness to run upstream test scripts directly

The previous harness tried to extract sed commands from GNU test scripts
via regex pattern matching, which produced false negatives (comparing
against empty expected output) and false positives. This led to inflated
test counts and unreliable pass/fail signals.

The new approach:
- Provides a lightweight shim for the gnulib test framework (init.sh)
  with implementations of compare_, returns_, skip_, framework_failure_,
  and all require_* functions
- Executes each .sh test script from the GNU testsuite directly,
  injecting our Rust sed binary via PATH
- Uses a clean srcdir with symlinks to real test data files
- Adds per-test timeout (10s) to catch infinite loops, with SIGTERM
  isolation so timeout signals don't kill the parent script
- Properly propagates exit codes (0=pass, 77=skip, 99=framework failure)

Results are now consistent with CI tracking (~12% pass rate) with
clear PASS/FAIL/SKIP/timeout categorization.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Kevin Burke
2026-03-19 22:04:56 +01:00
committed by Sylvestre Ledru
co-authored by Claude Opus 4.6
parent 81d98b9049
commit 16f11456b3
3 changed files with 284 additions and 319 deletions
+2 -2
View File
@@ -75,8 +75,8 @@ jobs:
run: |
## Run GNU sed testsuite using our script
cd 'sed'
# Set GNU testsuite directory
export GNU_TESTSUITE_DIR="../gnu.sed/testsuite"
# Set GNU sed source directory (testsuite is at gnu.sed/testsuite/)
export GNU_SED_DIR="../gnu.sed"
# Run tests with JSON output
./util/run-gnu-testsuite.sh --json-output "${{ env.TEST_FULL_SUMMARY_FILE }}" || true
+8 -2
View File
@@ -47,7 +47,8 @@ The binary is named `sed` in `target/release/sed`.
### GNU sed Compatibility Testing
Test compatibility against GNU sed using the comprehensive testsuite (47+ tests, ~10% pass rate):
Test compatibility against GNU sed by running the upstream testsuite shell scripts
with a lightweight gnulib test-framework shim:
```bash
# Clone GNU sed testsuite (one time setup)
@@ -56,11 +57,16 @@ git clone https://github.com/mirror/sed.git ../gnu.sed
# Run compatibility tests
./util/run-gnu-testsuite.sh
# Verbose mode shows failure details
./util/run-gnu-testsuite.sh -v
# Generate JSON results for CI
./util/run-gnu-testsuite.sh --json-output results.json
```
The testsuite extracts test cases from the GNU sed repository and tests them against expected outputs.
The harness executes each `.sh` test from the GNU sed testsuite directly, injecting
our Rust sed binary via `PATH` and providing shim implementations of the gnulib test
framework functions (`compare_`, `returns_`, `skip_`, etc.).
### Unit Tests
+274 -315
View File
File diff suppressed because it is too large Load Diff