improvements was "failing in the reference, not failing now", which is
also true of a test that produced no result because it hung. That is how
a hang in find got reported as "+ gnu/okdir_path_empty". List those
separately instead; they don't fail the job, since an upstream rename
looks the same.
The old comparison only commented when the PASS/FAIL/ERROR/SKIP *counts*
changed vs main, so it was blind to net-zero changes (one test fixed, one
broken). diff-gnu.sh also compared against an unextracted (empty) baseline,
flagging every current failure as a regression, and the published baseline
could be stale.
Adopt the sed/grep design:
- build-gnu.sh / build-bfs.sh now emit a per-test JSON summary
(findutils-gnu-full-result.json / bfs-full-result.json) listing every
test's name + PASS/FAIL/SKIP, via the new gnu_json_result.py /
bfs_json_result.py.
- compare_test_results.py diffs the set of failing test names against a
reference, writing a human-readable result file (empty when nothing
changed) and exiting non-zero only on new, non-intermittent failures.
- compat.yml fetches the reference *-full-result.json from the default
branch's latest run (dawidd6/action-download-artifact) and uploads a
comparison artifact per suite.
- comment.yml posts a PR comment whenever the comparison is non-empty,
i.e. on any per-test improvement or regression regardless of net count.
- ignore-intermittent.txt lists known-flaky tests, reported but not
failing the job.
Remove the superseded compare_gnu_result.py / compare_bfs_result.py /
diff-gnu.sh / diff-bfs.sh.
Now that -path/-ipath is implemented, no tests are skipped any more.
Fix CI by using zero instead of the empty string, which breaks the
compare_bfs_result.py script.
This includes much of the core xargs functionality, with the following
notable exceptions:
- Parallel execution (`-P`): This option currently just does
nothing, that way anything that passes -P can at least run without a
notable behavior shift (other than simply being slower).
- Replacement strings (`-I`): This can easily be worked around via an
intermediate shell invocation (e.g. `xargs -L1 sh -c 'do-things-with
$@' --`).
- EOF strings (`-E`): I've honestly never seen this actually used,
though it would not be particularly difficult to implement given the
current architecture.
Closes#37
Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
This adds the rest of the compatibility test workflow, in order to
automatically run against the GNU findutils & BFS test suites as part of
standard CI and compare the results to the latest from the 'main'
branch.
Closes#128
Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
This is the first half of the needed changes to set up automated
compatibility tests against GNU findutils and bfs, handling the uploads
of the build logs and JSON results. The workflow itself is heavily based
on the one from uutils/coreutils:
https://github.com/uutils/coreutils/blob/main/.github/workflows/GnuTests.yml
but with various cleanups & tweaks to better suit findutils.
This does *not* include the actual regression comparisons, because those
will only pass once archives of these files are up on the main branch.
Ref #128
Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
Sylvestre Ledru
oech3
Tavian Barnes
hanbings
Ryan Gonzalez