Author SHA1 Message Date
Sylvestre LedruandGitHub d0263b2aa6 Merge branch 'main' into grep-quiet-exit-after-error 2026-06-05 08:28:59 +02:00
Sylvestre LedruandGitHub b4980df814 Merge pull request #16 from uutils/literal-fast-path
perf: buffer-at-a-time search for literal patterns
2026-06-05 08:28:04 +02:00
Sylvestre LedruandGitHub e9825e3503 Merge pull request #12 from uutils/grep-initial-tab-empty-line
grep: don't emit the -T alignment tab on empty lines
2026-06-05 08:27:43 +02:00
Sylvestre LedruandGitHub f5d5f6c063 Merge pull request #40 from koopatroopa787/issue-34-perl-single-pattern
fix: reject multiple patterns when -P/--perl-regexp is used
2026-06-05 08:25:18 +02:00
Sylvestre LedruandGitHub 337b7c704b Merge pull request #52 from wondr-wclabs/codex/empty-match-word-line
grep: select zero-width matches under -w and -x
2026-06-05 08:23:03 +02:00
Wondr ad7595ffe6 grep: select zero-width matches under -w and -x 2026-06-05 04:39:22 +01:00
Kanishk Sachan e3d80f59e2 fix: reject multiple patterns when -P/--perl-regexp is used
GNU grep's PCRE backend supports only a single pattern. Supplying
multiple patterns via repeated -e flags, or a pattern string that
contains a literal newline, must exit 2 with the message:

    the -P option only supports a single pattern

Add the validation immediately after patterns are collected, before
regex-mode selection. Add a test covering:
  - two separate -e flags with -P
  - a newline-embedded pattern string with -P
  - single -e with -P still works normally

Closes #34
2026-06-05 01:25:22 +01:00
Sylvestre Ledru 56d774f576 test: cover slow-path modes that literal tests no longer reach
The buffer-at-a-time fast path now serves the literal patterns that the
existing -l/-L/-q and binary tests used, leaving the line-at-a-time
engine's equivalents uncovered. Add bracket-class (non-literal) tests
for -l/-L/-q and binary handling (notice, -a text, without-match bail,
and the finalize-time notice), plus a fast-path test for a NUL that is
only discovered after a line was already printed.

No dead code was found: the remaining uncovered lines are writer I/O
error-propagation arms and pre-existing filesystem error handlers.
2026-06-04 22:24:27 +02:00
Sylvestre Ledru 28186e9ec3 perf: buffer-at-a-time search for literal patterns
Literal searches were ~50-70x slower than GNU grep because every line
paid per-line costs (terminator scan, NUL scan, dispatch) even when a
buffer held no match. Add a buffer-at-a-time driver that scans whole
chunks with a substring searcher and only locates line boundaries
around the matches it finds; a chunk with no match costs a single
vectorized sweep and no per-line work.

The driver activates only for plain ASCII literal patterns (case
sensitive, no metacharacters) in the simpler output modes: -c, -l, -L,
-q, and plain line printing with -n/-b/filename/-m. Anything needing
match positions, context, inversion, color, or special binary handling
falls back to the unchanged line-at-a-time path. Output stays
byte-identical to that path, including binary/invalid-UTF-8 behavior.

- line_buffer: read_chunk() yields the largest span of complete lines.
- matcher: expose per-pattern memmem searchers when every pattern is a
  plain literal (plain_literal()).
- searcher: eligible_for_fast_path(), fast_locate(), fast_print().

All scanning rides on the memchr crate (SIMD memchr/memrchr/memmem).
Unit tests for read_chunk and plain_literal; integration tests for
prefixes, -m, and multi-chunk line-number correctness.

Benchmarks (31 MB corpus) vs prior release:
  -F (no match):  232ms -> 15ms  (15.9x; now faster than GNU)
  -c literal:     229ms -> 15ms  (15.2x)
  plain print:    248ms -> 18ms  (13.5x)
Regex and -i paths are unchanged (still the line-at-a-time engine).
2026-06-04 22:24:27 +02:00
rifatxandGitHub f4798cb6d0 fix -l and -L to be mutually exclusive so that last one wins (#30) 2026-06-04 22:14:12 +02:00
Sylvestre LedruandGitHub 4e6823a8b3 Add installation section to README (#29) 2026-06-03 17:56:23 +02:00
Sylvestre LedruandGitHub b0700b1d78 Merge pull request #21 from oech3/pub
GnuTests: publish binary from main
2026-06-02 21:04:50 +02:00
oech3 bc416c6d8b GnuTests: publish binary from main 2026-06-03 00:05:24 +09:00
Sylvestre LedruandGitHub c614a57a05 Merge pull request #17 from uutils/e2e-bench-only
E2e bench only
2026-05-31 11:17:38 +02:00
Sylvestre Ledru 6e6db248f1 bench: add e2e benchmarks for the grep tldr invocations
Cover the real-world grep usage shapes from the tldr page end-to-end
through uumain over a shared multi-MB corpus (plus a directory with a
binary file for -rI):

  search pattern, -F fixed string, -rI recursive ignoring binary,
  -C 3 context, -Hn --color=always, -o only-matching, -v invert,
  -Ei extended + ignore-case.

Kept alongside the pure-scan throughput benches (literal vs regex, no
match). A rare marker keeps matched output small so the full-file scan
dominates the timing.
2026-05-31 11:10:11 +02:00
Sylvestre Ledru b5816820ed bench: end-to-end search throughput only
Replace the matcher micro-benchmarks with a single end-to-end 'search'
group driven through uumain over a multi-megabyte file: a literal
pattern (which a buffer-at-a-time searcher can accelerate) and an
extended-regex control (which cannot). Matching pre-split lines in
isolation cannot reveal how the searcher feeds data to the matcher;
this does.
2026-05-31 11:05:28 +02:00
Sylvestre LedruandGitHub ede1676d1a Merge pull request #15 from uutils/bench-literal-throughput
bench: end-to-end search throughput via uumain
2026-05-31 10:55:17 +02:00
Sylvestre Ledru b46a86d48a bench: end-to-end search throughput via uumain
The existing match/throughput benches call Matcher::match_line on
pre-split lines, so they only measure matching in isolation and cannot
observe how the searcher feeds data to the matcher. Add a 'search'
group that drives the whole pipeline through uumain over a multi-MB
file: a literal pattern (which a buffer-at-a-time searcher can speed
up) and an extended-regex control (which it cannot). Uses -q with a
non-matching pattern for a silent full-file scan.
2026-05-31 10:41:52 +02:00
Sylvestre LedruandGitHub b0164440e3 Merge pull request #14 from uutils/fix
Add Default impl for GlobSet to satisfy clippy
2026-05-31 10:34:25 +02:00
Sylvestre Ledru 96762f26ca Add Default impl for GlobSet to satisfy clippy 2026-05-31 10:18:32 +02:00
Sylvestre Ledru c8dfef6563 Add pre-commit configuration 2026-05-31 10:11:57 +02:00
Sylvestre LedruandGitHub ddac723054 Merge pull request #13 from uutils/codspeed-wizard-1780213675759
Add CodSpeed performance benchmarks
2026-05-31 10:11:23 +02:00
codspeed-hq[bot]andGitHub 079619ee44 Add CodSpeed performance benchmarks 2026-05-31 07:59:22 +00:00
Sylvestre LedruandGitHub 2a6a3aba7e Add note about performance improvements needed 2026-05-30 19:25:27 +02:00
Sylvestre Ledru a7b15320af grep: don't emit the -T alignment tab on empty lines
With -T, grep pads the prefix with a tab so line content lands on a tab
stop. GNU omits that tab when the line has no content: an empty line
prints just its prefix (a whitespace-only line still gets the tab).
uu_grep always wrote the tab, so empty matched lines gained a spurious
trailing tab. Gate the tab on non-empty content. Fixes the GNU testsuite
'initial-tab' test.
2026-05-30 19:13:52 +02:00
Sylvestre Ledru 399d2d1192 grep: -q match yields exit 0 even after a file error
GNU grep documents that with -q the exit status is 0 as soon as a line
is selected, even if an error (such as a missing file) occurred. uu_grep
let had_error take precedence, so 'grep -q PAT missing -' returned 2
instead of 0 when stdin matched. Give quiet+match priority in finish();
without -q, or with -q and no match, a file error still yields exit 2.
Fixes the GNU testsuite 'status' test.
2026-05-30 18:55:04 +02:00
Sylvestre LedruandGitHub 98e6bb6f53 Merge pull request #10 from uutils/improv-cov
Improve the code coverage
2026-05-30 14:25:50 +02:00
Sylvestre Ledru da0ada8a37 test: use platform-specific expected message for nonexistent-file error 2026-05-30 11:33:47 +02:00
Sylvestre Ledru fcf46d8f56 test: cover -D skip on an explicit special-file argument 2026-05-30 11:28:04 +02:00
Sylvestre Ledru 55cb643545 test: cover strip_dot_prefix for implicit-cwd recursive search 2026-05-30 11:28:04 +02:00
Sylvestre Ledru 89aec4a45a test: cover -T line-number width on the file (not stdin) path 2026-05-30 11:28:03 +02:00
Sylvestre Ledru 6c5b7dc9e3 test: cover -L listing when the pattern matches one file 2026-05-30 11:28:03 +02:00
Sylvestre LedruandGitHub ff918e4a63 grep: strip trailing (os error N) from file error messages to match GNU (#6) 2026-05-29 22:24:20 +02:00
Sylvestre LedruandGitHub c50c0458cb fix: accept repeated options like GNU grep (#2)
GNU grep tolerates options given more than once (boolean flags are
idempotent, value options take the last occurrence), but clap rejected
them with "cannot be used multiple times" (exit 2). Set
args_override_self(true) so clap replaces rather than errors; args with
ArgAction::Append (-e/-f/--include/--exclude) still accumulate.

Found by the differential fuzzer (e.g. `grep -e .* -E -n -o -n -x`).

Add a regression test covering repeated booleans, repeated value options
(last wins), and that -e still accumulates patterns.
2026-05-29 22:21:27 +02:00
Sylvestre LedruandGitHub f7813500c9 Merge pull request #8 from uutils/gnu-test
run the gnu testsuite in the ci
2026-05-29 21:43:41 +02:00
Leonard Hecker 05d167b8d5 Fix support for Python-style named backreferences 2026-05-29 21:12:19 +02:00
Sylvestre LedruandGitHub 41b92b3cc1 Merge pull request #5 from uutils/fix-coverage-profraw-path
ci: use absolute LLVM_PROFILE_FILE path to fix 0% coverage
2026-05-29 20:46:10 +02:00
Sylvestre Ledru e885f66523 ci: post GNU grep testsuite comparison as a PR comment
Add a GnuComment workflow that runs after GnuTests completes on a pull
request, downloads the 'comment' artifact (PR number + comparison text), and
posts it as a PR comment. Mirrors ../sed's GnuComment workflow.
2026-05-29 18:43:11 +02:00
Sylvestre Ledru 10bfa405dc ci: add GnuTests workflow running the GNU grep testsuite
Add a GnuTests workflow that, on push and PR, fetches the GNU grep release
tarball, builds the Rust grep binary, runs util/run-gnu-testsuite.sh, and
uploads the JSON results. An aggregate job compares the run against the
reference summary from the default branch (util/compare_test_results.py,
borrowed from ../sed) and fails only on new, non-intermittent regressions;
known-flaky tests are listed in .github/workflows/ignore-intermittent.txt.
2026-05-29 18:43:11 +02:00
Sylvestre Ledru f0ae449d11 tests: run the GNU grep testsuite against uu_grep
Add util/fetch-gnu.sh (downloads the GNU grep 3.12 release tarball from
ftp.gnu.org) and util/run-gnu-testsuite.sh, which reuses the gnulib test
framework shipped in the tarball (tests/init.sh + init.cfg) and injects the
Rust grep binary via PATH, replicating tests/Makefile.am's TESTS_ENVIRONMENT.
Each test is classified by its gnulib exit code (0=PASS, 77=SKIP, else FAIL)
and results are emitted as JSON.

Modelled on ../sed (lightweight PATH-injection runner) and ../coreutils
(release-tarball fetch). Current baseline: 61 pass / 39 fail / 28 skip of 128
tests -- the failures quantify the remaining GNU-compatibility gap.
2026-05-29 18:43:11 +02:00
Sylvestre Ledru 79db36edbe add license headers 2026-05-29 10:12:38 +02:00
20 changed files with 2691 additions and 85 deletions
+83
View File
@@ -0,0 +1,83 @@
name: GnuComment
# Post the GNU grep testsuite comparison (produced by the GnuTests workflow)
# as a comment on the pull request.
on:
workflow_run:
workflows: ["GnuTests"]
types:
- completed
permissions: {}
jobs:
post-comment:
permissions:
actions: read # to list workflow run artifacts
pull-requests: write # to comment on the pr
runs-on: ubuntu-latest
if: >
github.event.workflow_run.event == 'pull_request'
steps:
- name: 'Download artifact'
uses: actions/github-script@v7
with:
script: |
// List all artifacts from GnuTests
var artifacts = await github.rest.actions.listWorkflowRunArtifacts({
owner: context.repo.owner,
repo: context.repo.repo,
run_id: ${{ github.event.workflow_run.id }},
});
// Download the "comment" artifact, which contains a PR number (NR) and result.txt
var matchArtifact = artifacts.data.artifacts.filter((artifact) => {
return artifact.name == "comment"
})[0];
if (!matchArtifact) {
console.log('No comment artifact found');
return;
}
var download = await github.rest.actions.downloadArtifact({
owner: context.repo.owner,
repo: context.repo.repo,
artifact_id: matchArtifact.id,
archive_format: 'zip',
});
var fs = require('fs');
fs.writeFileSync('${{ github.workspace }}/comment.zip', Buffer.from(download.data));
- run: unzip comment.zip || echo "Failed to unzip comment artifact"
- name: 'Comment on PR'
uses: actions/github-script@v7
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
var fs = require('fs');
// Check if files exist
if (!fs.existsSync('./NR')) {
console.log('No NR file found, skipping comment');
return;
}
if (!fs.existsSync('./result.txt')) {
console.log('No result.txt file found, skipping comment');
return;
}
var issue_number = Number(fs.readFileSync('./NR'));
var content = fs.readFileSync('./result.txt');
if (content.toString().trim().length > 7) { // 7 because we have backquote + \n
await github.rest.issues.createComment({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: issue_number,
body: 'GNU grep testsuite comparison:\n```\n' + content + '```'
});
} else {
console.log('Comment content too short, skipping');
}
+197
View File
@@ -0,0 +1,197 @@
name: GnuTests
# Run the upstream GNU grep testsuite against the Rust grep implementation to
# track and guard byte-for-byte compatibility. See util/run-gnu-testsuite.sh.
on:
pull_request:
push:
branches:
- '*'
permissions:
contents: write # Publish grep instead of discarding
# End the current execution if there is a new changeset in the PR.
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}
env:
DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}
TEST_FULL_SUMMARY_FILE: 'grep-gnu-full-result.json'
jobs:
native:
name: Run GNU grep testsuite
runs-on: ubuntu-24.04
steps:
- name: Checkout code (grep)
uses: actions/checkout@v4
with:
path: 'grep'
persist-credentials: false
- uses: dtolnay/rust-toolchain@master
with:
toolchain: stable
- uses: Swatinem/rust-cache@v2
with:
workspaces: "./grep -> target"
- name: Fetch GNU grep testsuite
shell: bash
run: |
## Download and extract the upstream GNU grep release tarball
mkdir -p gnu.grep
cd gnu.grep
bash ../grep/util/fetch-gnu.sh
- name: Build Rust grep binary
shell: bash
run: |
cd 'grep'
cargo build --release --config=profile.release.strip=true
tar -C target/release -cf - grep | zstd -19 -o ../grep-x86_64-unknown-linux-gnu.tar.zst
- name: Publish latest commit
uses: softprops/action-gh-release@v3
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
with:
tag_name: latest-commit
body: |
commit: ${{ github.sha }}
draft: false
prerelease: true
files: |
grep-x86_64-unknown-linux-gnu.tar.zst
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Run GNU grep testsuite
shell: bash
run: |
cd 'grep'
export GNU_GREP_DIR="../gnu.grep"
./util/run-gnu-testsuite.sh --json-output "${{ env.TEST_FULL_SUMMARY_FILE }}" || true
- name: Upload full json results
uses: actions/upload-artifact@v4
with:
name: grep-gnu-full-result
path: grep/${{ env.TEST_FULL_SUMMARY_FILE }}
if-no-files-found: warn
aggregate:
needs: [native]
permissions:
actions: read
contents: read
pull-requests: read
name: Aggregate GNU test results
runs-on: ubuntu-24.04
steps:
- name: Initialize workflow variables
id: vars
shell: bash
run: |
## VARs setup
outputs() { step_id="${{ github.action }}"; for var in "$@" ; do echo steps.${step_id}.outputs.${var}="${!var}"; echo "${var}=${!var}" >> $GITHUB_OUTPUT; done; }
TEST_SUMMARY_FILE='grep-gnu-result.json'
outputs TEST_SUMMARY_FILE
- name: Checkout code (grep)
uses: actions/checkout@v4
with:
path: 'grep'
persist-credentials: false
- name: Retrieve reference artifacts
uses: dawidd6/action-download-artifact@v6
continue-on-error: true
with:
workflow: GnuTests.yml
branch: "${{ env.DEFAULT_BRANCH }}"
workflow_conclusion: completed
path: "reference"
if_no_artifact_found: warn
- name: Download full json results
uses: actions/download-artifact@v4
with:
name: grep-gnu-full-result
path: results
- name: Extract/summarize testing info
id: summary
shell: bash
run: |
## Extract/summarize testing info
outputs() { step_id="${{ github.action }}"; for var in "$@" ; do echo steps.${step_id}.outputs.${var}="${!var}"; echo "${var}=${!var}" >> $GITHUB_OUTPUT; done; }
RESULT_FILE="results/${{ env.TEST_FULL_SUMMARY_FILE }}"
if [[ ! -f "$RESULT_FILE" ]]; then
echo "::error ::Result file $RESULT_FILE not found"
find results -type f || true
exit 1
fi
TOTAL=$(jq -r '.summary.total // 0' "$RESULT_FILE")
PASS=$(jq -r '.summary.passed // 0' "$RESULT_FILE")
FAIL=$(jq -r '.summary.failed // 0' "$RESULT_FILE")
SKIP=$(jq -r '.summary.skipped // 0' "$RESULT_FILE")
output="GNU grep tests summary = TOTAL: $TOTAL / PASS: $PASS / FAIL: $FAIL / SKIP: $SKIP"
echo "${output}"
if [[ "$FAIL" -gt 0 ]]; then
echo "::warning ::${output}"
fi
outputs TOTAL PASS FAIL SKIP
- name: Compare test failures VS reference
shell: bash
run: |
## Compare current results against the reference summary from the default branch
REF_SUMMARY_FILE='reference/grep-gnu-full-result/${{ env.TEST_FULL_SUMMARY_FILE }}'
CURRENT_SUMMARY_FILE="results/${{ env.TEST_FULL_SUMMARY_FILE }}"
IGNORE_INTERMITTENT="grep/.github/workflows/ignore-intermittent.txt"
# Set up comment directory for the GnuComment workflow.
COMMENT_DIR="reference/comment"
mkdir -p ${COMMENT_DIR}
echo ${{ github.event.number }} > ${COMMENT_DIR}/NR
COMMENT_LOG="${COMMENT_DIR}/result.txt"
: > "${COMMENT_LOG}"
COMPARISON_RESULT=0
if test -f "${REF_SUMMARY_FILE}"; then
python3 grep/util/compare_test_results.py \
--ignore-file "${IGNORE_INTERMITTENT}" \
--output "${COMMENT_LOG}" \
"${CURRENT_SUMMARY_FILE}" "${REF_SUMMARY_FILE}" || COMPARISON_RESULT=$?
else
echo "::warning ::Skipping test comparison; no prior reference summary at '${REF_SUMMARY_FILE}'."
fi
if [ ${COMPARISON_RESULT} -eq 1 ]; then
echo "::error ::Found new non-intermittent test failures"
UPLOAD_EXIT=1
else
echo "::notice ::No new test failures detected"
UPLOAD_EXIT=0
fi
echo "UPLOAD_EXIT=${UPLOAD_EXIT}" >> $GITHUB_ENV
- name: Upload comparison log (for GnuComment workflow)
if: success() || failure()
uses: actions/upload-artifact@v4
with:
name: comment
path: reference/comment/
- name: Report test results
if: success() || failure()
shell: bash
run: |
echo "::notice ::GNU grep testsuite: TOTAL ${{ steps.summary.outputs.TOTAL }} / PASS ${{ steps.summary.outputs.PASS }} / FAIL ${{ steps.summary.outputs.FAIL }} / SKIP ${{ steps.summary.outputs.SKIP }}"
# Fail the job if the comparison found new non-intermittent regressions.
exit "${UPLOAD_EXIT:-0}"
+37
View File
@@ -0,0 +1,37 @@
name: CodSpeed
on:
push:
branches:
- "main"
pull_request:
# `workflow_dispatch` allows CodSpeed to trigger backtest
# performance analysis in order to generate initial data.
workflow_dispatch:
permissions:
contents: read
id-token: write
jobs:
codspeed:
name: Run benchmarks
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup Rust toolchain, cache and cargo-codspeed binary
uses: moonrepo/setup-rust@v0
with:
channel: stable
cache-target: release
bins: cargo-codspeed
- name: Build the benchmark target(s)
run: cargo codspeed build
- name: Run the benchmarks
uses: CodSpeedHQ/action@v4
with:
mode: simulation
run: cargo codspeed run
@@ -0,0 +1,7 @@
# List of intermittent test names to ignore in result comparisons
# Format: one test name per line, lines starting with # are comments
#
# Add test names that are known to be flaky or environment-dependent
# Example:
# basic_substitution
# line_address_test
+55
View File
@@ -0,0 +1,55 @@
# See https://pre-commit.com for more information
# See https://pre-commit.com/hooks.html for more hooks
exclude: ^tests/fixtures/
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v6.0.0
hooks:
- id: check-added-large-files
- id: check-executables-have-shebangs
- id: check-json
exclude: '\.vscode/(cSpell|extensions)\.json' # cSpell.json and extensions.json use comments
- id: check-shebang-scripts-are-executable
exclude: '.+\.rs' # would be triggered by #![some_attribute]
- id: check-symlinks
- id: check-toml
- id: check-yaml
args: [ --allow-multiple-documents ]
- id: destroyed-symlinks
- id: end-of-file-fixer
- id: mixed-line-ending
args: [ --fix=lf ]
- id: trailing-whitespace
- repo: local
hooks:
- id: rust-linting
name: Rust linting
description: Run cargo fmt on files included in the commit.
entry: cargo +stable fmt --
pass_filenames: true
types: [file, rust]
language: system
- id: rust-clippy
name: Rust clippy
description: Run cargo clippy on files included in the commit.
entry: cargo +stable clippy --workspace --all-targets --all-features -- -D warnings
pass_filenames: false
types: [file, rust]
language: system
- id: cargo-lock-check
name: Cargo.lock sync check
description: Ensure Cargo.lock and fuzz/Cargo.lock are up-to-date.
entry: bash -c 'for dir in . fuzz; do if [ -d "$dir" ]; then ( cd "$dir" && cargo fetch --quiet ); fi; done'
pass_filenames: false
files: 'Cargo\.(toml|lock)$'
language: system
- id: cspell
name: Code spell checker (cspell)
description: Run cspell to check for spelling errors (if available).
entry: bash -c 'if command -v cspell >/dev/null 2>&1; then cspell --no-must-find-files -- "$@"; else echo "cspell not found, skipping spell check"; exit 0; fi' --
pass_filenames: true
language: system
ci:
skip: [rust-linting, rust-clippy, cargo-lock-check, cspell]
Generated
+532 -11
View File
File diff suppressed because it is too large Load Diff
+5
View File
@@ -27,5 +27,10 @@ onig_sys = { version = "*", default-features = false }
uucore = "0.8.0"
walkdir = "2.5"
[[bench]]
name = "grep_bench"
harness = false
[dev-dependencies]
criterion = { version = "4.7.0", package = "codspeed-criterion-compat" }
uutests = "0.8.0"
+12
View File
@@ -4,12 +4,19 @@
[![dependency status](https://deps.rs/repo/github/uutils/grep/status.svg)](https://deps.rs/repo/github/uutils/grep)
[![CodeCov](https://codecov.io/gh/uutils/grep/branch/main/graph/badge.svg)](https://codecov.io/gh/uutils/grep)
[![CodSpeed](https://img.shields.io/endpoint?url=https://codspeed.io/badge.json)](https://codspeed.io/uutils/grep?utm_source=badge)
# Grep, now in Rust
A Rust implementation of [GNU Grep](https://www.gnu.org/software/grep/).
This project is an initial release and may contain bugs.
## Install
```shell
cargo install uu_grep
```
## Building
Download Rust at: https://rustup.rs/
@@ -29,10 +36,15 @@ cargo build --release
cargo test
```
## Pre-commit hooks
This project uses [pre-commit](https://pre-commit.com); run `pre-commit install` to enable the git hooks.
## Known Issues
* Does not take `LANG`, etc., into account for handling file encodings (non-UTF8 matches are treated as binary)
* No localization support yet
* Performances need to be improved
## Contributing
+128
View File
@@ -0,0 +1,128 @@
// This file is part of the uutils grep package.
//
// For the full copyright and license information, please view the LICENSE
// file that was distributed with this source code.
use criterion::{Criterion, black_box, criterion_group, criterion_main};
use std::ffi::OsString;
use std::path::Path;
/// Run grep end-to-end through the real `uumain` entry point. `args` are the
/// arguments after the program name (flags, pattern, paths). The exit status is
/// ignored — we only care about the work performed.
fn run(args: &[&str]) {
let mut argv: Vec<OsString> = Vec::with_capacity(args.len() + 1);
argv.push(OsString::from("grep"));
argv.extend(args.iter().map(OsString::from));
let _ = uu_grep::uumain(argv.into_iter());
}
/// Build a multi-megabyte log-like corpus plus a directory holding it alongside
/// a binary file. Every line contains `worker-<n>` and a `2024-…` timestamp; a
/// rare `RAREHIT` marker appears on a handful of lines (≈ every 10000th).
/// Returns `(dir, log_file)`.
fn build_corpus() -> (std::path::PathBuf, std::path::PathBuf) {
let mut content = String::new();
for i in 0..80_000u32 {
if i % 10_000 == 0 {
content.push_str(&format!(
"2024-01-15 10:30:{:02} RAREHIT worker-{i} special marker seen\n",
i % 60
));
} else if i % 100 == 0 {
content.push_str(&format!(
"2024-01-15 10:30:{:02} ERROR worker-{i} connection reset\n",
i % 60
));
} else {
content.push_str(&format!(
"2024-01-15 10:30:{:02} INFO worker-{i} request handled in {}ms\n",
i % 60,
i % 1000
));
}
}
assert!(content.len() > 4 * 1024 * 1024);
let dir = std::env::temp_dir().join(format!("uu_grep_bench_{}", std::process::id()));
std::fs::create_dir_all(&dir).unwrap();
let log = dir.join("app.log");
std::fs::write(&log, &content).unwrap();
// A binary file (contains NUL) that also holds the marker, so `-I` has
// something to skip while recursing.
let mut binary = vec![0u8, 1, 2, 3];
binary.extend_from_slice(b"RAREHIT in binary blob");
binary.extend(std::iter::repeat_n(0u8, 4096));
std::fs::write(dir.join("data.bin"), &binary).unwrap();
(dir, log)
}
fn bench_e2e(c: &mut Criterion) {
let (dir, log) = build_corpus();
let file = log.to_str().unwrap();
let dir_str = dir.to_str().unwrap();
// Pure scanning throughput: `-q` with a pattern that never matches forces a
// full scan and produces no output. A literal (which a buffer-at-a-time
// searcher can accelerate) versus an extended-regex control (which cannot).
{
let mut group = c.benchmark_group("scan");
group.bench_function("literal_no_match", |b| {
b.iter(|| run(black_box(&["-q", "NONEXISTENT_TOKEN_XYZ", file])))
});
group.bench_function("regex_no_match", |b| {
b.iter(|| run(black_box(&["-q", "-E", "NON[0-9]EXISTENT_TOKEN", file])))
});
group.finish();
}
// Real invocation shapes from the `grep` tldr page, each scanning the whole
// corpus. The `RAREHIT` marker matches only a handful of lines, so output
// stays small while the full-file scan dominates.
{
let mut group = c.benchmark_group("usage");
// Search for a pattern within a file.
group.bench_function("search_pattern", |b| {
b.iter(|| run(black_box(&["RAREHIT", file])))
});
// Search for an exact string (-F).
group.bench_function("fixed_string", |b| {
b.iter(|| run(black_box(&["-F", "RAREHIT", file])))
});
// Recursive search ignoring binary files (-rI).
group.bench_function("recursive_no_binary", |b| {
b.iter(|| run(black_box(&["-rI", "RAREHIT", dir_str])))
});
// Print 3 lines of context (-C 3).
group.bench_function("context", |b| {
b.iter(|| run(black_box(&["-C", "3", "RAREHIT", file])))
});
// Filename + line number with forced color (-Hn --color=always).
group.bench_function("filename_lineno_color", |b| {
b.iter(|| run(black_box(&["-Hn", "--color=always", "RAREHIT", file])))
});
// Print only the matched text (-o).
group.bench_function("only_matching", |b| {
b.iter(|| run(black_box(&["-o", "RAREHIT", file])))
});
// Invert match (-v); `worker-` is on every line, so nothing is printed
// and this measures the full inverted scan.
group.bench_function("invert_match", |b| {
b.iter(|| run(black_box(&["-v", "worker-", file])))
});
// Extended regex, case-insensitive (-Ei).
group.bench_function("extended_icase", |b| {
b.iter(|| run(black_box(&["-Ei", "rarehit", file])))
});
group.finish();
}
let _ = std::fs::remove_dir_all(Path::new(dir_str));
}
criterion_group!(benches, bench_e2e);
criterion_main!(benches);
+5
View File
@@ -1,3 +1,8 @@
// This file is part of the uutils grep package.
//
// For the full copyright and license information, please view the LICENSE
// file that was distributed with this source code.
pub struct LineView<'a> {
/// Line content (without the terminator).
pub line: &'a [u8],
+101 -59
View File
@@ -1,6 +1,14 @@
mod context_buffer;
mod line_buffer;
mod matcher;
// This file is part of the uutils grep package.
//
// For the full copyright and license information, please view the LICENSE
// file that was distributed with this source code.
#[doc(hidden)]
pub mod context_buffer;
#[doc(hidden)]
pub mod line_buffer;
#[doc(hidden)]
pub mod matcher;
mod output;
mod searcher;
@@ -14,8 +22,9 @@ use std::io::{IsTerminal as _, Read};
use std::path::Path;
use uucore::error::{FromIo, UResult, USimpleError};
#[derive(Clone, Copy, PartialEq, Eq)]
enum RegexMode {
#[derive(Clone, Copy, PartialEq, Eq, Debug)]
#[doc(hidden)]
pub enum RegexMode {
Fixed,
Basic,
Extended,
@@ -23,7 +32,8 @@ enum RegexMode {
}
#[derive(Clone, Copy, PartialEq, Eq)]
enum BinaryMode {
#[doc(hidden)]
pub enum BinaryMode {
Binary,
Text,
WithoutMatch,
@@ -37,79 +47,84 @@ enum ColorMode {
}
#[derive(Clone, Copy, PartialEq, Eq)]
enum DirectoryMode {
#[doc(hidden)]
pub enum DirectoryMode {
Read,
Skip,
Recurse,
}
#[derive(Clone, Copy, PartialEq, Eq)]
enum DeviceMode {
#[doc(hidden)]
pub enum DeviceMode {
Default,
Read,
Skip,
}
struct ColorConfig<'a> {
matched_selected: &'a str,
matched_context: &'a str,
filename: &'a str,
line_number: &'a str,
byte_offset: &'a str,
separator: &'a str,
selected_line: &'a str,
context_line: &'a str,
#[doc(hidden)]
pub struct ColorConfig<'a> {
pub matched_selected: &'a str,
pub matched_context: &'a str,
pub filename: &'a str,
pub line_number: &'a str,
pub byte_offset: &'a str,
pub separator: &'a str,
pub selected_line: &'a str,
pub context_line: &'a str,
reverse_video: bool,
no_erase: bool,
pub reverse_video: bool,
pub no_erase: bool,
}
struct GlobSet {
#[doc(hidden)]
pub struct GlobSet {
patterns: Vec<glob::Pattern>,
}
struct Config<'a> {
#[doc(hidden)]
pub struct Config<'a> {
// Searcher
directory_mode: DirectoryMode,
device_mode: DeviceMode,
follow_symlinks: bool,
include_globs: GlobSet,
exclude_globs: GlobSet,
exclude_dir_globs: GlobSet,
label: &'a str,
pub directory_mode: DirectoryMode,
pub device_mode: DeviceMode,
pub follow_symlinks: bool,
pub include_globs: GlobSet,
pub exclude_globs: GlobSet,
pub exclude_dir_globs: GlobSet,
pub label: &'a str,
#[cfg(windows)]
strip_cr: bool,
binary_mode: BinaryMode,
max_count: Option<u64>,
before_context: usize,
after_context: usize,
has_context: bool,
pub strip_cr: bool,
pub binary_mode: BinaryMode,
pub max_count: Option<u64>,
pub before_context: usize,
pub after_context: usize,
pub has_context: bool,
// Matcher
patterns: &'a [&'a str],
regex_mode: RegexMode,
ignore_case: bool,
invert_match: bool,
word_regexp: bool,
line_regexp: bool,
pub patterns: &'a [&'a str],
pub regex_mode: RegexMode,
pub ignore_case: bool,
pub invert_match: bool,
pub word_regexp: bool,
pub line_regexp: bool,
// Output
quiet: bool,
count: bool,
show_filename: bool,
files_with_matches: bool,
files_without_match: bool,
only_matching: bool,
byte_offset: bool,
line_number: bool,
initial_tab: bool,
null_separator: bool,
null_data: bool,
line_buffered: bool,
no_messages: bool,
group_separator: Option<&'a str>,
use_color: bool,
color_config: ColorConfig<'a>,
pub quiet: bool,
pub count: bool,
pub show_filename: bool,
pub files_with_matches: bool,
pub files_without_match: bool,
pub only_matching: bool,
pub byte_offset: bool,
pub line_number: bool,
pub initial_tab: bool,
pub null_separator: bool,
pub null_data: bool,
pub line_buffered: bool,
pub no_messages: bool,
pub group_separator: Option<&'a str>,
pub use_color: bool,
pub color_config: ColorConfig<'a>,
}
#[uucore::main(no_signals)]
@@ -240,6 +255,14 @@ pub fn uumain(args: impl uucore::Args) -> UResult<()> {
));
}
// GNU grep's PCRE backend (-P) supports only a single pattern.
if perl_regexp && patterns.len() > 1 {
return Err(USimpleError::new(
2,
"the -P option only supports a single pattern".to_string(),
));
}
// Decoded options into enums
let regex_mode = if fixed_strings {
RegexMode::Fixed
@@ -416,6 +439,10 @@ pub fn uu_app() -> Command {
.about("Search for PATTERNS in each FILE.")
.disable_help_flag(true)
.disable_version_flag(true)
// GNU grep accepts repeated options (booleans are idempotent, value
// options take the last); make clap replace rather than error. Args
// with ArgAction::Append (e.g. -e/-f/--include) still accumulate.
.args_override_self(true)
.after_help(
"When FILE is '-', read standard input. If no FILE is given, read standard \
input, but with -r, recursively search the working directory instead. With \
@@ -688,14 +715,16 @@ pub fn uu_app() -> Command {
.short('L')
.long("files-without-match")
.help("print only names of FILEs with no selected lines")
.action(ArgAction::SetTrue),
.action(ArgAction::SetTrue)
.overrides_with("files_with_matches"),
)
.arg(
Arg::new("files_with_matches")
.short('l')
.long("files-with-matches")
.help("print only names of FILEs with selected lines")
.action(ArgAction::SetTrue),
.action(ArgAction::SetTrue)
.overrides_with("files_without_match"),
)
.arg(
Arg::new("count")
@@ -845,8 +874,21 @@ fn expand_num_shorthand(args: impl Iterator<Item = OsString>) -> Vec<OsString> {
out
}
impl Default for GlobSet {
fn default() -> Self {
Self::new()
}
}
impl GlobSet {
fn with_capacity(capacity: usize) -> Self {
/// Create an empty GlobSet.
pub fn new() -> Self {
Self {
patterns: Vec::new(),
}
}
pub fn with_capacity(capacity: usize) -> Self {
Self {
patterns: Vec::with_capacity(capacity),
}
+195 -1
View File
@@ -1,4 +1,9 @@
use memchr::memchr;
// This file is part of the uutils grep package.
//
// For the full copyright and license information, please view the LICENSE
// file that was distributed with this source code.
use memchr::{memchr, memrchr};
use std::fs::File;
use std::io::{self, Read as _};
@@ -106,4 +111,193 @@ impl LineBuffer {
self.end += n;
}
}
/// Read the next run of *complete* lines as a single slice.
///
/// Returns `Ok(None)` at end of input. Otherwise returns `Ok(Some((chunk,
/// chunk_start)))`, where `chunk` spans one or more whole lines (each ending
/// in the terminator) and `chunk_start` is the absolute byte offset of the
/// first byte of the chunk. The only exception is a final line lacking a
/// terminator, which is returned on its own as the last chunk.
///
/// This hands back as much buffered data as ends on a line boundary, so a
/// caller can scan many lines with one pass instead of line by line.
pub fn read_chunk(&mut self, file: &mut File) -> io::Result<Option<(&[u8], u64)>> {
loop {
// Hand back everything up to and including the last terminator.
if self.end > self.beg
&& let Some(off) = memrchr(self.line_terminator, &self.buffer[self.beg..self.end])
{
let beg = self.beg;
let lim = self.beg + off + 1;
let chunk_start = self.next_line_start;
self.next_line_start += (lim - beg) as u64;
self.beg = lim;
self.scan = lim;
return Ok(Some((&self.buffer[beg..lim], chunk_start)));
}
// No whole line buffered. At EOF, flush any unterminated remainder.
if self.eof {
if self.beg == self.end {
return Ok(None);
}
let beg = self.beg;
let chunk_start = self.next_line_start;
self.next_line_start += (self.end - beg) as u64;
self.beg = self.end;
self.scan = self.end;
return Ok(Some((&self.buffer[beg..self.end], chunk_start)));
}
// Slide the partial tail to the front to maximize room for reading.
if self.beg > 0 {
self.buffer.copy_within(self.beg..self.end, 0);
self.end -= self.beg;
self.beg = 0;
self.scan = 0;
}
if self.end == self.buffer.len() {
// A single line is longer than the whole buffer; grow it.
self.buffer.resize(self.buffer.len() * 2, 0);
}
let n = loop {
match file.read(&mut self.buffer[self.end..]) {
Ok(n) => break n,
Err(e) if e.kind() == io::ErrorKind::Interrupted => {}
Err(e) => return Err(e),
}
};
if n == 0 {
self.eof = true;
} else {
self.end += n;
}
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::io::{Seek as _, SeekFrom, Write as _};
use std::sync::atomic::{AtomicU32, Ordering};
static COUNTER: AtomicU32 = AtomicU32::new(0);
/// A temp file pre-loaded with `content`, rewound to the start, and removed
/// from disk when dropped.
struct TempInput {
file: File,
path: std::path::PathBuf,
}
impl Drop for TempInput {
fn drop(&mut self) {
let _ = std::fs::remove_file(&self.path);
}
}
fn temp_input(content: &[u8]) -> TempInput {
let mut path = std::env::temp_dir();
let n = COUNTER.fetch_add(1, Ordering::Relaxed);
path.push(format!("uu_grep_lb_{}_{n}.tmp", std::process::id()));
let mut file = std::fs::OpenOptions::new()
.read(true)
.write(true)
.create(true)
.truncate(true)
.open(&path)
.unwrap();
file.write_all(content).unwrap();
file.seek(SeekFrom::Start(0)).unwrap();
TempInput { file, path }
}
/// Drain `read_chunk` into a list of (owned bytes, start offset) pairs.
fn chunks(term: u8, content: &[u8]) -> Vec<(Vec<u8>, u64)> {
let mut lb = LineBuffer::new(term);
let mut input = temp_input(content);
let mut out = Vec::new();
while let Some((chunk, start)) = lb.read_chunk(&mut input.file).unwrap() {
out.push((chunk.to_vec(), start));
}
out
}
#[test]
fn empty_input_yields_nothing() {
assert!(chunks(b'\n', b"").is_empty());
}
#[test]
fn whole_complete_lines_come_back_as_one_chunk() {
// Small input arrives in a single read, so everything up to the final
// terminator is one chunk starting at offset 0.
assert_eq!(
chunks(b'\n', b"a\nbb\nccc\n"),
vec![(b"a\nbb\nccc\n".to_vec(), 0)]
);
}
#[test]
fn unterminated_tail_is_a_final_chunk_with_its_own_offset() {
// "a\n" is the complete-line chunk; "bb" is flushed at EOF at offset 2.
assert_eq!(
chunks(b'\n', b"a\nbb"),
vec![(b"a\n".to_vec(), 0), (b"bb".to_vec(), 2)]
);
}
#[test]
fn input_without_any_terminator_is_one_chunk() {
assert_eq!(chunks(b'\n', b"abc"), vec![(b"abc".to_vec(), 0)]);
}
#[test]
fn honors_a_custom_terminator() {
assert_eq!(
chunks(b'\0', b"a\0bb\0c"),
vec![(b"a\0bb\0".to_vec(), 0), (b"c".to_vec(), 5)]
);
}
#[test]
fn reassembles_input_larger_than_the_buffer() {
// Force many reads and at least one chunk boundary mid-file.
let mut content = Vec::new();
for i in 0..50_000u32 {
content.extend_from_slice(format!("line number {i}\n").as_bytes());
}
assert!(content.len() > 128 * 1024);
let got = chunks(b'\n', &content);
assert!(got.len() > 1, "expected multiple chunks, got {}", got.len());
// Chunks must tile the input exactly, contiguously, each ending on a
// line boundary (the input ends with a terminator).
let mut expected_start = 0u64;
let mut joined = Vec::new();
for (bytes, start) in &got {
assert_eq!(*start, expected_start);
assert_eq!(*bytes.last().unwrap(), b'\n');
expected_start += bytes.len() as u64;
joined.extend_from_slice(bytes);
}
assert_eq!(joined, content);
}
#[test]
fn grows_to_hold_a_single_overlong_line() {
// One line far bigger than the initial 128 KiB buffer, then a short one.
let mut content = vec![b'x'; 300 * 1024];
content.push(b'\n');
content.extend_from_slice(b"tail\n");
let got = chunks(b'\n', &content);
let joined: Vec<u8> = got.iter().flat_map(|(b, _)| b.clone()).collect();
assert_eq!(joined, content);
assert_eq!(got[0].1, 0);
}
}
+5
View File
@@ -1 +1,6 @@
// This file is part of the uutils grep package.
//
// For the full copyright and license information, please view the LICENSE
// file that was distributed with this source code.
uucore::bin!(uu_grep);
+131 -11
View File
@@ -1,11 +1,26 @@
// This file is part of the uutils grep package.
//
// For the full copyright and license information, please view the LICENSE
// file that was distributed with this source code.
use crate::{Config, RegexMode};
use onig::{EncodedBytes, Regex, RegexOptions, Region, SearchOptions, Syntax, SyntaxBehavior};
use memchr::memmem;
use onig::{
EncodedBytes, Regex, RegexOptions, Region, SearchOptions, Syntax, SyntaxBehavior,
SyntaxOperator,
};
use onig_sys::{OnigEncCtype_ONIGENC_CTYPE_WORD, OnigEncodingUTF8};
use uucore::error::{UResult, USimpleError};
pub struct Matcher<'a> {
config: &'a Config<'a>,
patterns: Vec<CompiledPattern>,
/// One substring searcher per pattern, present only when *every* pattern is
/// a plain literal that a raw byte search resolves exactly (see
/// [`plain_literal`]). When set, a caller can decide a line matches by
/// looking for any of these needles, bypassing the regex engine entirely.
/// `None` as soon as a single pattern needs real regex evaluation.
literal_searchers: Option<Vec<memmem::Finder<'static>>>,
}
impl<'a> Matcher<'a> {
@@ -14,19 +29,41 @@ impl<'a> Matcher<'a> {
for raw in config.patterns {
patterns.push(CompiledPattern::compile(raw, config)?);
}
Ok(Self { config, patterns })
// If we can reduce the whole pattern set to literal needles, keep a
// searcher for each so the driver can take a bulk substring-scan path.
let needles: Option<Vec<Vec<u8>>> = config
.patterns
.iter()
.map(|p| plain_literal(p, config.ignore_case, config.regex_mode))
.collect();
let literal_searchers = needles.filter(|n| !n.is_empty()).map(|n| {
n.iter()
.map(|w| memmem::Finder::new(w).into_owned())
.collect()
});
Ok(Self {
config,
patterns,
literal_searchers,
})
}
/// Per-pattern substring searchers, present only when the pattern set is a
/// pure set of literals (no regex needed). Used by the searcher to scan a
/// whole buffer at once instead of testing line by line.
pub fn literal_searchers(&self) -> Option<&[memmem::Finder<'static>]> {
self.literal_searchers.as_deref()
}
/// Decide whether `line` matches and return the positions to highlight.
pub fn match_line(&self, line: &[u8]) -> Option<Vec<(usize, usize)>> {
let mut any_seen = false;
let mut any_selected = false;
let positions: Vec<_> = MatchIter::new(&self.patterns, line)
.filter(|&(start, end)| {
any_seen = true;
// Drop zero-length matches from the output.
if start == end {
return false;
}
// Drop matches that don't span the whole line if `-x` was requested.
if self.config.line_regexp && !(start == 0 && end == line.len()) {
return false;
@@ -35,13 +72,19 @@ impl<'a> Matcher<'a> {
if self.config.word_regexp && !Self::is_word_match(line, start, end) {
return false;
}
any_selected = true;
// Drop zero-length matches from the output.
if start == end {
return false;
}
true
})
.collect();
let raw_matched = if self.config.line_regexp || self.config.word_regexp {
// -w / -x are authoritative once positions are filtered.
!positions.is_empty()
// -w / -x are authoritative once matches are filtered. Zero-length
// matches can select a line even though there is no span to output.
any_selected
} else {
any_seen
};
@@ -166,7 +209,7 @@ struct Cursor<'a> {
impl Cursor<'_> {
fn refill(&mut self) {
if self.offset >= self.line.len() {
if self.offset > self.line.len() {
self.pending = None;
return;
}
@@ -186,6 +229,25 @@ impl Cursor<'_> {
}
}
/// Return the literal bytes of `pattern` when a raw byte-for-byte substring
/// search is *exactly* equivalent to matching it, otherwise `None`.
///
/// We accept only ASCII, case-sensitive needles. That keeps the byte search in
/// agreement with the regex engine on every possible input, including bytes that
/// are not valid UTF-8: an ASCII byte can never be part of a multi-byte sequence,
/// so its presence is unambiguous. In the regex modes we also require that no
/// byte could ever act as a metacharacter; under `-F` the text is literal as-is.
fn plain_literal(pattern: &str, ignore_case: bool, mode: RegexMode) -> Option<Vec<u8>> {
if ignore_case || pattern.is_empty() || !pattern.is_ascii() {
return None;
}
// Every byte that carries special meaning in any of our regex syntaxes.
// A needle without these reads the same as a literal in Basic/Extended/Perl.
const SPECIAL: &[u8] = b".*[]^$\\+?{}()|";
let plain = mode == RegexMode::Fixed || !pattern.bytes().any(|b| SPECIAL.contains(&b));
plain.then(|| pattern.as_bytes().to_vec())
}
struct CompiledPattern {
/// Default semantics. It's decently fast and used for searching.
leftmost: Regex,
@@ -201,11 +263,23 @@ impl CompiledPattern {
RegexMode::Fixed => Syntax::asis(),
RegexMode::Basic => Syntax::grep(),
RegexMode::Extended => Syntax::gnu_regex(),
RegexMode::Perl => Syntax::perl(),
RegexMode::Perl => Syntax::perl_ng(),
};
if !matches!(config.regex_mode, RegexMode::Fixed) {
if config.regex_mode != RegexMode::Fixed {
// GNU grep supports `{,n}` as an alias for `{0,n}`.
syntax.enable_behavior(SyntaxBehavior::SYNTAX_BEHAVIOR_ALLOW_INTERVAL_LOW_ABBREV);
}
if config.regex_mode == RegexMode::Perl {
// GNU grep supports `(?P<name>...)`.
// Unfortunately, the onig crate defines the OP2 flag without the
// necessary <<32 bit shift, so we need to hotpatch that here.
const _: () =
assert!(SyntaxOperator::SYNTAX_OPERATOR_QMARK_CAPITAL_P_NAME.bits() == 0x80000000);
const FIXED: SyntaxOperator = SyntaxOperator::from_bits_retain(
SyntaxOperator::SYNTAX_OPERATOR_QMARK_CAPITAL_P_NAME.bits() << 32,
);
syntax.enable_operators(FIXED);
}
let mut options = RegexOptions::REGEX_OPTION_NONE;
if config.ignore_case {
@@ -269,3 +343,49 @@ impl CompiledPattern {
.is_some()
}
}
#[cfg(test)]
mod tests {
use super::plain_literal;
use crate::RegexMode;
fn lit(p: &str, ic: bool, mode: RegexMode) -> Option<Vec<u8>> {
plain_literal(p, ic, mode)
}
#[test]
fn fixed_mode_takes_any_ascii_verbatim() {
// Under -F every byte is literal, even regex metacharacters.
assert_eq!(lit("abc", false, RegexMode::Fixed), Some(b"abc".to_vec()));
assert_eq!(lit("a.*b", false, RegexMode::Fixed), Some(b"a.*b".to_vec()));
assert_eq!(lit("a+b", false, RegexMode::Fixed), Some(b"a+b".to_vec()));
}
#[test]
fn regex_modes_accept_metacharacter_free_literals() {
for mode in [RegexMode::Basic, RegexMode::Extended, RegexMode::Perl] {
assert_eq!(lit("ing", false, mode), Some(b"ing".to_vec()));
assert_eq!(lit("Hello123", false, mode), Some(b"Hello123".to_vec()));
}
}
#[test]
fn regex_modes_reject_anything_with_a_metacharacter() {
for mode in [RegexMode::Basic, RegexMode::Extended, RegexMode::Perl] {
for p in [
"a.b", "a*", "[ab]", "^a", "a$", "a\\b", "a+", "a?", "(a)", "a|b", "a{2}",
] {
assert_eq!(lit(p, false, mode), None, "pattern {p:?} in {mode:?}");
}
}
}
#[test]
fn rejects_empty_case_insensitive_and_non_ascii() {
assert_eq!(lit("", false, RegexMode::Fixed), None);
assert_eq!(lit("abc", true, RegexMode::Fixed), None); // -i
assert_eq!(lit("abc", true, RegexMode::Basic), None);
assert_eq!(lit("café", false, RegexMode::Fixed), None); // non-ASCII
assert_eq!(lit("naïve", false, RegexMode::Basic), None);
}
}
+18 -1
View File
@@ -1,8 +1,14 @@
// This file is part of the uutils grep package.
//
// For the full copyright and license information, please view the LICENSE
// file that was distributed with this source code.
use crate::Config;
use crate::context_buffer::LineView;
use std::ffi::OsStr;
use std::io::{self, BufWriter, StdoutLock, Write};
use std::path::Path;
use uucore::error::strip_errno;
#[cfg(target_pointer_width = "64")]
const BUF_SIZE: usize = 128 * 1024;
@@ -65,6 +71,7 @@ impl<'a> OutputWriter<'a> {
view.line_number,
view.byte_offset + start as u64,
b':',
false,
)?;
self.write_colored_bytes(
@@ -84,6 +91,7 @@ impl<'a> OutputWriter<'a> {
view.line_number,
view.byte_offset,
if view.is_match { b':' } else { b'-' },
view.line.is_empty(),
)?;
let mut last_end = 0;
@@ -119,6 +127,7 @@ impl<'a> OutputWriter<'a> {
line_number: u64,
byte_offset: u64,
sep_char: u8,
content_empty: bool,
) -> io::Result<()> {
if self.config.show_filename {
self.write_colored_fmt(
@@ -149,7 +158,10 @@ impl<'a> OutputWriter<'a> {
self.write_separator(sep_char)?;
}
// GNU grep aligns content with a tab under -T, but only when there is
// content to align: an empty line keeps just its prefix (no tab).
if self.config.initial_tab
&& !content_empty
&& (self.config.line_number || self.config.byte_offset || self.config.show_filename)
{
self.out.write_all(b"\t")?;
@@ -190,7 +202,12 @@ impl<'a> OutputWriter<'a> {
/// Write an IO error to stderr.
pub fn report_io_error(&self, label: &OsStr, err: &io::Error) {
if !self.config.no_messages && !self.config.quiet {
eprintln!("grep: {label}: {err}", label = label.to_string_lossy());
// Strip the trailing " (os error XX)" so the message matches GNU grep.
eprintln!(
"grep: {label}: {err}",
label = label.to_string_lossy(),
err = strip_errno(err)
);
}
}
+250 -2
View File
@@ -1,9 +1,15 @@
// This file is part of the uutils grep package.
//
// For the full copyright and license information, please view the LICENSE
// file that was distributed with this source code.
use crate::context_buffer::{ContextBuffer, LineView};
use crate::line_buffer::LineBuffer;
use crate::matcher::Matcher;
use crate::output::OutputWriter;
use crate::{BinaryMode, Config, DeviceMode, DirectoryMode};
use memchr::memchr;
use memchr::memmem::Finder;
use memchr::{memchr, memchr_iter, memrchr};
use std::ffi::OsStr;
use std::fs::File;
use std::io;
@@ -112,7 +118,12 @@ impl<'a> Searcher<'a> {
.flush()
.map_err_context(|| "(standard output)".to_string())?;
if self.had_error {
// With -q, a match yields exit status 0 even if an error (e.g. a
// missing file) occurred earlier: GNU exits as soon as a line is
// selected, so the error never affects the status.
if self.config.quiet && self.any_match {
Ok(())
} else if self.had_error {
Err(ExitCode::new(2))
} else if self.any_match {
Ok(())
@@ -243,12 +254,221 @@ impl<'a> Searcher<'a> {
self.binary_notice_enabled && self.session_binary_detected && self.session_any_match()
}
/// Whether the current configuration can use the buffer-at-a-time fast
/// path. It applies only to pure-literal patterns and the simpler output
/// modes — anything needing match positions, context, inversion, or special
/// binary handling falls back to the line-at-a-time [`Self::session_run`].
fn eligible_for_fast_path(&self) -> bool {
// On Windows the line-at-a-time path strips a trailing CR before
// matching; the fast path mirrors that only for printed output, so a
// literal needle still behaves the same. Nothing else differs.
self.matcher.literal_searchers().is_some()
&& !self.config.invert_match
&& !self.config.word_regexp
&& !self.config.line_regexp
&& !self.config.only_matching
&& !self.config.use_color
// `has_context` also covers `-C 0`, which still emits `--` separators.
&& !self.config.has_context
&& !self.config.null_data
&& self.config.binary_mode != BinaryMode::WithoutMatch
}
/// Buffer-at-a-time driver for literal patterns. Instead of testing every
/// line, it scans whole chunks with a substring searcher and only locates
/// line boundaries around the matches it finds.
fn session_run_fast(
&mut self,
lb: &mut LineBuffer,
path: &Path,
reader: &mut File,
) -> io::Result<bool> {
lb.reset();
if self.config.quiet
|| self.config.files_with_matches
|| self.config.files_without_match
|| self.config.count
{
self.fast_locate(lb, path, reader)
} else {
self.fast_print(lb, path, reader)
}
}
/// Fast path for modes that only need to know *whether* / *how many* lines
/// match: `-c`, `-l`, `-L`, `-q`. No per-line rendering, so no line numbers,
/// byte offsets, or binary bookkeeping are required (the count of matching
/// lines is unaffected by binary detection, and `-l`/`-L`/`-q` list files
/// regardless).
fn fast_locate(
&mut self,
lb: &mut LineBuffer,
path: &Path,
reader: &mut File,
) -> io::Result<bool> {
let finders = self
.matcher
.literal_searchers()
.expect("eligibility guarantees literal searchers");
let max = self.config.max_count;
// Existence is enough for these three; only `-c` needs the full tally.
let stop_at_first =
self.config.quiet || self.config.files_with_matches || self.config.files_without_match;
let mut count: u64 = 0;
let mut matched = false;
'outer: while let Some((chunk, _)) = lb.read_chunk(reader)? {
let mut p = 0;
while p < chunk.len() {
let Some(rel) = leftmost_match(finders, &chunk[p..]) else {
break;
};
if max.is_some_and(|mx| count >= mx) {
break 'outer;
}
let (_, line_end) = line_bounds(chunk, p + rel);
count += 1;
matched = true;
if stop_at_first {
break 'outer;
}
// Each line counts once: resume past this line's terminator.
p = line_end + 1;
}
}
// `-l`/`-L` take precedence over `-c`, matching the line-at-a-time path.
if self.config.quiet {
// Exit status only.
} else if self.config.files_with_matches {
if matched {
self.writer.write_filename(path)?;
}
} else if self.config.files_without_match {
if !matched {
self.writer.write_filename(path)?;
}
} else if self.config.count {
self.writer.write_count(count, path)?;
}
Ok(matched)
}
/// Fast path that prints whole matching lines (optionally with `-n`, `-b`,
/// filename prefixes, `-m`). Binary files are detected per chunk and reported
/// with the usual notice instead of dumping their lines.
fn fast_print(
&mut self,
lb: &mut LineBuffer,
path: &Path,
reader: &mut File,
) -> io::Result<bool> {
let finders = self
.matcher
.literal_searchers()
.expect("eligibility guarantees literal searchers");
let max = self.config.max_count;
let want_lineno = self.config.line_number;
let detect_binary = self.config.binary_mode != BinaryMode::Text;
let notice_enabled = self.binary_notice_enabled;
let mut count: u64 = 0;
let mut matched = false;
let mut binary = false;
// Number of terminators in all previously consumed chunks (for `-n`).
let mut base_lines: u64 = 0;
'outer: while let Some((chunk, chunk_off)) = lb.read_chunk(reader)? {
let mut p = 0;
// NUL scanned up to here; terminators counted up to `nl_cursor`.
let mut nul_scanned = 0;
let mut nl_cursor = 0;
let mut nl_before = 0u64;
while p < chunk.len() {
let Some(rel) = leftmost_match(finders, &chunk[p..]) else {
break;
};
if max.is_some_and(|mx| count >= mx) {
break 'outer;
}
let (line_beg, line_end) = line_bounds(chunk, p + rel);
// A NUL anywhere up to this line marks the file binary, as does
// an invalid-UTF-8 matching line.
if detect_binary && !binary {
if memchr(0, &chunk[nul_scanned..line_end]).is_some() {
binary = true;
}
nul_scanned = line_end;
}
let line = &chunk[line_beg..line_end];
#[cfg(windows)]
let line = if self.config.strip_cr && line.last() == Some(&b'\r') {
&line[..line.len() - 1]
} else {
line
};
if detect_binary && !binary && std::str::from_utf8(line).is_err() {
binary = true;
}
if binary {
// First match in a binary file: stop and emit the notice
// once at the end instead of dumping the line.
matched = true;
break 'outer;
}
let line_number = if want_lineno {
nl_before += count_terminators(&chunk[nl_cursor..line_beg]);
nl_cursor = line_beg;
base_lines + nl_before + 1
} else {
0
};
self.writer.write_line(
&LineView {
line,
line_number,
byte_offset: chunk_off + line_beg as u64,
is_match: true,
match_positions: &[],
},
path,
)?;
count += 1;
matched = true;
p = line_end + 1;
}
// Carry NUL detection and the line tally across the chunk boundary.
if detect_binary && !binary && memchr(0, &chunk[nul_scanned..]).is_some() {
binary = true;
}
if want_lineno {
base_lines += nl_before + count_terminators(&chunk[nl_cursor..]);
}
}
if binary && notice_enabled && matched {
self.writer.report_binary_match(path);
}
Ok(matched)
}
fn session_run(
&mut self,
lb: &mut LineBuffer,
path: &Path,
reader: &mut File,
) -> io::Result<bool> {
if self.eligible_for_fast_path() {
return self.session_run_fast(lb, path, reader);
}
// Reset all session (per-file) state.
self.session_context_buf.clear();
self.session_match_count = 0;
@@ -465,3 +685,31 @@ impl<'a> Searcher<'a> {
}
}
}
/// Offset of the earliest occurrence of any needle in `hay`, or `None`.
fn leftmost_match(finders: &[Finder<'static>], hay: &[u8]) -> Option<usize> {
let mut best: Option<usize> = None;
for finder in finders {
if let Some(pos) = finder.find(hay) {
best = Some(best.map_or(pos, |b| b.min(pos)));
if best == Some(0) {
break; // Can't start any earlier.
}
}
}
best
}
/// Count line terminators in `bytes`.
fn count_terminators(bytes: &[u8]) -> u64 {
memchr_iter(b'\n', bytes).count() as u64
}
/// Byte range `[start, end)` of the line containing `pos` in `buf`, excluding
/// the trailing terminator. `start` follows the previous terminator (or 0);
/// `end` is the next terminator (or end of buffer).
fn line_bounds(buf: &[u8], pos: usize) -> (usize, usize) {
let start = memrchr(b'\n', &buf[..pos]).map_or(0, |i| i + 1);
let end = memchr(b'\n', &buf[pos..]).map_or(buf.len(), |i| pos + i);
(start, end)
}
+381
View File
@@ -126,6 +126,46 @@ fn ere_invalid_pattern_is_error() {
.stderr_contains("invalid pattern");
}
#[test]
fn quiet_match_overrides_file_error() {
// With -q, a match makes grep exit 0 even if an earlier file could not be
// opened. Without -q the missing file still yields exit 2, and -q with no
// match keeps the error status.
let (_s, mut c) = ucmd();
c.args(&["-q", "abc", "no-such-file", "-"])
.pipe_in("abcd\n")
.succeeds()
.no_output();
let (_s, mut c) = ucmd();
c.args(&["abc", "no-such-file", "-"])
.pipe_in("abcd\n")
.fails_with_code(2);
let (_s, mut c) = ucmd();
c.args(&["-q", "zzz", "no-such-file", "-"])
.pipe_in("abcd\n")
.fails_with_code(2);
}
#[test]
fn initial_tab_skips_empty_lines() {
// -T aligns content with a tab, but GNU omits the tab for an empty line
// (a whitespace-only line still gets one). -H forces the filename prefix
// on, so the tab is exercised.
let (s, mut c) = ucmd();
s.fixtures.write("in", "x\n\n");
c.args(&["-T", "-H", "^", "in"])
.succeeds()
.stdout_is("in:\tx\nin:\n");
let (s, mut c) = ucmd();
s.fixtures.write("in", "x\n \n");
c.args(&["-T", "-H", "^", "in"])
.succeeds()
.stdout_is("in:\tx\nin:\t \n");
}
#[test]
fn fixed_string_is_literal() {
// Metacharacters are not interpreted.
@@ -170,6 +210,34 @@ fn pcre_features() {
.stdout_only("42\n7\n");
}
#[test]
fn perl_regexp_rejects_multiple_patterns() {
// GNU grep's PCRE backend (-P) only supports a single pattern.
// Multiple -e patterns must produce exit 2 and the canonical error message.
// See: https://github.com/uutils/grep/issues/34
// Two separate -e flags.
let (_s, mut c) = ucmd();
c.args(&["-P", "-e", "foo", "-e", "bar"])
.pipe_in("foo\nbar\n")
.fails_with_code(2)
.stderr_contains("the -P option only supports a single pattern");
// A newline inside the pattern string is split into multiple patterns.
let (_s, mut c) = ucmd();
c.args(&["-P", "-e", "foo\nbar"])
.pipe_in("foo\nbar\n")
.fails_with_code(2)
.stderr_contains("the -P option only supports a single pattern");
// A single pattern with -P must still work normally.
let (_s, mut c) = ucmd();
c.args(&["-P", "-e", r"\d+"])
.pipe_in("abc\n42\n")
.succeeds()
.stdout_only("42\n");
}
#[test]
fn posix_character_classes() {
let (_s, mut c) = ucmd();
@@ -251,6 +319,12 @@ fn word_regexp() {
.pipe_in("foo bar\nfoobar\n")
.succeeds()
.stdout_only("foo bar\n");
let (_s, mut c) = ucmd();
c.args(&["-w", "$"])
.pipe_in("abc\n\nx\n")
.succeeds()
.stdout_only("\n");
}
#[test]
@@ -260,6 +334,12 @@ fn line_regexp() {
.pipe_in("foo bar\nfoo bar!\nx foo bar\n")
.succeeds()
.stdout_only("foo bar\n");
let (_s, mut c) = ucmd();
c.args(&["-x", "$"])
.pipe_in("abc\n\nx\n")
.succeeds()
.stdout_only("\n");
}
#[test]
@@ -440,6 +520,17 @@ fn files_with_and_without_matches() {
.fails_with_code(1)
.stdout_is("hit\nmiss\n");
// -L with a pattern that DOES match in one file: the matching file is
// excluded from the listing, so only the non-matching file is printed.
// This exercises the early-return in `session_handle_match` taken when
// `files_without_match` is set and a match is found (src/searcher.rs).
let (scene, mut c) = ucmd();
scene.fixtures.write("hit", "yes\n");
scene.fixtures.write("miss", "no\n");
c.args(&["-L", "yes", "hit", "miss"])
.succeeds()
.stdout_is("miss\n");
// -l early-exits after the first match. Verify it doesn't print twice.
// when the file has many.
let (scene, mut c) = ucmd();
@@ -449,6 +540,25 @@ fn files_with_and_without_matches() {
.stdout_only("many\n");
}
#[test]
fn files_with_and_without_matches_mutually_exclusive() {
// Test that -l and -L are mutually exclusive with last-one-wins semantics
let (scene, mut c) = ucmd();
scene.fixtures.write("file", "match\n");
// -l -L: last flag (-L) wins, so no output (file has match, -L excludes it)
c.args(&["-l", "-L", "match", "file"])
.succeeds()
.stdout_only("");
// -L -l: last flag (-l) wins, so filename is printed
let (scene, mut c) = ucmd();
scene.fixtures.write("file", "match\n");
c.args(&["-L", "-l", "match", "file"])
.succeeds()
.stdout_only("file\n");
}
#[test]
fn count_combined_with_listing_flags() {
let (scene, _) = ucmd();
@@ -615,6 +725,15 @@ fn line_number_and_byte_offset_prefixes() {
.pipe_in("x\n")
.succeeds()
.stdout_contains("1:\tx\n");
// -T against a real file (not stdin): the line-number field width is
// derived from the file size, which only happens on the `File` path in
// `process_file` (src/searcher.rs), not the stdin path.
let (scene, mut c) = ucmd();
scene.fixtures.write("f", "x\n");
c.args(&["-T", "-n", "x", "f"])
.succeeds()
.stdout_contains("1:\tx\n");
}
#[test]
@@ -883,6 +1002,25 @@ fn recursive_no_file_defaults_to_cwd_not_stdin() {
.stdout_contains("a.txt");
}
#[test]
fn recursive_implicit_cwd_strips_dot_prefix() {
// `-r` with no path argument searches the implicit ".", so reported paths
// come back as "./only.txt" (or ".\\only.txt" on Windows). GNU strips that
// leading prefix; `strip_dot_prefix` in src/searcher.rs must do the same.
// A single file keeps the output deterministic and lets us assert the exact
// line, which `stdout_contains` in `recursive_no_file_defaults_to_cwd_not_stdin`
// cannot (it would also pass with a leaked "./" prefix).
let (scene, _) = ucmd();
scene.fixtures.mkdir_all("flat");
scene.fixtures.write("flat/only.txt", "grep me\n");
let mut c = scene.cmd(env!("CARGO_BIN_EXE_grep"));
c.current_dir(scene.fixtures.plus("flat"))
.args(&["-r", "grep"])
.succeeds()
.stdout_is("only.txt:grep me\n");
}
#[test]
fn recursive_with_include_exclude() {
let (scene, _) = ucmd();
@@ -1022,6 +1160,30 @@ fn recursive_skips_fifos_by_default() {
.stdout_does_not_contain("fifo");
}
#[cfg(unix)]
#[test]
fn device_skip_on_explicit_special_file_arg() {
use std::process::Command;
// A special file (FIFO) named *directly* as an argument, not via recursion.
// With `-D skip` it must be dropped without reading (reading would block
// forever, so the test returning at all proves it was skipped). This covers
// the top-level special-file branch in `process_path` and `is_special_file`
// (src/searcher.rs), distinct from the recursive FIFO path.
let (scene, _) = ucmd();
let fifo_path = scene.fixtures.plus("fifo");
let status = Command::new("mkfifo")
.arg(&fifo_path)
.status()
.expect("mkfifo failed");
assert!(status.success(), "could not create FIFO");
let mut c = scene.cmd(env!("CARGO_BIN_EXE_grep"));
c.args(&["-D", "skip", "grep", "fifo"])
.fails_with_code(1)
.no_output();
}
#[test]
fn nonexistent_file_is_error() {
let (_s, mut c) = ucmd();
@@ -1030,6 +1192,23 @@ fn nonexistent_file_is_error() {
.stderr_contains("does-not-exist");
}
#[test]
fn nonexistent_file_error_has_no_os_error_suffix() {
// GNU prints "grep: <file>: No such file or directory" with no
// " (os error 2)" suffix; strip_errno keeps us byte-compatible. The
// underlying OS message text differs on Windows, but in both cases the
// trailing " (os error N)" must be absent.
#[cfg(not(windows))]
let expected = "grep: does-not-exist: No such file or directory\n";
#[cfg(windows)]
let expected = "grep: does-not-exist: The system cannot find the file specified.\n";
let (_s, mut c) = ucmd();
c.args(&["x", "does-not-exist"])
.fails_with_code(2)
.stderr_is(expected);
}
#[test]
fn dash_argument_means_stdin() {
let (_s, mut c) = ucmd();
@@ -1138,3 +1317,205 @@ fn help_and_version() {
.succeeds()
.stdout_contains(env!("CARGO_PKG_VERSION"));
}
#[test]
fn repeated_options_are_accepted() {
// GNU grep tolerates options given more than once: boolean flags are
// idempotent and value options take the last occurrence. clap would
// otherwise error with "cannot be used multiple times".
// Repeated boolean flags are a no-op (not an error).
let (_s, mut c) = ucmd();
c.args(&["-n", "-n", "a"])
.pipe_in("abc\n")
.succeeds()
.stdout_only("1:abc\n");
// Mixed repeated booleans behave like a single occurrence.
let (_s, mut c) = ucmd();
c.args(&["-i", "-i", "abc"])
.pipe_in("ABC\n")
.succeeds()
.stdout_only("ABC\n");
// Repeated value options take the last value (here: -m 1 wins).
let (_s, mut c) = ucmd();
c.args(&["-m", "5", "-m", "1", "x"])
.pipe_in("x\nx\nx\n")
.succeeds()
.stdout_only("x\n");
// -e (ArgAction::Append) must still accumulate every pattern.
let (_s, mut c) = ucmd();
c.args(&["-e", "a", "-e", "b"])
.pipe_in("a\nb\nc\n")
.succeeds()
.stdout_only("a\nb\n");
}
#[test]
fn literal_buffer_path_prefixes_and_max() {
// Plain literals are served by the buffer-at-a-time engine; the line/byte
// prefixes and -m must still be byte-identical to the line-at-a-time path.
// -n and -b together: "lineno:byteoffset:line".
let (_s, mut c) = ucmd();
c.args(&["-nb", "foo"])
.pipe_in("foo\nbar\nfoobar\n")
.succeeds()
.stdout_only("1:0:foo\n3:8:foobar\n");
// A line matched more than once is still emitted once.
let (_s, mut c) = ucmd();
c.args(&["-c", "oo"])
.pipe_in("oooo\nbar\noo\n")
.succeeds()
.stdout_only("2\n");
// -m caps printed matches.
let (_s, mut c) = ucmd();
c.args(&["-m", "2", "x"])
.pipe_in("x\ny\nx\nz\nx\n")
.succeeds()
.stdout_only("x\nx\n");
// Final line without a trailing terminator still matches and is printed
// with an added newline.
let (_s, mut c) = ucmd();
c.args(&["foo"])
.pipe_in("bar\nfoo")
.succeeds()
.stdout_only("foo\n");
}
#[test]
fn literal_buffer_path_spans_many_chunks() {
// Build an input far larger than the read buffer so the buffer-at-a-time
// engine crosses several chunk boundaries, and check that line numbers and
// counts stay correct across them.
let mut input = String::new();
let mut expected_n = String::new();
let mut count = 0u32;
for i in 1..=100_000u32 {
if i % 7 == 0 {
input.push_str("needle\n");
expected_n.push_str(&format!("{i}:needle\n"));
count += 1;
} else {
input.push_str("some filler text\n");
}
}
assert!(input.len() > 512 * 1024, "input must exceed several chunks");
let (_s, mut c) = ucmd();
c.args(&["-c", "needle"])
.pipe_in(input.clone())
.succeeds()
.stdout_only(format!("{count}\n"));
let (_s, mut c) = ucmd();
c.args(&["-n", "needle"])
.pipe_in(input)
.succeeds()
.stdout_only(expected_n);
}
// Plain literals run on the buffer-at-a-time fast path, so the following tests
// use bracket-class patterns (non-literal) to keep the line-at-a-time engine's
// `-l` / `-L` / `-q` and binary-handling paths exercised too.
#[test]
fn slow_path_list_and_quiet_modes() {
let (scene, _) = ucmd();
scene.fixtures.write("hit", "yes\n");
scene.fixtures.write("miss", "no\n");
// -l: list matching files.
scene
.cmd(env!("CARGO_BIN_EXE_grep"))
.args(&["-l", "[y]es", "hit", "miss"])
.succeeds()
.stdout_is("hit\n");
// -L with a match in one file: only the non-matching file is listed.
scene
.cmd(env!("CARGO_BIN_EXE_grep"))
.args(&["-L", "[y]es", "hit", "miss"])
.succeeds()
.stdout_is("miss\n");
// -L with no match anywhere: both files listed, exit 1.
scene
.cmd(env!("CARGO_BIN_EXE_grep"))
.args(&["-L", "[z]z", "hit", "miss"])
.fails_with_code(1)
.stdout_is("hit\nmiss\n");
// -q stops at the first match (exit 0) or reports no match (exit 1).
scene
.cmd(env!("CARGO_BIN_EXE_grep"))
.args(&["-q", "[y]es", "hit"])
.succeeds()
.no_output();
scene
.cmd(env!("CARGO_BIN_EXE_grep"))
.args(&["-q", "[z]z", "hit"])
.fails_with_code(1)
.no_output();
}
#[test]
fn slow_path_binary_handling() {
let (scene, _) = ucmd();
// NOTE: avoid the name "nul" here — it's a reserved device name on Windows,
// so writing/reading it hits the null device instead of a real file.
scene.fixtures.write_bytes("nulbin", b"hit\0\n");
scene.fixtures.write_bytes("bad", b"a\x9d\n");
// Binary notice on the line-at-a-time engine (regex pattern).
scene
.cmd(env!("CARGO_BIN_EXE_grep"))
.args(&["[h]it", "nulbin"])
.succeeds()
.no_stdout()
.stderr_contains("binary file matches");
// -a forces text mode: the NUL line is printed verbatim.
scene
.cmd(env!("CARGO_BIN_EXE_grep"))
.args(&["-a", "[h]it", "nulbin"])
.succeeds()
.stdout_is_bytes(b"hit\0\n");
// --binary-files=without-match bails out on an invalid-UTF-8 match.
scene
.cmd(env!("CARGO_BIN_EXE_grep"))
.args(&["--binary-files=without-match", "[a]", "bad"])
.fails_with_code(1)
.no_output();
// A NUL after the matched line means binariness is discovered at EOF, so
// the line is printed first and the notice is emitted during finalization.
scene.fixtures.write_bytes("late", b"hit\nno\0\n");
scene
.cmd(env!("CARGO_BIN_EXE_grep"))
.args(&["[h]it", "late"])
.succeeds()
.stdout_is("hit\n")
.stderr_contains("binary file matches");
}
#[test]
fn fast_path_binary_detected_after_a_printed_line() {
// A NUL that appears only after the last match in the buffer marks the file
// binary on the fast path *after* an earlier match was already printed: the
// printed line stays and the trailing notice is still emitted.
let (scene, _) = ucmd();
scene.fixtures.write_bytes("b", b"hit\nno\0\n");
scene
.cmd(env!("CARGO_BIN_EXE_grep"))
.args(&["hit", "b"])
.succeeds()
.stdout_is("hit\n")
.stderr_contains("binary file matches");
}
+173
View File
@@ -0,0 +1,173 @@
#!/usr/bin/env python3
"""
Compare the current GNU test results to the last results gathered from the main branch to
highlight if a PR is making the results better/worse.
Don't exit with error code if all failing tests are in the ignore-intermittent.txt list.
"""
import json
import sys
import argparse
from pathlib import Path
def load_ignore_list(ignore_file):
"""Load list of intermittent test names to ignore from file."""
ignore_set = set()
if ignore_file and Path(ignore_file).exists():
with open(ignore_file, "r") as f:
for line in f:
line = line.strip()
if line and not line.startswith("#"):
ignore_set.add(line)
return ignore_set
def extract_test_results(json_data):
"""Extract test results from JSON data."""
if not json_data or "summary" not in json_data:
return {"total": 0, "passed": 0, "failed": 0, "skipped": 0}, []
summary = json_data["summary"]
tests = json_data.get("tests", [])
# Extract failed test names
failed_tests = []
for test in tests:
if test.get("status") == "FAIL":
failed_tests.append(test.get("name", "unknown"))
return summary, failed_tests
def compare_results(current_file, reference_file, ignore_file=None, output_file=None):
"""Compare current results with reference results."""
# Load ignore list
ignore_set = load_ignore_list(ignore_file)
# Load JSON files
try:
with open(current_file, "r") as f:
current_data = json.load(f)
current_summary, current_failed = extract_test_results(current_data)
except Exception as e:
print(f"Error loading current results: {e}")
return 1
try:
with open(reference_file, "r") as f:
reference_data = json.load(f)
reference_summary, reference_failed = extract_test_results(reference_data)
except Exception as e:
print(f"Error loading reference results: {e}")
return 1
# Calculate differences
pass_diff = int(current_summary.get("passed", 0)) - int(
reference_summary.get("passed", 0)
)
fail_diff = int(current_summary.get("failed", 0)) - int(
reference_summary.get("failed", 0)
)
total_diff = int(current_summary.get("total", 0)) - int(
reference_summary.get("total", 0)
)
# Find new failures and improvements
current_failed_set = set(current_failed)
reference_failed_set = set(reference_failed)
new_failures = current_failed_set - reference_failed_set
improvements = reference_failed_set - current_failed_set
# Filter out intermittent failures
non_intermittent_new_failures = new_failures - ignore_set
# Check if results are identical (no changes)
no_changes = (
pass_diff == 0
and fail_diff == 0
and total_diff == 0
and not new_failures
and not improvements
)
# If no changes, write empty output to prevent comment posting
if no_changes:
with open(output_file, "w") as f:
f.write("")
return 0
# Prepare output message
output_lines = []
# Show current vs reference numbers for debugging
output_lines.append("Test results comparison:")
output_lines.append(
f" Current: TOTAL: {current_summary.get('total', 0)} / PASSED: {current_summary.get('passed', 0)} / FAILED: {current_summary.get('failed', 0)} / SKIPPED: {current_summary.get('skipped', 0)}"
)
output_lines.append(
f" Reference: TOTAL: {reference_summary.get('total', 0)} / PASSED: {reference_summary.get('passed', 0)} / FAILED: {reference_summary.get('failed', 0)} / SKIPPED: {reference_summary.get('skipped', 0)}"
)
output_lines.append("")
# Summary of changes
if pass_diff != 0 or fail_diff != 0 or total_diff != 0:
output_lines.append("Changes from main branch:")
output_lines.append(f" TOTAL: {total_diff:+d}")
output_lines.append(f" PASSED: {pass_diff:+d}")
output_lines.append(f" FAILED: {fail_diff:+d}")
output_lines.append("")
# New failures
if new_failures:
output_lines.append(f"New test failures ({len(new_failures)}):")
for test in sorted(new_failures):
if test in ignore_set:
output_lines.append(f" - {test} (intermittent)")
else:
output_lines.append(f" - {test}")
output_lines.append("")
# Improvements
if improvements:
output_lines.append(f"Test improvements ({len(improvements)}):")
for test in sorted(improvements):
output_lines.append(f" + {test}")
output_lines.append("")
# Write output
output_text = "\n".join(output_lines)
if output_file:
with open(output_file, "w") as f:
f.write(output_text)
else:
print(output_text)
# Return appropriate exit code
if non_intermittent_new_failures:
print(
f"ERROR: Found {len(non_intermittent_new_failures)} new non-intermittent test failures"
)
return 1
return 0
def main():
parser = argparse.ArgumentParser(description="Compare GNU test results")
parser.add_argument("current", help="Current test results JSON file")
parser.add_argument("reference", help="Reference test results JSON file")
parser.add_argument(
"--ignore-file", help="File containing intermittent test names to ignore"
)
parser.add_argument("--output", help="Output file for comparison results")
args = parser.parse_args()
return compare_results(args.current, args.reference, args.ignore_file, args.output)
if __name__ == "__main__":
sys.exit(main())
+17
View File
@@ -0,0 +1,17 @@
#!/bin/bash -e
# This file is part of the uutils grep package.
#
# For the full copyright and license information, please view the LICENSE
# file that was distributed with this source code.
#
# Download and extract the upstream GNU grep release tarball into the current
# directory. Run it from an (empty) directory that will hold the GNU grep tree,
# e.g.:
#
# mkdir -p ../gnu.grep && (cd ../gnu.grep && bash ../grep/util/fetch-gnu.sh)
#
# The extracted tree ships a ready-to-use gnulib test framework under tests/
# (init.sh + init.cfg + the extensionless test scripts), which
# util/run-gnu-testsuite.sh drives against the Rust grep binary.
ver="3.12"
curl -L "https://ftp.gnu.org/gnu/grep/grep-${ver}.tar.xz" | tar --strip-components=1 -xJf -
+359
View File
@@ -0,0 +1,359 @@
#!/bin/bash
# This file is part of the uutils grep package.
#
# For the full copyright and license information, please view the LICENSE
# file that was distributed with this source code.
#
# Run the upstream GNU grep testsuite against the Rust grep implementation.
#
# Unlike GNU coreutils, we do *not* build GNU grep here. Instead we reuse the
# gnulib test framework (tests/init.sh + tests/init.cfg) shipped in the GNU grep
# release tarball and inject our Rust `grep` binary via PATH, replicating the
# environment that tests/Makefile.am's TESTS_ENVIRONMENT would normally set up.
# Each test is classified by its gnulib exit code: 0 = PASS, 77 = SKIP, anything
# else = FAIL (timeouts and framework failures count as FAIL).
#
# Get the GNU grep sources with:
# mkdir -p ../gnu.grep && (cd ../gnu.grep && bash ../grep/util/fetch-gnu.sh)
#
# Usage: ./util/run-gnu-testsuite.sh [options]
#
# Options:
# -h, --help Show this help message
# -v, --verbose Show diagnostics for failing/skipped tests
# -q, --quiet Only print failures and the final summary
# --json-output FILE Write results to FILE as JSON
#
# Environment variables:
# GNU_GREP_DIR Path to the extracted GNU grep source tree
# (default: ../gnu.grep)
# RUN_EXPENSIVE_TESTS Set to "yes" to run expensive tests (default: no)
# PER_TEST_TIMEOUT Per-test timeout in seconds (default: 30)
# Don't exit on failure since test failures are expected.
set -o pipefail
# Configuration
RUST_GREP_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
GNU_GREP_DIR="${GNU_GREP_DIR:-${RUST_GREP_DIR}/../gnu.grep}"
GNU_TESTS_DIR=""
VERBOSE=false
QUIET=false
JSON_OUTPUT_FILE=""
PER_TEST_TIMEOUT="${PER_TEST_TIMEOUT:-30}"
DETAILED_RESULTS=()
# Statistics
TOTAL_TESTS=0
PASSED_TESTS=0
FAILED_TESTS=0
SKIPPED_TESTS=0
usage() {
echo "Usage: $0 [options]"
echo
echo "Options:"
echo " -h, --help Show this help message"
echo " -v, --verbose Show diagnostics for failing/skipped tests"
echo " -q, --quiet Only print failures and the final summary"
echo " --json-output FILE Write results to FILE as JSON"
echo
echo "Environment variables:"
echo " GNU_GREP_DIR Path to the extracted GNU grep source tree"
echo " (default: ../gnu.grep)"
echo " RUN_EXPENSIVE_TESTS Set to 'yes' to run expensive tests"
echo " PER_TEST_TIMEOUT Per-test timeout in seconds (default: 30)"
echo
echo "Setup:"
echo " mkdir -p ../gnu.grep && (cd ../gnu.grep && bash ../grep/util/fetch-gnu.sh)"
}
log_info() { [[ "$QUIET" != "true" ]] && echo "[INFO] $1"; return 0; }
log_success() { [[ "$QUIET" != "true" ]] && echo "[PASS] $1"; return 0; }
log_skip() { [[ "$QUIET" != "true" ]] && echo "[SKIP] $1"; return 0; }
log_warning() { echo "[WARN] $1"; }
log_error() { echo "[FAIL] $1"; }
# Generate JSON output (schema shared with ../sed so compare_test_results.py works).
generate_json_output() {
cd "$RUST_GREP_DIR" || return
local timestamp
timestamp=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
local rust_version
rust_version=$(cargo metadata --no-deps --format-version 1 2>/dev/null | jq -r '.packages[0].version // "unknown"')
local tests_json="[]"
if [[ ${#DETAILED_RESULTS[@]} -gt 0 ]]; then
local temp_file
temp_file=$(mktemp)
printf "%s\n" "${DETAILED_RESULTS[@]}" > "$temp_file"
tests_json=$(jq -s '.' < "$temp_file" 2>/dev/null) || tests_json="[]"
rm -f "$temp_file"
fi
jq -n \
--arg timestamp "$timestamp" \
--argjson total "$TOTAL_TESTS" \
--argjson passed "$PASSED_TESTS" \
--argjson failed "$FAILED_TESTS" \
--argjson skipped "$SKIPPED_TESTS" \
--argjson duration "$duration" \
--arg rust_version "$rust_version" \
--arg gnu_testsuite_dir "$GNU_TESTS_DIR" \
--argjson tests "$tests_json" \
'{
timestamp: $timestamp,
summary: {
total: $total,
passed: $passed,
failed: $failed,
skipped: $skipped,
duration_seconds: $duration
},
environment: {
rust_grep_version: $rust_version,
gnu_testsuite_dir: $gnu_testsuite_dir
},
tests: $tests
}' > "$JSON_OUTPUT_FILE"
log_info "JSON results written to: $JSON_OUTPUT_FILE"
}
# Parse command line arguments
while [[ $# -gt 0 ]]; do
case $1 in
-h|--help) usage; exit 0 ;;
-v|--verbose) VERBOSE=true; shift ;;
-q|--quiet) QUIET=true; shift ;;
--json-output) JSON_OUTPUT_FILE="$2"; shift 2 ;;
*) echo "Unknown argument: $1"; usage; exit 1 ;;
esac
done
# Validate environment
if [[ -d "$GNU_GREP_DIR" ]]; then
GNU_GREP_DIR="$(cd "$GNU_GREP_DIR" && pwd)"
GNU_TESTS_DIR="$GNU_GREP_DIR/tests"
fi
if [[ ! -f "$GNU_TESTS_DIR/init.sh" ]]; then
log_error "GNU grep testsuite not found at: $GNU_GREP_DIR"
log_error "Fetch it with:"
log_error " mkdir -p ${RUST_GREP_DIR}/../gnu.grep && (cd ${RUST_GREP_DIR}/../gnu.grep && bash ${RUST_GREP_DIR}/util/fetch-gnu.sh)"
exit 1
fi
if [[ ! -f "$RUST_GREP_DIR/Cargo.toml" ]]; then
log_error "Not in a Rust project directory: $RUST_GREP_DIR"
exit 1
fi
# Build the Rust grep implementation
log_info "Building Rust grep implementation..."
cd "$RUST_GREP_DIR" || exit 1
if ! cargo build --release --quiet; then
log_error "Failed to build Rust grep implementation"
exit 1
fi
RUST_GREP_BIN="$RUST_GREP_DIR/target/release/grep"
if [[ ! -x "$RUST_GREP_BIN" ]]; then
log_error "Built grep binary not found at: $RUST_GREP_BIN"
exit 1
fi
log_info "Using Rust grep binary: $RUST_GREP_BIN"
# Create a temporary work tree that mimics a GNU grep build directory.
TEST_WORK_DIR=$(mktemp -d)
trap 'rm -rf "$TEST_WORK_DIR"' EXIT
log_info "Test working directory: $TEST_WORK_DIR"
# A fake $abs_top_builddir whose src/ holds the binaries the tests expect.
BUILD_DIR="$TEST_WORK_DIR/build"
BIN_DIR="$BUILD_DIR/src"
mkdir -p "$BIN_DIR"
# grep, plus the egrep/fgrep wrappers a handful of tests rely on.
cat > "$BIN_DIR/grep" <<WRAPPER_EOF
#!/bin/sh
exec "$RUST_GREP_BIN" "\$@"
WRAPPER_EOF
cat > "$BIN_DIR/egrep" <<WRAPPER_EOF
#!/bin/sh
exec "$RUST_GREP_BIN" -E "\$@"
WRAPPER_EOF
cat > "$BIN_DIR/fgrep" <<WRAPPER_EOF
#!/bin/sh
exec "$RUST_GREP_BIN" -F "\$@"
WRAPPER_EOF
chmod +x "$BIN_DIR/grep" "$BIN_DIR/egrep" "$BIN_DIR/fgrep"
# Empty config.h: tests that probe it for build-time features just skip.
: > "$BUILD_DIR/config.h"
# get-mb-cur-max is a tiny standalone helper used by the locale require_ checks.
if [[ -f "$GNU_TESTS_DIR/get-mb-cur-max.c" ]]; then
if cc -I"$BUILD_DIR" -o "$BIN_DIR/get-mb-cur-max" "$GNU_TESTS_DIR/get-mb-cur-max.c" 2>/dev/null; then
log_info "Built get-mb-cur-max helper"
else
log_warning "Could not build get-mb-cur-max; multibyte/locale tests may skip"
fi
fi
# Replicate the PCRE_WORKS probe from tests/Makefile.am's TESTS_ENVIRONMENT.
PCRE_WORKS=0
if err=$(echo . | "$BIN_DIR/grep" -Pq . 2>&1); then
[[ -z "$err" ]] && PCRE_WORKS=1
fi
log_info "PCRE_WORKS=$PCRE_WORKS"
GREP_VERSION=$(basename "$GNU_GREP_DIR" | sed 's/^grep-//')
[[ "$GREP_VERSION" == "$(basename "$GNU_GREP_DIR")" ]] && GREP_VERSION="unknown"
HOST_TRIPLET="$(uname -m)-pc-linux-gnu"
# Record a test result (for JSON output)
record_result() {
if [[ -n "$JSON_OUTPUT_FILE" ]]; then
DETAILED_RESULTS+=("$(jq -n \
--arg name "$1" --arg status "$2" --arg error "$3" \
'{name: $name, status: $status, error: $error}')")
fi
}
# Run a single GNU testsuite script with the Rust grep on PATH.
run_gnu_test() {
local test_script="$1"
local test_name
test_name=$(basename "$test_script")
TOTAL_TESTS=$((TOTAL_TESTS + 1))
local test_output_file="$TEST_WORK_DIR/test_output_$$"
local test_exit_code=0
# When not the process-group leader (e.g. in CI), GNU timeout falls back to
# "foreground" mode and SIGTERMs the whole group on timeout. Shield the
# parent script so a single hung test doesn't take the run down.
trap '' TERM
(
cd "$TEST_WORK_DIR" || exit 99
# init.cfg refuses to run if these are set.
unset GREP_COLOR GREP_COLORS TERM CDPATH
export PATH="$BIN_DIR:$PATH"
export srcdir="$GNU_TESTS_DIR" abs_srcdir="$GNU_TESTS_DIR"
export abs_top_srcdir="$GNU_GREP_DIR" top_srcdir="$GNU_GREP_DIR"
export abs_top_builddir="$BUILD_DIR"
export CONFIG_HEADER="$BUILD_DIR/config.h"
export built_programs="grep egrep fgrep"
export AWK=awk PERL=perl SHELL=/bin/sh MAKE=make CC=cc
export LC_ALL=C MALLOC_PERTURB_=87
export VERSION="$GREP_VERSION" PACKAGE_VERSION="$GREP_VERSION"
export host_triplet="$HOST_TRIPLET"
export PCRE_WORKS="$PCRE_WORKS"
export GREP_TEST_NAME="$test_name"
export RUN_EXPENSIVE_TESTS="${RUN_EXPENSIVE_TESTS:-no}"
# fd 9 is the framework's stderr (init.cfg's stderr_fileno_=9).
if [[ "$test_name" == *.pl ]]; then
exec timeout --kill-after=5 "$PER_TEST_TIMEOUT" \
perl -w -I"$GNU_TESTS_DIR" -MCoreutils -MCuSkip "$test_script" 9>&2
else
exec timeout --kill-after=5 "$PER_TEST_TIMEOUT" \
/bin/sh "$test_script" 9>&2
fi
) </dev/null >"$test_output_file" 2>&1
test_exit_code=$?
trap - TERM
# Strip NUL bytes: some tests (e.g. z-anchor-newline) emit binary output,
# which would otherwise trigger a "ignored null byte" warning from $(...).
local test_output=""
[[ -f "$test_output_file" ]] && test_output=$(tr -d '\0' < "$test_output_file")
rm -f "$test_output_file"
# 124 = GNU timeout, 125 = uutils timeout, >=128 = killed by signal.
if [[ $test_exit_code -eq 124 || $test_exit_code -eq 125 || $test_exit_code -ge 128 ]]; then
log_error "$test_name (timeout)"
FAILED_TESTS=$((FAILED_TESTS + 1))
record_result "$test_name" "FAIL" "Test timed out after ${PER_TEST_TIMEOUT}s"
return
fi
case $test_exit_code in
0)
log_success "$test_name"
PASSED_TESTS=$((PASSED_TESTS + 1))
record_result "$test_name" "PASS" ""
;;
77)
log_skip "$test_name"
SKIPPED_TESTS=$((SKIPPED_TESTS + 1))
[[ "$VERBOSE" == "true" ]] && echo "$test_output" | head -3 | sed 's/^/ | /'
record_result "$test_name" "SKIP" "$test_output"
;;
*)
log_error "$test_name (exit $test_exit_code)"
FAILED_TESTS=$((FAILED_TESTS + 1))
[[ "$VERBOSE" == "true" ]] && echo "$test_output" | head -10 | sed 's/^/ | /'
record_result "$test_name" "FAIL" "Exit code: $test_exit_code"
;;
esac
}
# Discover the canonical test list from tests/Makefile.am's TESTS variable.
collect_tests() {
awk '
/^TESTS *\+?=/ { collect=1; sub(/^TESTS *\+?=/, "") }
collect {
line=$0
cont=sub(/\\[ \t]*$/, "", line)
n=split(line, a, /[ \t]+/)
for (i=1; i<=n; i++) if (a[i] != "") print a[i]
if (!cont) collect=0
}
' "$GNU_TESTS_DIR/Makefile.am"
}
log_info "Discovering tests from $GNU_TESTS_DIR/Makefile.am"
mapfile -t TEST_LIST < <(collect_tests | sort -u)
log_info "Found ${#TEST_LIST[@]} tests"
log_info "Starting test execution..."
start_time=$(date +%s)
for t in "${TEST_LIST[@]}"; do
[[ -z "$t" ]] && continue
test_path="$GNU_TESTS_DIR/$t"
[[ -f "$test_path" ]] || { log_warning "Listed test not found: $t"; continue; }
run_gnu_test "$test_path"
done
end_time=$(date +%s)
duration=$((end_time - start_time))
# Print summary
echo
echo "========================================="
echo "GNU grep testsuite results"
echo "========================================="
echo "Total tests: $TOTAL_TESTS"
echo "Passed: $PASSED_TESTS"
echo "Failed: $FAILED_TESTS"
echo "Skipped: $SKIPPED_TESTS"
echo "Duration: ${duration}s"
if [[ -n "$JSON_OUTPUT_FILE" ]]; then
generate_json_output
fi
if [[ $((PASSED_TESTS + FAILED_TESTS)) -gt 0 ]]; then
pass_rate=$(( (PASSED_TESTS * 100) / (PASSED_TESTS + FAILED_TESTS) ))
echo "Pass rate: ${pass_rate}%"
fi
# Mirror the script's exit convention to ../sed: nonzero if anything failed.
[[ $FAILED_TESTS -eq 0 ]] && exit 0 || exit 1