Author SHA1 Message Date
Sylvestre Ledru c099152810 sed: underline the offending script character at a terminal
When stderr is a terminal (or UUTILS_DIAG=always), quote the offending
script line and underline the faulty character, in the layout uucore's
diagnostics use. Pipes and the test suite still get the single-line
message.

Both error constructors handle this, so no call site changes.
ScriptLocation now carries the line text, captured only when diagnostics
are on and shared between commands from the same line, so semantic
errors raised after compilation can still be drawn.

Anything that cannot be drawn (non-UTF-8 script, empty line, no line
recorded) leaves the message unchanged.
2026-09-27 20:03:49 +02:00
Sylvestre Ledru 72ed709088 sed: fix panics and zero steps in ~ and , addresses
A range address that ended early made the parser panic: `1~` read past
the end of the line, and `1~p`, `1,~p` or `1,p` handed a command letter
to the address parser, which only knew how to panic on it. A missing
~step now counts as 0, and a comma with no address after it is reported
as an unexpected `,'. The address parser returns an error instead of
panicking.

A step of 0 was not handled either: `2,~0` ran to the end of the input
instead of selecting line 2 alone, and `0~0`, which can select no line,
was accepted.
2026-09-27 16:44:36 +02:00
Sylvestre Ledru 7f9ce84f34 docs: move extensions and incompatibilities to docs/src/extensions.md
Start an mdBook under docs/, laid out as in coreutils, and move the list of
extensions and incompatibilities out of the README into it. The README keeps
a pointer.

A Documentation workflow builds the book with mdbook 0.5.0, the version the
uutils website uses, whenever docs/ changes, and uploads it as an artifact.
mdbook 0.5 rejects the `multilingual` field coreutils still carries (the
website strips it before building), so the book leaves it out.
2026-09-27 11:42:09 +02:00
Sylvestre Ledru cb9135ddac prepare version 0.3.0 2026-09-27 08:14:38 +02:00
renovate[bot] 76086c8d9a Update dawidd6/action-download-artifact action to v25 2026-09-26 07:14:18 +02:00
Sichen ab2f0a82d8 in-place editing: treat each file as a separate input
sed -i with more than one file treated them as one continuous stream:
line numbers continued across files, $ matched only the last line of
the last file, and N could join lines from different files. GNU sed
documents -i as implying -s, so enable separate-file processing for
--in-place as well as --separate.

There was a second bug when automatic output was suppressed. At the end
of a file, process_file cleared a pending N action only when
!context.quiet. With -n, the action carried into the next file and
joined with its first line. Clear the action in either mode, but write
the pattern space only when automatic output is enabled.

The same bug can be reproduced without -i. With f1 containing a1, a2,
a3 and f2 containing b1, b2, b3:

sed -n -s -e 'N;s/\n/+/;p' f1 f2

Before this change it prints a1+a2, a3+b1, b2+b3. GNU prints a1+a2,
b1+b2.

Add coverage for line numbers, $, ranges, N across file boundaries,
hold space, Q, backup suffixes, and -n.
2026-09-25 15:16:35 +02:00
Jorge Polanco e5474abccf sed: implement the GNU R command (read one line from a file)
The R command reads one line at a time from a named file. All R commands
naming the same file share a single cursor so they read successive lines,
matching GNU sed. Rejected under --sandbox like the other file commands.

Includes unit and integration tests. Makes the GNU sed testsuite `cmd-R`
test pass.
2026-09-25 09:00:25 +02:00
renovate[bot] 961835e509 Update Rust crate rustix to v1.1.5 2026-09-25 07:14:17 +02:00
renovate[bot] 4a73dfb60c Update Rust crate clap_complete to v4.6.11 2026-09-25 07:13:41 +02:00
renovate[bot] a1a146a720 Update Rust crate clap to v4.6.7 2026-09-25 07:12:53 +02:00
Daniel Hofstetter c46dd6dab6 Bump cargo-dist from 0.30.3 to 0.33.0 (#556) 2026-09-21 14:17:18 +02:00
renovate[bot] fcab8f1ca7 Update Rust crate rand to v0.10.3 2026-09-21 07:34:32 +02:00
renovate[bot] cffecd151f Update Rust crate uutests to 0.12.0 2026-09-18 07:40:05 +02:00
renovate[bot] 0339313d49 Update Rust crate uucore to 0.12.0 2026-09-18 07:27:53 +02:00
renovate[bot] cffcc0c806 Update Rust crate divan to v5.0.2 2026-09-18 07:18:34 +02:00
renovate[bot] 643266bc93 Update Rust crate uufuzz to 0.12.0 2026-09-18 07:11:50 +02:00
renovate[bot] 81cf88e601 Update Rust crate console to v0.16.6 2026-09-13 09:44:23 +02:00
Daniel Hofstetter cc5e214878 Bump chacha20 from 0.10.0 to 0.10.2 (#553)
because version 0.10.0 got yanked
2026-09-09 11:32:26 +02:00
Daniel Hofstetter e4a923ac98 Bump getrandom from 0.4.2 to 0.4.3 (#554) 2026-09-09 11:31:56 +02:00
renovate[bot] 77f442ddc2 Update Rust crate uutests to 0.11.0 2026-09-01 15:18:47 +02:00
renovate[bot] dc2ec59ce7 Update Rust crate uufuzz to 0.11.0 2026-09-01 15:05:30 +02:00
renovate[bot] 08b8594355 Update Rust crate uucore to 0.11.0 2026-09-01 13:54:34 +02:00
Diomidis Spinellis f1c5813aa4 Make Unicode listing GNU-sed compatible (#537)
Adjust `l` command to list Unicode characters by default as
octal byte sequences of the corresponding UTF-8 bytes.
A new -U --uutil-extensions option allows listing them
with the common Unicode \u and \U hex escapes.
2026-08-29 23:24:09 +02:00
Sylvestre Ledru 32537dd5d9 ci: generate multibyte locales for the GNU testsuite (#539)
Without en_US.UTF-8, ru_RU.UTF-8 and el_GR.iso88597 the testsuite silently
skips eight multibyte tests, hiding both regressions and improvements.
Also make require_el_iso88597_locale_ probe locale -a instead of always
skipping.
2026-08-27 08:22:48 +02:00
Diomidis SpinellisandMukunda Katta b038f8fb2d Implement GNU W (write pattern first line) command (#531)
While at it, also fix the w command on patterns missing a newline.

Co-authored-by: Mukunda Katta <mukunda.vjcs6@gmail.com>
2026-08-26 16:29:57 +02:00
Diomidis Spinellis b39a9eaa05 Add mandelbrot.sed to integration tests and update goals
Updated README to reflect the addition of mandelbrot.sed in integration tests and improved GNU sed compatibility goals.
2026-08-23 09:37:22 +03:00
renovate[bot] 7be1f77dda Update dawidd6/action-download-artifact action to v24 2026-08-21 10:02:43 +02:00
Diomidis SpinellisandSylvestre Ledru e858951814 Add tests for non-pattern print on deletion
Co-authored-by: Sylvestre Ledru <sylvestre@debian.org>
2026-08-21 09:51:57 +02:00
xtqqczze a7a303352e fix(deps): limit the feature of sha2 (#534) 2026-08-20 21:08:37 +02:00
Diomidis Spinellis 76efad4aa1 Fix failing test under Windows (#529)
Replace `NamedTempFile` with a regular file inside a `TempDir`, so the
file is closed and can be renamed or replaced by in-place editing on
Windows. `TempDir` also simplifies cleanup by automatically removing
the file, backup, and directory when dropped.
2026-08-20 10:07:47 +02:00
Diomidis Spinellis 79e5df0df8 Document F command extension 2026-08-19 14:32:26 +03:00
Diomidis Spinellis 3f72f93c5d Support F (output filename) command (#528)
This also requires storing input_name as PathBuf to avoid lossy
conversions when outputting an OS-specific byte sequence representing
the file name.
2026-08-19 12:20:27 +02:00
renovate[bot] d837ddf398 Update Rust crate similar to v3.2.0 2026-08-17 13:53:54 +02:00
renovate[bot] b1ca27e82d Update dawidd6/action-download-artifact action to v23 2026-08-16 10:06:49 +02:00
renovate[bot] ce711ac531 Update Rust crate clap_mangen to v0.3.3 2026-08-13 08:11:59 +02:00
renovate[bot] 618a3f7c39 Update Rust crate ctor to v1.0.13 2026-08-10 08:16:11 +02:00
Sylvestre Ledru 079eb9340e Add cargo-binstall support (#520)
Point cargo-binstall at the archives published by dist in the GitHub
releases, so 'cargo binstall sed' fetches a prebuilt binary instead of
building from source.
2026-08-09 10:00:46 +02:00
Sylvestre Ledru 2ce633cb8d prepare version 0.2.0 (#519) 2026-08-09 01:57:25 +02:00
Mukunda Rao Katta 91465e566b fix: identify uutils in cli output (#435) 2026-08-09 00:55:37 +02:00
Sylvestre Ledru d842aa305e sed: fix clippy cast_possible_truncation errors (#518) 2026-08-09 00:52:26 +02:00
xtqqczze d28a1f8e73 chore(cargo): remove deprecated package authors field (#444) 2026-08-08 23:28:22 +02:00
renovate[bot] af5f8afdcb Update Rust crate clap_complete to v4.6.9 2026-08-07 08:07:56 +02:00
renovate[bot] 8f3f126816 Update Rust crate clap_mangen to v0.3.2 2026-08-07 08:05:08 +02:00
renovate[bot] c771318a7a Update Rust crate clap to v4.6.6 2026-08-07 08:03:52 +02:00
renovate[bot] 36a1cf4954 Update Rust crate uucore to 0.10.0 2026-08-06 09:02:46 +02:00
renovate[bot] 13bb82ba13 Update Rust crate clap_mangen to v0.3.1 2026-08-06 08:32:10 +02:00
renovate[bot] cb9eb1da42 Update Rust crate uufuzz to 0.10.0 2026-08-06 08:15:55 +02:00
renovate[bot] b101462d1b Update Rust crate uutests to 0.10.0 2026-08-06 08:10:31 +02:00
oech3 9f2e1a1ad4 remove sysinfo dep 2026-08-05 14:18:30 +02:00
Franklin-Qi 2b092d53b0 fast_io.rs: replace the unsafe libc::copy_file_range call by rustix (#458)
Replace the unsafe `libc::copy_file_range` call with the safe `rustix::fs::copy_file_range`.

Closes: #442
2026-08-05 13:32:45 +02:00
renovate[bot] 550d545d72 Update Rust crate similar to v3.1.2 2026-08-05 08:01:27 +02:00
Diomidis Spinellisandmukunda katta a5d219764f feat: add T command (branch on no-substitution) (#508)
* feat: add T command (branch on no-substitution)

Implements the GNU `T LABEL` command: branch to LABEL if no s///
substitution has succeeded since the last input line was read or the
last t/T was taken. It is the inverse of `t` and shares the same
address and label parsing.

Wires 'T' into get_cmd_spec, resolve_branch_targets, and the processor
branch logic, resetting the substitution flag on evaluation just like
't'. Adds integration tests (simple branch + flag-clear) and a cmd-spec
unit test.

Closes #395

Co-authored-by: Diomidis Spinellis <dds@aueb.gr>

* ci: re-trigger checks (windows runner crates.io download flake)

* Disable T in POSIX mode

---------

Co-authored-by: mukunda katta <mukunda.vjcs6@gmail.com>
2026-08-02 17:47:18 +03:00
Develandpre-commit-ci[bot] 397ab86302 Implement v command #397 (#420)
* implement version command

* add tests for version option

* fix patch comparsion

* absurd extreme comparsion fix

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* accept missing minor or patch

* guard 'v' with if posix

* add a few tests

* add test_default_version

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* check if minor is empty and push zero instead of shadowing

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2026-08-01 18:18:05 +03:00
31 changed files with 1981 additions and 980 deletions

No files matched your search

+14 -1
View File
@@ -49,6 +49,19 @@ jobs:
# Omit installing part of https://github.com/actions/runner-images/tree/main/images/ubuntu
### Locales
# Several testsuite scripts guard themselves with require_*_locale_ and
# silently skip when the locale is missing, which hides both regressions
# and improvements in the multibyte code paths.
- name: Generate the locales required by the testsuite
shell: bash
run: |
## Generate the locales required by the testsuite
sudo localedef -i en_US -f UTF-8 en_US.UTF-8
sudo localedef -i ru_RU -f UTF-8 ru_RU.UTF-8
sudo localedef -i el_GR -f ISO-8859-7 el_GR.iso88597
locale -a
### Build
- name: Build Rust sed binary
shell: bash
@@ -134,7 +147,7 @@ jobs:
persist-credentials: false
- name: Retrieve reference artifacts
uses: dawidd6/action-download-artifact@v21
uses: dawidd6/action-download-artifact@v25
continue-on-error: true
with:
workflow: GnuTests.yml
+41
View File
@@ -0,0 +1,41 @@
name: Documentation
on:
pull_request:
paths:
- 'docs/**'
- '.github/workflows/documentation.yml'
push:
branches:
- main
paths:
- 'docs/**'
- '.github/workflows/documentation.yml'
permissions:
contents: read
jobs:
build-doc:
name: Build the mdBook documentation
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v7
with:
persist-credentials: false
- name: Install mdbook
uses: taiki-e/install-action@v2
with:
tool: mdbook@0.5.0
- name: Build documentation
run: mdbook build docs
- name: Upload documentation
uses: actions/upload-artifact@v7
with:
name: documentation
path: docs/book
+1 -1
View File
@@ -64,7 +64,7 @@ jobs:
# we specify bash to get pipefail; it guards against the `curl` command
# failing. otherwise `sh` won't catch that `curl` returned non-0
shell: bash
run: "curl --proto '=https' --tlsv1.2 -LsSf https://github.com/axodotdev/cargo-dist/releases/download/v0.30.3/cargo-dist-installer.sh | sh"
run: "curl --proto '=https' --tlsv1.2 -LsSf https://github.com/axodotdev/cargo-dist/releases/download/v0.33.0/cargo-dist-installer.sh | sh"
- name: Cache dist
uses: actions/upload-artifact@v7
with:
Generated
+88 -624
View File
File diff suppressed because it is too large. Load diff
+28 -10
View File
@@ -1,12 +1,12 @@
# sed (uutils)
# * see the repository LICENSE, README, and CONTRIBUTING files for more information
# spell-checker:ignore (libs) mangen sed
# spell-checker:ignore (libs) mangen sed binstall chrono codspeed libc memchr memmap rlimit rustix textwrap uucore uutests xattr
# spell-checker:ignore (misc) clippy codegen msvc startswith uninlined uutils
[package]
name = "sed"
version = "0.1.1"
authors = ["uutils developers"]
version = "0.3.0"
license = "MIT"
description = "sed ~ implemented as universal (cross-platform) utils, written in Rust"
default-run = "sed"
@@ -21,6 +21,19 @@ rust-version = "1.88"
build = "build.rs"
# cargo-binstall: fetch the archives published by `dist` in the GitHub releases
# instead of building from source. The layout below has to stay in sync with
# `dist-workspace.toml` (targets) and the archives it produces:
# sed-<target>.tar.xz (zip on Windows), containing a sed-<target>/ directory.
# Note that the release tags carry no `v` prefix, hence `{ version }`.
[package.metadata.binstall]
pkg-url = "{ repo }/releases/download/{ version }/{ name }-{ target }.{ archive-suffix }"
bin-dir = "{ name }-{ target }/{ bin }{ binary-ext }"
pkg-fmt = "txz"
[package.metadata.binstall.overrides.x86_64-pc-windows-msvc]
pkg-fmt = "zip"
[features]
default = ["feat_common_core"]
@@ -29,7 +42,6 @@ sed = []
[workspace.dependencies]
assert_fs = "1.1.3"
bytesize = "2.0.0"
chrono = { version = "0.4.37", default-features = false, features = [
"clock",
] }
@@ -47,12 +59,13 @@ phf_codegen = "0.14.0"
predicates = "3.1.3"
rand = { version = "0.10.0" }
regex = "1.10.4"
sha2 = "0.11"
sysinfo = "0.38"
sha2 = { version = "0.11.0", default-features = false, features = ["alloc"] }
tempfile = "3.10.1"
terminal_size = "0.4.2"
textwrap = { version = "0.16.1", features = ["terminal_size"] }
uucore = { version = "0.9.0", features = ["libc"] }
unicode-width = "0.2.2"
uucore = { version = "0.12.0", features = ["diagnostics", "libc"] }
rustix = "1.1.4"
xattr = "1.3.1"
@@ -66,11 +79,12 @@ memmap2.workspace = true
phf = { workspace = true }
predicates = { workspace = true }
regex = { workspace = true }
sysinfo = { workspace = true }
tempfile = { workspace = true }
terminal_size = { workspace = true }
textwrap = { workspace = true }
unicode-width = { workspace = true }
uucore = { workspace = true }
rustix = { workspace = true }
[dev-dependencies]
assert_fs = { workspace = true }
@@ -85,7 +99,7 @@ regex = { workspace = true }
sha2 = { workspace = true }
tempfile = { workspace = true }
uucore = { workspace = true, features = ["entries", "process", "signals", "benchmark"] }
uutests = "0.9.0"
uutests = "0.12.0"
[target.'cfg(unix)'.dev-dependencies]
xattr = { workspace = true }
@@ -119,6 +133,10 @@ inherits = "release"
opt-level = "z"
strip = true
# The profile that 'dist' will build with
[profile.dist]
inherits = "release"
[lints.clippy]
all = { level = "warn", priority = -1 }
@@ -135,7 +153,7 @@ must_use_candidate = "allow" # 56
needless_raw_string_hashes = "allow" # 20
needless_pass_by_value = "allow" # 15
missing_panics_doc = "allow" # 12
cast_possible_truncation = "allow" # 7
# cast_possible_truncation = "allow" # 7
unnecessary_wraps = "allow" # 6
match_wildcard_for_single_variants = "allow" # 4
cast_sign_loss = "allow" # 4
+15 -64
View File
@@ -15,22 +15,30 @@ and other extensions.
## Status
At this state _sed_ implements all [POSIX features](https://pubs.opengroup.org/onlinepubs/9799919799/)
and can run correctly the two complex scripts of its integration tests:
[hanoi.sed](https://github.com/uutils/sed/blob/main/tests/fixtures/sed/script/hanoi.sed) (solves the Towers of Hanoi puzzle) and
and can run correctly the three complex scripts of its integration tests:
[hanoi.sed](https://github.com/uutils/sed/blob/main/tests/fixtures/sed/script/hanoi.sed) (solves the Towers of Hanoi puzzle),
[mandelbrot.sed](https://github.com/uutils/sed/blob/main/tests/fixtures/sed/script/mandelbrot.sed), (draws the Mandelbrot set) and
[math.sed](https://github.com/uutils/sed/blob/main/tests/fixtures/sed/script/math.sed) (implements an arbitrary precision integer math calculator).
The performance of this Rust implementation is now better than the GNU and FreeBSD implementations for most benchmarked cases.
Further work aims to:
* Adjust buffering on terminal output to match current implementations,
* Implement more GNU extensions,
* Improve performance where possible.
* improve GNU _sed_ compatibility, especially on the regular expression front,
* implement more GNU extensions, and
* improve performance where possible.
## Installation and Use
We provide a Linux x86_64 binary archive from the main branch at
https://github.com/uutils/sed/releases/tag/latest-commit .
If you have [cargo-binstall](https://github.com/cargo-bins/cargo-binstall),
the released binaries can be installed directly with:
```bash
cargo binstall sed
```
For other platforms, ensure you have Rust installed on your system. You can install Rust through [rustup](https://rustup.rs/).
Clone the repository and build the project using Cargo:
@@ -80,65 +88,8 @@ cargo test
```
## Extensions and incompatibilities
### Supported GNU extensions
* Command-line arguments can be specified in long (`--`) form.
* Spaces can precede a regular expression modifier.
* `I` can be used in as a synonym for the `i` (case insensitive) substitution
flag.
* `M` and `m` substitution flags allow multi-line matching.
* In addition to `\n`, other escape sequences (octal, hex, C) are supported
in the strings of the `y` command.
Under POSIX these yield undefined behavior.
* The `a`, `c`, and `i` commands do not require an initial backslash,
allow text to appear on the same line, and support escape sequences
in the specified text.
* The `a`, `i`, `=`, `l`, `q` and `r` commands support address range as an extension to POSIX.
* The substitution command replacement group `\0` is a synonym for &.
* A `Q` command (optionally followed by an exit code) quits immediately.
* The `q` command can be optionally followed by an exit code.
* The `l` command can be optionally followed by the output width.
* The `--follow-symlinks` option for in-place editing.
* The `--sandbox` option that limits potentially destructive commands.
* Address 0 can be used to specify an address range that is already
active on line 1 and can finish with the specified regular expression.
* Address steps can be specified in the form of start~step and start,~step
ranges.
* Address 0 can be used in the `r` command to prepend a file.
### Supported BSD and GNU extensions
* The second address in a range can be specified as a relative address with +N.
* In-place editing of file with the `-i` flag.
### New extensions
* Unicode characters can be specified in regular expression pattern, replacement
and transliteration sequences using `\uXXXX` or `\UXXXXXXXX` sequences.
* The `l` command lists Unicode characters using the `\uXXXX` and `\UXXXXXXXX`
sequences.
### Incompatibilities
* Similarly to GNU _sed_, input is processed as raw bytes or as valid UTF-8
(this includes 7-bit ASCII) based on the locale as specified by the
`LC_ALL`, `LC_CTYPE`, and `LANG` environment variables,
with the default being byte processing.
However, in contrast with GNU _sed_, other locales (e.g. ISO-8859-1)
are not supported. If the input is in another code page or encoding
and requires locale-specific processing (e.g. ignore/map case,
character classes), consider converting it through UTF-8 to ensure
the correct handling of locale-specific regular expressions.
This _sed_ program can also handle arbitrary byte sequences
if no part of the input requires treating it as a Rust String.
* Back-references aren't supported when input is processed as bytes
(`LC_ALL=C`).
* The command will report an error and fail if duplicate labels are found
in the script.
This matches the BSD behavior. The GNU version accepts duplicate labels.
* The last line (`$`) address is interpreted as the last non-empty line of
the last file. If files specified in subsequent arguments until the last
one are empty, then the last line condition will never be triggered.
This behavior is consistent with the
[original implementation](https://github.com/dspinellis/unix-history-repo/blob/Research-V7/usr/src/cmd/sed/sed1.c#L665).
* Labels are parsed for alphanumeric characters. The BSD version parses them
until the end of the line, preventing ; to be used as a separator.
The GNU, BSD and new extensions _sed_ supports, and where it differs from GNU
_sed_, are listed in [docs/src/extensions.md](docs/src/extensions.md).
## GNU test suite compatibility
+2 -2
View File
@@ -4,7 +4,7 @@ members = ["cargo:."]
# Config for 'dist'
[dist]
# The preferred dist version to use in CI (Cargo.toml SemVer syntax)
cargo-dist-version = "0.30.3"
cargo-dist-version = "0.33.0"
# CI backends to support
ci = "github"
# The installers to generate for each app
@@ -13,5 +13,5 @@ installers = []
targets = ["aarch64-apple-darwin", "aarch64-unknown-linux-gnu", "x86_64-apple-darwin", "x86_64-unknown-linux-gnu", "x86_64-unknown-linux-musl", "x86_64-pc-windows-msvc"]
# Which actions to run on pull requests
pr-run-mode = "plan"
# Allow dirty workflows to ignore out-of-date contents
# Skip checking whether the specified configuration files are up to date
allow-dirty = ["ci"]
+1
View File
@@ -0,0 +1 @@
book
+7
View File
@@ -0,0 +1,7 @@
[book]
language = "en"
src = "src"
title = "uutils sed Documentation"
[output.html]
git-repository-url = "https://github.com/uutils/sed/tree/main/docs/src"
+5
View File
@@ -0,0 +1,5 @@
# Summary
[Introduction](index.md)
* [Extensions and incompatibilities](extensions.md)
+84
View File
@@ -0,0 +1,84 @@
# Extensions and incompatibilities
The main goal of the project is compatibility with GNU _sed_, but _sed_ also
supports features that GNU _sed_ does not, and differs from it in a few places.
Below is a list of these extensions and incompatibilities.
## Supported GNU extensions
* Command-line arguments can be specified in long (`--`) form.
* Spaces can precede a regular expression modifier.
* `I` can be used in as a synonym for the `i` (case insensitive) substitution
flag.
* `M` and `m` substitution flags allow multi-line matching.
* In addition to `\n`, other escape sequences (octal, hex, C) are supported
in the strings of the `y` command.
Under POSIX these yield undefined behavior.
* The `a`, `c`, and `i` commands do not require an initial backslash,
allow text to appear on the same line, and support escape sequences
in the specified text.
* The `a`, `i`, `=`, `l`, `q` and `r` commands support address range as an extension to POSIX.
* The substitution command replacement group `\0` is a synonym for &.
* An `F` command outputs the name of the file currently being processed.
* A `Q` command (optionally followed by an exit code) quits immediately.
* The `q` command can be optionally followed by an exit code.
* A `W` command writes to a file the pattern's first line.
* An `R` command reads one line at a time from a file.
* The `l` command can be optionally followed by the output width.
* The `--follow-symlinks` option for in-place editing.
* The `--sandbox` option that limits potentially destructive commands.
* Address 0 can be used to specify an address range that is already
active on line 1 and can finish with the specified regular expression.
* Address steps can be specified in the form of start~step and start,~step
ranges.
* Address 0 can be used in the `r` command to prepend a file.
## Supported BSD and GNU extensions
* The second address in a range can be specified as a relative address with +N.
* In-place editing of file with the `-i` flag.
## New extensions
* Unicode characters can be specified in regular expression pattern, replacement
and transliteration sequences using `\uXXXX` or `\UXXXXXXXX` sequences.
* On a terminal, script errors also quote the script line and underline the
character at fault (`UUTILS_DIAG=always` or `never` overrides this):
```
$ sed 's/a/b/q'
sed: <script argument 1>:1:7: error: invalid substitute flag: 'q'
╭─[ <script argument 1>:1:7 ]
│
1 │ s/a/b/q
│ ─
───╯
```
## Incompatible extensions
The `-U` or `--uutil-extensions` option enables useful extensions or bug fixes
that aren't compatible with GNU sed or POSIX.
* The `l` command lists Unicode characters using the `\uXXXX` and `\UXXXXXXXX`
escapes rather than as octal UTF-8 byte sequences.
## Incompatibilities
* Similarly to GNU _sed_, input is processed as raw bytes or as valid UTF-8
(this includes 7-bit ASCII) based on the locale as specified by the
`LC_ALL`, `LC_CTYPE`, and `LANG` environment variables,
with the default being byte processing.
However, in contrast with GNU _sed_, other locales (e.g. ISO-8859-1)
are not supported. If the input is in another code page or encoding
and requires locale-specific processing (e.g. ignore/map case,
character classes), consider converting it through UTF-8 to ensure
the correct handling of locale-specific regular expressions.
This _sed_ program can also handle arbitrary byte sequences
if no part of the input requires treating it as a Rust String.
* Back-references aren't supported when input is processed as bytes
(`LC_ALL=C`).
* The command will report an error and fail if duplicate labels are found
in the script.
This matches the BSD behavior. The GNU version accepts duplicate labels.
* The last line (`$`) address is interpreted as the last non-empty line of
the last file. If files specified in subsequent arguments until the last
one are empty, then the last line condition will never be triggered.
This behavior is consistent with the
[original implementation](https://github.com/dspinellis/unix-history-repo/blob/Research-V7/usr/src/cmd/sed/sed1.c#L665).
* Labels are parsed for alphanumeric characters. The BSD version parses them
until the end of the line, preventing ; to be used as a separator.
+10
View File
@@ -0,0 +1,10 @@
# uutils sed
_sed_ is a Rust reimplementation of the
[sed utility](https://pubs.opengroup.org/onlinepubs/9799919799/utilities/sed.html)
with some [GNU sed](https://www.gnu.org/software/sed/manual/sed.html),
[FreeBSD sed](https://man.freebsd.org/cgi/man.cgi?sed(1)),
and other extensions.
It is part of the [uutils](https://uutils.github.io/) project. The source code
is on [GitHub](https://github.com/uutils/sed).
+15 -130
View File
@@ -245,9 +245,9 @@ checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570"
[[package]]
name = "console"
version = "0.16.4"
version = "0.16.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4fe5f465a4f6fee88fad41b85d990f84c835335e85b5d9e6e63e0d06d28cba7c"
checksum = "e96a4956774c13c126a8b5af4daa79384f4d826534c95a02d76afb39e2ab64e3"
dependencies = [
"encode_unicode",
"libc",
@@ -646,15 +646,6 @@ version = "0.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "61807f77802ff30975e01f4f071c8ba10c022052f98b3294119f3e615d13e5be"
[[package]]
name = "ntapi"
version = "0.4.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c3b335231dfd352ffb0f8017f3b6027a4917f7df785ea2143d8af2adc66980ae"
dependencies = [
"winapi",
]
[[package]]
name = "num-bigint"
version = "0.4.6"
@@ -683,25 +674,6 @@ dependencies = [
"autocfg",
]
[[package]]
name = "objc2-core-foundation"
version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2a180dd8642fa45cdb7dd721cd4c11b1cadd4929ce112ebd8b9f5803cc79d536"
dependencies = [
"bitflags",
]
[[package]]
name = "objc2-io-kit"
version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "33fafba39597d6dc1fb709123dfa8289d39406734be322956a69f0931c73bb15"
dependencies = [
"libc",
"objc2-core-foundation",
]
[[package]]
name = "once_cell"
version = "1.21.4"
@@ -848,9 +820,9 @@ checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf"
[[package]]
name = "rand"
version = "0.10.2"
version = "0.10.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c7f5fa3a058cd35567ef9bfa5e75732bee0f9e4c55fa90477bef2dfcdbc4be80"
checksum = "65c9fb96cbc91e3478eaae79a69fcd3f1ae4ad052e471fe6732fff548984b4af"
dependencies = [
"chacha20",
"getrandom 0.4.2",
@@ -925,7 +897,7 @@ checksum = "b39cdef0fa800fc44525c84ccb54a029961a8215f9619753635a9c0d2538d46d"
[[package]]
name = "sed"
version = "0.1.1"
version = "0.3.0"
dependencies = [
"clap",
"clap_complete",
@@ -937,7 +909,7 @@ dependencies = [
"phf_codegen",
"predicates",
"regex",
"sysinfo",
"rustix",
"tempfile",
"terminal_size",
"textwrap",
@@ -1013,9 +985,9 @@ checksum = "703d5c7ef118737c72f1af64ad2f6f8c5e1921f818cdcb97b8fe6fc69bf66214"
[[package]]
name = "similar"
version = "3.1.1"
version = "3.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e6505efef05804732ed8a3f2d4f279429eb485bd69d5b0cc6b19cc02005cda16"
checksum = "4f66ca1f7aca2474dc10c942eb22feffc897735f54cd1db90138c2fddb490987"
dependencies = [
"bstr",
]
@@ -1055,20 +1027,6 @@ dependencies = [
"unicode-ident",
]
[[package]]
name = "sysinfo"
version = "0.38.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "92ab6a2f8bfe508deb3c6406578252e491d299cbbf3bc0529ecc3313aee4a52f"
dependencies = [
"libc",
"memchr",
"ntapi",
"objc2-core-foundation",
"objc2-io-kit",
"windows",
]
[[package]]
name = "tempfile"
version = "3.27.0"
@@ -1194,9 +1152,9 @@ checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821"
[[package]]
name = "uucore"
version = "0.9.0"
version = "0.12.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "069b34217c27f611e1589f540f58118dbf226a9e407d38ab472052ff075a1dc2"
checksum = "f24a5910ebbf2c0baf486a9125705f8e026a8f7741a3ea8cf5eca3c9257c9bdf"
dependencies = [
"bigdecimal",
"clap",
@@ -1209,9 +1167,9 @@ dependencies = [
"os_display",
"procfs",
"rustc-hash",
"rustix",
"thiserror",
"unic-langid",
"unicode-width",
"uucore_procs",
"wild",
]
@@ -1233,9 +1191,9 @@ dependencies = [
[[package]]
name = "uucore_procs"
version = "0.9.0"
version = "0.12.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "34337da211e7abfff7189b794afb3b5018fe356fb36474b73645458fc1201350"
checksum = "50aa1f3bba55168888ae6f1f2714ce58897d8020d742d21f474f01e86468fedd"
dependencies = [
"proc-macro2",
"quote",
@@ -1243,9 +1201,9 @@ dependencies = [
[[package]]
name = "uufuzz"
version = "0.9.0"
version = "0.12.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8e4192b67d5e7f0a3300db353ec3217beee89442f02c11d8ae108867eee392d0"
checksum = "7147da7bd2de08e6647116e005b0992cc602e34c34972793844a45c4e8648dd9"
dependencies = [
"console",
"rand",
@@ -1361,49 +1319,6 @@ dependencies = [
"glob",
]
[[package]]
name = "winapi"
version = "0.3.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5c839a674fcd7a98952e593242ea400abe93992746761e38641405d28b00f419"
dependencies = [
"winapi-i686-pc-windows-gnu",
"winapi-x86_64-pc-windows-gnu",
]
[[package]]
name = "winapi-i686-pc-windows-gnu"
version = "0.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ac3b87c63620426dd9b991e5ce0329eff545bccbbb34f3be09ff6fb6ab51b7b6"
[[package]]
name = "winapi-x86_64-pc-windows-gnu"
version = "0.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f"
[[package]]
name = "windows"
version = "0.62.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "527fadee13e0c05939a6a05d5bd6eec6cd2e3dbd648b9f8e447c6518133d8580"
dependencies = [
"windows-collections",
"windows-core",
"windows-future",
"windows-numerics",
]
[[package]]
name = "windows-collections"
version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "23b2d95af1a8a14a3c7367e1ed4fc9c20e0a26e79551b1454d72583c97cc6610"
dependencies = [
"windows-core",
]
[[package]]
name = "windows-core"
version = "0.62.2"
@@ -1417,17 +1332,6 @@ dependencies = [
"windows-strings",
]
[[package]]
name = "windows-future"
version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e1d6f90251fe18a279739e78025bd6ddc52a7e22f921070ccdc67dde84c605cb"
dependencies = [
"windows-core",
"windows-link",
"windows-threading",
]
[[package]]
name = "windows-implement"
version = "0.60.2"
@@ -1456,16 +1360,6 @@ version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5"
[[package]]
name = "windows-numerics"
version = "0.3.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6e2e40844ac143cdb44aead537bbf727de9b044e107a0f1220392177d15b0f26"
dependencies = [
"windows-core",
"windows-link",
]
[[package]]
name = "windows-result"
version = "0.4.1"
@@ -1493,15 +1387,6 @@ dependencies = [
"windows-link",
]
[[package]]
name = "windows-threading"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3949bd5b99cafdf1c7ca86b43ca564028dfe27d66958f2470940f73d86d75b37"
dependencies = [
"windows-link",
]
[[package]]
name = "wit-bindgen"
version = "0.51.0"
+2 -2
View File
@@ -14,8 +14,8 @@ libc = "0.2.153"
tempfile = "3.15.0"
rand = { version = "0.10.0" }
similar = "3.0.0"
uucore = { version = "0.9.0", features = ["libc"] }
uufuzz = "0.9.0"
uucore = { version = "0.12.0", features = ["libc"] }
uufuzz = "0.12.0"
sed = { path = ".." }
+16 -10
View File
@@ -10,6 +10,7 @@
use crate::sed::error_handling::{ScriptLocation, runtime_error};
use crate::sed::fast_regex::{Captures, Match, Regex};
use crate::sed::named_reader::NamedReader;
use crate::sed::named_writer::NamedWriter;
use crate::sed::script_char_provider::ScriptCharProvider;
use crate::sed::script_line_provider::ScriptLineProvider;
@@ -38,10 +39,11 @@ pub struct ProcessingContext {
pub sandbox: bool,
pub unbuffered: bool,
pub null_data: bool,
pub uutil_extensions: bool,
// Other context
/// Currently processed input file name (not script) in quoted form
pub input_name: String,
/// Currently processed input file name (not script)
pub input_name: PathBuf,
/// Current input line number
pub line_number: usize,
/// True if this is the last address of a range
@@ -66,6 +68,9 @@ pub struct ProcessingContext {
pub parsed_block_nesting: usize,
/// Command associated with each label
pub label_to_command_map: HashMap<String, Rc<RefCell<Command>>>,
/// Shared `R` readers keyed by file path, so that all `R` commands naming
/// the same file advance a single cursor, matching GNU sed.
pub named_readers: HashMap<PathBuf, Rc<RefCell<NamedReader>>>,
/// Commands with a (latchable and resetable) address range
pub range_commands: Vec<Rc<RefCell<Command>>>,
/// True if a substitution was made as specified in the t command
@@ -255,12 +260,12 @@ impl Default for Transliteration {
/// Create a new Transliteration with identity mapping for the fast-path.
fn default() -> Self {
let mut fast = [0u8; 256];
for (i, slot) in fast.iter_mut().enumerate() {
*slot = i as u8;
for (slot, value) in fast.iter_mut().zip(0..=u8::MAX) {
*slot = value;
}
let mut unicode_fast = ['\0'; COMMON_UNICODE];
for (i, slot) in unicode_fast.iter_mut().enumerate() {
*slot = char::from_u32(i as u32).unwrap_or('\0');
for (slot, cp) in unicode_fast.iter_mut().zip(0u32..) {
*slot = char::from_u32(cp).unwrap_or('\0');
}
Self {
byte_fast: fast,
@@ -361,12 +366,13 @@ impl Command {
#[derive(Debug)]
/// Command-specific data
/// After parsing, t, b Label elements are converted into BranchTarget ones.
/// After parsing, t, T, b Label elements are converted into BranchTarget ones.
pub enum CommandData {
None,
BranchTarget(Option<Rc<RefCell<Command>>>), // Commands for 'b', 't', '{'
Label(Option<String>), // Label name for 'b', 't', ':'
BranchTarget(Option<Rc<RefCell<Command>>>), // Commands for 'b', 't', 'T', '{'
Label(Option<String>), // Label name for 'b', 't', 'T', ':'
Path(PathBuf), // File path for 'r'
NamedReader(Rc<RefCell<NamedReader>>), // Successive file lines for 'R'
NamedWriter(Rc<RefCell<NamedWriter>>), // File output for 'w'
Number(usize), // Number for 'l', 'q', 'Q' (GNU)
Substitution(Box<Substitution>), // Substitute command 's'
@@ -593,7 +599,7 @@ mod tests {
fn test_all_fast_path_mapped_to_space() {
let mut t = Transliteration::default();
for cp in 0..COMMON_UNICODE {
if let Some(ch) = char::from_u32(cp as u32) {
if let Some(ch) = u32::try_from(cp).ok().and_then(char::from_u32) {
t.insert(ch, ' ');
}
}
+187 -36
View File
@@ -18,6 +18,7 @@ use crate::sed::delimited_parser::{
};
use crate::sed::error_handling::{ScriptLocation, compilation_error, semantic_error};
use crate::sed::fast_regex::Regex;
use crate::sed::named_reader::NamedReader;
use crate::sed::named_writer::NamedWriter;
use crate::sed::script_char_provider::ScriptCharProvider;
use crate::sed::script_line_provider::{ScriptLineProvider, ScriptValue};
@@ -213,8 +214,8 @@ fn resolve_branch_targets(
resolve_branch_targets(Some(sub_head.clone()), context)?;
}
// Only for 't' or 'b' commands:
if matches!(cmd.code, 't' | 'b') {
// Only for 't', 'T', or 'b' commands:
if matches!(cmd.code, 't' | 'T' | 'b') {
// Take ownership of the current data
let old_data = mem::replace(&mut cmd.data, CommandData::None);
@@ -358,7 +359,7 @@ fn compile_address_range(
let is_step_match = line.current() == '~'; // E.g. 0~2: Pick even-numbered lines
line.advance();
line.eat_spaces();
let is_step_end = if line.current() == '~' {
let is_step_end = if !line.eol() && line.current() == '~' {
// E.g. /foo/,~10: Start at foo, include all lines until multiple of 10 is reached.
line.advance();
line.eat_spaces();
@@ -372,39 +373,46 @@ fn compile_address_range(
return compilation_error(lines, line, "~step is invalid in POSIX mode");
}
// Look for second address.
if !line.eol() {
let addr2 = compile_address(lines, line, context)?;
// Set step_n to the number specified in the (required numeric) address.
let step_n = if is_step_match || is_step_end {
match addr2 {
Address::Line(n) => n,
_ => {
return compilation_error(
lines,
line,
"~step can only be specified through numeric values",
);
}
}
} else {
0 // dummy, not used
};
if is_line0 && !matches!(addr2, Address::Re(_)) && !is_step_match {
return compilation_error(lines, line, ERR_ADDRESS_0_USAGE);
}
// If needed, transform Address::Line into Address::Step*.
cmd.addr2 = if is_step_match {
Some(Address::StepMatch(step_n))
} else if is_step_end {
Some(Address::StepEnd(step_n))
} else {
Some(addr2)
};
n_addr += 1;
// Look for second address; a missing ~step counts as 0.
let has_addr2 = !line.eol() && (is_address_char(line.current()) || line.current() == '+');
if !(has_addr2 || is_step_match || is_step_end) {
return compilation_error(lines, line, "unexpected `,'");
}
let addr2 = if has_addr2 {
compile_address(lines, line, context)?
} else {
Address::Line(0)
};
// Set step_n to the number specified in the (required numeric) address.
let step_n = if is_step_match || is_step_end {
match addr2 {
Address::Line(n) => n,
_ => {
return compilation_error(
lines,
line,
"~step can only be specified through numeric values",
);
}
}
} else {
0 // dummy, not used
};
// Address 0 needs a regex end or a positive ~step (0~0 selects no line).
if is_line0 && !matches!(addr2, Address::Re(_)) && !(is_step_match && step_n > 0) {
return compilation_error(lines, line, ERR_ADDRESS_0_USAGE);
}
// If needed, transform Address::Line into Address::Step*.
cmd.addr2 = if is_step_match {
Some(Address::StepMatch(step_n))
} else if is_step_end {
Some(Address::StepEnd(step_n))
} else {
Some(addr2)
};
n_addr += 1;
}
// Zero-address read command check
@@ -495,7 +503,7 @@ fn compile_address(
let number = parse_number(lines, line, true)?.unwrap();
Ok(Address::Line(number))
}
_ => panic!("invalid context address"),
_ => compilation_error(lines, line, "expected context address"),
}
}
@@ -1122,6 +1130,28 @@ fn compile_read_file_command(
Ok(CommandHandling::Continue)
}
// Handles R
fn compile_read_line_command(
lines: &mut ScriptLineProvider,
line: &mut ScriptCharProvider,
cmd: &mut Command,
context: &mut ProcessingContext,
) -> UResult<CommandHandling> {
if context.sandbox {
return compilation_error(lines, line, ERR_SANDBOX);
}
let path = read_file_path(lines, line)?;
// Reuse a reader keyed by path so all `R` commands on the same file share
// one cursor (see `ProcessingContext::named_readers`).
let reader = context
.named_readers
.entry(path.clone())
.or_insert_with(|| NamedReader::new(path))
.clone();
cmd.data = CommandData::NamedReader(reader);
Ok(CommandHandling::Continue)
}
// Handles w
fn compile_write_file_command(
lines: &mut ScriptLineProvider,
@@ -1373,6 +1403,72 @@ fn compile_text_command_posix(
Ok(CommandHandling::Continue)
}
// Handle v
fn compile_version_command(
lines: &mut ScriptLineProvider,
line: &mut ScriptCharProvider,
_cmd: &mut Command,
_context: &mut ProcessingContext,
) -> UResult<CommandHandling> {
// Claim version partify with GNU sed 4.9
const GNU_MAJOR: u8 = 4;
const GNU_MINOR: u8 = 9;
const GNU_PATCH: u8 = 0;
line.advance();
line.eat_spaces(); // Skip any leading whitespace.
let mut major = String::new();
let mut minor = String::new();
let mut patch = String::new();
let mut ver_semantic = 0;
while !line.eol() {
if line.current() == '.' {
ver_semantic += 1;
line.advance();
} else {
match ver_semantic {
0 => major.push(line.current()),
1 => minor.push(line.current()),
2 => patch.push(line.current()),
_ => return compilation_error(lines, line, "invalid version of sed"),
}
line.advance();
}
}
if major.is_empty() {
major = GNU_MAJOR.to_string();
minor = GNU_MINOR.to_string();
patch = GNU_PATCH.to_string();
}
if minor.is_empty() {
minor.push('0');
}
if patch.is_empty() {
patch.push('0');
}
match major.parse::<u8>() {
Ok(major_int) => match minor.parse::<u8>() {
Ok(minor_int) => match patch.parse::<u8>() {
Ok(patch_int) => {
if minor_int <= GNU_MINOR && major_int <= GNU_MAJOR && patch_int == GNU_PATCH {
return Ok(CommandHandling::Continue);
}
compilation_error(lines, line, "expected newer version of sed")
}
Err(_) => compilation_error(lines, line, "invalid version of sed"),
},
Err(_) => compilation_error(lines, line, "invalid version of sed"),
},
Err(_) => compilation_error(lines, line, "invalid version of sed"),
}
}
// Handles e
// With no argument, the command executes the pattern space as a shell
// command at runtime. With an argument, the rest of the line is the
@@ -1566,6 +1662,14 @@ fn get_cmd_spec(
n_addr: 2,
handler: compile_execute_command,
}),
'F' if !posix => Ok(CommandSpec {
n_addr: 2,
handler: compile_empty_command,
}),
'R' if !posix => Ok(CommandSpec {
n_addr: 2,
handler: compile_read_line_command,
}),
'r' => Ok(CommandSpec {
n_addr: if posix { 1 } else { 2 },
handler: compile_read_file_command,
@@ -1574,6 +1678,14 @@ fn get_cmd_spec(
n_addr: 2,
handler: compile_subst_command,
}),
'T' if !posix => Ok(CommandSpec {
n_addr: 2,
handler: compile_label_command,
}),
'W' if !posix => Ok(CommandSpec {
n_addr: 2,
handler: compile_write_file_command,
}),
'w' => Ok(CommandSpec {
n_addr: 2,
handler: compile_write_file_command,
@@ -1582,6 +1694,10 @@ fn get_cmd_spec(
n_addr: 2,
handler: compile_trans_command,
}),
'v' if !posix => Ok(CommandSpec {
n_addr: 0,
handler: compile_version_command,
}),
_ => compilation_error(lines, line, format!("invalid command code `{cmd_code}'")),
}
}
@@ -1665,6 +1781,16 @@ mod tests {
);
}
#[test]
fn test_lookup_branch_commands() {
// b, t, and T all share compile_label_command and accept 2 addresses.
for code in ['b', 't', 'T'] {
let (lines, line) = make_providers("123abc");
let cmd = get_cmd_spec(&lines, &line, code, false).unwrap();
assert_eq!(cmd.n_addr, 2, "command `{code}` should accept 2 addresses");
}
}
// Utility to create a ScriptCharProvider from a &str
fn char_provider_from(s: &str) -> ScriptCharProvider {
ScriptCharProvider::new(s)
@@ -2929,6 +3055,31 @@ mod tests {
assert!(err.to_string().contains(ERR_SANDBOX));
}
// compile_read_line_command (R)
#[test]
fn test_compile_read_line_command_rejected_under_sandbox() {
let (mut lines, mut chars) = make_providers("R input.txt");
let mut cmd = Command::default();
let mut context = ctx();
context.sandbox = true;
let err =
compile_read_line_command(&mut lines, &mut chars, &mut cmd, &mut context).unwrap_err();
assert!(err.to_string().contains(ERR_SANDBOX));
}
#[test]
fn test_compile_read_line_command_sets_named_reader() {
let (mut lines, mut chars) = make_providers("R input.txt");
let mut cmd = Command::default();
let mut context = ctx();
let handling =
compile_read_line_command(&mut lines, &mut chars, &mut cmd, &mut context).unwrap();
assert!(matches!(handling, CommandHandling::Continue));
assert!(matches!(cmd.data, CommandData::NamedReader(_)));
}
// compile_write_file_command
#[test]
fn test_compile_write_file_command_rejected_under_sandbox() {
+153 -8
View File
@@ -12,8 +12,14 @@ use crate::sed::command::ProcessingContext;
use crate::sed::script_char_provider::ScriptCharProvider;
use crate::sed::script_line_provider::ScriptLineProvider;
use std::env;
use std::io::{IsTerminal, stderr};
use std::rc::Rc;
use std::sync::OnceLock;
use unicode_width::UnicodeWidthChar;
use uucore::diagnostics;
use uucore::display::Quotable;
use uucore::error::{UResult, USimpleError};
#[derive(Clone, Debug)]
@@ -22,6 +28,9 @@ pub struct ScriptLocation {
pub input_name: Rc<str>, // Shared input name
pub line_number: usize, // 1-based line number
pub column_number: usize, // 1-based column number
/// The script line, for errors raised after it was consumed.
/// `None` when diagnostics are off.
pub line_text: Option<Rc<str>>,
}
impl Default for ScriptLocation {
@@ -30,6 +39,7 @@ impl Default for ScriptLocation {
input_name: Rc::from("<unknown>"),
line_number: 1,
column_number: 1,
line_text: None,
}
}
}
@@ -41,10 +51,123 @@ impl ScriptLocation {
line_number: lines.get_line_number(),
column_number: line.get_pos() + 1,
input_name: Rc::from(lines.get_input_name()),
line_text: if diagnostics_enabled() {
line.line_text()
} else {
None
},
}
}
}
/// Whether script errors get a snippet. Evaluated once: a location is
/// recorded for every compiled command, and the terminal check is a syscall.
fn diagnostics_enabled() -> bool {
static ENABLED: OnceLock<bool> = OnceLock::new();
*ENABLED.get_or_init(diagnostics::enabled)
}
/// Format `msg` as an error at the given script position, followed by the
/// offending script line, underlined, when diagnostics are enabled.
fn script_error(
input_name: &str,
line_number: usize,
column_number: usize,
line_text: Option<&str>,
msg: impl ToString,
) -> String {
let message = format!(
"{input_name}:{line_number}:{column_number}: error: {}",
msg.to_string()
);
// Line 0: the script was exhausted, so there is no line to show.
if !diagnostics_enabled() || line_number == 0 {
return message;
}
// A script line that is not valid UTF-8 cannot be drawn.
let Some(text) = line_text.filter(|text| !text.is_empty()) else {
return message;
};
// `column_number` is a 1-based byte column; errors at end of line point
// one past the last character.
let Some(prefix) = text.get(..(column_number - 1).min(text.len())) else {
return message;
};
let drawn = render_snippet(
input_name,
line_number,
column_number,
text,
prefix,
use_color(),
);
format!("{message}\n{drawn}")
}
/// Whether the underline is colored. uucore's diagnostics make the same
/// check but keep it private.
fn use_color() -> bool {
env::var_os("NO_COLOR").is_none() && stderr().is_terminal()
}
/// Draw `text` as line `line_number` of `input_name`, underlining the
/// character that follows `prefix`, in the layout uucore's diagnostics use:
///
/// ```text
/// ╭─[ <script argument 1>:1:7 ]
/// │
/// 1 │ s/a/b/q
/// │ ─
/// ───╯
/// ```
fn render_snippet(
input_name: &str,
line_number: usize,
column_number: usize,
text: &str,
prefix: &str,
color: bool,
) -> String {
let number = line_number.to_string();
let margin = " ".repeat(number.len() + 2);
// Pad by display width so the underline lines up with the character
// above it; tabs are kept as they are, since their width depends on the
// terminal.
let indent: String = prefix
.chars()
.flat_map(|c| {
let (fill, width) = if c == '\t' {
('\t', 1)
} else {
(' ', c.width().unwrap_or(0))
};
std::iter::repeat_n(fill, width)
})
.collect();
// As wide as the offending character; one column past the end of line.
let width = text[prefix.len()..]
.chars()
.next()
.and_then(UnicodeWidthChar::width)
.unwrap_or(1)
.max(1);
let underline = "─".repeat(width);
let underline = if color {
format!("\x1b[31m{underline}\x1b[0m")
} else {
underline
};
format!(
"{margin}╭─[ {input_name}:{line_number}:{column_number} ]\n\
{margin}│\n\
\x20{number} │ {text}\n\
{margin}│ {indent}{underline}\n\
{rule}╯",
rule = "─".repeat(margin.len()),
)
}
/// Fail with msg as a compile error at the provider location.
/// The error's exit code is 1 (compilation phase).
pub fn compilation_error<T>(
@@ -54,12 +177,12 @@ pub fn compilation_error<T>(
) -> UResult<T> {
Err(USimpleError::new(
1,
format!(
"{}:{}:{}: error: {}",
script_error(
lines.get_input_name(),
lines.get_line_number(),
line.get_pos() + 1,
msg.to_string()
str::from_utf8(line.get_line()).ok(),
msg,
),
))
}
@@ -69,12 +192,12 @@ pub fn compilation_error<T>(
fn location_error<T>(location: &ScriptLocation, msg: impl ToString, exit_code: i32) -> UResult<T> {
Err(USimpleError::new(
exit_code,
format!(
"{}:{}:{}: error: {}",
location.input_name,
script_error(
&location.input_name,
location.line_number,
location.column_number,
msg.to_string()
location.line_text.as_deref(),
msg,
),
))
}
@@ -108,9 +231,31 @@ pub fn input_runtime_error<T>(
location.input_name,
location.line_number,
location.column_number,
context.input_name,
context.input_name.quote(),
context.line_number,
msg.to_string()
),
))
}
#[cfg(test)]
mod tests {
use super::*;
// The integration tests check the layout; they cannot get color, as
// their stderr is not a terminal.
#[test]
fn the_underline_is_red_when_colored() {
let drawn = render_snippet("f.sed", 1, 7, "s/a/b/q", "s/a/b/", true);
assert_eq!(
drawn,
concat!(
" ╭─[ f.sed:1:7 ]\n",
" │\n",
" 1 │ s/a/b/q\n",
" │ \x1b[31m─\x1b[0m\n",
"───╯",
)
);
}
}
+156 -46
View File
@@ -29,6 +29,11 @@ use std::marker::PhantomData;
#[cfg(unix)]
use std::os::fd::RawFd;
#[cfg(unix)]
use rustix::fd::BorrowedFd;
#[cfg(target_os = "linux")]
use rustix::fs::copy_file_range as rustix_copy_file_range;
#[cfg(unix)]
use std::os::unix::io::AsRawFd;
@@ -402,9 +407,24 @@ impl FastCopy {
Self {
fd,
is_regular: ftype as libc::mode_t == libc::S_IFREG,
block_size: st.st_blksize as usize,
// st_blksize is always positive and small; fall back to a common
// block size if it ever doesn't fit in usize.
block_size: usize::try_from(st.st_blksize).unwrap_or(4096),
}
}
/// Return a borrowed file descriptor.
///
/// # Safety invariant
///
/// `FastCopy` only stores file descriptors obtained from open
/// [`File`](std::fs::File) objects. Those `File`s are kept alive
/// by the owning [`MmapLineCursor`] / output-buffer, guaranteeing
/// the fd remains valid for the duration of any borrow.
pub fn as_fd(&self) -> BorrowedFd<'_> {
// SAFETY: self.fd is a valid file descriptor owned by a live File.
unsafe { BorrowedFd::borrow_raw(self.fd) }
}
}
#[cfg(test)]
@@ -798,10 +818,10 @@ impl OutputBuffer {
if chunk.in_fast_copy.is_regular && self.fast_copy.is_regular {
portable_copy_file_range(
chunk.out_ptr,
chunk.in_fast_copy.fd,
chunk.in_fast_copy.as_fd(),
// Input file offset
unsafe { chunk.out_ptr.offset_from(chunk.base_ptr) } as libc::off_t,
self.fast_copy.fd,
self.fast_copy.as_fd(),
chunk.len,
// Alignment block size: the largest of the two
chunk.in_fast_copy.block_size.max(self.fast_copy.block_size),
@@ -916,9 +936,9 @@ fn reliable_write(fd: i32, ptr: *const u8, len: usize) -> std::io::Result<usize>
#[allow(unused_variables)]
fn portable_copy_file_range(
in_ptr: *const u8,
in_fd: i32,
in_fd: BorrowedFd<'_>,
in_off: libc::off_t,
out_fd: i32,
out_fd: BorrowedFd<'_>,
len: usize,
block_size: usize,
cover: WriteRange,
@@ -933,7 +953,7 @@ fn portable_copy_file_range(
}
#[cfg(not(all(target_os = "linux", target_env = "gnu")))]
{
reliable_write(out_fd, in_ptr, len)
reliable_write(out_fd.as_raw_fd(), in_ptr, len)
}
}
@@ -944,37 +964,34 @@ fn portable_copy_file_range(
#[cfg(all(target_os = "linux", target_env = "gnu"))]
fn reliable_copy_file_range(
in_ptr: *const u8,
in_fd: i32,
in_fd: BorrowedFd<'_>,
mut in_off: libc::off_t,
out_fd: i32,
out_fd: BorrowedFd<'_>,
len: usize,
) -> std::io::Result<usize> {
let mut pending = len;
while pending > 0 {
let ret = unsafe {
libc::copy_file_range(
in_fd,
&raw mut in_off,
out_fd,
std::ptr::null_mut(), // Use and update output offset
pending,
0,
)
};
if ret < 0 {
let err = io::Error::last_os_error();
return match err.raw_os_error() {
Some(libc::ENOSYS) | Some(libc::EOPNOTSUPP) | Some(libc::EXDEV) => {
// Fallback to write(2).
reliable_write(out_fd, in_ptr, pending)
}
_ => Err(err),
};
} else if ret == 0 {
// EOF reached
break;
let mut in_off_u64 = in_off as u64;
let result: std::io::Result<usize> =
rustix_copy_file_range(in_fd, Some(&mut in_off_u64), out_fd, None, pending)
.map_err(std::io::Error::from);
match result {
Ok(0) => break,
Ok(ret) => {
pending -= ret;
in_off = in_off_u64 as libc::off_t;
}
Err(err) => {
return match err.raw_os_error() {
Some(libc::ENOSYS) | Some(libc::EOPNOTSUPP) | Some(libc::EXDEV) => {
// Fallback to write(2).
reliable_write(out_fd.as_raw_fd(), in_ptr, pending)
}
_ => Err(err),
};
}
}
pending -= ret as usize;
}
Ok(len)
}
@@ -986,24 +1003,27 @@ fn reliable_copy_file_range(
#[cfg(all(target_os = "linux", target_env = "gnu"))]
fn aligned_copy_file_range(
mut in_ptr: *const u8,
in_fd: i32,
in_fd: BorrowedFd<'_>,
mut in_off: libc::off_t,
out_fd: i32,
out_fd: BorrowedFd<'_>,
len: usize,
block_size: usize,
cover: WriteRange,
) -> std::io::Result<usize> {
// 1. Get current output offset.
let res = unsafe { libc::lseek(out_fd, 0, libc::SEEK_CUR) as i64 };
if res < 0 {
return Err(std::io::Error::last_os_error());
}
let out_off = res as usize;
// Get current output offset.
// If either offset doesn't fit in usize (only possible on 32-bit
// targets), give up on alignment and just copy everything.
let (Ok(out_off), Ok(in_off_bytes)) = (
usize::try_from(rustix::fs::tell(out_fd)?),
usize::try_from(in_off),
) else {
return reliable_copy_file_range(in_ptr, in_fd, in_off, out_fd, len);
};
let mut pending = len;
// Obtain head alignment.
// Bytes to end of block:
let remainder = in_off as usize % block_size;
let remainder = in_off_bytes % block_size;
// Bytes to write (block_size becomes 0):
let head_align = (block_size - remainder) % block_size;
@@ -1015,7 +1035,7 @@ fn aligned_copy_file_range(
if head_align > 0 {
// Align the two files on a block boundary.
let head_len = head_align.min(pending);
reliable_write(out_fd, in_ptr, head_len)?;
reliable_write(out_fd.as_raw_fd(), in_ptr, head_len)?;
in_ptr = unsafe { in_ptr.add(head_len) };
in_off += head_len as i64;
pending -= head_len;
@@ -1029,7 +1049,7 @@ fn aligned_copy_file_range(
// Copy tail if needed.
if pending > 0 && cover == WriteRange::Complete {
in_ptr = unsafe { in_ptr.add(aligned_len) };
pending -= reliable_write(out_fd, in_ptr, pending)?;
pending -= reliable_write(out_fd.as_raw_fd(), in_ptr, pending)?;
}
Ok(len - pending)
@@ -1062,6 +1082,8 @@ mod tests {
#[cfg(all(target_os = "linux", target_env = "gnu"))]
use std::io::{self, Write};
use std::io::{Seek, SeekFrom};
#[cfg(target_os = "linux")]
use std::os::unix::io::AsFd;
use tempfile::NamedTempFile;
use tempfile::tempfile;
@@ -1793,8 +1815,8 @@ mod tests {
infile.write_all(data).unwrap();
infile.rewind().unwrap();
let in_fd = infile.as_raw_fd();
let out_fd = outfile.as_raw_fd();
let in_fd = infile.as_fd();
let out_fd = outfile.as_fd();
let in_ptr = data.as_ptr();
// Copy with block size 4, cover = Complete
@@ -1829,8 +1851,8 @@ mod tests {
infile.write_all(data).unwrap();
infile.rewind().unwrap();
let in_fd = infile.as_raw_fd();
let out_fd = outfile.as_raw_fd();
let in_fd = infile.as_fd();
let out_fd = outfile.as_fd();
let in_ptr = data.as_ptr();
// Copy with block size 4, cover = Blocks
@@ -1848,6 +1870,94 @@ mod tests {
assert_eq!(&buf, b"abcdefgh");
}
#[test]
#[cfg(target_os = "linux")]
fn test_aligned_copy_respects_existing_out_offset() {
let mut infile = tempfile().unwrap();
let mut outfile = tempfile().unwrap();
// Write a prefix into the output file so tell() returns a non-zero offset.
let prefix = b"HEADER";
outfile.write_all(prefix).unwrap();
outfile.flush().unwrap();
// Input data = 12 bytes (3 blocks of 4 when block_size=4).
let data = b"abcdefghijkl";
infile.write_all(data).unwrap();
infile.rewind().unwrap();
let in_fd = infile.as_fd();
let out_fd = outfile.as_fd();
let in_ptr = data.as_ptr();
let copied = aligned_copy_file_range(
in_ptr,
in_fd,
0,
out_fd,
data.len(),
4,
WriteRange::Complete,
)
.unwrap();
assert_eq!(copied, data.len());
outfile.rewind().unwrap();
let mut buf = Vec::new();
outfile.read_to_end(&mut buf).unwrap();
assert_eq!(&buf, b"HEADERabcdefghijkl");
}
#[test]
#[cfg(target_os = "linux")]
fn test_reliable_copy_file_range_rustix_path() {
let mut infile = tempfile().unwrap();
let mut outfile = tempfile().unwrap();
let data = b"0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ";
infile.write_all(data).unwrap();
infile.rewind().unwrap();
let in_fd = infile.as_fd();
let out_fd = outfile.as_fd();
let in_ptr = data.as_ptr();
// Copy a subrange starting at offset 10.
let copied = reliable_copy_file_range(in_ptr, in_fd, 10, out_fd, 16).unwrap();
assert_eq!(copied, 16);
outfile.rewind().unwrap();
let mut buf = Vec::new();
outfile.read_to_end(&mut buf).unwrap();
assert_eq!(&buf, b"ABCDEFGHIJKLMNOP");
// Now verify that copying into a file with existing data appends correctly.
let mut infile2 = tempfile().unwrap();
let mut outfile2 = tempfile().unwrap();
let input_data = b"abcdefghijklmnopqrstuvwxyz";
infile2.write_all(input_data).unwrap();
infile2.rewind().unwrap();
outfile2.write_all(b"PRE:").unwrap();
outfile2.flush().unwrap();
let in_fd2 = infile2.as_fd();
let out_fd2 = outfile2.as_fd();
let in_ptr2 = input_data.as_ptr();
let copied2 = reliable_copy_file_range(in_ptr2, in_fd2, 5, out_fd2, 10).unwrap();
assert_eq!(copied2, 10);
outfile2.rewind().unwrap();
let mut buf2 = Vec::new();
outfile2.read_to_end(&mut buf2).unwrap();
assert_eq!(&buf2, b"PRE:fghijklmno");
}
///////////////////////////////
// Unit tests for write_chunk()
///////////////////////////////
+17 -5
View File
@@ -15,6 +15,7 @@ pub mod error_handling;
pub mod fast_io;
pub mod fast_regex;
pub mod in_place;
pub mod named_reader;
pub mod named_writer;
pub mod processor;
pub mod script_char_provider;
@@ -24,15 +25,16 @@ use crate::sed::command::{ByteSpace, CharacterMode, ProcessingContext};
use crate::sed::compiler::compile;
use crate::sed::processor::process_all_files;
use crate::sed::script_line_provider::ScriptValue;
use clap::{Arg, ArgMatches, Command, arg, crate_version};
use clap::{Arg, ArgMatches, Command, arg};
use std::collections::HashMap;
use std::env;
use std::path::PathBuf;
use uucore::error::{UResult, USimpleError, UUsageError};
use uucore::format_usage;
const ABOUT: &str = "Stream editor for filtering and transforming text";
const ABOUT: &str = "Stream editor for filtering and transforming text (part of uutils)";
const USAGE: &str = "sed [OPTION]... [script] [file]...";
const VERSION: &str = concat!(env!("CARGO_PKG_VERSION"), " (uutils)");
#[uucore::main]
pub fn uumain(args: impl uucore::Args) -> UResult<()> {
@@ -62,7 +64,7 @@ pub fn uu_app() -> Command {
let util_name = uucore::util_name();
Command::new(util_name)
.version(crate_version!())
.version(VERSION)
.about(ABOUT)
.override_usage(format_usage(USAGE))
.args_override_self(true)
@@ -113,6 +115,11 @@ pub fn uu_app() -> Command {
arg!(-s --separate "Consider files as separate rather than as a long stream."),
arg!(--sandbox "Operate in a sandbox by disabling e/r/w commands."),
arg!(-u --unbuffered "Load minimal input data and flush output buffers regularly."),
Arg::new("uutil-extensions")
.short('U')
.long("uutil-extensions")
.help("Enable incompatible extensions.")
.action(clap::ArgAction::SetTrue),
Arg::new("null-data")
.short('z')
.long("null-data")
@@ -238,16 +245,17 @@ fn build_context(matches: &ArgMatches) -> UResult<ProcessingContext> {
length: matches.get_one::<u32>("length").map_or(70, |v| *v as usize),
quiet: matches.get_flag("quiet"),
posix: matches.get_flag("posix"),
separate: matches.get_flag("separate"),
separate: matches.get_flag("separate") || matches.contains_id("in-place"),
sandbox: matches.get_flag("sandbox"),
unbuffered: matches.get_flag("unbuffered"),
null_data: matches.get_flag("null-data"),
uutil_extensions: matches.get_flag("uutil-extensions"),
// Environment
character_mode: character_mode_for_locale(&locale)?,
// Other context
input_name: "<stdin>".to_string(),
input_name: PathBuf::from("-"),
line_number: 0,
last_address: false,
last_line: false,
@@ -261,6 +269,7 @@ fn build_context(matches: &ArgMatches) -> UResult<ProcessingContext> {
},
parsed_block_nesting: 0,
label_to_command_map: HashMap::new(),
named_readers: HashMap::new(),
range_commands: Vec::new(),
substitution_made: false,
append_elements: Vec::new(),
@@ -388,6 +397,7 @@ mod tests {
assert!(!ctx.sandbox);
assert!(!ctx.unbuffered);
assert!(!ctx.null_data);
assert!(!ctx.uutil_extensions);
}
#[test]
@@ -405,6 +415,7 @@ mod tests {
"-s",
"--sandbox",
"-u",
"-U",
"-z",
]);
@@ -423,6 +434,7 @@ mod tests {
assert!(ctx.sandbox);
assert!(ctx.unbuffered);
assert!(ctx.null_data);
assert!(ctx.uutil_extensions);
}
#[test]
+101
View File
@@ -0,0 +1,101 @@
// An abstraction for input files read one line at a time by the `R` command
//
// SPDX-License-Identifier: MIT
// Copyright (c) 2025 Diomidis Spinellis
//
// This file is part of the uutils sed package.
// It is licensed under the MIT License.
// For the full copyright and license information, please view the LICENSE
// file that was distributed with this source code.
use std::cell::RefCell;
use std::fs::File;
use std::io::{BufRead, BufReader};
use std::path::PathBuf;
use std::rc::Rc;
#[derive(Debug)]
/// State of the file backing an `R` command, opened lazily on first use.
enum State {
Unopened,
Open(BufReader<File>),
Exhausted,
}
#[derive(Debug)]
/// Reader that yields successive lines of a file for the GNU `R` command.
/// The file is opened on first use; a file that cannot be opened or read is
/// treated as having no more lines, matching GNU sed (no error is raised).
pub struct NamedReader {
path: PathBuf,
state: State,
}
impl NamedReader {
/// Create a reader for `path` without opening it yet.
pub fn new(path: PathBuf) -> Rc<RefCell<Self>> {
Rc::new(RefCell::new(NamedReader {
path,
state: State::Unopened,
}))
}
/// Return the next line of the file, including its trailing newline if
/// present, or `None` once the file is exhausted or could not be read.
pub fn next_line(&mut self) -> Option<Vec<u8>> {
if matches!(self.state, State::Unopened) {
self.state = match File::open(&self.path) {
Ok(file) => State::Open(BufReader::new(file)),
Err(_) => State::Exhausted,
};
}
let State::Open(reader) = &mut self.state else {
return None;
};
let mut line = Vec::new();
match reader.read_until(b'\n', &mut line) {
Ok(0) | Err(_) => {
self.state = State::Exhausted;
None
}
Ok(_) => Some(line),
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::io::Write;
use tempfile::NamedTempFile;
#[test]
fn yields_successive_lines_then_none() {
let mut file = NamedTempFile::new().unwrap();
file.write_all(b"one\ntwo\n").unwrap();
let reader = NamedReader::new(file.path().to_path_buf());
assert_eq!(reader.borrow_mut().next_line(), Some(b"one\n".to_vec()));
assert_eq!(reader.borrow_mut().next_line(), Some(b"two\n".to_vec()));
assert_eq!(reader.borrow_mut().next_line(), None);
assert_eq!(reader.borrow_mut().next_line(), None);
}
#[test]
fn last_line_without_newline_is_preserved() {
let mut file = NamedTempFile::new().unwrap();
file.write_all(b"abc").unwrap();
let reader = NamedReader::new(file.path().to_path_buf());
assert_eq!(reader.borrow_mut().next_line(), Some(b"abc".to_vec()));
assert_eq!(reader.borrow_mut().next_line(), None);
}
#[test]
fn missing_file_yields_no_lines() {
let reader = NamedReader::new(PathBuf::from("/nonexistent/xyzzy-42-does-not-exist"));
assert_eq!(reader.borrow_mut().next_line(), None);
}
}
Loaded 20 of 31 files, more files were not shown because too many files have changed in this diff. Show more