The "Build website artifacts" job spends most of its time here: the
l10n step builds ~36 language books one after another, each taking its
turn at mutating the shared coreutils checkout (swap the locale into
en-US.ftl, run uudoc, build mdbook, restore en-US.ftl). Nothing in it
uses more than one core.
Everything uudoc touches is relative to its working directory, so give
each language a throwaway copy of the checkout to work in and run
DOC_JOBS of them at a time (default nproc). The checkout itself becomes
read-only, which is what makes this safe and also removes the need to
restore en-US.ftl between languages. Copying the checkout costs ~0.2s
per language; target/, .git/ and docs/book*/ are left out, which is
almost all of its weight.
The per-language tldr download and repack moves into the worker too:
it was a second serial loop, ~36 archives fetched one at a time before
any book was built.
Measured on a 30-locale tree with DOC_JOBS=4, to match a GitHub runner:
10m04s at 107% CPU -> 1m59s at 387%. The 30 resulting books are
byte-for-byte identical to the ones the serial script produces.
Two notes on what this deliberately avoids, both found by running it on
a machine where uutils shadows GNU coreutils/findutils:
- The fan-out is a bash job pool rather than `xargs -P`. uutils' xargs
accepts -P and silently ignores it ("[NOT IMPLEMENTED]"), so the
parallel build would quietly stay serial there.
- The tldr merge keeps the original direction (English base, translated
pages overlaid on top). Overlaying English onto the translation with
`cp -rn` reads as equivalent, but uutils' cp stops merging a directory
after the first existing file and still exits 0 — it copied 296 of
4673 pages, and the Arabic book lost every Examples section.
Upstream coreutils changed `feat_wasm` from an explicit list of utilities
to `feat_wasm = ["feat_common_core", "nproc"]`. The workflow scraped that
list with a sed range, which cannot resolve feature groups and, now that
feat_wasm is a one-liner, ran on to the next `]` and swallowed the body of
feat_os_unix. The deployed commands.js ended up as
const WASM_COMMANDS = ["feat_Tier1","feat_common_core","feat_os_unix",
"feat_require_unix","feat_require_unix_utmpx","nproc","grep",...];
and since executeCommandLine() gates dispatch on that array, every
coreutils command answered "command not found" in the playground without
ever reaching the WASM binary, which was fine all along.
Replace the scrape with scripts/wasm-command-list.py, which resolves the
feature through the [features] table down to the leaf optional
dependencies (the utility crates), drops `dep:`/`crate/feat` entries,
strips the `uu_` prefix (uu_test -> test) and fails loudly if the feature
resolves to no utilities.
The unit tests stayed green through all of this because the test page only
loaded wasm-terminal.js and therefore always exercised FALLBACK_COMMANDS,
never the generated file. Load /wasm/commands.js there too and assert that
the core utilities are present and that no feature group leaks in; against
the currently deployed commands.js that now fails 72 of 185 tests instead
of passing. Also refresh FALLBACK_COMMANDS with the utilities feat_wasm
gained (hostid, mktemp, nice, test, tty).
A shared link can now carry the language alongside the command, e.g.
/playground?lang=fr-FR&cmd=cut+-f+1,4-2,9-12+fruits.txt to show a
localized error message.
The value accepts both the short (fr) and full (fr-FR) form, is
validated against the locales the build ships, and is applied before
the ?cmd= commands run. The share-link button appends it when the
locale is not the default, and the Language dropdown stays in sync
via a new uutils:locale-changed event.
The caret diagnostics are the most demo-able thing in 0.11.0 and the
playground never showed them. Add a section of arguments that fail to parse
(tr set, cut range, sort key, expr and test operands, numfmt and printf
formats, head SIZE, csplit regex), all on files the playground ships, and
point at the blog post for the rest.
chmod, mkfifo, mknod, install, stat, env, du, df and stdbuf are not in
feat_wasm, so their "try it" links could only ever answer "command not
found". Show those commands as plain code, drop the two inline playground
links for chmod and env, and say that the linked examples are the ones the
WebAssembly build ships.
feat_diagnostics only comes in through coreutils' default feature set, and
the playground binary is built with --no-default-features, so uucore
compiled diagnostics_stub.rs: enabled() was a hardcoded false and every
utility printed the plain one-line error. `expr 9 + foo` in the playground
showed no caret at all.
browser_wasi_shim reports stderr as a character device with FD_WRITE and
neither FD_SEEK nor FD_TELL, which is what Rust's WASI is_terminal() takes
for a tty, so the reports render once the feature is on.
renovate[bot]
Sylvestre Ledru