uutils/uutils-args
0
0
Fork 0
mirror of https://github.com/uutils/uutils-args.git synced 2026-06-10 16:13:08 -07:00
Code Issues Packages Projects Releases Wiki Activity

add design docs on problems with clap and argument types

Browse Source
This commit is contained in:
Terts Diepraam
2022-12-20 12:20:33 +01:00
parent 830f391359
commit d6a34350fa
3 changed files with 339 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
design/README.md
+23
View File
@@ -0,0 +1,23 @@
# `uutils-args` Design Docs
This is a series of design documents, explaining the various design goals and
decisions. Before diving in, let's lay out the design goals of this project.
- Must support all options in GNU coreutils.
- Must support a many-to-many relationship between options and settings.
- Must have a convenient derive API.
- Must support help strings from file.
- Code must be "greppable" (e.g. search file for `--all` to find the code for
that argument).
- Maintainability is more important than terseness.
- With a bit of luck, it will be smaller and faster than `clap`, because we have
fewer features to support.
- Use outside uutils is possible but not prioritized. Hence, configurability
beyond the coreutils is not necessary.
- Errors must be at least as good as GNU's, but may be different (hopefully improved).
## Pages
1. [Arguments in coreutils](arguments_in_coreutils.md)
2. [Problems with `clap` and other parsers](problems_with_clap.md)
3. [Library design](design.md) (TODO once the design settles)
design/arguments_in_coreutils.md
+112
View File
@@ -0,0 +1,112 @@
# Argument Types and Behaviour in Coreutils
The coreutils are specified by POSIX and have various implementations. We want
to be compatible with the GNU implementation. Generally, these utils use
`getopt_long` function provided by GNUlib. This is a fairly simple parser, that
you can repeatedly call to iterate over the options passed to the util.
## Default Behaviour
This construction gives the follow default behaviours:
- `--help` and `--version` are used as the flags for, well, help and version,
respectively.
- `-h` and `-V` are NOT excepted and sometimes even used for other purposes than
showing help and version.
- Values with leading hyphens are accepted by default.
- `getopt_long` does not do any checking of conflicting arguments. Hence, all
arguments have overriding behaviour, including overriding themselves.
- Long options are inferred from unambiguous prefixes. For example, `ls --group`
is inferred to `ls --group-directories-first` because there is no other long
option starting with `group`.
- The help string is written by hand and not provided by `getopt_long`. To their
credit, the GNU authors have put great effort into standardizing these
strings.
## Many-to-many relationship
In the coreutils, there is a very loose coupling between the arguments and their
effect in the program. Take the snippet from `cat` below, for example. Settings
can be changed by multiple options (e.g. `show_nonprinting` is set by `-t`, `-v`
and `-A`). This leads to a many-to-many relationship: each option can change
multiple settings and each settings can be changed by multiple options.
```C
switch (c) {
case 't':
show_tabs = true;
show_nonprinting = true;
break;
case 'v':
show_nonprinting = true;
break;
case 'A':
show_nonprinting = true;
show_ends = true;
show_tabs = true;
break;
case 'E':
show_ends = true;
break;
case 'T':
show_tabs = true;
break;
}
```
## Argument Types
### Flags
There are many simple flags that do not take any values. For example, the flags
from `cat` above are all flags. They can have both long and short versions
(`--show-nonprinting` & `-v`), but they can also have just one of the two.
Some flags are hidden, like `tail`'s `---presume-input-pipe` option. These
hidden arguments also have 3 leading hyphens.
### Options with values
Some options take values. Most of the time, this is just a long options. Some
examples:
- `ls` has `-w, --width=COLS`, where the value is required for both the short
and long option.
- `ls` has `-F, --classify[=WHEN]`, where the value is optional for the long
option and the short option does not take a value.
- `ls` has `--hyperlink[=WHEN]`, which does not have a short version (and an
optional value).
- `mktemp` has `-p DIR, --tmpdir[=DIR]`, where the value is required for the
short option and optional for the long option.
- `date` has `-I[FMT], --iso-8601[=FMT]`, where the value is optional for both
the short and long option.
If the option takes one of several possible values, these values are inferred
from unambiguous prefixes. For example, `ls --color=y` can be used as shorthand
for `ls --color=yes`.
### Positional arguments
Some utils take positional arguments, which might be required.
- `arch` takes no positional arguments.
- `comm FILE1 FILE2` takes 2 required positional arguments.
- `tr SET1 [SET2]` has 1 required and 1 optional positional argument.
- `uniq [INPUT [OUTPUT]]` takes 2 optional positional arguments.
- `ls [FILE]...` takes 0 or more positional arguments.
- `cp SOURCE... DEST` take 1 or more source arguments and 1 required destination
argument, however, `cp -t DIRECTORY SOURCE...` does not have the destination
argument.
- `timeout DURATION COMMAND...` takes one 1 required duration and a trailing
argument of minimal 1 value. Any options appearing after the first value of
`COMMAND` should be parsed as part of `COMMAND`.
- `who [ FILE | ARG1 ARG2 ]` either takes 1 `FILE` argument or 2 `ARG`
arguments.
### Deprecated syntax `+N` and `-N`
Some utils (e.g. `head`, `tail` and `uniq`) support an old deprecated syntax where numbers can be directly passed as arguments as a shorthand. For example, `uniq +5` is a shorthand for `uniq -s 5` and `uniq -5` is short for `uniq -f 5`.
design/problems_with_clap.md
+204
View File
@@ -0,0 +1,204 @@
# Problems with `clap` and other parsers
To ensure that this library is an improvement over the current situation, we
need to investigate what we want to change and what to keep from `clap`. In the
process, I'll also discuss some other parsers to see if we can take some
inspiration from them.
Before I continue, I want to note that these are not (always) general problems
with `clap`. They are problems that show up when you want to implement the
coreutils with it. The coreutils have some weird behaviour that you won't have
to deal with in a new project. `clap` is still a really good library and you
should probably use it over this library, unless you need compatibility with GNU
utils.
## Problem 1: No many-to-many relationship between arguments and settings
This is the biggest issue we have with `clap`. In `clap`, it is assumed that
options do not interfere with each other. This means that _partially overriding_
options are really hard to support. `rm` has `--interactive` and `-f`, which
mostly just override each other, because they set the interactive mode and
decide whether to print warnings. However, `--interactive=never` does nog change
whether warnings are printed. Hence, they cannot override completely, because
then these two are **not** identical:
```bash
rm -f --interactive=never
rm --interactive=never
```
The only way we've come up with to support this in `clap` is by manually
comparing the indices between these options, which is not very nice.
This can get very complicated, as is the case in `ls`, where the
[parsing of the format is very strange and error-prone](https://github.com/uutils/coreutils/blob/03710a180eb273e566b52c59ac54715844380e0c/src/uu/ls/src/ls.rs#L432-L505).
## Problem 2: We can't use the derive API
This is mostly due to the previous problem, but because arguments usually change
multiple settings, we cannot use the derive API of `clap` in most cases. We
could go for a hybrid between the derive and builder APIs, which `clap` does
support, but that feels overly complicated. Hence, we stuck with the builder
API.
## Problem 3: Wrong defaults
The defaults of clap are often not what we want them to be. One might argue
`clap`'s defaults are better, but we're aiming for compatibility with coreutils,
so we have no choice but to override them. Here are a few examples:
| | `clap` defaults | coreutils |
| --------------------- | ---------------------------- | --------------- |
| help flags | `-h` and `--help` | `--help` |
| version flags | `-V` and `--version` | `--version` |
| Long option inference | Optional | Always |
| Conflicting options | Must be set to override | Always override |
| Leading hyphens | Must be set per argument[^1] | Always accepted |
Changing these defaults is sometimes just a single line, but other times it
becomes quite verbose. In particular, setting the options to override becomes
quite verbose in some cases.
[^1]: There is a setting to set it for all arguments, but it behaves differently
than setting it individually and leads to some troubles, due to the differences
mentioned in the next section.
## Problem 4: Subtle differences
`clap` parses differently than `getopt`. This can be seen with optional values:
`clap` does not require a `=` between the flag and the value, but `getopt` does.
Instead, `clap` checks whether the next argument starts with a hyphen to check
whether the value is the value of the previous option or a new option.
Therefore, unless we tell `clap` that a `=` is required it will parse `foo.txt`
as the value to `--color` instead of as a file here:
```bash
ls --color foo.txt
```
Now assume there is some argument `-f`, `--foo` with an optional value. If we do
require `=`, then the behaviour is still not correct, because now `clap` also
requires a `=` for the short option. In the coreutils, however, `=` is never
used for a short option. Hence, the only way to get the desired behaviour is to
create multiple arguments.
But even then, there is no way to tell `clap` to consider the `=` as part of the value. E.g. `cut -d=` will be parsed as `cut -d''`, which we have to work around.
It happens quite often that we miss these subtle differences and therefore end
up not being compatible with GNU coreutils. If we do want to do this correctly,
it usually takes changing multiple settings to get the desired result.
## Problem 5: Deprecated syntax of `head`, `tail` and `uniq`
As discussed in the argument types document, these utils support a shorthand
syntax for some options (e.g. `-5` is short for `-s 5`). We have not managed to
implement these nicely with `clap`. Our best efforts try to filter these values
out of the arguments before passing them to `clap`, but it is extremely
difficult to handle all edge-cases.
## Problem 6: It's stringly typed
`clap`'s arguments are identified by strings. This leads to code like this:
```rust
const OPT_NAME: &'static str = "name";
// -- snip --
fn main() {
let cmd = Command::new(...)
.arg(
Arg::new(OPT_NAME)
);
// -- snip --
let name = matches.get_one<String>(OPT_NAME);
}
```
There is no checking at compile time whether `OPT_NAME` has been registered as
an argument and if we wouldn't use a constant, it would be prone to typos. It
also leads to a big list of strings at the top of the file, which is not a big
deal, but a bit annoying.
Of course, we wouldn't have this problem if we were able to use the derive API.
## Problem 7: Reading help string from a file
In `uutils` our help strings can get quite long. Therefore, we like to extract
those to an external file. With `clap` this means that we need to do some custom
preprocessing on this file to extract the information for the several pieces of
the help string that `clap` supports.
## Problem 8: No markdown support
Granted, this is not really a problem, but more of a nice-to-have. We have
online documentation for the utils, based on the help strings and these are
rendered from markdown. Ideally, our argument parser supports markdown too, so
that we can have nicely rendered help strings which have (roughly) the same
appearance in the terminal and online.
## Good things about `clap`
Alright, enough problems. Let's praise `clap` a bit, because it's an excellent
library.
- The help text looks great (although I think we should turn off the textwrap
feature).
- The error messages are very informative and provide a lot of context.
- It has no trouble dealing with invalid UTF-8.
- It is very configurable. The fact that we were able to work around most of our
issues at all, even though it might have been quite verbose is a great
accomplishment of `clap`'s developers.
- It has support for generating completion information for many shells.
- We can access its internals to generate our online docs.
## Other parsers
I'll do a quick rundown of other parsers and why they are not well-suited to the
uutils project. I've included anything I could find, including obscure
libraries.
- [`lexopt`](https://github.com/blyxxyz/lexopt)
- Great but very low-level.
- No help generation or other fancy features.
- `uutils-args` actually uses `lexopt` under the hood.
- [`clap_lex`](https://github.com/clap-rs/clap/tree/master/clap_lex)
- As discussed above, `clap`'s lexing is slightly different from coreutils.
- Otherwise, it would be interesting to build on top of.
- [`argh`](https://github.com/google/argh)
- Does not handle invalid UTF-8.
- It is also not configurable enough.
- Does not support a many-to-many relationship.
- [`bpaf`](https://github.com/pacak/bpaf)
- Very configurable, even supports `dd`-style.
- No different configuration between short and long options (as far as I can
find).
- Does not have the many-to-many relationship (options map directly to fields
in a struct).
- [`gumdrop`](https://github.com/murarth/gumdrop)
- Does not handle invalid UTF-8.
- Not configurable enough.
- Does not have the many-to-many relationship (options map directly to fields
in a struct).
- [`pico_args`](https://github.com/razrfalcon/pico-args)
- Interesting, but does not seem to provide much over `lexopt`.
- [`xflags`](https://github.com/matklad/xflags)
- No different configuration between short and long options.
- Does not have the many-to-many relationship (options map directly to
fields).
- [`getopts`](https://github.com/rust-lang/getopts)
- Was once used by uutils.
- No help generation.
- No many-to-many relationship.
- [`getopt`](https://docs.rs/getopt/latest/getopt/) and
[`libc::getopt`](https://docs.rs/libc/latest/libc/fn.getopt.html)
- No long options.
- [`getopt_long`](https://docs.rs/getopt-long/0.3.0/getopt_long/)
- Cumbersome to use.
- Seems unmaintained.
- No license.
- [`getopt_long`](https://www.gnu.org/software/gnulib/manual/html_node/getopt_005flong.html)
from GNUlib
- We can't use GNU code, because of the GPL.
- We also do not want to do this, because we don't want to depend on GNUlib.