diff --git a/design/README.md b/design/README.md new file mode 100644 index 0000000..164bca4 --- /dev/null +++ b/design/README.md @@ -0,0 +1,23 @@ +# `uutils-args` Design Docs + +This is a series of design documents, explaining the various design goals and +decisions. Before diving in, let's lay out the design goals of this project. + +- Must support all options in GNU coreutils. +- Must support a many-to-many relationship between options and settings. +- Must have a convenient derive API. +- Must support help strings from file. +- Code must be "greppable" (e.g. search file for `--all` to find the code for + that argument). +- Maintainability is more important than terseness. +- With a bit of luck, it will be smaller and faster than `clap`, because we have + fewer features to support. +- Use outside uutils is possible but not prioritized. Hence, configurability + beyond the coreutils is not necessary. +- Errors must be at least as good as GNU's, but may be different (hopefully improved). + +## Pages + +1. [Arguments in coreutils](arguments_in_coreutils.md) +2. [Problems with `clap` and other parsers](problems_with_clap.md) +3. [Library design](design.md) (TODO once the design settles) diff --git a/design/arguments_in_coreutils.md b/design/arguments_in_coreutils.md new file mode 100644 index 0000000..b0c794e --- /dev/null +++ b/design/arguments_in_coreutils.md @@ -0,0 +1,112 @@ +# Argument Types and Behaviour in Coreutils + +The coreutils are specified by POSIX and have various implementations. We want +to be compatible with the GNU implementation. Generally, these utils use +`getopt_long` function provided by GNUlib. This is a fairly simple parser, that +you can repeatedly call to iterate over the options passed to the util. + +## Default Behaviour + +This construction gives the follow default behaviours: + +- `--help` and `--version` are used as the flags for, well, help and version, + respectively. +- `-h` and `-V` are NOT excepted and sometimes even used for other purposes than + showing help and version. +- Values with leading hyphens are accepted by default. +- `getopt_long` does not do any checking of conflicting arguments. Hence, all + arguments have overriding behaviour, including overriding themselves. +- Long options are inferred from unambiguous prefixes. For example, `ls --group` + is inferred to `ls --group-directories-first` because there is no other long + option starting with `group`. +- The help string is written by hand and not provided by `getopt_long`. To their + credit, the GNU authors have put great effort into standardizing these + strings. + +## Many-to-many relationship + +In the coreutils, there is a very loose coupling between the arguments and their +effect in the program. Take the snippet from `cat` below, for example. Settings +can be changed by multiple options (e.g. `show_nonprinting` is set by `-t`, `-v` +and `-A`). This leads to a many-to-many relationship: each option can change +multiple settings and each settings can be changed by multiple options. + +```C +switch (c) { + case 't': + show_tabs = true; + show_nonprinting = true; + break; + + case 'v': + show_nonprinting = true; + break; + + case 'A': + show_nonprinting = true; + show_ends = true; + show_tabs = true; + break; + + case 'E': + show_ends = true; + break; + + case 'T': + show_tabs = true; + break; +} +``` + +## Argument Types + +### Flags + +There are many simple flags that do not take any values. For example, the flags +from `cat` above are all flags. They can have both long and short versions +(`--show-nonprinting` & `-v`), but they can also have just one of the two. + +Some flags are hidden, like `tail`'s `---presume-input-pipe` option. These +hidden arguments also have 3 leading hyphens. + +### Options with values + +Some options take values. Most of the time, this is just a long options. Some +examples: + +- `ls` has `-w, --width=COLS`, where the value is required for both the short + and long option. +- `ls` has `-F, --classify[=WHEN]`, where the value is optional for the long + option and the short option does not take a value. +- `ls` has `--hyperlink[=WHEN]`, which does not have a short version (and an + optional value). +- `mktemp` has `-p DIR, --tmpdir[=DIR]`, where the value is required for the + short option and optional for the long option. +- `date` has `-I[FMT], --iso-8601[=FMT]`, where the value is optional for both + the short and long option. + +If the option takes one of several possible values, these values are inferred +from unambiguous prefixes. For example, `ls --color=y` can be used as shorthand +for `ls --color=yes`. + +### Positional arguments + +Some utils take positional arguments, which might be required. + +- `arch` takes no positional arguments. +- `comm FILE1 FILE2` takes 2 required positional arguments. +- `tr SET1 [SET2]` has 1 required and 1 optional positional argument. +- `uniq [INPUT [OUTPUT]]` takes 2 optional positional arguments. +- `ls [FILE]...` takes 0 or more positional arguments. +- `cp SOURCE... DEST` take 1 or more source arguments and 1 required destination + argument, however, `cp -t DIRECTORY SOURCE...` does not have the destination + argument. +- `timeout DURATION COMMAND...` takes one 1 required duration and a trailing + argument of minimal 1 value. Any options appearing after the first value of + `COMMAND` should be parsed as part of `COMMAND`. +- `who [ FILE | ARG1 ARG2 ]` either takes 1 `FILE` argument or 2 `ARG` + arguments. + +### Deprecated syntax `+N` and `-N` + +Some utils (e.g. `head`, `tail` and `uniq`) support an old deprecated syntax where numbers can be directly passed as arguments as a shorthand. For example, `uniq +5` is a shorthand for `uniq -s 5` and `uniq -5` is short for `uniq -f 5`. \ No newline at end of file diff --git a/design/problems_with_clap.md b/design/problems_with_clap.md new file mode 100644 index 0000000..c6fa593 --- /dev/null +++ b/design/problems_with_clap.md @@ -0,0 +1,204 @@ +# Problems with `clap` and other parsers + +To ensure that this library is an improvement over the current situation, we +need to investigate what we want to change and what to keep from `clap`. In the +process, I'll also discuss some other parsers to see if we can take some +inspiration from them. + +Before I continue, I want to note that these are not (always) general problems +with `clap`. They are problems that show up when you want to implement the +coreutils with it. The coreutils have some weird behaviour that you won't have +to deal with in a new project. `clap` is still a really good library and you +should probably use it over this library, unless you need compatibility with GNU +utils. + +## Problem 1: No many-to-many relationship between arguments and settings + +This is the biggest issue we have with `clap`. In `clap`, it is assumed that +options do not interfere with each other. This means that _partially overriding_ +options are really hard to support. `rm` has `--interactive` and `-f`, which +mostly just override each other, because they set the interactive mode and +decide whether to print warnings. However, `--interactive=never` does nog change +whether warnings are printed. Hence, they cannot override completely, because +then these two are **not** identical: + +```bash +rm -f --interactive=never +rm --interactive=never +``` + +The only way we've come up with to support this in `clap` is by manually +comparing the indices between these options, which is not very nice. + +This can get very complicated, as is the case in `ls`, where the +[parsing of the format is very strange and error-prone](https://github.com/uutils/coreutils/blob/03710a180eb273e566b52c59ac54715844380e0c/src/uu/ls/src/ls.rs#L432-L505). + +## Problem 2: We can't use the derive API + +This is mostly due to the previous problem, but because arguments usually change +multiple settings, we cannot use the derive API of `clap` in most cases. We +could go for a hybrid between the derive and builder APIs, which `clap` does +support, but that feels overly complicated. Hence, we stuck with the builder +API. + +## Problem 3: Wrong defaults + +The defaults of clap are often not what we want them to be. One might argue +`clap`'s defaults are better, but we're aiming for compatibility with coreutils, +so we have no choice but to override them. Here are a few examples: + +| | `clap` defaults | coreutils | +| --------------------- | ---------------------------- | --------------- | +| help flags | `-h` and `--help` | `--help` | +| version flags | `-V` and `--version` | `--version` | +| Long option inference | Optional | Always | +| Conflicting options | Must be set to override | Always override | +| Leading hyphens | Must be set per argument[^1] | Always accepted | + +Changing these defaults is sometimes just a single line, but other times it +becomes quite verbose. In particular, setting the options to override becomes +quite verbose in some cases. + +[^1]: There is a setting to set it for all arguments, but it behaves differently +than setting it individually and leads to some troubles, due to the differences +mentioned in the next section. + +## Problem 4: Subtle differences + +`clap` parses differently than `getopt`. This can be seen with optional values: +`clap` does not require a `=` between the flag and the value, but `getopt` does. +Instead, `clap` checks whether the next argument starts with a hyphen to check +whether the value is the value of the previous option or a new option. +Therefore, unless we tell `clap` that a `=` is required it will parse `foo.txt` +as the value to `--color` instead of as a file here: + +```bash +ls --color foo.txt +``` + +Now assume there is some argument `-f`, `--foo` with an optional value. If we do +require `=`, then the behaviour is still not correct, because now `clap` also +requires a `=` for the short option. In the coreutils, however, `=` is never +used for a short option. Hence, the only way to get the desired behaviour is to +create multiple arguments. + +But even then, there is no way to tell `clap` to consider the `=` as part of the value. E.g. `cut -d=` will be parsed as `cut -d''`, which we have to work around. + +It happens quite often that we miss these subtle differences and therefore end +up not being compatible with GNU coreutils. If we do want to do this correctly, +it usually takes changing multiple settings to get the desired result. + +## Problem 5: Deprecated syntax of `head`, `tail` and `uniq` + +As discussed in the argument types document, these utils support a shorthand +syntax for some options (e.g. `-5` is short for `-s 5`). We have not managed to +implement these nicely with `clap`. Our best efforts try to filter these values +out of the arguments before passing them to `clap`, but it is extremely +difficult to handle all edge-cases. + +## Problem 6: It's stringly typed + +`clap`'s arguments are identified by strings. This leads to code like this: + +```rust +const OPT_NAME: &'static str = "name"; + +// -- snip -- + +fn main() { + let cmd = Command::new(...) + .arg( + Arg::new(OPT_NAME) + ); + + // -- snip -- + let name = matches.get_one(OPT_NAME); +} +``` + +There is no checking at compile time whether `OPT_NAME` has been registered as +an argument and if we wouldn't use a constant, it would be prone to typos. It +also leads to a big list of strings at the top of the file, which is not a big +deal, but a bit annoying. + +Of course, we wouldn't have this problem if we were able to use the derive API. + +## Problem 7: Reading help string from a file + +In `uutils` our help strings can get quite long. Therefore, we like to extract +those to an external file. With `clap` this means that we need to do some custom +preprocessing on this file to extract the information for the several pieces of +the help string that `clap` supports. + +## Problem 8: No markdown support + +Granted, this is not really a problem, but more of a nice-to-have. We have +online documentation for the utils, based on the help strings and these are +rendered from markdown. Ideally, our argument parser supports markdown too, so +that we can have nicely rendered help strings which have (roughly) the same +appearance in the terminal and online. + +## Good things about `clap` + +Alright, enough problems. Let's praise `clap` a bit, because it's an excellent +library. + +- The help text looks great (although I think we should turn off the textwrap + feature). +- The error messages are very informative and provide a lot of context. +- It has no trouble dealing with invalid UTF-8. +- It is very configurable. The fact that we were able to work around most of our + issues at all, even though it might have been quite verbose is a great + accomplishment of `clap`'s developers. +- It has support for generating completion information for many shells. +- We can access its internals to generate our online docs. + +## Other parsers + +I'll do a quick rundown of other parsers and why they are not well-suited to the +uutils project. I've included anything I could find, including obscure +libraries. + +- [`lexopt`](https://github.com/blyxxyz/lexopt) + - Great but very low-level. + - No help generation or other fancy features. + - `uutils-args` actually uses `lexopt` under the hood. +- [`clap_lex`](https://github.com/clap-rs/clap/tree/master/clap_lex) + - As discussed above, `clap`'s lexing is slightly different from coreutils. + - Otherwise, it would be interesting to build on top of. +- [`argh`](https://github.com/google/argh) + - Does not handle invalid UTF-8. + - It is also not configurable enough. + - Does not support a many-to-many relationship. +- [`bpaf`](https://github.com/pacak/bpaf) + - Very configurable, even supports `dd`-style. + - No different configuration between short and long options (as far as I can + find). + - Does not have the many-to-many relationship (options map directly to fields + in a struct). +- [`gumdrop`](https://github.com/murarth/gumdrop) + - Does not handle invalid UTF-8. + - Not configurable enough. + - Does not have the many-to-many relationship (options map directly to fields + in a struct). +- [`pico_args`](https://github.com/razrfalcon/pico-args) + - Interesting, but does not seem to provide much over `lexopt`. +- [`xflags`](https://github.com/matklad/xflags) + - No different configuration between short and long options. + - Does not have the many-to-many relationship (options map directly to + fields). +- [`getopts`](https://github.com/rust-lang/getopts) + - Was once used by uutils. + - No help generation. + - No many-to-many relationship. +- [`getopt`](https://docs.rs/getopt/latest/getopt/) and + [`libc::getopt`](https://docs.rs/libc/latest/libc/fn.getopt.html) + - No long options. +- [`getopt_long`](https://docs.rs/getopt-long/0.3.0/getopt_long/) + - Cumbersome to use. + - Seems unmaintained. + - No license. +- [`getopt_long`](https://www.gnu.org/software/gnulib/manual/html_node/getopt_005flong.html) + from GNUlib + - We can't use GNU code, because of the GPL. + - We also do not want to do this, because we don't want to depend on GNUlib.