diff --git a/design/coreutils_arg.md b/design/coreutils_arg.md deleted file mode 100644 index 7ec700e..0000000 --- a/design/coreutils_arg.md +++ /dev/null @@ -1,485 +0,0 @@ -TODO list for design: - -- Parsing - - Flags - - [x] Flags based on `bool` - - [ ] Flags based on `enum` - - [ ] Inferring long flags/options - - Options - - [ ] Options (with `Option`) - - [ ] Options with default values - - [ ] Options with optional values - - [ ] Options with multiple values/occurrences - - [ ] Options based on `enum` - - [ ] Option values parsing? - - [ ] Default values from environment variables - - [ ] Error handling - - [ ] Position arguments - - [ ] Trailing var arg - - [ ] Conflicts - - [ ] `dd` style arguments? (Probably not worth it) -- Help & Version - - [ ] Help - - [ ] Usage - - [ ] Version - - [ ] Help and version override default flags - - [ ] Defining metadata (author, license, etc.) - - [ ] Defining short/long help - - [ ] Help strings from external markdown file -- [ ] Completion -- [ ] `mdbook` generation -- [ ] `man` generation - ---- - -In uutils, we have been using `clap` as our argument parser. `clap` is an -excellent library and still improving at a steady rate. However, our needs are -extremely specific and we have to work around several limitations of `clap`, -e.g. by manually checking certain indices. - -One such example is the `-f` flag of `rm`, which does two things: - -1. It suppresses warnings. -2. It disables interactive prompts. - -This means that the behaviour or `rm -fi` (`-i` is interactive), is different -from `rm -if`. In the first one, warnings are suppressed and we have -interactive prompts, in the second one, we have the behaviour of `rm -f`. -I'm gonna call this "partial overrides". - -Let's imagine a shiny future where we have a derive API for argument parsing -that supports this case and other weird cases in `coreutils`. This is purely -hypothetical for now, as both implementation and migration to this new API, -would take a long time. - -I'll be distinguishing three types of parameters: -1. Flags: arguments without a value, just present or not, like `--opt`. -2. Options: arguments with a value like `--opt=value` or `--opt value`. -3. Positional arguments. - -# The big idea - -`clap` and other `derive`-based API's use an approach to the definition of the -arguments where each field is an argument. This leads to a 1-1 relation between -arguments and settings (n-1 if you count aliases). I think this is great for most -applications, but it leads to friction when implementing the `coreutils`. In -the GNU coreutils, each arguments is lexed and then applied to the settings. -Hence, some utils have a n-m relation between arguments and settings. - -The solution is then to decouple arguments and how they map to settings. -Instead, we define a `Settings` struct and define through attributes how the -flags map to these settings. - -As a small teaser, consider this example: - -```rust -struct Settings { - #[flag("-x", "-z")] - a: bool, - #[flag("-x", "-y", "-z")] - b: bool, - #[flag("-y", "-z")] - c: bool, -} -``` - -The flags `-x`, `-y` and `-z` all set multiple values, hence we have a n-m -relationship. The equivalent code in `clap` would require a manual mapping -from the parsed arguments to the settings. - -Arguably, the example above is bad API design and `clap`'s approach is -therefore better. In uutils, however, we don't have the luxury to define our -own API; we have to follow the GNU utils to maintain compatibility. - -# Simple Flags - -I'm calling our library `magic` for now, because it does not exist. Let's start -by supporting flags. - -```rust -use magic::{Options}; - -#[derive(Options, Default)] -struct Settings { - // -o and --one are implicit - // if the field name has only one character, then only the short flag is - // implied - #[flag] - one: bool, - - // Only -t, not --two - #[flag("-t")] - two: bool, - - // Only --three, not -t - #[flag("--three")] - three: bool, - - // Explicit form of `#[flag]` - #[flag("-f", "--four")] - four: bool, - - // We can define as many flags as we want - #[flag("-a", "-b", "--five", "--six")] - five: bool, - - // Cannot be set by an argument - six: bool, -} -``` - -This would expand to: - -```rust -struct Settings { - one: bool, - two: bool, - three: bool, - four: bool, - five: bool, - six: bool, -} - -impl Options for Settings { - fn parse(args: impl Iterator<&OsStr>) -> Result { - let mut self = Self::default(); - - use magic::Token::{Short, Long}; - for s in lexer(args) { - match s { - Short('o') | Long("one") => self.one = true, - Short('t') => self.two = true, - Long("three") => self.three = true, - Short('f') | Long("four") => self.four = true, - Short('a') | Short('b') | Long("five") | Long("six") => { - self.five = true, - } - } - } - self - } -} -``` - -Now here is the nice part for our `rm` usecase. Nothing is stopping us for -specifying the same flag multiple times. `rm` with just those flags is simply: - -```rust -struct Settings { - #[flag] - force: bool, - - #[flag] - #[hidden_flag("-f", "--force", value = false)] - interactive: bool -} -``` - -Where `hidden_flag` functions just like `flag`, but will be hidden in the -`--help` (more on that later). - -> **Note**: the rules for the arguments of `(hidden_)flag` are `-` specifies -> a short flag, `--` a long flag and another value is a value to set. - -Which expands to something along these lines (though probably less pretty -because it will be autogenerated): - -```rust -impl Options for Settings { - fn parse(args: impl Iterator<&OsStr>) -> Result { - let mut self = Self::default(); - for s in lex(args) { - match s { - Short('f') | Long("force") => { - self.force = true; - self.interactive = false; - } - Short('i') | Long("interactive") => { - self.interactive = true; - } - } - } - self - } -} -``` - -# Help generation - -We can do help generation like `clap`'s derive API, but we could build in -markdown support from the start. Note that help is not added by default and -must by added explicitly. - -```rust -/// help for the entire application goes here -#[derive(Options, Default)] -#[help(-h, --help)] // or #[help] -struct Settings { - /// Some help in *markdown* - #[flag] - force: bool -} -``` - -The markdown is parsed at compile-time, but rendered at runtime to fit the -terminal width. The markdown parser will therefore not be part of the -executable. - -> **Note**: markdown help should probably be feature gated. The alternative is -> just printing the string. - -But, because we have the n-m relationship, not each flag has a nicely defined -field in the struct. To document these flags we can create dummy fields, which -will be optimized away at runtime by the compiler. The type can be anything, -but making it zero-size is probablty a good idea (so either `()` or -`PhantomData`). A `dummy_flag` attribute then specifies to treat this as a -`flag` for help generation, but not for parsing. - -```rust -#[derive(Options, Default)] -struct Settings { - /// Some help for `-a` - #[flag] - #[hidden_flag("-c")] - a: bool, - - /// Some help for `-b` - #[flag] - #[hidden_flag("-c")] - b: bool, - - /// Some help for `-c` - #[dummy_flag] - c: (), -} -``` - -So the `--help` would look roughly like this - -```txt - -a Some help for `-a` - -b Some help for `-b` - -c Some help for `-c` -``` - -Similarly, we can do something like this: - -```rust -#[derive(Options, Default)] -struct Settings { - /// Enable long format - #[flag("-l", "--long")] - #[hidden_flag("-s", "--short", value = false)] - long_format: bool, - - /// Disable long format - #[dummy_flag("-s", "--short")] - short_format: () -} -``` - -Note that this has overriding behaviour by default. That is, `-sl` will use the -long format and `-ls` will use the short format. I'm leaving conflicts for -later because most coreutils have overriding behaviour. - -There's also a chance for -this to be error prone, because there's no syncing of the flags between the -different attributes. Another possible representation could be: - -```rust -#[derive(Options)] -struct Settings { - /// Enable long format - #[flag("-l", "--long")] - #[flag("-s", "--short", value = false, help = "Disable long format")] - long_format: bool, -} -``` - -We'll have to see what works best in practice. - -# Options - -Often options take a few possible values, which we can model with enums: - -```rust -#[derive(FromOpt, Default)] -enum Color { - #[value] - Always, - - #[default] - #[value] - Auto, - - #[value] - Never -} - -#[derive(Options, Default)] -struct Settings { - #[option("--color")] - color: Color; -} -``` - -Sometimes we need flags that are shortcuts for option values, we can support -this by adding `flag` attributes to the `enum`. - -```rust -#[derive(FromOpt, Default)] -enum Sort { - #[flag("-U")] - #[value] - None, - - #[default] - Name, - - #[flag("-S")] - #[value] - Size, - - #[flag("-t")] - #[value] - Time, - - #[flag("-v")] - #[value] - Version, - - #[flag("-X")] - #[value] - Extension, - - #[value] - Width, -} - -#[derive(Options, Default)] -struct Settings { - #[option("--sort")] - sort: Sort; -} -``` - -This models the `--sort`, `-U`, `-S`, `-t`, `-v` `-X` options of `ls`, and -though it may look complex, I would argue it is very efficient for 5 flags and -an option with 6 possible values and a default value. Note that we were able to -also specify that `Width` does not have a flag and that `Name` cannot be -expressed at all, apart from being the default. - -We can also express multiple values per variant, take for instance the `format` -argument of `ls`: - -```rust -#[derive(FromOpt, Default)] -enum Format { - #[value("long", "verbose")] - Long, - - #[value] - SingleColumn, - - #[default] - #[value("columns", "vertical")] - Columns, - - #[value("across", "horizontal")] - Across, - - #[value] - Commas, -} - -#[derive(Options, Default)] -struct Settings { - #[option(--format)] - format: Format, -} -``` - -Let's expand on this example with `-o`, `-n` and `-g`: - -``` - -g like -l, but do not list owner - -o like -l, but do not list group information - -n, --numeric-uid-gid like -l, but list numeric user and group IDs -``` - -The text here is deceptively simple, because they can stack and they are not -overridden by `-l`. Here's how we can handle them: - -```rust - -#[derive(FromOpt, Default)] -enum Format { - #[flag("-l")] - #[hidden_flag("-g", "-o", "-n", "--numeric-uid-gid")] - #[value("long", "verbose")] - Long, - - #[value] - SingleColumn, - - #[default] - #[flag("-C")] - #[value("columns", "vertical")] - Columns, - - #[flag("-x")] - #[value("across", "horizontal")] - Across, - - #[value] - #[flag("-m")] - Commas, -} - -#[derive(Options, Default)] -struct Settings { - #[flag("-g")] - long_hide_owner: bool, - - #[flag("-o")] - long_hide_group: bool, - - #[flag("-n", "--numeric-uid-gid")] - long_numeric_id: bool, - - #[option("--format")] - format: Format, -} -``` - -There's one final complication: `-1` is ineffective after `--long`, which is -actually distinct from `--format=single-column`. Here's a way to handle that, -even though I'm not very happy with it: - -```rust -enum Format { - #[value] - #[flag("-l")] - Long, - - #[default] - #[value] - #[flag("-C")] - Columns, -} - -struct Settings { - #[flag("-1", "-l")] - #[hidden_flag("-C", value = false)] - #[option("--format=single-column")] - single_column: bool, - - #[option("--format")] - format: Format, -} -``` - -We would now display a single column if `format == Format::SingleColumn` or -`single_column == true`. Luckily, there's only one such case that I know of -that needs this. - diff --git a/design/design.md b/design/design.md new file mode 100644 index 0000000..9754b42 --- /dev/null +++ b/design/design.md @@ -0,0 +1,213 @@ +# Library design + +In this document, I explain how this library solves the problems with `clap` and +how it accomplishes the design goals. + +## Basic API + +This library only has a derive API. In most derive-based argument parsers, the +arguments are based on a `struct`, but in this library they are based on `enum` +variants, which then get mapped to a `struct`. The parsing happens in two stages + +1. Arguments get mapped to an `enum` +2. The `enum` variants are matched and update `struct` fields. + +This gives us a separation of concerns: the `enum` determines how the arguments +get parsed and the `struct` determines how they map to the program settings. +This gives us a lot of freedom in defining our mapping from arguments to +settings. + +Here is a simple example comparing `clap` and `uutils_args`. + +> **Note**: There are differences in behaviour between these two. E.g. +> uutils_args allows options to appear multiple times, remembering only the last +> one. + +```rust +// Clap +#[derive(Parser)] +struct Args { + /// Name of the person to greet + #[arg(short, long)] + name: String, + + /// Number of times to greet + #[arg(short, long)] + say_goodbye: bool, +} + +// Uutils args +#[derive(Arguments, Clone)] +enum Arg { + /// Name of the person to greet + #[option("-n NAME", "--name=NAME")] + Name(String), + + /// Number of times to greet + #[option("-g", "--goodbye")] + SayGoodbye +} + +#[derive(Options, Default)] +#[arg_type(Arg)] +struct Settings { + #[set(Arg::Name)] + name: String + + #[map(Arg::SayGoodbye => true)] + goodbye: bool, +} +``` + +> **Note**: `uutils_args` is more explicit than `clap`, you have to explicitly +> state the names of the flags and values. This helps maintainability because it +> is always obvious where an argument is defined. + +As part of the `Options` derive, we get a `Settings::parse` method that returns +a `Settings` from a `OsString` iterator. The implementation of +this is defined by the `set` and `map` attributes. `map` just says: "if we +encounter this value in the iterator set this value", using a match-like syntax +(it expands to a match). And the `#[set(Arg::Name)]` is just short for +`#[map(Arg::Name(name) => name)]`, because that is a commonly appearing pattern. + +Importantly, arguments can appear in the attributes for multiple fields. We +could for instance do this: + +```rust +#[derive(Arguments, Clone)] +enum Arg { + #[option("-a")] + A, + + #[option("--a-and-b")] + B +} + +#[derive(Options, Default)] +#[arg_type(Arg)] +struct Settings { + #[map(Arg::A | Arg::B => true)] + a: bool + + #[map(Arg::B => true)] + b: bool, +} +``` + +## Argument types + +```rust +#[derive(Arguments, Clone)] +#[help("--help")] // help and version must be explicitly defined +#[version("--version")] +enum Arg { + // Note: You can have as many flags as you want for each variable + #[option("-f", "--foo")] + Flag, + + // Note: The value name is required and will be used in `--help` + #[option("-r VALUE", "--required=VALUE")] + OptionWithRequiredValue(String), + + // Note: The value name is again required. + // Note: If no `default` is specified, `Default::default` is used. + #[option("-o[VALUE]", "--optional[=VALUE]", default = "DEFAULT".into())] + OptionWithOptionalValue(String), + + // Note: `-l` will use the default value. + #[option("-l", "--long=VALUE", default = "SHORT VALUE")] + ValueOnlyForLongOption(String), + + // Any combination of required, optional and no arguments is possible. + #[option("-t VAL", "--test[=VAL]", default = "")] + ValueOptionalForLongOption(String), + + // Positional arguments take a range of the number of arguments they + // take. The default is 1..=1, i.e. exactly 1 argument. + #[positional] + SinglePositionalArgument(String), + + #[positional(0..=1)] + OptionalPositionalArgument(String), + + // Range is open on both sides so 0..=MAX + #[positional(..)] + AnyNumberOfPositionalArguments(String), + + // All remaining arguments are collected into a `Vec`. + #[position(last)] + TrailingVarArg(Vec), + + // Same range can still be applied even though there can only ever + // be 1 trailing var arg. + #[position(last, 0..=1)] + OptionalTrailingVarArg(Vec), +} +``` + +## Options struct + +The options struct has just one fundamental attribute: `map`. It works much like a `match` expression (in fact, that's what it expands to). Furthermore, it's possible to define defaults on fields. + +```rust +#[derive(Options, Default)] +struct Settings { + // When a Arg::Foo is parsed, set this field to `true`. + // Any expression is possible. + // Any field starts with `Default::default()`. + #[map(Arg::Foo => true)] + foo: bool + + // Arg::BarTrue sets this to true, Arg::BarFalse sets this to false. + // We can have as many arms as we want. For each field, the first + // matching arm is applied and the rest is ignored. + #[map( + Arg::BarTrue => true, + Arg::BarFalse => false, + )] + bar: bool, + + // We can set a default value with the field attribute. + #[map(Arg::Baz => false)] + #[field(default = true)] + baz: bool, + + // We can also define a env var to read from if available, else + // the default value will be used. + #[map(Arg::SomeVar => true)] + #[field(env = "SOME_VAR", default = false)] + some_var: bool, +} +``` + +As a shorthand, there is also a `set` attribute. These fields behave identically: + +```rust +#[derive(Options, Default)] +struct Settings { + #[map(Arg::Foo(f) => f)] + bar: u64, + + #[set(Arg::Foo)] + baz: u64 +} +``` + +## `FromValue` enums + +We often want to map values to some enum, we can define this mapping by deriving `FromValue`: + +```rust +#[derive(Default, FromValue)] +enum Color { + #[value("always", "yes", "force")] + Always, + + #[default] + #[value("auto", "tty", "if-tty")] + Auto, + + #[value("never", "no", "none")] + Never, +} +``` \ No newline at end of file diff --git a/design/problems_with_clap.md b/design/problems_with_clap.md index c6fa593..101e654 100644 --- a/design/problems_with_clap.md +++ b/design/problems_with_clap.md @@ -82,7 +82,9 @@ requires a `=` for the short option. In the coreutils, however, `=` is never used for a short option. Hence, the only way to get the desired behaviour is to create multiple arguments. -But even then, there is no way to tell `clap` to consider the `=` as part of the value. E.g. `cut -d=` will be parsed as `cut -d''`, which we have to work around. +But even then, there is no way to tell `clap` to consider the `=` as part of the +value. E.g. `cut -d=` will be parsed as `cut -d''`, which we have to work +around. It happens quite often that we miss these subtle differences and therefore end up not being compatible with GNU coreutils. If we do want to do this correctly, @@ -96,6 +98,15 @@ implement these nicely with `clap`. Our best efforts try to filter these values out of the arguments before passing them to `clap`, but it is extremely difficult to handle all edge-cases. +## Problem 6: Exit codes + +In coreutils, different utils have different exit codes when they fail to parse. +For example, `timeout` returns `125`, because the command it calls probably uses +`1` or `2`. There is no way to customize this in `clap`, so we work around it in +uutils and when we opened as issue for it, it was discarded. This makes sense +from `clap`'s perspective, but it shows that the priorities between `clap` and +uutils diverge. + ## Problem 6: It's stringly typed `clap`'s arguments are identified by strings. This leads to code like this: