snapd/tests/lib/tools/README

ONCE THINGS ARE MADE UNIFORM THIS SHOULD BECOME A README WITHOUT THE TODO PART!

This "tools" directory collects useful utility commands that can be
used in our spread test suites. (They are separate from the shell
function libraries *.sh in the level above.)

Commands here should have names that clarify their purpose, longish
names using '-' as word separator are expected. Commands that take
themselves a full command to execute should usually end in '-do'.

The commands might take one or multiple subcommands themselves if
meaningful, subcommands should also use '-' as internal word
separator.

If a subcommand executes other commands it should usually be called
"exec".

All the commands invoked without arguments should either do something
informative/non-destructive or print usage.

All the commands should support at least a -h help option.

To invoke a command here tests can do "$TESTSTOOLS"/cmd.

Commands that are used very often or that are particularly useful to
debug at a spread -debug shell prompt can be promoted to be on the
PATH during tests, if an acceptable name can be found for them.

This is done by putting symlinks to here from tests/bin which itself
will be on the PATH.

Commands expected to be on the PATH should follow more specific naming
conventions, the name should either hint at testing or trade that off
for brevity, but only when appropriate:

* really frequently used commands can have short/pithy names (not, retry...)

* commands that manage or deal with cross-test/cross-task state should be
  prefixed with "tests." (tests.session)

* otherwise the command should take some form of suffix like ".query", ".check"
  that hints at observing, checking the system

In general if none of these apply they should continue to be invoked
via $TESTSTOOLS, or possibly have a tests/lib/*.sh wrapping them if
appropriate.

TODO:

[done] rename tests/lib/bin to tests/lib/tools, define TESTSTOOLS in
spread.yaml pointing at it

[done] create tests/bin, have ultimately this on the PATH and not tools:
  drop tests/lib/tools from PATH in spread.yaml and debian/tests/integrationtests

symlinks and renames (plus fixes):

[done] tests/bin/MATCH -> tests/lib/tools/MATCH
[done] tests/bin/REBOOT -> tests/lib/tools/REBOOT
[done] tests/bin/not -> tests/lib/tools/not
[done] tests/bin/retry -> tests/lib/tools/retry (was retry-tool)

[done] tests/bin/tests.session -> tests/lib/tools/tests.session (was session-tool)
  [done] fix to have -h
  [done] convert --ACTION into subcommands
  [done] require "exec" subcommand to execute a command

[done] tests/lib/tools/memory-observe-do (was memory-tool)
[done] tests/lib/tools/version-compare (was version-tool)

[done] tests/bin/mountinfo.query -> tests/lib/tools/mountinfo.query (was mountinfo-tool)
  (we might not do this and keep just tests/lib/tools/mountinfo and call it the long way)

[done] tests/bin/snapd.tool -> tests/lib/tools/snapd.tool (was snap-tool)
  [done] fix no args
  [done] fix to have -h
  (we might not do this, this would be a special case, the name has
   snapd in itself, it makes sense in /bin mostly for debugging,
   otherwise it could stay just in tools as in itself it is not called
   that often by tests, it would just become snapd-tool)

We are then left with a mixture of commands with maybe one or a couple
subcommands dealing with system state:

* simple option rename them with a -state suffix in most cases:

[done] user-tool -> test/lib/tools/user-state
  [done] fix with no args
  [done] fix -h help

[done] apt-tool -> tests/lib/tools/apt-state
  [done] fix no args
  [done] fix to have -h

[done] lxd-tool -> tests/lib/tools/lxd-state
  [done] fix no args
  [done] fix to have -h

[done] invariant-tool -> tests/lib/tools/tests.invariant? then it could also live in /bin
  [done] fix to have -h

* more complicated option, have a tests.state trampoline command in tests/bin
  that given `tests.state WHAT ARGS` invokes `tests/lib/tools/state-WHAT ARGS`

  then we would have in tools:

  state-apt (was apt-tool)
  state-lxd-undo-mount-changes (was lxd-tool)
  state-remove-user-with-group (was user-tool)

  tests.state without arguments or with -h would invoke all the state-* commands
  appropriately to show usage info

  invariant-tool could possibly become `tests.state invariant` or be
  dealt with separately