Commander

Define language independent tests for your command line scripts and programs in simple yaml files.

• It runs on windows, osx and linux
• It can validate local machines, ssh hosts and docker containers
• It is a self-contained binary - no need to install a heavy lib or language
• It is easy and fast to write

For more information take a look at the quick start, the examples or the integration tests.

Table of contents

• Installation
• Any system with Go installed
• Linux & osx
• Windows
• Quick start
• Complete YAML file
• Executing
• Adding tests
• Documentation
• Usage
• Tests
  • command
  • config
  • exit-code
  • stdout
  • contains
  • exactly
  • json
  • lines
  • line-count
  • not-contains
  • xml
  • file
  • stderr
  • skip
• Config
  • dir
  • env
  • inherit-env
  • interval
  • retries
  • timeout
  • nodes
• Nodes
  • local
  • ssh
  • docker
• Development
• Misc

Installation

Any system with Go installed

Probably the easiest way to install commander is by using go get to download and install it in one simple command:

go get github.com/commander-cli/commander/v2/cmd/commander

This works on any OS, as long as go is installed. If go is not installed on your system, follow one of the methods below.

Linux & osx

Visit the release page to get the binary for you system.

curl -L https://github.com/commander-cli/commander/releases/download/v2.5.0/commander-linux-amd64 -o commander
chmod +x commander

Windows

• Download the current release
• Add the path to your path environment variable
• Test it: commander --version

Quick start

A commander test suite consists of a config and tests root element. To start quickly you can use the following examples.

# You can even let commander add tests for you!
$ ./commander add --stdout --file=/tmp/commander.yaml echo hello
tests:
  echo hello:
    exit-code: 0
    stdout: hello

written to /tmp/commander.yaml

# ... and execute!
$ ./commander test /tmp/commander.yaml
Starting test file /tmp/commander.yaml...

✓ echo hello

Duration: 0.002s
Count: 1, Failed: 0

Complete YAML file

Here you can see an example with all features for a quick reference

nodes:
  ssh-host1:
    type: ssh
    addr: 192.168.0.1:22
    user: root
    pass: pass
  ssh-host2:
    type: ssh
    addr: 192.168.0.1:22
    user: root
    identity-file: /home/user/id_rsa.pub
  docker-host1:
    type: docker
    image: alpine:2.4
  docker-host2:
    type: docker
    instance: alpine_instance_1

config: # Config for all executed tests
    dir: /tmp #Set working directory
    env: # Environment variables
        KEY: global
    timeout: 50s # Define a timeout for a command under test
    retries: 2 # Define retries for each test
    nodes:
    - ssh-host1 # define default hosts

tests:
    echo hello: # Define command as title
        stdout: hello # Default is to check if it contains the given characters
        exit-code: 0 # Assert exit-code

    it should skip:
        command: echo "I should be skipped"
        stdout: I should be skipped
        skip: true

    it should fail:
        command: invalid
        stderr:
            contains: 
                - invalid # Assert only contain work
            not-contains:
                - not in there # Validate that a string does not occur in stdout
            exactly: "/bin/sh: 1: invalid: not found"
            line-count: 1 # Assert amount of lines
            lines: # Assert specific lines
                1: "/bin/sh: 1: invalid: not found"
            json:
                object.attr: hello # Make assertions on json objects
            xml:
                "//book//auhtor": Steven King # Make assertions on xml documents
            file: correct-output.txt
        exit-code: 127
        skip: false

    it has configs:
        command: echo hello
        stdout:
            contains: 
              - hello #See test "it should fail"
            exactly: hello
            line-count: 1
        config:
            inherit-env: true # You can inherit the parent shells env variables
            dir: /home/user # Overwrite working dir
            env:
                KEY: local # Overwrite env variable
                ANOTHER: yeah # Add another env variable
            timeout: 1s # Overwrite timeout
            retries: 5
            nodes: # overwrite default nodes
              - docker-host1
              - docker-host2
        exit-code: 0

Executing

# Execute file commander.yaml in current directory
$ ./commander test 

# Execute a specific suite
$ ./commander test /tmp/test.yaml

# Execute a single test
$ ./commander test /tmp/test.yaml --filter "my test"

# Execute suite from stdin
$ cat /tmp/test.yaml | ./commander test -

# Execute suite from url
$ ./commander test https://your-url/commander_test.yaml

# Execute suites within a test directory
$ ./commander test --dir /tmp

# Execute suites in a different working directory 
$ ./commander test --workdir /examples minimal_test.yaml

Adding tests

You can use the add argument if you want to commander to create your tests.

# Add a test to the default commander.yaml
$ ./commander add echo hello
written to /tmp/commander.yaml

# Write to a given file
$ ./commander add --file=test.yaml echo hello
written to test.yaml

# Write to stdout and file
$ ./commander add --stdout echo hello
tests:
  echo hello:
    exit-code: 0
    stdout: hello

written to /tmp/commander.yaml

# Only to stdout
$ ./commander add --stdout --no-file echo hello
tests:
  echo hello:
    exit-code: 0
    stdout: hello

Documentation

Usage

NAME:
   Commander - CLI app testing

USAGE:
   commander [global options] command [command options] [arguments...]

COMMANDS:
     test     Execute the test suite
     add      Automatically add a test to your test suite
     help, h  Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --help, -h     show help
   --version, -v  print the version

Tests

Tests are defined in the tests root element. Every test consists of a command and an expected result, i.e. an exit-code.

tests: # root element
  echo test: # test case - can either be the command or a given title
    stdout: test
    exit-code: 0

A test is a map which configures the test. The key (echo test in the example above) of the test can either be the command itself or the title of the test which will be displayed in the test execution.

If the same command is tested multiple times it is useful to set the title of the test manually and use the command property. Further the title can be useful to describe tests better. See the commander test suite as an example.

• name: title or command under test
• type: map
• default: {}

Examples:

tests:
  echo test: # command and title will be the same
    stdout: test
    exit-code: 0

  my title: # custom title
    command: exit 1 # set command manually
    exit-code: 1

command

command is a string containing the command to be tested. Further the command property is automatically parsed from the key if no command property was given.

• name: command
• type: string
• default: can't be empty
• notes: Will be parsed from the key if no command property was provided and used as the title too

echo test: # use command as key and title
  exit-code: 0

it should print hello world: # use a more descriptive title...
  command: echo hello world  # ... and set the command in the property manually
  stdout: hello world
  exit-code: 0

config

config sets configuration for the test. config can overwrite global configurations.

• name: config
• type: map
• default: {}
• notes:
• for more information look at config

echo test:
  config:
    timeout: 5s

exit-code

exit-code is an int type and compares the given code to the exit-code of the given command.

• name: exit-code
• type: int
• default: 0

exit 1: # will pass
  exit-code: 1
exit 0: # will fail
  exit-code: 1

stdout

stdout and stderr allow to make assertions on the output of the command. The type can either be a string or a map of different assertions.

If only a string is provided it will check if the given string is contained in the output.

• name: stdout
• type: string or map
• default:
• notes: stderr works the same way

echo test:
  stdout: test # make a contains assertion

echo hello world:
  stdout:
    line-count: 1 # assert the amount of lines and use stdout as a map

contains

contains is an array or string. It checks if a string is contained in the output. It is the default if a string is directly assigned to stdout or stderr.

• name: contains
• type: string or array
• default: []
• notes: default assertion if directly assigned to stdout or stderr

echo hello world:
  stdout: hello # Default is a contains assertion

echo more output:
  stdout:
    contains:
      - more
      - output

exactly

exactly is a string type which matches the exact output.

• name: exactly
• type: string
• default:

echo test:
  stdout:
    exactly: test

json

json is a map type and allows to parse json documents with a given GJSON syntax to query for specific data. The key represents the query, the value the expected value.

• name: json
• type: map
• default: {}
• notes: Syntax taken from GJSON

cat some.json: # print json file to stdout
  name.last: Anderson # assert on name.last, see document below

some.json file:

{
  "name": {"first": "Tom", "last": "Anderson"},
  "age":37,
  "children": ["Sara","Alex","Jack"],
  "fav.movie": "Deer Hunter",
  "friends": [
    {"first": "Dale", "last": "Murphy", "age": 44, "nets": ["ig", "fb", "tw"]},
    {"first": "Roger", "last": "Craig", "age": 68, "nets": ["fb", "tw"]},
    {"first": "Jane", "last": "Murphy", "age": 47, "nets": ["ig", "tw"]}
  ]
}

More examples queries:

"name.last"          >> "Anderson"
"age"                >> 37
"children"           >> ["Sara","Alex","Jack"]
"children.#"         >> 3
"children.1"         >> "Alex"
"child*.2"           >> "Jack"
"c?ildren.0"         >> "Sara"
"fav\.movie"         >> "Deer Hunter"
"friends.#.first"    >> ["Dale","Roger","Jane"]
"friends.1.last"     >> "Craig"

lines

lines is a map which makes exact assertions on a given line by line number.

• name: lines
• type: map
• default: {}
• note: starts counting at 1 ;-)

echo test\nline 2:
  stdout:
    lines:
      2: line 2 # asserts only the second line

line-count

line-count asserts the amount of lines printed to the output. If set to 0 this property is ignored.

• name: line-count
• type: int
• default: 0

echo test\nline 2:
  stdout:
    line-count: 2

not-contains

not-contains is a array of elements which are not allowed to be contained in the output. It is the inversion of contains.

• name: not-contains
• type: array
• default: []

echo hello:
  stdout:
    not-contains: bonjour # test passes because bonjour does not occur in the output

echo bonjour:
  stdout:
    not-contains: bonjour # test fails because bonjour occurs in the output

xml

xml is a map which allows to query xml documents viá xpath queries. Like the [json][#json] assertion this uses the key of the map as the query parameter to, the value is the expected value.

• name: xml
• type: map
• default: {}
• notes: Used library xmlquery

cat some.xml:
  stdout:
    xml:
      //book//author: J. R. R. Tolkien

some.xml file:

<book>
    <author>J. R. R. Tolkien</author>
</book>

file

file