A short getting-started guide to shelltestrunner, which lets you test command-line tools given a stdin, expecting a stdout and stderr, and an exit status.

I have this tendency of getting carried away when writing command-line tools, presumably because some environments such as Python and Node.js make it so easy to write incredibly useful utilities which seamlessly read complex configuration files, and parse sophisticated command-line arguments. So there's no stopping me adding new features. And when I get to test them, I realise that I didn't put sufficient effort to organise the code into stand-alone functions which could have given way to unit test frameworks. At which point I'm bound to refactor all the code.

However, there are utilities for testing command-line tools regardless of their implementation. These utilities expect you to specify how to run them, i.e. with which arguments or options and in which environment, what's on stdin, and what should be eventually expected on stdout, stderr and as exit status. One such utility is shelltestrunner.

1 Installation

Because shelltestrunner is written in Haskell, I was at first afraid that installing it might involve downloading a gigabyte of Haskell libraries, as is the case with xmonad. But not so, it's pretty much self-standing. I quite like the idea, by the way, that the author chose to use a language helping you so much to write correct code for something as crucial as testing.

2 Sample Test

The files which shelltestrunner loads test definitions from bear the .test extension by default and can conveniently be stored in directories which you can supply as argument. There are 3 formats which the official documentation refers to as 1, 2 and 3. Format 1 is rather verbose. It has been in use since the beginning. Formats 2 and 3 offer some interesting shortcuts and means to avoid repetition. They were introduced since version 1.9. I'm using format 1 here, which typically looks like this:

# Sample test
ls /foo
>>>
bar
>>>= 0

This test will probably fail since I'm expecting exit status 0 and some output that's unlikely to match. That is, unless you do have a /foo directory on your system which just contains the bar file. You can define multiple tests in a given file. However, it's hard to have shelltestrunner refer to a specific test if you only want this one run. If you do split up your tests into separate files, the -i and -x options will enable you to choose them. But in turn, this will get in the way of limiting code repetition. It's probably just best to comment out tests. I see a little Vim script coming up...

Speaking of comments, while # can be used to describe each one of your tests, which is good enough for many practical purposes, I think I would have liked shelltestrunner to let me label them for being able to target them right from its command-line interface.

3 Testing the Way of the Shell

The commands defining your tests are run by /bin/sh and can in fact be complete command lines, including control operators such as |, ;, &. They're literally command lines, then. And as such, you probably want to specify the executable you're in the process of developing with ./, like you would in a real shell script, to ensure it's the executable you're writing which you're testing, not some previous version that's already deployed system-wide.

There's no support for pre- and post-sections for your tests, for respectively setting up a testing environment and tearing down – as in tidy up. And while I missed these for a few seconds, I quickly realised it's not how you should regard shelltestrunner: it was written with the UNIX style of doing things in mind in that it blends in perfectly with other tools which might be part of your workflow. If you need pre- and post-sections, why don't you run them as commands too and whack them e.g. in a Makefile which will also be left in charge of running shelltestrunner in between? Or a shell script? That's what I once did for a project of mine, actually:

#!/usr/bin/zsh

mkdir -p tests/tmp
touch tests/tmp/{a,b,c}

shelltest tests

rm -rf tests/tmp

The pre-section entails making directories and creating a few files. Then, shelltestrunner runs. Finally, the post-section recursively removes the previously-created files and directories. That's one of the many ways you can get stuff done with a good shell.

Likewise, shelltestrunner doesn't have a configuration file. Again, I briefly missed one so I could have colour reports enabled by default. But for such a simple use case, what do I need a full-fledged configuration file when I use a shell that lets me define alias? If I want colour reports by default, all I need is kindly ask my shell to sort it out for me:

alias shelltest='shelltest -c'

4 Reporting and Debugging

I found using the --debug option useful for printing stdout and stderr beyond what shelltestrunner does by default e.g. when the output doesn't match. In truth, it's easy to end up in a situation where it all becomes rather unreadable, which is only made worth by the fact you can't conveniently select which tests to run from inside a file. I really do have to write this little Vim script to quickly comment whole tests out.

5 Regular Expressions

One extremely useful feature I was pleased to see shelltestrunner offer is that instead of matching what's on stdout and stderr as a whole, you can use regular expressions. Not to mention that they become essential when you expect formats rather than exact values, or when part of your output is only relevant to specific environments.

Regular expressions find their place between / and multiple lines are supported. Does lsb_release correctly include the Distributor, Description, Release and Codename? Let's find out:

lsb_release -a
>>> /Distributor ID:.*
Description:.*
Release:.*
Codename:.*
/
>>>= 0

Because regular expressions are delimited between /, using / in your regular expressions is a bit of a faff as you need to escape them. Is xorg.conf.d located under a subdirectory /X11/ of /etc?

find /etc -name xorg.conf.d
>>> /\/X11\//
>>>= //

On a side note, since I do have to specify the expected exit status in format 1 but I don't care anyway, I used a regular expression there too. An empty one, one that matches anything. It so happens that find is likely to exit non-zero searching /etc if you're not root because some files are for the administrator's eyes only.

One word of caution when specifying expected output: while you do have to start your regular expression in line with >>>, you mustn't do when you supply verbatim output – start on the next line in this case. This got me a few times.

6 Beyond Testing Command-Line Tools

Again, following the UNIX way of getting about, since shelltestrunner is only interested in what you expect from your various output streams and exit status of whichever command line you supply, you can naturally use it beyond its expected usage to test not the command-line itself, but indirectly its object. You could for instance very easily add a pylint run of the software you're developing as part of your tests. Something as concise as:

pylint foo
>>>= 0

It's not about testing pylint, it's about using pylint to test your software. Another even more creative example is to test a REST interface by using curl against different endpoints and check the results, possibly using jq:

curl https://jsonplaceholder.typicode.com/todos | jq '. | length'
>>>
200
>>>= //

I'm literally wondering whether or not the data set provided by various REST API example services is always the same. Using shelltestrunner, I could establish that it's the case for https://jsonplaceholder.typicode.com which must then be read-only but not for http://dummy.restapiexample.com which presumably lets anyone push their own data. I even had to suppress the urge to periodically make sure of this by setting up a systemd timer to do so automatically. When UNIX gives you wings...

7 Unit Tests?

But what about unit tests, if you still need them? That probably depends on how flexible your development environment is, but if you're writing Python, it's a doddle. Does len() work with arrays?

python -c 'print len([1,2,3])'
>>>
3
>>>= //

Before long, you will want to use shelltestrunner for just about everything, just because there's no complex setup or syntax involved to quickly test something out.

8 Reference

• shelltestrunner: Easy, repeatable testing of CLI programs/commands on Hackage
• Easy, repeatable testing of CLI programs/commands on GitHub, which offers a more readable README file presumably because its parsing didn't go belly up the way it did on Hackage
• JSONPlaceholder and Dummy sample rest api
• The hdupes tool for which I used shelltestrunner