Bash Testing Tips and Tricks

Helpful Structure
Test Doubles and Dynamic Scope
Unix, Files, and Containers: Can I make something unmockable?
- Hack 1: temporarily replace the script
- Hack 2: chroot
But WHY?
- Everywhere
- Close to the OS

When we're in the business of messing with files and installing things on unix systems, bash can be a really useful programming language to have around.

I like to be able to test my programs. Below I'm going to talk about the technology I commonly use for testing bash programs. I'll cover:

Helpful Structure
Test Doubles and Dynamic Scope
Unix, Files, and Containers: Can I make something unmockable?
But WHY?

Helpful Structure

As a piece of system automation grows, I'll often find clusters of three files start to appear:

-rwxrwxr-x util
-rw-rw-r-- util_libs.bash
-rw-rw-r-- util_libs_test.bash

The util file is executable. It is the script I call when I want to use this utility. The logic for this utility is defined in functions which live in util_libs.bash. The tests for those functions live in util_libs_test.bash. I make a point of having no top-level commands in util_libs.bash. It is intended to be loaded with source util_libs.bash, and contains only function definitions.

I like to use basht for my tests, and can run them all with basht *_test.bash. basht is a pretty minimal testing tool. The basht README will tell you how to write and run a basic test. That's all it gives you, and that's all we need from it. The rest we can get from bash and unix.

Test Doubles and Dynamic Scope

I sometimes like to be able to use (something like) test doubles in my tests. Of course, bash isn't object-oriented, so the metaphor is a little streched… But it turns out that we can get quite close to what I think of as the spirit of test-doubles, just using pure bash, with no need for frameworks.

If you haven't come across test doubles before, a good place to start reading is Martin Fowler's definitions. Sorts of test doubles include dummies, stubs, spies, fakes, and mocks. In any case, the rough idea will hopefully become clear enough in the stuff below.

The key idea in almost everything below is to use the dynamic scoping of bash functions and variables to inject hand-written doubles. These tricks work easily in bash, elisp, powershell and so on. They are much harder to make work in most modern lexically scoped languages.

(It's probably worth noting that I'm not arguing that dynamic scoping is a good thing. I'm just taking my advantages where I can get them.)

A Basic Stub

Suppose we have a basic basht test file that looks something like this:

source ./file_that_contains_functions_we_want_to_test.bash

T_some_test() {
    if [[ "$(some_function_we_want_to_test)" != "some output we expect to see" ]]
    then
        $T_fail "I expected to see something different from what I saw"
        return
    fi
}

But what if some_function_we_want_to_test is quite big, and we want to mock out some part of it?

Let us suppose that part of ./file_that_contains_functions_we_want_to_test.bash looks like this:

get_a_bunch_of_data() {
    ...
}
some_function_we_want_to_test() {
    local mydata
    mydata="$(get_a_bunch_of_data)"
    if somethingsomething "$mydata"
    then
        otherthings "$mydata"
        ...
    fi
}

If we just want to test the logic in some_function_we_want_to_test, we can provide a stub of get_a_bunch_of_data in our test file like so:

source ./file_that_contains_functions_we_want_to_test.bash

get_a_bunch_of_data() {
    echo "some fake data"
}
T_some_test() {
    if [[ "$(some_function_we_want_to_test)" != "some output we expect to see" ]]
    then
        $T_fail "I expected to see something different from what I saw"
        return
    fi
}

Notice that we source the file containing the real definition of get_a_bunch_of_data before we declare the stub version of get_a_bunch_of_data. Because bash is dynamically scoped, the later declaration of the function overwrites the earlier one. By the time the code under test is called, our stub function is in place, and is used by the code we're testing.

Tiny Variant 1: Injecting Commands

Now let's imagine that ./file_that_contains_functions_we_want_to_test.bash is a bit different:

some_function_we_want_to_test() {
    local mydata
    mydata="$(curl http://some.url.com/some/page)"
    if somethingsomething "$mydata"
    then
        otherthings "$mydata"
        ...
    fi
}

We've taken away the handy mockable-function get_a_bunch_of_data and replaced it with curl, which is a binary not a function.

Fortunately, in bash, functions take precedence over binaries!

So we can do exactly the same testing trick like this:

source ./file_that_contains_functions_we_want_to_test.bash

curl() {
    echo "some fake data"
}

T_some_test() {
    if [[ "$(some_function_we_want_to_test)" != "some output we expect to see" ]]
    then
        $T_fail "I expected to see something different from what I saw"
        return
    fi
}

Tiny Variant 1: Working with Scripts

All the stuff we did above only works because the function under test has access to the functions we define in our test code. But what if the thing we want to test isn't a function in a bash library but is a bash script?

Let us suppose that we have a bash script called script_to_test. It contains the following:

#!/usr/bin/env bash

mydata="$(curl http://some.url.com/some/page)"
if somethingsomething "$mydata"
then
  otherthings "$mydata"
  ...
fi
...

We'd like to be able to do the same trick, but this won't work:

curl() {
  echo "some fake data"
}

T_completely_broken_test() {
  if [[ "$(./script_to_test)" != "some output we expect to see" ]]
  then
    $T_fail "I expected to see something different from what I saw"
    return
  fi
}

!!! That test will fail to inject our stub curl into the script we want to test. The real curl will be used instead, and will make the network request we wanted to avoid. !!!

Fortunately, we can fix this problem with one line:

curl() {
  echo "some fake data"
}

T_fixed_test() {
  export -f curl   # <-------------- totally vital fix
  if [[ "$(./script_to_test)" != "some output we expect to see" ]]
  then
    $T_fail "I expected to see something different from what I saw"
    return
  fi
}

The reason the broken test doesn't work is because when we call ./script_to_test, a second instance of the bash interpreter starts, for the second script.

That second instance of bash doesn't have all the variables and functions defined that we have in our first instance. In particular, it doesn't know about our curl function. So when the script asks for curl, all it can find is the binary on the disk.

The line export -f curl says "from now on, any time you start a new bash instance, I want that instance to have this function available too"

You can do the same thing without the -f if you want to export regular variables. The -f here is because our curl isn't a variable, it's a function. In bash it's possible to have a function with the same name as a variable.

Functions and Variables

Oh, and if "it's possible to have a function with the same name as a variable" feels weird, watch what happens when we create a variable named foo and a function named foo in an interactive bash session:

; foo="bar"
; echo $foo
bar
; foo() { echo "blort" ; }
; echo $foo
bar
; foo
blort
; echo $foo
bar

That's why export needs the -f argument if you want it to export a function, rather than a variable.

Unix, Files, and Containers: Can I make something unmockable?

All of this business about using dynamic scoping to reach into a program and re-write bits of it before we run it might be a little uncomfortable. What if I want to write a bash script that makes it hard for someone else to do this?

Well, I don't think it's practical to do so. If you want a lexically scoped language, choose a lexically scoped language. However, there's fun stuff to be learned by trying!

Here's an option:

#!/usr/bin/env bash

my_immutable_function() {
    echo "Probably impossible to override this"
}

mydata="$(my_immutable_function)"
if somethingsomething "$mydata"
then
    otherthings "$mydata"
    ...
fi

...

Notice that the function is declared immediately before it's used. There's no time for any other context to nip in and re-define it. It feels like this will do the trick, but it means your functions can only be declared and used in this one script. We lose the possible advantages of pulling functions out into libraries of their own.

Another option is to stop using functions altogether, and only ever use full paths to scripts:

mydata="$(/home/gds/scripts/myscript)"

That's harder to inject into, because we're not referring to anything by name, we're referring by a full path.

If I wanted to mess with that in a test, I'd probably have to do either of the following hacks:

Hack 1: temporarily replace the script

T_my_test() {
    mv /home/gds/scripts/myscript{,.bak}
    cp /home/gds/script/{stub_script,myscript}
    if somethingsomething "$(function_under_test)"
    then
        mv /home/gds/scripts/myscript{.bak,}
        $T_fail "my failure message"
        return
    else
        mv /home/gds/scripts/myscript{.bak,}
    fi
}

The disadvantage is that this messes with the global state of the system. If anyone else is using /home/gds/scripts/myscript at the time, their stuff will break while our test runs.

Hack 2: chroot

T_my_test() {
    mkdir -p /tmp/testing_jail/home/gds/scripts
    cp /home/gds/script/stub_script /tmp/testing_jail/home/gds/scripts
    if somethingsomething "$(chroot /tmp/testing_jail function_under_test)"
    then
        $T_fail "my failure message"
        return
    fi
}

Look at man 8 chroot and man 2 chroot for details on how that one works. The key is that the command being run thinks that / refers to the directory that we previously thought of as /tmp/testing_jail.

This is a part of the technology behind containers, where we tend to use the slightly more powerful pivot_root. You can find out about that with man 8 pivot_root to learn about the shell command, or man 2 pivot_root to learn about the C function.

There's also a golang function called pivot_root, which I'm pretty sure literally just calls the C one. You can see that function used in practice in Julz Freidman's excellent Build a container in less than 100 lines of go article. I think Julz may have been the first person to live-code a container in a tech talk, but when I search youtube for the video I see lots of other people doing so, and not him. Perhaps he started a trend?

But WHY?

There are lots of nice lexicaly scoped languages out there that aren't bash. Object oriented languages, functional languages, and even other scripting languages. Why do I insist on TDDing bash?

It's everywhere
It's close to the OS

Everywhere

Most of the tricks above work even if my final scripts are intended to be run in a busybox container with a tiny implementation of /bin/sh. Bash is on (almost) all linux and mac computers by default, and we can write sh-compatible scripts if we want to run in even more places. Sure, I can totally install other language environments in my fleet of cloud machines, but the thing that installs them in the first place is almost always bash.

Close to the OS

When it comes to doing things in and with unix-like operating systems, I think almost nothing can match the tight integration that bash has. Notice that injecting a test double for a "shelled out" command was exactly the same as injecting a test double for a "function". Notice that a discussion about bash unit testing started straying into describing container technology. The UNIX filesystem is a core data structure in the bash language, much like lists in lisp and structs in C.

Unix is written in C, but it's operated in the shell. If you're interested in automating the operation of unix machines, I think bash is a great place to start.