general-tests

Last updated: 2019-01-22 19:29:23 +0000

Upstream URL: git clone http://chriswarbo.net/git/general-tests.git

Repo

View repository

View issue tracker

Contents of README follows


Simple test suite

I use this as a high-level test harness, testing as many of my projects as possible; like a poor man’s continuous integration server.

Architecture

Tests are defined in the *.nix files in tests/ (except for tests/default.nix). Each test is a Nix derivation which produces a script: running that script runs the test. Each file in tests/ can either define a single test or an attrset of tests (nested to arbitrary depth).

For example, tests/hlint.nix defines an attrset, where each value is a script for running hlint on a different Haskell project.

Running

The top-level default.nix file defines a script which runs all of these tests and counts passes/fails. Each test script is wrapped in a script which writes either PASS or FAIL to an appropriately named file in /tmp, so we can find out which tests have failed, etc. The main runner will delete any of these files which don’t correspond to a known test.

To build the main runner, and hence all of the tests scripts and their wrappers, just run nix-build in the top-level directory: the path to the runner script will be printed on stdout, and a symlink called result will also point to it. Running that script will invoke the whole test suite. If you want to run an individual test (via its wrapper), look for the tests variable in the main runner script: that’s where all of the individual tests can be found.

The advantage to this two step build-then-run approach is that we can have the build step perform some time-consuming tasks, like using find to look for shell scripts, git repos, etc. and generate test scripts based on what’s found. Once they’re built, we can run those scripts again and again without having to perform the slow tasks again. When we want to “flush the cache” we just run nix-build to generate new scripts.

For convenience, the included run script will build and run the test suite.

Debugging

The top-level script prints out the path to each test script as it goes. To investigate why a particular test is failing, just run it individually to see the stdout, stderr and exit code.

There are two ways to find an individual test without running the whole suite. One way is to look at the env vars defined in the top-level script, which include the directory storing the generated test scripts.

Another way is to look in the Nix derivations, for example using the nix-repl command. The top-level default.nix file takes a packageOnly parameter, defaulting to true: when this is true only the top-level script is produced; when false it returns an attrset containing the top-level script, all of the individual test derivations and the utilities from helpers/.

Extras

The release.nix file allows tests to be run on the Hydra continuous integration server. This is nice to have, but some tests only make sense on my development machine; for example tests/all-committed.nix checks that my git repositories don’t contain any uncommitted things. This wouldn’t make sense on a continuous integration server, since it will only see committed things.