Last updated: 2019-01-22 19:29:23 +0000
Upstream URL: git clone http://chriswarbo.net/git/general-tests.git
Contents of README follows
I use this as a high-level test harness, testing as many of my projects as possible; like a poor man’s continuous integration server.
Tests are defined in the <code>*.nix</code> files in <code>tests/</code> (except for <code>tests/default.nix</code>). Each test is a Nix derivation which produces a script: running that script runs the test. Each file in <code>tests/</code> can either define a single test or an attrset of tests (nested to arbitrary depth).
For example, <code>tests/hlint.nix</code> defines an attrset, where each value is a script for running hlint on a different Haskell project.
The top-level <code>default.nix</code> file defines a script which runs all of these tests and counts passes/fails. Each test script is wrapped in a script which writes either <code>PASS</code> or <code>FAIL</code> to an appropriately named file in <code>/tmp</code>, so we can find out which tests have failed, etc. The main runner will delete any of these files which don’t correspond to a known test.
To build the main runner, and hence all of the tests scripts and their wrappers, just run <code>nix-build</code> in the top-level directory: the path to the runner script will be printed on stdout, and a symlink called <code>result</code> will also point to it. Running that script will invoke the whole test suite. If you want to run an individual test (via its wrapper), look for the <code>tests</code> variable in the main runner script: that’s where all of the individual tests can be found.
The advantage to this two step build-then-run approach is that we can have the build step perform some time-consuming tasks, like using <code>find</code> to look for shell scripts, git repos, etc. and generate test scripts based on what’s found. Once they’re built, we can run those scripts again and again without having to perform the slow tasks again. When we want to “flush the cache” we just run <code>nix-build</code> to generate new scripts.
For convenience, the included <code>run</code> script will build and run the test suite.
The top-level script prints out the path to each test script as it goes. To investigate why a particular test is failing, just run it individually to see the stdout, stderr and exit code.
There are two ways to find an individual test without running the whole suite. One way is to look at the env vars defined in the top-level script, which include the directory storing the generated test scripts.
Another way is to look in the Nix derivations, for example using the <code>nix-repl</code> command. The top-level <code>default.nix</code> file takes a <code>packageOnly</code> parameter, defaulting to <code>true</code>: when this is <code>true</code> only the top-level script is produced; when <code>false</code> it returns an attrset containing the top-level script, all of the individual test derivations and the utilities from <code>helpers/</code>.
The <code>release.nix</code> file allows tests to be run on the Hydra continuous integration server. This is nice to have, but some tests only make sense on my development machine; for example <code>tests/all-committed.nix</code> checks that my git repositories don’t contain any uncommitted things. This wouldn’t make sense on a continuous integration server, since it will <em>only</em> see committed things.