--- title: Continuous Integration --- Whilst Nix's strict control of software versions and dependencies is excellent when handling *other people's* programs, it can be a little frustrating when developing one's own software. In particular, version control systems like Git have their own notion of immutable state (commits) and dependencies (each commit's hash includes that of its parent, just like Nix derivations include their dependencies' hashes). This similarity can often lead to overlaps, where I need to synchronise Nix packages with git updates, or vice versa. I've tried several methods of resolving this, but have finally found one that I'm happy with. Here I'll explain these approaches. ## Manual Git Revisions ## This is the most tedious, but it ensures you're using a known-good configuration. For example, this is the way you'd manage an official package in nixpkgs. Let's say we're maintaining the following packages `foo` and `bar`. For the sake of argument, let's say we're maintaining these definitions in our `~/.nixpkgs/config.nix` file: ``` foo = stdenv.mkDerivation { name = "foo"; src = fetchgit { url = "https://example.com/foo.git"; rev = "f000001"; sha256 = "f000000000000000000000000000000000000000000000000001"; }; }; bar = stdenv.mkDerivation { name = "bar"; src = fetchgit { url = "https://example.com/bar.git"; rev = "b000001"; sha256 = "b000000000000000000000000000000000000000000000000001"; }; buildInputs = [ foo ]; }; ``` Now let's say we make a change to `foo`, which lives in a new git revision `f000002`. We need to go and edit our package definition: ``` foo = stdenv.mkDerivation { name = "foo"; src = fetchgit { url = "https://example.com/foo.git"; rev = "f000002"; sha256 = "f000000000000000000000000000000000000000000000000001"; }; }; ``` In fact, this package will not build, since the SHA256 hash will not match. Keep in mind that the SHA sum of the git commit doesn't have anything to do with the SHA sum of the nix derivation, so we can't just copy it over. Instead, we must ask Nix what the new SHA sum should be, and the easiest way to do that is to attempt to build the package: ``` $ nix-shell -p foo ... output path ‘/nix/store/...foo’ should have r:sha256 hash ‘f000000000000000000000000000000000000000000000000001’, instead has ‘f000000000000000000000000000000000000000000000000002’ ... ``` This is what we were expecting, so we can now safely copy that new hash in place of the old one: ``` foo = stdenv.mkDerivation { name = "foo"; src = fetchgit { url = "https://example.com/foo.git"; rev = "f000002"; sha256 = "f000000000000000000000000000000000000000000000000002"; }; }; ``` Clearly, this is a tedious process. The worst part is that we need to do the same thing even for *tiny* changes. Let's say that `bar` requires some small change to the API of `foo`, e.g. changing a private function into a public one (before you scoff, read on for a treatment of the software engineering aspects!). Even if we have [a nice development workflow for `bar`](developing_on_nixos.html), with this setup we will still need to bump the version of `foo` *globally*, just to have the new version available to `bar`. This is problematic in a couple of ways: firstly, *all* package definitions will now point to the new version of `foo`. Of course, thanks to the way Nix works this won't affect any *existing* derivations (i.e. all of our installed packages will carry on working as-is), however it will affect *new* derivations, like those instantiated by `nix-shell`. Symptoms might include a load of expensive rebuilds, or subtle breakages that we don't want to contend with *right now* (software engineers, read on!). The other problem is that our change might not work! With this approach, we must go to all this effort and bump the global version of `foo` *just to try something out*! As for the software engineering perspective, it's certainly true that any change in public API should lead to a version increase, and is not considered "tiny". It's also true that we should try not to release new API versions if they're known to break existing clients, at least without understanding whether the breakages are acceptable. *However*, our problem shouldn't have anything to do with releases and versions! Even at the stage of *experimenting* and *prototyping*, to see if the proposed change will even work in the first place, we are suddenly forced to deal with a release management scenario! This is obviously a bad state of affairs, in particular because it penalises modularity: these headaches in tying packages together will subconsciously bias us against re-using existing components, or splitting up our task into smaller sub-tasks. ## Ignoring Git ## We can also go to the opposite extreme and ignore version control altogether! Consider the following: ``` foo = stdenv.mkDerivation { name = "foo"; src = ~/Programming/foo; }; bar = stdenv.mkDerivation { name = "bar"; src = ~/Programming/bar; buildInputs = [ foo ]; }; ``` Using hard-coded paths is clearly a bit of a code smell, as these package definitions are no longer portable to other machines. However, look at what we've gained! We no longer need to specify a git revision or an SHA256 checksum: Nix will scan the contents of the directories, and if they've changed from the last build it will copy them into the store and make a new derivation. Unfortunately, there are obvious problems with this approach. In particular, we've still got the issue that any changes we make will have a global effect. In fact, we no longer have to commit something in order to alter our system! If we have unstaged changes in the `foo` or `bar` directories, any new derivations depending on these packages will get those changes, regardless of whether we intended them to or not. This undermines a lot of the purpose of Nix, since we want to have confidence in our system integrity *and* the freedom to develop software without fear of breaking anything with half-finished changes. ## Automating Updates ## Another solution I tried is to automate the process of updating revisions and hashes. I wrote two scripts, called `bumpCommit` and `bumpEverything`, to aid this process. First, the packages would be defined to load their revision and hash attributes from external files: ``` foo = stdenv.mkDerivation { name = "foo"; src = fetchgit { url = "https://example.com/foo.git"; rev = import "./foo.rev.nix"; sha256 = import "./foo.sha256.nix"; }; }; bar = stdenv.mkDerivation { name = "bar"; src = fetchgit { url = "https://example.com/bar.git"; rev = import "./bar.rev.nix"; sha256 = import "./bar.sha256.nix"; }; buildInputs = [ foo ]; }; ``` `bumpCommit` would look at the current working directory, and use a look up table to find the corresponding `rev.nix` and `sha256.nix` files, which it would overwrite with the new versions. Since these package definitions *themselves* are stored in git repositories, running `bumpCommit` on a project may cause changes in other projects, which requires more commits, and so on. That's where `bumpEverything` came in: it would loop through all projects in dependency order, commit any outstanding changes to `rev.nix` or `sha256.nix` files, then invoke `bumpCommit`. This was clearly a hack, and was very fragile in the face of changing projects, e.g. splitting a project into two parts. It also generated a large number of git commits, which would do nothing other than bump revision numbers and hashes. ## latestGit ## My current solution abandons the `bumpCommit` and `bumpEverything` commands. Instead, we try to combine the best features of the git approach *and* the no-git approach. The aim is to treat a git URL as if it were a local directory: no need to specify a particular revision (just fetch a sensible default like `HEAD`, `master`, etc.) and no need to specify a particular hash (just like with local directories, if it's changed then build a new derivation; of course, in practice a particular git revision *will not* change, and is hence safe to cache). To do this, we abuse Nix's idea of derivations in order to run arbitrary scripts and cache the results in the Nix store for subsequent access. First, we write a "package" which is just a single file, containing the latest git revision of a particular repository: ``` with import {}; with builtins; getHeadRev = { url, ref ? "HEAD" }: stdenv.mkDerivation { inherit url ref; name = "repo-head-${hashString "sha256" url}"; version = toString currentTime; # Required for SSL GIT_SSL_CAINFO = "${cacert}/etc/ssl/certs/ca-bundle.crt"; buildInputs = [ git gnused ]; buildCommand = '' source $stdenv/setup # printf is an ugly way to avoid trailing newlines printf "%s" $(git ls-remote "$url" "$ref" | head -n1 | sed -e 's/\s.*//g') > "$out" ''; }; ``` The `getHeadRev` function takes a `url` parameter and, optionally, a `ref` (defaulting to `HEAD`). It then generates a package with a name derived from the given URL, and a version based on the current time; this ensures that we avoid the Nix cache. The contents of the package, stored in the file at location `$out`, are generated by the `buildCommand`, which queries the given URL for the latest revision ID. Next, we need a way to access this revision information from within Nix. We do this by coercing the packages generated by `getHeadRev` into strings, which will correspond the the `$out` path at which they're stored. We use `readFile` to read the contents of the generated files: ``` rev = args: unsafeDiscardStringContext (readFile "${getHeadRev args}"); ``` We use `unsafeDiscardStringContext` to ignore the "dependencies" of this string, i.e. the particular invocation of the `git ls-remote` command; all we care about is the revision, not the time at which it was looked up. Now that we have a git revision, we just need to avoid the hash requirement of `fetchgit`, to prevent it from being a so-called "fixed-output derivation". We do this in two steps, utilising Nix's lazy evaluation. First, we do a regular `fetchgit` invocation: ``` fg = args: fetchgit { url = args.url; rev = rev args; # Dummy hash sha256 = hashString "sha256" args.url; }; ``` Finally, we override this derivation to strip out all of the hashing information: ``` latestGit = args: stdenv.lib.overrideDerivation (fg args) (old: { outputHash = null; outputHashAlgo = null; outputHashMode = null; sha256 = null; }); ``` Without these details, Nix will use the hashes it calculates from the source. Now we can use this `latestGit` function to specify our package sources: ``` foo = stdenv.mkDerivation { name = "foo"; src = latestGit { url = "https://example.com/foo.git"; }; }; bar = stdenv.mkDerivation { name = "bar"; src = latestGit { url = "https://example.com/bar.git"; }; buildInputs = [ foo ]; }; ``` This ensures that Nix and git are always synchronised, since git revisions are now the canonical form of package versions, and Nix will ask Git if these have changed when a new derivation is being instantiated. From a development point of view, this gives us the freedom to tinker and experiment without fear of breaking our system. When our local changes are suitable for wider use, we can do the usual `git push` to make them available to the world; which now includes *our* installation of Nix! Our packages are also portable, as long as `latestGit` is made available somewhere. This doesn't *quite* solve the problem of using experimental versions of `foo` from within `bar`, however we're free to define *extra* packages, like `foo-unstable`, which use the local directory as their source. If we make `foo` a *parameter* of the `bar` package, we can override it with `nix-shell` to get a one-off development shell using `foo-unstable`, without breaking anything, without exposing our dodgy prototypes to the world, and without building a mountain of trivial git commits.