haskell-te: 033e935507a877cd26359b66c6de1b4ea5e8b1ba

     1: From: Chris Warburton
     2: Date: Tue, 01 Aug 2017 17:47:12 +0100
     3: State: resolved
     4: Subject: See if our benchmark setup could be built by Nix
     5: Message-Id: <a48cf44569ff379d-0-artemis@nixos>
     6: resolution: fixed
     7: 
     8: The setup_cache functions in our benchmarks are getting quite
     9: complicated. There's a lot this that we could probably do with Nix,
    10: rather than just giving access to a few env vars.
    11: 
    12: We specifically DON'T want to run the benchmarking tools via Nix, since
    13: that can cause them to interfere (e.g. if Nix builds a bunch of stuff in
    14: parallel).
    15: 
    16: We can do the sampling, translation, etc. up front in Nix.
    17: 
    18: We can turn the parameters.py file into a JSON file, which can be read
    19: by Nix (and maybe passed in to Python via the environment?)
    20: 
    21: Analysing the results will still need to be done from Python, since that
    22: requires the stdout of the benchmarked runs. We might be able to package
    23: this up a little better though.
    24: 
    25: I think there's a balance to be struck in general:
    26: 
    27:  - Nix is good at running programs, reading files, setting up
    28:    environments, etc. so it should be used for that.
    29:  - Python is good for manipulating data (e.g. JSON or otherwise).
    30: 
    31: We should focus each on their strengths. In particular, we can use
    32: Python instead of jq in Nix, and we can use Nix instead of subprocess32
    33: in Python.