haskell-te: 64a61dfb1c33e52f0f541b032be999049c6450e9

     1: From: Chris Warburton
     2: Date: Tue, 01 Aug 2017 17:29:08 +0100
     3: State: resolved
     4: Subject: Tease out independent components again.
     5: Message-Id: <bdc1e1f920a1ecfa-0-artemis@nixos>
     6: resolution: fixed
     7: 
     8: This began as a collection of bash scripts (oh how times have changed!),
     9: which were at least runnable as standalone commands.
    10: 
    11: The closest we have right now are the quickspecBench, hashspecBench and
    12: mlspecBench commands, which are awful for the following reasons:
    13: 
    14:  - Their primary purpose is to benchmark these systems, rather than just
    15:    run them.
    16:  - They're FAR too complicated, mostly because of the benchmarking
    17:    mentioned above.
    18:  - They try to do too much, for example they include one mode which
    19:    samples from TEB and one mode which takes TIP from stdin. They
    20:    include TIP to Haskell package translation (from TEB). And so on.
    21:  - Their functionality is reasonably modular, but it's invoked in weird
    22:    ways (e.g. calling out to nix-build).
    23: 
    24: A lot of this is due to the requirements of benchmarking: we need to
    25: repeat one bunch of work (the exploration) without repeating other
    26: expensive work (like AST extraction); we need to be flexible enough to
    27: support arbitrary packages, whilst ensuring the benchmarked sections
    28: don't invoke any compilers; and so on.
    29: 
    30: Using asv lets us offload a bunch of these concerns:
    31: 
    32:  - Benchmarking and repetition are handled by our Python benchmark
    33:    scripts.
    34:  - Preparing the input (translation, etc.) can be taken care of by Nix
    35:    and/or Python.
    36:  - Compiling, setting up an appropriate environment, etc. can be taken
    37:    care of by Nix, in benchmarks/default.nix.
    38: 
    39: We should try, as much as possible, to turn the actual exploration
    40: systems into reasonably-standalone scripts. These should probably be
    41: something like:
    42: 
    43:  - A TIP to haskell package script (taken from TEB)
    44:  - A Haskell package to ASTs script
    45:  - A TIP sample to ASTs script
    46:  - A script to explore ASTs with QuickSpec (no reduction)
    47:  - Ditto for HashSpec (no reduction)
    48:  - Ditto for MLSpec (no reduction)
    49: 
    50: These should, as far as possible, avoid doing stuff like caching to the
    51: Nix store. They don't need to, and they're modular enough that we can do
    52: so in our benchmark environments.