haskell-te: 4263a8b42715e8cbec01fbce4d667239b830b863

     1: From: Chris Warburton
     2: Date: Wed, 12 Jul 2017 13:30:44 +0100
     3: State: resolved
     4: Subject: Use asv for running benchmarks
     5: Message-Id: <f535fc881e4491fe-0-artemis@nixos>
     6: resolution: fixed
     7: 
     8: We should benchmark:
     9: 
    10:  - hashspec
    11:  - quickspec
    12:  - mlspec
    13: 
    14: These should probably be different dependencies.
    15: 
    16: We should add benchmarks for:
    17: 
    18:  - Runtime
    19:  - Precision
    20:  - Recall
    21: 
    22: It would be nice to share the same execution for the results of these,
    23: e.g. running `choose_sample 1 1` once, running it through quickspec
    24: once, then spitting out a runtime, precision and recall from those
    25: results.
    26: 
    27: What we could do is run it in a setup_cache function, which would be
    28: re-used across runs for the same version. Unfortunately that's not
    29: parameterised, so it would all have to be done up front. Similarly,
    30: doing the run inside a dependency would require all results to be
    31: generated up front. Also, these wouldn't let us use asv's built-in
    32: support for repetitions and parameters, which match what we want quite
    33: closely.
    34: 
    35: How about, for now, we just implement these as standalone things? We can
    36: worry about optimising them through re-use later.