[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Use asv for running benchmarks



We should benchmark:

 - hashspec
 - quickspec
 - mlspec

These should probably be different dependencies.

We should add benchmarks for:

 - Runtime
 - Precision
 - Recall

It would be nice to share the same execution for the results of these,
e.g. running `choose_sample 1 1` once, running it through quickspec
once, then spitting out a runtime, precision and recall from those
results.

What we could do is run it in a setup_cache function, which would be
re-used across runs for the same version. Unfortunately that's not
parameterised, so it would all have to be done up front. Similarly,
doing the run inside a dependency would require all results to be
generated up front. Also, these wouldn't let us use asv's built-in
support for repetitions and parameters, which match what we want quite
closely.

How about, for now, we just implement these as standalone things? We can
worry about optimising them through re-use later.