Work log 2016-08-23

Store IsaCoSy results for Nat, and times.

Problem with MLSpec exploring list-extras. Maybe unpatched ifcxt, tried pulling mikeizbicki master, which our patches are now merged into. Seems to have fixed it.

Problem with reduce.equations. Looks like we’re merging each cluster’s equations improperly:

$ cat x y z
[]
[{ ... },
 { ... }]
[{ ... }]

This is not an array of objects or one object per line. Rather than cat, we now loop over the filenames and do:

$ jq -c '.[]'
{ ... }
{ ... }
...

reduce-equations is also giving out equations in a rendered form:

f x y == g p
h i == j k l
...

Adjusted our counting logic to reflect this.

All haskell-te tests now pass!

:)

I’ve ripped out a load of TIP stuff from haskell-te, since I think it’s invoking stuff at the wrong level.

What we want:

-- Haskell pkg --> single command -- equations & metrics -->

We can then implement the same API for QuickSpec, and a similar one for IsaCoSy.

Then we can do sampling at the TIP end:

                                   +------------------+
                                   | Experiment:      |
TIP Benchmarks ---> Test Data ---> |   Sampler        | ---> Data
                                   |   ML4HS-runner   |
                                   |   IsaCoSy-runner |
                                   |   etc.           |
                                   +------------------+

We can then graph, etc. afterwards. I think doing this in Nix would be a bad idea. Just provide a command which we can invoke from a nix-shell to ensure the machine isn’t doing other stuff in parallel.

Are there existing ‘experiment runners’ like this? What about that of PyBrain? ‘A Python Experiment Suite’, not too useful; just provides iteration and logging, but requires wrapping everything in Python.

‘Sacred’ seems better, but stores results in MongoDB (?!!).

Maybe just write a script for now…