~/projects/model-comparison-cannon $ cat README.md

Does model comparison find the real cause?

Training networks with a known flaw on a human learning task, to test whether standard modelling can tell what went wrong.

context
Personal research project, next to my MSc
when
2026
status
Ongoing
  • Python
  • PyTorch
  • GRU networks
  • Bayesian ideal observer
  • Model comparison (BIC)
  • Open human data

// the problem

In cognitive science, a common way to explain why people learn worse than they could is to write down a few candidate models, fit them to the data and pick the one that fits best. But with people you never know the true cause, so you can't check whether the method picked the right one.

// what I built

I use a well-studied task where people catch cannonballs with a shield while the cannon jumps or drifts (Nassar, Bruckner & Frank, 2019), with their open data from 39 people. An exact Bayesian learner is the yardstick. Then I train small recurrent networks on the same task and build a flaw into them that I know: a wrong belief about the world (pretraining in a drifting or noisier world) or faulty computation (noise inside the network, weight decay). Finally I run the standard model-comparison pipeline on the networks and check whether it names the flaw I put in.

// result so far

On data simulated from the candidate models themselves, the pipeline works: it picks the right family of explanations 84% of the time, above the 70% bar I set beforehand. On the networks it doesn't: the right family wins only 43–50% of the time, and one explanation, noisy detection of jumps, wins by default. Even a large margin in the fit doesn't mean the answer is right. So reliability measured on simulated data overstates how well the method works on learners that aren't formula models. Each condition rests on only two networks so far, so this is a first result, not a final one.

// what I took away

Setting the pass marks before running anything made the negative result easy to trust. And it is the same question as in my bachelor's thesis, from the other side: when a network and a person get the same score, what is going on inside?