Computational models for imaging analyses Zurich SPM Course February 14, 2014 Christoph Mathys What the brain is about What do our imaging methods measure? Brain activity. But when does the brain become active? When predictions have to be adjusted. So where do the brains predictions come from?

From a model. Feb 14, 2014 Model-based imaging, Zurich SPM Course, Christoph Mathys Page 2 What does this mean for neuroimaging? If brain activity reflects model updating, we need to understand what model is updated in what way to make sense of brain activity. Feb 14, 2014 Model-based imaging, Zurich SPM Course, Christoph Mathys

Page 3 The Bayesian brain and predictive coding Model-based prediction updating is described by Bayes theorem. the Bayesian brain Hermann von Helmholtz This is implemented by predictive coding. Feb 14, 2014 Model-based imaging, Zurich SPM Course, Christoph Mathys Page 4

Advantages of model-based imaging Model-based imaging permits us to infer the computational (predictive) mechanisms underlying neuronal activity. to localize such mechanisms. to compare different models. Feb 14, 2014 Model-based imaging, Zurich SPM Course, Christoph Mathys Page 5 How to build a model Fundamental ingredients: Prediction

Sensory input Hidden states Inference based on prediction errors Feb 14, 2014 Model-based imaging, Zurich SPM Course, Christoph Mathys Page 6

Example of a simple learning model Rescorla-Wagner learning: Learning rate Previous value (prediction) () =( 1) + ( ( ) ( 1) ) Prediction error () Inferred value of New input ( 1) Feb 14, 2014

() Model-based imaging, Zurich SPM Course, Christoph Mathys ( ) Page 7 From perception to action Agent Sensory input Inversion of perceptual generative model Inferred hidden states World

Generative process True hidden states Decision model Feb 14, 2014 Action

Model-based imaging, Zurich SPM Course, Christoph Mathys Page 8 From perception to action Agent Sensory input Inferred hidden states World

True hidden states Action In behavioral tasks, we observe actions (). How do we use them to infer beliefs ()? We introduce a decision model. Feb 14, 2014

Model-based imaging, Zurich SPM Course, Christoph Mathys Page 9 Example of a simple decision model Say 3 options A, B, and C have values , , and . Then we can translate these values into action probabilities via a softmax function: The parameter determines the sensitivity to value differences

=0.1 Feb 14, 2014 Model-based imaging, Zurich SPM Course, Christoph Mathys =0.6 Page 10 All the necessary ingredients Perceptual model (updates based on prediction errors) Value function (inferred state -> action value) Decision model (value -> action probability)

Feb 14, 2014 Model-based imaging, Zurich SPM Course, Christoph Mathys Page 11 Reinforcement learning example (ODoherty et al., 2003) ODoherty et al. (2003), Glscher et al. (2010) Feb 14, 2014 Model-based imaging, Zurich SPM Course, Christoph Mathys Page 12 Reinforcement learning example

Significant effects of prediction error with fixed learning rate ODoherty et al. (2003) Feb 14, 2014 Model-based imaging, Zurich SPM Course, Christoph Mathys Page 13 Bayesian models for the Bayesian brain Agent World Sensory input Inferred hidden states

True hidden states Action Includes uncertainty about hidden states.

I.e., beliefs have precisions. But how can we make them computationally tractable? Feb 14, 2014 Model-based imaging, Zurich SPM Course, Christoph Mathys Page 14 The hierarchical Gaussian filter (HGF): a computationally tractable model for individual learning under uncertainty State of the world

Model p(x3(k)) ~ N(x3(k-1),) ( 1) 3 ( ) 3 , Gaussian random walk

with constant step size Log-volatility x3 of tendency p(x3(k)) x3(k-1) p(x2(k)) ~ N(x2(k-1), exp(xx3+)) ( 1) 2

( 1) 1 ( ) 2 ( ) 1 Tendency x2 towards category 1 Stimulus

category x1 (0 or 1) Gaussian random walk with step size exp(xx3+) p(x2(k)) x2(k-1) Sigmoid transformation of x2 p(x1=1) = s(x2) p(x1=0) = 1-s(x2) 1 p(x1=1)

x2 0 Feb 14, 2014 Model-based imaging, Zurich SPM Course, Christoph Mathys Page 15 HGF: variational inversion and update equations Inversion proceeds by introducing a mean field approximation and fitting quadratic approximations to the resulting variational energies (Mathys et al., 2011). This leads to simple one-step update equations for the sufficient statistics (mean and precision) of the approximate Gaussian posteriors of the states . The updates of the means have the same structure as value updates in Rescorla-Wagner learning:

Prediction error ^ 1 1 Precisions determine learning rate Furthermore, the updates are precision-weighted prediction errors. Feb 14, 2014 Model-based imaging, Zurich SPM Course, Christoph Mathys Page 16 HGF: adaptive learning rate

Simulation: Feb 14, 2014 0.5, 2.2, 1.4 Model-based imaging, Zurich SPM Course, Christoph Mathys Page 17 Individual model-based regressors Uncertainty-weighted prediction error Feb 14, 2014 Model-based imaging, Zurich SPM Course, Christoph Mathys Page 18

Example: Iglesias et al. (2013) Feb 14, 2014 Model-based imaging, Zurich SPM Course, Christoph Mathys Page 19 Example: Iglesias et al. (2013) Model comparison: Feb 14, 2014 Model-based imaging, Zurich SPM Course, Christoph Mathys Page 20

Example: Iglesias et al. (2013) Feb 14, 2014 Model-based imaging, Zurich SPM Course, Christoph Mathys Page 21 Example: Iglesias et al. (2013) Feb 14, 2014 Model-based imaging, Zurich SPM Course, Christoph Mathys Page 22 Example: Iglesias et al. (2013)

Feb 14, 2014 Model-based imaging, Zurich SPM Course, Christoph Mathys Page 23 How to estimate and compare models: the HGF Toolbox Available at http://www.tranlsationalneuromodeling.org/tapas Start with README and tutorial there Modular, extensible Matlab-based Feb 14, 2014

Model-based imaging, Zurich SPM Course, Christoph Mathys Page 24 How its done in SPM Feb 14, 2014 Model-based imaging, Zurich SPM Course, Christoph Mathys Page 25 How its done in SPM Feb 14, 2014 Model-based imaging, Zurich SPM Course, Christoph Mathys

Page 26 How its done in SPM Feb 14, 2014 Model-based imaging, Zurich SPM Course, Christoph Mathys Page 27 How its done in SPM Feb 14, 2014 Model-based imaging, Zurich SPM Course, Christoph Mathys Page 30

How its done in SPM Feb 14, 2014 Model-based imaging, Zurich SPM Course, Christoph Mathys Page 31 Take home The brain is an organ whose job is prediction. To make its predictions, it needs a model. Model-based imaging infers the model at work in the brain. It enables inference on mechanisms, localization of mechanisms, and model comparison. Agent Sensory input Inferred

hidden states World True hidden states Action Feb 14, 2014 Model-based imaging, Zurich SPM Course, Christoph Mathys

Page 32 Thank you Feb 14, 2014 Model-based imaging, Zurich SPM Course, Christoph Mathys Page 33