---
title: "Neural mechanisms of learning and decision-making"
author: "Shany Grossman and Ondrej Zika"
institute: "Max Planck Research Group NeuroCode"
date: "Last update: `r format(Sys.time(), '%d %B, %Y')`"
output: 
  xaringan::moon_reader:
      self_contained: true
      yolo: false
      lib_dir: libs
      nature:
        highlightStyle: solarized-dark
        highlightLines: true
        countIncrementalSlides: false
        #css: xaringan-themer.css
        #css: [default, metropolis, metropolis-fonts]
---
## Overview
### Part 1: Basic RL and dopamine (Ondrej)
- one very good point

### Part 2: Avanced RL (Shany)
- an excellent point

- yalla! 

---


## Part 1: Basic RL and dopamine
### Introduction 
- Learning by reinforcement - Pavlovian/Operant conditioning 
- Hebbian learning as biological counterpart - mention single cell recordings in Amygdala? acquired predicatbility 
- What is RL and where it comes from? 

---

### Associative (Pavlovian) learning 
- forming stimulus-outcome associations

---

### Rescorla-Wagner model 
$V_t$ ... expected reward for the current trial  
$V_{t+1}$ ... expected reward for next trial  
$r$ ... reward received on the current trial  
$\alpha$ ... learning rate (i.e. "how much do you learn")

--

$V_{t+1} = V_t + \alpha(r - V_t)$  

--

$\alpha=0.3$ $V_0=5$
![](../../simulation/img/rw_1/1_11_2.png)


---

### Rescorla-Wagner model 
$V_t$ ... expected reward for the current trial  
$V_{t+1}$ ... expected reward for next trial  
$r$ ... reward received on the current trial  
$\alpha$ ... learning rate (i.e. "how much do you learn")

$V_{t+1} = V_t + \alpha(r - V_t)$  

$\alpha=0.3$ $V_0=5$
![](../../simulation/img/rw_1/1_21_2.png)


---

### Rescorla-Wagner model 
$V_t$ ... expected reward for the current trial  
$V_{t+1}$ ... expected reward for next trial  
$r$ ... reward received on the current trial  
$\alpha$ ... learning rate (i.e. "how much do you learn")

$V_{t+1} = V_t + \alpha(r - V_t)$  

$\alpha=0.3$ $V_0=5$
![](../../simulation/img/rw_1/1_31_2.png)


---

### Rescorla-Wagner model 
$V_t$ ... expected reward for the current trial  
$V_{t+1}$ ... expected reward for next trial  
$r$ ... reward received on the current trial  
$\alpha$ ... learning rate (i.e. "how much do you learn")


$V_{t+1} = V_t + \alpha(r - V_t)$  

$\alpha=0.3$ $V_0=5$
![](../../simulation/img/rw_1/2_12_0.png)


---

### Rescorla-Wagner model 
$V_t$ ... expected reward for the current trial  
$V_{t+1}$ ... expected reward for next trial  
$r$ ... reward received on the current trial  
$\alpha$ ... learning rate (i.e. "how much do you learn")


$V_{t+1} = V_t + \alpha(r - V_t)$  

$\alpha=0.3$ $V_0=5$
![](../../simulation/img/rw_1.gif)


---

### Rescorla-Wagner model 
$V_t$ ... expected reward for the current trial  
$V_{t+1}$ ... expected reward for next trial  
$r$ ... reward received on the current trial  
$\alpha$ ... learning rate (i.e. "how much do you learn")


$V_{t+1} = V_t + \alpha(r - V_t)$  

$\alpha=0.3$ $V_0=5$
![](../../simulation/img/rw_1/29_329_2.png)


---


- description and example of different alpha simulations
- pitfalls (reinstatement, second order conditioning) 
 
---

### Dopamine
- anatomy of dopaminergic afferents and BG
- large variabilty in terms of computation
- involved in movement control (Prkinson's) and vigor, go/nogo pathways 
- involvement in reward (believed to signal obtained reward) 
- timing (Walton), state signalling (Starkweather),  distributed processing RPE (Daw recent) 

---

### DA and RPE
- activity associated with obtaining food -> reward? 
- maybe not reward, but the difference between expectation and obtained (Schultz) 
- asymmetry in firing rate potential (relevant later for aversive learning) 
- replicated in many single-cell recording, fMRI and optogenetic studies 
- make distinction of value and reward 

---

### how rare this is 
- a link between a normative strategy and a feature of the nervous system
- yes but also if you think about it it's not a surprise since evolution is based on trial-and-error learning


---

### TD-learning
- taking temporal information into account, predicting aggregate reward over actions
- stimuli predicting highest average reward elicit highest DA responses


---

### Value 
- predicting future rewards
- keeping track of *relative* values of different stimuli/features 
- useful framework to study decision making - softmax 
- vmPFC/OFC role in representing value as a common currency across categories 
- maybe mention metaRL

---

### Appetitive versus aversive learning
- issue with dopamine floor 
- absolute prediction error, end/avoidance of aversive stimulus is rewarding (Moutoussis 2008)  
- major differences between appetitive and aversive learning (discuss fight/flight pathway PAG etc) 

---

### Learning rates and uncertianty

- how to decide how much to learn? 
- noise in observations
- changeability of the environment 
- learning the volatility 

---






---