BOTVINICK, Matthew, Hierarchical Reinforcement Learning, Princeton, Dec 4, 2013
<p class="MsoNormal" >BOTVINICK, Matthew, Hierarchical Reinforcement Learning, Princeton, Dec 4, 2013</p>
<p>
he opens with dopamine = DA response to a reward and then
condition stimulus causes DA release if reward, then
dip in DA if no reward.</p>
<p>
(from 1997);</p>
<p>
Matumoto in Nature 2007;</p>
<p>
RL = reinforcement learning...</p>
<p>
prospection...
I've learned that this lever leads to cheese, but
now I'm really sated, so why bother (says the mouse);</p>
<p>
but these mechs do not scale well to more and more complex situations...</p>
<p>
stick with a toy domain...</p>
<p>
branching maze with a pile of money at only one leaf...
"the curse of dimensionality</p>
<p>
how are the comp sci guys dealing with this complexity of tasks?</p>
<p>
think about “subroutines”</p>
<p>
suppose instead of primitive actions, it has high level task (that are composed of many subtasks)</p>
<p>
thinking abt planning a trip, so you think...
yes, I will use CheapTickets.com for step 1...
this is reducing the size of the decision tree via abstraction
(ie abstracting and reducing the size of the tree by using hierarchies)...</p>
<p>
pi is policy that the agent considers.</p>
<p>
temporal difference learning...</p>
<p>
the reward can percolate backward thru all the steps.</p>
<p>
HRL = hierarchical reinforecement learning...</p>
<p>
this is from Botvininck, Niv and Barto in Cognition in 2009;</p>
<p>
tasks are broken down into subtasks....</p>
<p>
flat RL (no subtasks...
HRL has subtasks...</p>
<p>
doorways are good subgoals
(he shows a complex maze divided into roughly 4 square areas each with doorways
that you must go thru (that simplify the task)</p>
<p>
reward prediciton error signals...</p>
<p>
(his example is expectation of reward from going to coffee shop)</p>
<p>
agent must be able to distinguish high level goal (getting somewhere) versus
opening the door to get there (a pseudo-reward for a subgoal)</p>
<p>
how to do credit assignment...</p>
<p>
his last name is Botvinick (only one n)</p>
<p>
task goal prepare a snack
immediate goal take a cookie and then pour the milk</p>
<p>
action: reach to location and, grasp item
movements extend the arm, preshape the hand, close fingers, maintain fingertip grip...</p>
<p>
Actor critic architecture.... mappings proposed to neuroanatomy (eg to ventral striatum)</p>
<p>
what correlates when we go from flat RL to HRL?</p>
<p>
(do not need to add much)</p>
<p>
orbitofronto PFC and lateral PFC need to be added;</p>
<p>
Carlos Diuk post doc did this work in his lab...</p>
<p>
slot machines (a bandit task) decide which slot machine is better...</p>
<p>
this is Diuk in J Neurosci in 2013.</p>
<p>
learning abt the casino and abt the indiv slot machines...</p>
<p>
ventral striatum tracked the machines and the casino rewards...</p>
<p>
RPE = reward prediction errors...
as opposed to PPEs pseudo prediction errors (for subtasks)</p>
<p>
task is the courier task; pick up a pkg and deliver it to a home and get a reward</p>
<p>
Ribase-Fernades inNeuron in 2011...</p>
<p>
ellipse is the location of all points that are the same total distance to
the pkg and then to the house...</p>
<p>
now suppose the agent himself has to figure out where the subgoals are (the doorways to other
tasks)</p>
<p>
yes, abstractions help... Buuuuut... the wrong abstractions can hurt you
(he calls that the "burden of abstraction")</p>
<p>
What should be learned? (is there some normative target of learning?
what they should be learning)</p>
<p>
bottlenecks....
similar to groups or clusters that are highly internally connected
by being sparsely connected to other clusters....</p>
<p>
Next... do people actually learn these decompositions?
(carving the overall task at its joints)</p>
<p>
a SimCity type task with locations (hospital, computer store, , drugstore, etc)
with adjacency graph with a bottleneck between 2 sections of town.</p>
<p>
leaping to the bottleneck is v important ...
even better than just HRL...
he calls this hierarchical/ saltatory (can JUMP right to the doorways (the bottlenecks)</p>
<p>
How do they learn?</p>
<p>
work of Anna Schapiro (started at Stfd)</p>
<p>
Schapiro in Nature Neurosci in 2013;</p>
<p>
L infer frontal cortex and anterior temp cortex...</p>
<p>
she did find similarity structure in connected regions of the graph ;</p>
<p>
similar to Rogers and McClelland in 2003</p>
<p>
similar to clusters of objects in PDP (eg grouping flowers together;
trees together; (ie objects related to one another)</p>
<p>
see Dayan in 1993...</p>
<p>
Tower of Hanoi...</p>
<p>
collab Yael Niv at Princeton
Andy Barto in U Mass and
Tim Rogers at U Wisc.</p>
<p>
comment: clusters form basis for abstraction</p>
<p>
looking at a video of somebody cleaning a bedroom;
first they are making a bed (and its easy to prdict the steps) then they finish
and then don't know what they're going to do... but then they start
vacuuming the rug and you know what the steps are.</p>
<p>
Jay: are you just parsing based on the repr. similarities?</p>
<p>
( eg saying eagle, sparrow, robin... and then saying dog, fox, hyena
and you would know (and press a button to signal that a transition had occurred.)</p>
<p>
ie predictions are stronger within clusters than between clusters.</p>
<p>
</p>
<p>
</p>