YAMINS, Dan, Visual Processing in Cortex, MIT, Feb 4 2015
<p class="MsoNormal" >YAMINS, Dan, Visual Processing in Cortex, MIT, Feb 4 2015</p>
<p>
This is at 3:45PM in Jordan Hall Psych Dept at Stanford.
He is a post doc with Jim DiCarlo at MIT.
(This is job talk for a jr. faculty position.)
Human behavior is founded on the ability to <b>identify meaningful entities</b> in complex noisy data streams that constantly bombard the senses. For example, in vision, retinal input is transformed into <b>rich object-based scenes</b>; in audition, sound waves are transformed into words and sentences. In this talk, I will describe my work using computational models to help uncover how sensory cortex accomplishes these enormous computational feats.</p>
<p>
The core observation underlying my work is that <b>optimizing neural networks</b> to solve challenging real-world tasks can yield <b>predictive models of the cortical neurons</b> that support these tasks.
I will first describe how we leveraged recent advances in high-performance computing
to<b> train a neural network that approaches human-level performance on a challenging visual object recognition task.
</b>Critically, even though this network was not explicitly fit to neural data, it is nonetheless<b> predictive of neural response patterns of neurons in multiple areas of the ventral visual pathway, </b>including higher cortical areas that have long resisted modeling attempts.
This model also makes two counterintuitive but testable predictions. One is that<b> inferior temporal (IT) cortex</b>, an area generally thought to specialize in ‘high-level’ categorization of objects, <b>also represents ‘low-level’ visual properties (e.g., position, size, pose, etc.) </b>— and in fact, represents better than low-level visual areas.
The other is that <b>face selectivity</b>, a property of some high-level neurons commonly believed to require extensive experience with faces to achieve<b>, can emerge just from the model architecture </b>itself.</p>
<p>
Intriguingly, some of these same ideas turn out be helpful for studying <b>audition</b>. We have recently found that <b>neural networks optimized for word recognition</b> and speaker identification tasks naturally exhibit <b>high predictivity for fMRI BOLD responses in human auditory cortex</b> to a wide spectrum of natural sound stimuli, and help differentiate poorly understood non-primary auditory cortical regions. I’ll discuss the similarities and differences between these models and those that perform well on visual tasks, assessing the extent to which they provide the beginnings of a general approach to understanding sensory cortex.
He begins: there is a
cascade of processing in vision and audit cortex.</p>
<p>
<b>behavior-driven modeling</b>... his approach.</p>
<p>
bio plausibility is important...</p>
<p>
1) does optimizing for object recognition lead to better models of visual cortex?</p>
<p>
2) what novel insights can be gleaned from such models?</p>
<p>
3) can similar principles work for auditory cortex.</p>
<p>
key collabs with Ha Hong... grad stud. for vision
and Jim DiCarlo.</p>
<p>
how to do vision despite transformations in position, size, pose, illumination
distortion, noise, clutter and occlusion...</p>
<p>
but we are great at doing this.</p>
<p>
Ventral visual stream... as in Holmes and Gross 1984 etc.</p>
<p>
hierarchy of cortical reas.</p>
<p>
pixel to RGC to LGN to V1, v2, v4 , IT.</p>
<p>
<b>goal is a predictive model of ventral stream</b>...</p>
<p>
he wants a predictive model of this...
meaning...</p>
<p>
CNNs = convol neural netowrks
as in Lecun 95 and Fukushima 90..</p>
<p>
multi-layer NN etc.</p>
<p>
filtering, pooling normalization.</p>
<p>
filtering: multiply (as a matrix against certain pattern templates</p>
<p>
threshold and saturate</p>
<p>
clipping values above and below etc</p>
<p>
pooling as in complex cells...</p>
<p>
normalize as in competitive inhibition.</p>
<p>
all operations are local... approx retinotopy.</p>
<p>
so... each layer has image like output (locally)</p>
<p>
CNNs:</p>
<p>
lower areas like RGC LGN and V1 have been reas. captured by single layer models (50% of variance explained)</p>
<p>
can stack these...</p>
<p>
BUT (problem) huge # of params consistent with the Hierachical CNN concept...</p>
<p>
architec params (#layers, #filters, recp field sizes) etc
and filter params...</p>
<p>
obvious strategy is fit neural data to constrain the model class
as in Gallant 2007 and Russ and Movshon in 2009.</p>
<p>
(but... not enough data exists)</p>
<p>
so, much less success in mapping higher areas like V1 and IT (perhaps only 10% of variance)</p>
<p>
our key idea:</p>
<p>
<b>test against neurons</b>...</p>
<p>
our method:</p>
<p>
step 1 optimize the performance of the task and
step 2: optimize for match to neural data (neural predictivity)</p>
<p>
my hypothesis is that performance (1) implies neural predictivity (step 2)</p>
<p>
<b>macaque V4 and IT</b></p>
<p>
we recorded 300 sites...
5760 images
64 objects
8 categories...
animals, boats, cars, chairs, faces, fruits, planes and tables...</p>
<p>
we did low variation (in the images) , med var, and high variation...</p>
<p>
we recorded from 300 neurons...</p>
<p>
<b>images sorted first by category then by variation level</b></p>
<p>
at high variation... V4 drops dramatically.</p>
<p>
but IT well tracks V4</p>
<p>
ie <b>at high variation levels, IT much better tracks human behavior </b>than does V4.</p>
<p>
each layer is a "UNIT" in his terminology (want to map these to neural units)</p>
<p>
accuracy = goodness of fit on held-out testing images.</p>
<p>
he shows scattergram of</p>
<p>
performance versus IT predictivity...
r = 0.55 on n= 2000.</p>
<p>
next,, we try to optimize the performance...
and we got (the blue dots)
with r = 0.79.... much better.</p>
<p>
now red dots r = 0.8
(but this had over-fitring problems)
(this is a model in which we optimized neural predictivity rather than behavioral performance)</p>
<p>
our conclusion... best to optimize behav performance...</p>
<p>
so, HMAX was great, PLOS09 great....</p>
<p>
But (he jokes) we really want to be way way better... shows a dot off the graph to the upper R.</p>
<p>
so, we really wanted better modelsl...</p>
<p>
we looked at ImageNet 2012... thousands of images and thousands of categories.</p>
<p>
we trained on real photos from ImageNet and tested on our same 8 category test set (with uncorrel bkgrounds)</p>
<p>
(low variation is a cheat; where this really gets tested is with high variation)</p>
<p>
But.. does it predict neurons better?</p>
<p>
we were able to go off to the right with HMO...
with IT variation explained... got much better... toward 50% var explained.</p>
<p>
but what abt intermediate layers?</p>
<p>
now compares layer 1 to 2 to 3 with eg the faces</p>
<p>
predicticity of neural response improves as you get toward the top</p>
<p>
ie HMO top explained 50% of IT variance
and HMO L3 was at about 38%
and on down for lower layers.</p>
<p>
at top layer we have 1250 units...</p>
<p>
600 faces are in the face category and
this is plotting performance on indiv faces on some population of IT neurons</p>
<p>
but we also want to predict intermed layers as well..</p>
<p>
now how abt V4 neural repsonses...
what's going on?</p>
<p>
he shows L3 of V4 tracks v well...
why? (he shows that HMO L3 is best (and even better than top alyer in V4)</p>
<p>
Layer 1 best explained Gabor wavelets...</p>
<p>
so, issue 1: does optim for object recog lead to better models of vis cortex... YES.</p>
<p>
2) what novel predictions...</p>
<p>
beyond categorization... so, you know plane is an F16 and where it is</p>
<p>
but we can also talk about its aspect ratio...the wings etc.</p>
<p>
*<b>Auditory </b>Cortex</p>
<p>
"please enter your 6 digit number after the tone" (an automatic voice attendant)</p>
<p>
well known tonotopy in audit cortex...</p>
<p>
then around A1 is a belt (he shows monkey cortex)</p>
<p>
600 way word recog task</p>
<p>
words spoken 20 times...</p>
<p>
which word was in the sentence...</p>
<p>
earlier this week defense Secr Weinberger told reproters at a... etc.</p>
<p>
he shows fMRI...</p>
<p>
stim set 165 natural sounds...
music , car horns, crying baby, typing etc.</p>
<p>
how well did model optimized for perform.. predict the voxel responses on fMRI.</p>
<p>
r = 0.94
word recog performance and audit predictivity...</p>
<p>
we took the best model on word recog perform...</p>
<p>
and we showed signif gain in neural predictivity...</p>
<p>
A1 = core/ belt/ parabelt</p>
<p>
higher layers correl best with outer annulus. (going from center to periphery)</p>
<p>
AP axis of auditory cortex A1... anterior are lower levels..
and posterior is higher levels (of hierarchy)</p>
<p>
key results</p>
<p>
behavior driven modeling can make greatly improved quantitative models of high-level cortex.
etc.</p>
<p>
Future work</p>
<p>
ventral visual cortex...want to characterize intermed levels like V2...</p>
<p>
and richer visual task eg explciit 3d geometry...</p>
<p>
auditory cortex...
word recog task versus speaker ID task. (they are not correlated)
how do we carve up that task in brain real estate.</p>
<p>
Learning and Development...
how does experience change representation...</p>
<p>
more bio realistic... less heavily supervised</p>
<p>
and cortical heterogeneity..</p>
<p>
faces, bodies, palces, color eg what is their origin
what is their func significance?</p>
<p>
eg are face-selective units "special"...
are they really sepearate>
and two explanatory hypotheses...
eg nature vs nurtue</p>
<p>
we ran an "in-silica face deprivation expt with NO faces and no animate objects...
but nonetheless as signif fraction of face-selective units emerged.</p>
<p>
(ie maybe its neither nature vs nurute)</p>
<p>
virtuous cycle of AI and brain understanding...</p>
<p>
Jay Q: face selecitv neurons..
A he shows IT site 150 liked chairs...</p>
<p>
is there cross validation?</p>
<p>
what he shows... v strong generalization...
half of faces for training and then test on other half of faces etc (for each of 8 categories)</p>
<p>
Surya Q: recent results...models are v sensitive to examples that screw up categories...</p>
<p>
A categoriz is too brittle a task (ie too easy to break)
perhaps if we capture robustly 3d geometr perhaps that brittleness will go away.</p>
<p>
Q are there limits to the notion that brain works in an optimal way
(eg to do categorization) Mike's Q.
A perhaps categoriztion only brings us so far... but maybe doing 3d geometry may take us even further....</p>
<p>
Brian W Q: how can position info be preserved in IT...
(despite enlargement of recep. field at higher levels... and loss of locality)</p>
<p>
in submission: model lowest layer is best explantion of imaging data in V1.
</p>