DiCarlo, Jim, Reverse Engineering Brain, MIT, Oct 11, 2018

<p class="MsoNormal" >DiCarlo, Jim,&nbsp; Reverse Engineering Brain,&nbsp; MIT,&nbsp; Oct 11, 2018</p>

<p>

cites Dan Yamins&nbsp; (now here as asst prof)

Here, Jim cites how to relate brain to mind and AI ... first...</p>

<p>

specification 1:&nbsp; specify&nbsp; domain of interest...</p>

<p>

is it performance,&nbsp; or power consumption , or size...</p>

<p>

then spec 2:&nbsp; must measure the internal system components...</p>

<p>

do you want spiking activity,&nbsp; blood flow, etc.</p>

<p>

next&nbsp; make a model...</p>

<p>

model must capture all those measurements and predict the held out measurements.</p>

<p>

this usu means&nbsp; ANNs =&nbsp; artif neural net models...</p>

<p>

we started around 2002... reverse engr&nbsp; visual scene understanding...</p>

<p>

eg label cars, bldgs,&nbsp; pedestrians...</p>

<p>

but&nbsp; when we look at an image we really mainly see the <b>10 degree center of gaze</b>...</p>

<p>

and we get 200 msec snapshots&nbsp; (and&nbsp; don't see in between)

&nbsp;and dwell for just a few hundred msec.</p>

<p>

so,&nbsp; <b>200 msec snapshots&nbsp; of the central ten degrees</b>.</p>

<p>

we call that <b>core object recognition</b>...</p>

<p>

eg&nbsp; look at an image of an artificially big face in the woods;&nbsp; or&nbsp; car&nbsp; above a field...</p>

<p>

humans&nbsp; around 2009&nbsp; were far better than machines on core recognition...</p>

<p>

from Pinto, Cox,&nbsp; DiCarlo&nbsp; in PLOS&nbsp; Comp Bio in 2008...</p>

<p>

different pose of objects...</p>

<p>

shows rhesus monkey&nbsp; in&nbsp; cage with plexiglass floor reaching thru

and touching images of 30 different objects....</p>

<p>

among&nbsp; say&nbsp;&nbsp; guitar,&nbsp; dog, rhino,&nbsp; elephant,&nbsp; wrench,&nbsp; knife,&nbsp; hanger,&nbsp; fork, etc.</p>

<p>

monkey accuracy chart looks quite like that in humans

with 276&nbsp; pairwise&nbsp; comparisons of objects.</p>

<p>

lesions in&nbsp; IT&nbsp; inferior temporal cortex... those lesions create a visual recognition deficit.</p>

<p>

<b>pixel =>&nbsp; RGC,&nbsp; LGN, v1, v2, v4,&nbsp; IT...</b></p>

<p>

<b>ventral visual stream... </b>from&nbsp; v1 to IT and back...</p>

<p>

at each station&nbsp; there is a&nbsp; nonlinear transform....</p>

<p>

he shows that the neural response in IT is reproducible (with

repetitions of the same image)</p>

<p>

shows action potentials&nbsp; (spikes)... similar from trial to trial for a given object.</p>

<p>

spikes per second....</p>

<p>

we&nbsp; use three Utah arrays...&nbsp; three&nbsp; 96 trode arrays.</p>

<p>

put in three at once in the IT.</p>

<p>

show animal thousands of images... with many repetitions...</p>

<p>

get a vector of neural responses...</p>

<p>

eg&nbsp; neuron 1 versus neuron 2 versus neuron 3&nbsp; and.... neuron N</p>

<p>

this creates a high D&nbsp; point for each&nbsp; object...</p>

<p>

then&nbsp; separate these points... eg to discriminate&nbsp; face versus not a face.</p>

<p>

then&nbsp; map these to an upstream neuron that decodes the pattern&nbsp; (the feature vector pattern)</p>

<p>

why are humans so good at this... we have a rich feature set...</p>

<p>

breakthru in 2013...</p>

<p>

so,&nbsp;&nbsp; before 2013&nbsp; it was like egyptian hieroglyphics... (the pattern of IT neurons)</p>

<p>

what are the intermediate&nbsp; features...</p>

<p>

Charlie Gross... early pioneer; and later Doris Tsao... etc.</p>

<p>

(Tanya... mentions&nbsp; Wu Tsai... here and in Hong Kong... will rename&nbsp; SNI)</p>

<p>

1980&nbsp; neocognitron...work by Fukushima</p>

<p>

then&nbsp; Tommy Poggio...at MIT;</p>

<p>

HMO&nbsp; by Dan Yamins&nbsp; and Ha Hong... 2012.</p>

<p>

we&nbsp; tested this again on the&nbsp; core object recog&nbsp; task.</p>

<p>

IT is a softmax layer...</p>

<p>

we hope that our models&nbsp;&nbsp; fit the spiking&nbsp; in the brain&nbsp; (but that is what it's optimizee on)</p>

<p>

the match is quite good...

result is that our evolved model's hidden neurons&nbsp; match the brain's hidden neurons.</p>

<p>

we explain about&nbsp; <b>50% of variance&nbsp; with our&nbsp; HMO ANN.</b></p>

<p>

even the intermediate layers&nbsp; match.</p>

<p>

related work by&nbsp; Kriegeskorte&nbsp; with fMRI.</p>

<p>

get 80% match with object recog at the top level&nbsp; (IT)</p>

<p>

Y axis is&nbsp; neurosci goal&nbsp; (matching spiking)

and X axis is the&nbsp; AI / behavior goal...&nbsp;&nbsp; correct recog..</p>

<p>

yes correlated&nbsp; with HMO in 2012&nbsp; at&nbsp; 50% of variance.</p>

<p>

so,&nbsp; can brain sci&nbsp; just sit back and wait for AI/ comp sci&nbsp; to find even better models...</p>

<p>

AlexNet in 2012&nbsp; was even better.</p>

<p>

BUT,,, it leveled off after&nbsp; 2012&nbsp;&nbsp;&nbsp; ie&nbsp; image recog&nbsp; performance improved but

NOT the match to&nbsp; neural spiking...</p>

<p>

but&nbsp; still problems...</p>

<p>

he shows a glass half full...</p>

<p>

only ability we've modeling is core object recognition.</p>

<p>

(not spatial layout and not learning, plasticity)</p>

<p>

try to drive a particular neural site even higher...&nbsp; yes can do that with synthetic images...</p>

<p>

next... try to perturb system...&nbsp; eg with BMI or etc.&nbsp; and predict degradation etc.</p>

<p>

put in wiring cost,&nbsp; size, and power... try to simplify it.</p>

<p>

which recurrence models&nbsp; really capture&nbsp; the brain score/&nbsp; neural match...</p>

<p>

want to automate this process.

so, what do we do next?</p>

<p>

recurrence;&nbsp; cell diversity;&nbsp; realistic&nbsp; biology&nbsp; (other than backprop), etc.

(asked by a questioner in the audience)...</p>

<p>

Jay...&nbsp; Q... JIm answer:&nbsp; go image by image...

which images do they NOT get right....&nbsp; probably the brain uses recurrence...</p>

<p>

Jay...&nbsp; is this work a contrib to our&nbsp; real understanding of how brain works...</p>

<p>

Jim... if want to do human health&nbsp; must understand&nbsp; feathers of the bird

&nbsp; (in addition to aerodynamics required for airplanes)</p>

<p>

&nbsp;

&nbsp;</p>