Dicarlo, Zoccolan, Rust, Visual Object Recognition, Neuron, 2012

Nov 26, 2013 08:25 PM

open access;  from Jim Dicarlo  (MIT  ,  McGovern Instit) and from Penn. (Rust)

core object recog subserved by largely reflexive, feedfwd compuation =>

powerful neuronal repr in IT cortex  (inferior temporal).

Abstract:

Mounting evidence suggests that ‘core object recognition,’ the ability to rapidly recognize objects despite

substantial appearance variation, is solved in the brain via a cascade of reflexive, largely feedforward

computations that culminate in a powerful neuronal representation in the inferior temporal cortex. However,

the algorithm that produces this solution remains poorly understood. Here we review evidence ranging

from individual neurons and neuronal populations to behavior and computational models. We propose

that understanding this algorithm will require using neuronal and psychophysical data to sift through many

computational models, each based on building blocks of small, canonical subnetworks with a common

functional goal....

first page = page 415;

see review by Logothetis and Sheinberg 1996...

half of the nonhuman primate neocortex is devoted to visual processing  (Felleman and Van Essen 91)

speaks to the comp. complexity of object recog...

we do recog within a frac of a second:  effortless:

coffee cup, printed words, people.

finding food, social interax, tool selection, reading...

half of the nonhuman primate neocortex is devoted to vision  (Felleman Van Essen 91);

discov of these algos  may extend to disco of meaning in highD  artif sense data.

What does it mean?  could be similar to a Turing test for vision (1950).

agreed upon sets of images and tasks.

We now have hi-performing artif vision systems  eg Pinto 08 and Serre 07.

visual phenomena at one level are explained by mechs to a level below (eg spiking population in IT)

page 416  What is Object REcog and Why is it Challenging?

want to explain phenomena at one level of abstraction eg successful performance in

recog benchmark tests... based one mechs one level below eg by neural spikes...

which themselves must be explained by connectivity and ion flow etc.

We look at three levels:

Core Object Recog;    the IT population repr and  IT single unit responses;

1)  CORE OBJECT RECOG:

vision:  besides recog also includes object tracking, segmentation, obstacle avoidance,

object grasping etc.;

import of dorsal stream...  guides eyes and allocates  spatial attention toward objects;

and to shape hand to grasp objects.

Here we define object recog as ability to assign labels to partic objects:

either precise labels  (identificaiton)  or coarse labels  (categories).

over change in position, size, pose, and bkgrnd clutter/ context.

behavior times for monkeys:  abt  250 ms and for humans abt 350 ms.

can present images at rates less than 100 ms per image.

central visual image can support recog in < 200 ms.

ERPs for object categorization in < 150 ms.

Primates explore their visual world with REMs with short fixations of

200 to 500 ms and determine central field objects (< 10 deg) rapidly.

page 417:  CRUX  computat. problem:  CORE RECOG requires Invariance

imagine a type of world where objects all present the same way (eg barcodes)

determine ID just from the retina.

but  in the real world  each object appears almost unique:

map all those images to just CAR  despite location on retina, scale, lighting conditions,

pose, some nonrigid distortion, illumination and clutter variability.

deformable objects:  faces and bodies.

and despite intraclass variation:  cars, faces, dogs.

ID them as an equiv class and not confuse with other classes.

Invariance of core object recog is a good cetnral problem:

it engages the CRUX computat difficulty.

page 417 col 2:

graphical intuition:

the ventral visual path gradualy UNTANGLES info abt object identity.

consider response vector in N-space (where N = # of neurons)

identity-preserving image transformations.

manifolds of difft objects are tangled together like paper crumpled into a ball.

a decision boundary that separates manifolds.

(Bob note:  but shouldn't subsets of the image corresponds to subsets of the manifolds???

And do they or not?  eg  is the TIRE of a CAR a subset of the CAR maniforld or not?

or is it entirely separate?)

a higher order neuron that "looks down" on the population and comutes a weighted sum of each neuron's responses followed by a threshold.

This is akin to studying aerodynamics, rather than feathers to understand flight.

Untangling reprs is the key intermediate step... it is what must be explained

at the singel neuron level.

p418  lesions in anterior IT produce selective deficits in ability to distinguish complex objects.

see figure p 419:  # of neurons in each area;

see Doris Tsao:  3 to 6 or more face regions in IT. suggests that

behavioral gols may be better organizing principles than retinotopy at IT level.

Ascending feedfwd is received in layer 4 and ascending fdfwd originates in upper layers (eg 2/3);

descneding feedback originates lower layers and is received in the upper and lower layers of

the "lower" cortical areas..

p 420:  mean  visual Evoked REsponse in each successive layer is lagged by 10 ms.

Just 100 ms after first photons, a first wave of image-slective neuronal activity is present thruout IT.

Ventral Stream Cortical Code:

so,  the neural repr in IT is the spatiotemp pattern of spikes produced by the set of pyramidal

neurons projecting out of IT.

there is some rate variability due to Poisson-like stochastic spike generation

with an underlying rate determined by each particualr image.

But one can infer the identity of the object (despite the variability)...

Relevant underlying decoding window is about 50 msec...

v reliable object info is found in IT in the first 50 ms AFTER a 100 to 150 ms image onset.

peak spike rates are higher in tat window than in later windows.

p 420  simple weighted sums of IT spike counts over short time intervals (50 ms) lead to

high rates of cross-validated performance for randonly slected pops of only a few 100 neurons.

IT is much better at object ID and categorization than earlier ventral stages.

p 421  we work under the null hypothesis that core object recog is

well descsrbied by a laregely feedfwd cascade of nonlinear filtering operations

and is expressed as a population rate code at abt 50 ms time scle.

(erformacne of this model exceeds artifical vision systems... see Pnto or Serre.

andd appears suffic to expalin human object recog performance.  Majaj 2012.;

p 422: A Comtep View of IT single neurons

how do single IT neurons do it?...  unknown  (but some progress)

activated by moder complex combos of features...  can

keep object preference over sm to moder changes in position, size, pose, illum and clutter.

They have diverse selectivity... not narrowly tune  (Desimone quote 1984).

typical IT neuron responds to many difft images and objects.

IT pop. is diverse in shape selectivyt and tolerance.

IT neurons with the highest shape selectivity are the least tolerant to changes in postion etc....

a finding  NOT consistent with grandmother cells... but arises natl from

feedfwd models.

this argues for DSITRIBUTED REPR of visual object in IT...  ie POPULATION CODING...

ie single IT neurons do NOT act sparsely as invariant detectors of spec objects but rather

as elements of a population that as a whole support object recog.

the key single unti property is neuronal tolerance (eg to position change)

single-unit tolerance to change is the key needed and observedproerty.

the resulting pop. repr is powerful cuz it simult gives object identity AND info abt

positon, size, pose context... so it avoids need to re-bind info later.

correspons p 423 to  aligning sheets of unfurled crumpled paper to anser whether

an object was on the left of another.

p 423:  WAHT IS THE ALGO to do this?

probably UNTANGLED using cortical circuits...

retina, LGN:  deal with luminance and contrast  but these are

point-wise sensors

so  V1 is also a tangeld pixel repr.

V1  had  30X  the # of output neurons;  so object maniforls are more "spread out"

p 423  Deep Stack of Cortical Areas:  each vis areas adds proc pwoer to solve incr

complex tasks;

ANALOGY:  CAR ASSEMBLY LINE WORKER can only do one small set of oerpations;

but a serial assembly line can build something much more complex  (a whole car or

a good object repr)

2nd idea:  interctions in the difft levels of the heirarch:  similar to US Army.

key notion:  local feedback btwn difft coricl areas is critical.

enable inference about weak or noisy or missing or occluded data  using

a hierarchical Bayes framework.

foot soldiers may "see an edge"  in V1... passed to seargents in V2 who pass acum info

to lieutenants... so higher agents can see the forest for the trees..

the  FOREST in turn directs the lowest level on how to optimize proc of this weak sens evidence to help the higher agents eg IT...

p 423:  CENTRAL ISSUE is whether re-entrant  ffedbk is neces for building explicit object repr in IT within time scale of nal vision.   perhaps expts  like Boydens (turning off feedback) will settle this debate.

Compromise view may be correct.;

binoc reivalry prob involves feedbacka nd interax  as does recog in visual clutter (Where's Waldo?);

But... large class of object recog  uses just CORE recog  using the first spikes in IT (first 150 ms);

But..  v short time  re-entrant proc within loca l circuits  (< 1 ms) is likely an integral part of even

the fast IT response.

p 424  one key idea is abstraction:  guy installing engine does not need to know details

of putting enging together;  job secription of each worker is locally specified and maintained.

p 425  he repeats the (Kurzweil FPGA/  Mountcastle) notion of repeating units of cortex

implementing the same algo.

canonical meta job description is to solve a microcosm of the genl untangling problem.

works locally:  a small aperture on LGN by V1,  a small apert on V1 by V2 etc.

424  Bottom Up  Encoding Model;

p425  Hubel and Wiesel:  Simple cells use  AND-like operation on LGN inputs...

orientation tuned response

V1 compelx cells  form an invariance by making OR-like combos of simpel cells

this is cental to current encoding models.

formalized into LN  linear non-linear class of encdoing mdoels...

each neurons adds and subtraxts inputs followed by thresholding...

they can accoutn for a substant amt of single neuron patterns in early visual, somatosens and auditor areas...

simplex extensions if the NLN  normalized  LN models... orig used normalization to capture luminance and contrast adaption  (divide by weighted sum of a nearby pool.

the field implictly assumes this is the case

but is the transfer function from V4 to one IT neuron similar (just with difft weights and pooling?)

So,  what is the specific image-based encoding function that predicts the response of a given IT neuron

(as the end of my electrode today!!!)... is impossible with current methods.

But... look at the problem for each IT (NLN) neuron:  from which V4 neurons whoudl I receive inputs...

with what wieghts and waht normaliza pool and with what statis nonlinearity>

Dicarlo, Zoccolan, Rust, Visual Object Recognition, Neuron, 2012

Attachments