Hayworth Ken Thesis 2009 Visual Encoding of Spatial Relations

Feb 07, 2011 09:05 PM

Ken's  thesis is available online  (Ken Hayworth)  also stored in my Neurosci files.

Thesis directly addresses my central issue:  encoding of visual perception.

Beautifully clear.

USC PhD thesis  finished in Dec 2009  with  Irving Biederman. (his thesis advisor)

neurosci PhD,  now in Jeff Lichtman's lab at Hvd.

Acknowledge  Irv Biederman theory:  Recognition by Components:

we understand visual structure by a neural rep that is itself structured.

JIM  computation model  by John Hummel showed how neural circuits could create such a structure rep.

He was also influenced by Allen Newell model of cog architecture.

His theory is  MS2

He worked at USC's Image Understanding Lab.

Abstract

converging evidence (ERP, single units, comp models) 

leads to  a  "STANDARD MODEL" of visual system in which

ventral cortical areas produce a final visual repres consisting of cells having

signif translational invariance and tuned to features of intermediate complexity.

this feature list is useful for object recognition but

it seems at odds with our ability to understand and parse visual structure and spatial relns btwn them.

He did fMRI-adaptation experiments  to check standard model.

look at  elephant above bus    vs  bus above elephant ;  this is an example of swapping relational roles

swamping the objects result in  BIG  release from  fMRI-adaptation in  LOC = lateral occip complex

then when objects were merely translated.

It is best characterized  by a  STRUCTURAL DESCRIPTION MODEL

where object's feature list is assigned a separate SLOT in LOC.

In such a slot model, a  SWAP of relational roles produces  a  WHOLESALE SWAPPING of feature lists

between LOC slots.   Results of  fMRI  support  THIS.

He proposed  MS2 =  Multiple Slots,  Multiple Spotlights model  (based on the standard model)

with the added assumption that the ventral visual stream contains multiple semi-independent feature hierarchies

and can thus create multiple simultaneously active feature lists.

MS2  explains all the phys data and also explains a host of cog psych results which standard model cannot.

He also  provides  an  ANN = artif neural net implementation of his MS2 model ,  which

correctly simulates all results!

Page 1  INTRO  (this is stunningly clear;  v high signal to noise ratio...  just reread this, many times)

What is nature of neural repres underlying  visual object recog, shape understanding and scene perception?

many theoretical models:  Neocognitron, Visnet,  HMAX,  Standard Model of Serre,  etc.

propose that  final stage of  ventral visual hierarchy  =  AIT in monkey = anterior, inferior temporal

or  anterior  LOC = lateral occip complex in human

consists of  a set of neurons with large receptive fields tune to features of intermediate complexity.

Such models are called  FEATURE HIERARCHY models  (hierarchy emphasizes the progression of neural layers)

He refers to them as  a  FEATURE LIST  b/c  they are  an UNORDERED listing of the features in the attended field.

page 2

Feature list models seem to be the ONLY class of models for doing very fast object recognition! (Serre 2005).

This model is what  EVERYBODY BELIEVES but the final repres it advocates:

an UNORDERED list of features -  is  starkly at odds with  a host

of behavioral expts in cog psych which support  STRUCTURAL DESCRIPTIONS (SD) as the final visual repres

as in Biederman 95.

SD reps:  some visual features are explicitly grouped together and segrated from other features;

ie  SD models  REQUIRE a NEURAL BINDING MECH  that can tag visual features belonging to the same group.

page 3:  this grouping and segregation, parses a scene into higher-oder visual entitites (objects or parts)

SD models :  elephant features bound together and included with info that that are at the TOP of the scene.

page 4

His fMRI expts  support the hypth that the final visual repres is NOT a simple feature list.

His results support the core assumptions of the SD model: 

 MULTIPLE visual entities simultaneously and separately!  (including  explicit spatial arrangement info)

His model is a full-fledged SD including neural binding and explicit spatial relations.

page 6:  Chapter 2  Bkgrd

He reviews the STND Model including Neocognitron and VisNet;  only feedforward.

Lateral and feedback provide the substrate for VISUAL ATTENTION:  isolate a single object from bkrgd.

This is a SINGLE SPOTLIGHT model of attn.

Serial shifting accomplishes   multiobject scene encoding  (according to this theory).

But...    its INADEQUATE!  does not answer how we repres visual structure!

Evidence for feed-forward feature hierarchy!  felleman  and van essen 91;

= ventral visual stream in occipital and temporal lobes;

neurons in V1  tuned to simple visual features... oriented edges  with small fields  abt  0-.5  degrees.

neurons in later visual areas like V4 or PIT  more complex features  eg  multiple curved contours

over wide range of positions.

AIT is the most anterior in the primate ventral visual stream..  even greater complexity.

anterior cells respons more slowly, as if they are gatherin ginput from poseterior areas.

How does it work?  only  well studied in  V1.

Simple cells are just a  roughly  CONJUNCTIVE  AND pooling of cells from  LGN afferents  to

produce simple cells tuned to  oritented edges and bars at partic locations,

whereas the the projections into V1 from LGN  are only tuned for points of light or dark with opposite surround.

COMPLEX CELLS in V1  are a  DISJUNCTIVE OR pool inputs from V1 simple cells...

also tuned to oriented edges but over a wider range of retinal locations.

(CONJUNCTIVE can do with a dot-product on neurons inputs

and disjunctive can do with a soft MAX operation).

Can  chain these kinds of operations up the hierarchy to do complex features,  eg  an entire face in  AIT

in a largley translational invariant manner.

page 11

researchers focus on  feedforward (rather then bkwd)  cuz  time frames  require great speed.

Also,  there are comput model  Neocognitron, HMAX  that have a degree of competence doing only feedforward.

Evidence for object-recog vis feedforward

can do  animal vs no-animal in 150 ms !!

In monkey in  IT,  neurons in IT - inferior temporal cortex respond in 100 ms.  (and pfc)

(too fast for feedback/ lateral effects, which are needed to compute a strucutre description of a multi-object scene).

His major point:  since feedforward is the degenerate case (ie used when great speed is required for a single object)

does NOT imply that it ALONE is used for ordinary vision (leisurely with mutiple complex objects).

page 12

Computational models based on feedfwd hierarchy

He reviews three of these models:  Neocognitron, VisNet and the Std Model;

Neocog was first;

VisNet  showed how it can self-organize, with a learning rule

Std Model  tied key model parameters to known ventral stream physo.

Neocognitron was first;  Fukushima 1980:  derived from Hubel, Wiesel.

S and C cells completely tile field;

final set of C-cells constituted a feature list repres.

page 17

VisNET  Learning receptive field tuning properties;

learns vis the TRACE RULE:  learning based on the visual working memory fromthe past few hundred msecs

as  objects move across the gaze of an infant.

VisNet  Rolls and Stringer 2006;

The Standard Model  Risenhuber and Poggio 1999... HMAX, similar to Fukushima Neocog.

HMAX now called the Model or the Std Model.

page 18

S1  is hard coded to precprocess the pixilated retinal image into a set of separate

spatial frequency channels.  =   filtering by a set of oriented  Gabor filters.

Gabors have 17 difft scales (ie spatial frequencies) and 4 difft orientations.

page 20  The 'final"  visual repres

The final C-layer of the Std Model consists of a set of cells tuned to features of intermediate complexity.

fields span the entire viz field and are responsive to their target feature over wide spatial scales.

So,  final C-layer signals presence or a simple 3D feature (like a corner of a box) anywhere in the field.

This IS the FEATURE LIST repre he talked about above.  purely feedfwd;  cells  INDEPENDENTLY signaling

presence of thier target feature.

page 21:  millions of neurons in LOC; therefore, each object cud have a unique fingerprint, neural firing pattern.

Real power is that this is a SPARSE rep.  Any particular object can be decomposed into hundreds of features

see page 22:  Example feature list repres:  AIRPLANE:   great SLIDE:

vertices  (ie corners and lines)  parts  (tail section, wing)  global shaped blob.

page 23:  Turtle  vs  JET   great slide   features map to semantic areas.

VTU = view tuned units are the ultimate neurons that finally NAME the object.

this is done b/c large roations in depth of an object can present difft sets of features.

A VTU is created for each of these views.

Stnd model works well  (see Serre et al 2005)...  tested on a large set of real world objects and

drawing of objects, some in cluttered bkgrnds.   Decision = car present or not.

Model's performance was 95%.  impressive and bio plausible.

But  2AFC = 2 alternative, forced choice is  a far cry from what we humans do.

We can id an object as being one of tens of thousand sof discriminable object types.

page 24  WE also UNDERSTAND the visual structure of what we see.

page 25

but see  Fig 2.3  Parsing vis sturcutre at many levels:

look at the truncated pyramid in fig 2.3 D... ev3en tho is may be 'primitive"

we still SEE it...  are its sides parallel or converging?  how many corners are visibile?

is a point inside or out?

such products are not reasoned... they are built into the system.

how about selective attention?

page 26:  how abt  ATTENTIONAL SELECTION and other functions of the DORSAL visual stream?

The dorsal stream  KNOWS where in the visual field it has allocated attention ...

it ENCODES WHERE the current object of attention is located:  giving rise to the WHAT vs WHERE.

page 27:  This is CRUCIAL!

The feature list rep on the ventral vis stram THOWS away position and scale info.

It does so to create a rep that is optimal for view-invariatn object recognition.

It makes an amazing clinical prediction:  If a pt. had only the ventral stram rep,

he could recognize objects but now know WHERE they were.  This  is BALINT's syndrome:

caused by bilateral destruction of the parietal lobes... knocking out the  dorsal visual stream

BAlint pts have simultagnosia ...  they can perceive only ONE single object at a time

with no understanding of WHERE it is in space or in relation to  other object...

but they can readily name the object!!!

A patient could READ a word but not tell whether it was at the top or bottom of the screen!

page 28

OR looking at a tic-tac-doe  board  could not judge realtive postion of an X relative to an O.

a consequence of simultagnosia assoc with Balint's syndrome.  Robertson, Tresiman 97.

When shown two objects, he ofter only saw one.

This Balint pt is acting like the Stnd Model only!

the firing pattern in his LOC allows his frontal neurons to decode the name of the object but

because of the big receptive fields of the LOC neurons the position is not available to the

frontal cortex.

page 29:  it is known that shifting visual attention (both covert and overt (saccading)) is controlled by

circuits in the parietal lob,  one might expect tha the spotlight of atten would be destroyed.

the result is NOT a loww of focused attention but instead

hyperattention to one object at the exclusion of all others (even ones visually overlapping)

Pareital inputs into ventral stream is only needed to SWITCH attention to a new object.

So how does the ventral stream accomplish object segregation and suppression of other objects.

This is wehre lateral and feedback connectivity comes in.

page 30... this circuitry gives rise to attractor dynamics in the entire hierarchy which allows the repres of

one coherent object to win out over the others.

Modulatory projection from the parietal lob nee donly bias this inherrent ventral stream dynamics from

one attractor state to another to switch attention btwn objects  (Serences and Yantis 06).

How abt normal people?  how do they project two objects?

one hypothesis is that dorsal circuits enable 2 simultaneous objects to be locked onto by

producing a final rep which is a listing of both objects features.

page 31:  postions wud be decode by looking at inter-object features.

OR this alternative:  SERIAL SHIFTING OF attention:

first look at one object, then shift attn to the next  and record this info.

He tests these  (they both flunk!).

page 32:  SUMMARY of  STND MODEL  good summary.

page 34

Open research question:  how do we repres structure?  dictionary def of structure:

the relationship of parts to the whole.

eg is the coffee cup closer to the lamp or the chair?

In this simple discussion, Ken destroys his opposition!!!

vision is NOT abstract reasoning!  We SEE the relative positions of the cup, lamp, and chair!

shfits of attn would throw that info away as would feature lists.

the feature list repres is STRUCTURELESS!  it does not contain the info!

task:  we must 1)  segment the parts  and  2)  represent the spatial relationships of those parts.

page 36:  STRUCTURAL DESCRIPTIONS

cognitive psychologist  do behavioral experiments and propose  SDs  rather than the

feature hierarchy models proposed by vis physiolgisits.

Must group features together.  Synchronous firing is the usual mechansim proposed.

The grouping and segregation of features , parses a scene into higher-order entititees.

Elephant above a bus:  elephant would fire as a group in phase.

Bus neurons would also fire in phase as a group...

but crucially, these two groups would be OUT OF PHASE with respect to each other.

This temproal syncrony binding mech parses the scene into vis entities.

page 37:  JIM  struc descrip  also  hypothes.  a set of cells  explicitly signaling

aptail relns   aBOVE, BELOW, beside.

As  elephant cells are firing in phase, the realtion cell signaling ABOVE would also fir in phase with it...

similarly iwth bas.

page 38  monkey IT cortex  =  human LOC...  the final site of final feature list repres.

39  are the hierarchy of features in the ventral stream mapped onto an SD or not??? big question.

SDs  make explicit a distinction btwn the shape of a scene's entitiies and the relations among those entities.

SD  offer a baiss for reasoning abt objects and scenes  and may be REASONING in general!

page 43:  bold adaptation basis is being debated  (also called  repetition suppression)... is neural fatigue, etc.

however,  all interpretations agree that a difference in neural reps btwn stim1 and stim2 is what leads to 

an increase inbold response.  it is not just a global surprise effect.

page 44:  Expertiment 1 REsults:

page 66:  Whole Brain Analysis:  see Fig 3.5A  voxels meeting criterion at p <0.016

in  pFs  LOC,  IPS and dlPFC

page69    results were unambiguous:  pFs showed a greater release from adaptation in the relation change condition in

everyone of the 30 subjects.

and separately!

Hayworth Ken Thesis 2009 Visual Encoding of Spatial Relations

Attachments