Ganguli Sompolinsky Compressed Sensing Sparsity Dimensionality Ann Rev Neuro 2012

Dec 02, 2012 07:33 PM

in my pdf collection...  online (from Haim's website);  prob at Surya's also.

Abstract

The curse of dimensionality poses severe challenges to both techni-cal and conceptual progress in neuroscience. In particular, it plagues

our ability to acquire, process, and model high-dimensional data sets.

Moreover, neural systems must cope with the challenge of processing

data in high dimensions to learn and operate successfully within a com-plex world.We review recent mathematical advances that provide ways

to combat dimensionality in specific situations. These advances shed

light on two dual questions in neuroscience.

 First, how can we as neu-roscientists rapidly acquire high-dimensional data from the brain and

subsequently extract meaningful models from limited amounts of these

data? And second, how do brains themselves process information in

their intrinsically high-dimensional patterns of neural activity as well

as learn meaningful, generalizable models of the external world from

limited experience?

INTRO  486

HH model in 1952:  reln btwn volts and current across membrane.

gray-scale image requires N variables = pixel intensities;  where N is about 1 M.

OR  number of firing rate patterns in a given brain area = # of neurons in that area.

487   curse of dimensionality:  harder to do, the more the d's.

approach:  REDUCE the # of vars to describe the pattern... dim reduction.

Can do this with NATURAL IMAGES;  highly restricted subset of all images:

natl images are SPARSE... if u view them in the WAVELET DOMAIN

(as a superposition of edges) (bb: ie  take Hubel/Wiesel wavelets and superimpose);

then a verysmall  # of K wavelet coeffs can do it...

K around 20,000 for a 1M pixel image.....

this underlies  JPEG compression... computes all poss wavelet coeffs and keeps on the K largest.

Taubman 2002... see that ref  Jpeg2000 image compression fundamentals...

p 487 col 1 cont....

simil... neural activity patterns  are often a highly restricted subset of all patterns...

see Chris Machens ref and Ganguli ref...

LOW K-MANIFOLD  embedded in N dim firing rate space...

ie  only K numbers are required to specify any observed pattern across N neurons,

where K can be much smaller than N.

example:  N neurons presented with a bar at various orienttions....

as orientation varies... firing rates trace out a circle or  1d manifold in

N dim space.

how can we or nerve systems extract a small # of vars w/o too much loss.

page 487 col 2:

Donoho ref and Baraniuk 2011 ref..

one can randly project patterns into low d space.

SHINE A FLASHLIGHT at a WIRE FRAME OBJECT randomly

(xc in degen cases)  it will reproduce all info on object in its SHADOW !!!

RP = random projection...

Field of COMPRESSED SENSING shows that  the shadow has enuf info to reconstruct

original image:  all N pixels  (if original is sparse enuf).

ie (bottom of page) is  M is sl larger than K then  CS = compressed sensing

is an algo to reconstruct image  =  L1 minimization.

thus, eg  we can simult sense and compress  1M pixel images with

20K random measurements.   Big implications for neurosci.

p 488: col 1:   

an RP of neuronal activ-ity in an upstream brain region consisting ofN

neurons can be achieved by synaptic mapping

to a downstream region consisting of M<N

neurons, where the downstreamneurons’ firing

rates are obtained by linearly summing the fir-ing rates of the upstreamneurons through a set

of random synaptic weights. Thus the down-stream activity constitutes a shadow of the up-stream activity through an RP determined by

the synaptic weights (i.e., angle of light).

Leaarning:  (also good to reduce dimensions before learning)...

Neural systems face a sim challenge in searching a high d synap wt space to 

LEARN generalizable rules from lim experience.

we review regulariztion techs  see Brad Efron ref 2004 !!! (old friend)...

closely related to CS  allow stats and neural systems alike to rapily learn

sparse models of high d data from lim examples.

col 2:  overivew of  CS = compressed sensing  framework.

Compressed Computation:

altho CS remphasize reconstruciton of sparse high d signals from low d projections...

many impt probs in learning and signal proc can be done direclty inlow d space.

Manyof thedse task rely on the DISTANCES btwn data points...

which are preseved by RPs !!

Approximate Sparsity and Noise....

in the above we assume  a defn of SPARSITY in which  the other elements of the N d space

have exactly ZERO coeffic... in reality many are just small but not exactly zero.

they are just well approx by a K-sparse vector...

Compressed Sensing by the brain:

The problem of storing, communicating, and

processing high-dimensional neural activity

patterns, or external stimuli, presents a funda-mental challenge to any neural system. This

challenge is complicated by the widespread

existence of convergent pathways, or bottle-necks, in which information stored in a large

number of neurons is often compressed into

a small number of axons, or neurons in

eg 1M  optic N fibers carry info abt activity of 100X photoreceptors.

and only 1M pyram tract fibers carry info from cortex to spinal cord.

And... cortico- basal ganglia pathways undergo a 10 o 1000 fold convergence.

SEMANTIC SIMILARIY and random projections

Compression and Expansion in Long-Range Brain Communication

see figure 5  (exaclty like 2 people talking:  everything compressed down to

a sequence of phonemes  (er even better analogyis with email)  entire msg

compressed and then re-expanded at other end.

Local sparse represen-tations are first compressed to communicate

them using a small number of axons and

potentially re-expanded in a downstream area.

Where in the brain might these transforma-tions occur? Coulter et al. (2010) predict that

this could occur within every cortical column,

with compressive projections, possibly random,

occurring between more superficial cortical

layers and the output layer 5. A key testable

physiological prediction would then be that

activity in more superficial layers is sparser than

activity in deeper output layers. Another pos-sibility is the transformation from sparse high-dimensional representations of space in the

CA3/CA1 fields of the hippocampus to denser,

lower-dimensional representations of space in

the subiculum, which constitutes themajor out-put structure of the hippocampus. A functional

explanation for this representational dichotomy

couldbe that the hippocampus is performing an

RP from CA3/CA1 to the subiculum, thereby

minimizing the number of axons required

to communicate the results of hippocampal

computations to the rest of the brain.

Overall, these works suggest more generally

that random compressionand sparse coding can

be combined to yield computational strategies

for efficient use of the limited bandwidth avail-able for long-range brain communication.

LEARNING IN

HIGH-DIMENSIONAL

SYNAPTIC WEIGHT SPACES

Learning new skills and knowledge is thought

to be achieved by continuous synaptic mod-ifications that explore the space of possible

neuronal circuits, selecting through experience

those that are well adapted to the given task.

We review how regularization techniques

used by statisticians to learn high-dimensional

statistical models from limited amounts of data

can also be employed by synaptic learning

rules to search efficiently the high-dimensional

space of synaptic patterns to learn appropriate

rules from limited experience.

NEURLA LEARNING of CLASSIFICATION

similar to classic percepron.. Frank Rosenblatt 1958...

DISCUSSION:  Dimensionality Reduction:  CS vs Efficient Coding

Efficient coding theories  BARLOW refs!

suggest info bottlenecks...

suggest brain does  optimal dimension reduc...

infomax theories of early vision predict that high-d spatiotemp patterns

shud project onto the linear subspace of their largest princip components.

the individual projection vectors,

i.e., retinal ganglion cell (RGC) RFs, depend

on the stimulus SNR; in particular, at a high

SNR, RFs should decorrelate or whiten the

stimulus. This is consistent with the center-surround arrangement of RFs, which removes

much of the low-frequency correlations in

natural images (Atick 1992, Atick & Redlich

1992, Borghuis et al. 2008

Although the algorithms and results of

ICA and source extraction by CS are often

similar, there are important differences. First,

CS results in signals that are truly sparse,

i.e., most of the coefficients are zero, whereas

ICA algorithms generally yield signals with

many small values, i.e., distributions with high

kurtosis but no coefficients vanish (Olshausen

et al. 1996, Bell & Sejnowski 1997, Hyvarinen

2010). Second, ICA emphasizes the statistical

independence of the unmixed sources (Barlow

2001). Sparseness is a special case; ICA can be

applied to reconstruct dense sources as well.

In contrast, signal extraction by CS relies only

on the assumed approximate sparseness of

the signal, and not on any statistical priors,

and is similar in spirit to the seminal work of

Olshausen et al. (1996). Indeed, a recent study

suggests that sparseness may be a more useful

notion than independence and that the success

of ICA in some applications is due to its ability

to generate sparse representations rather than

to discover statistically independent features

(Daubechies et al. 2009

Ganguli Sompolinsky Compressed Sensing Sparsity Dimensionality Ann Rev Neuro 2012

Attachments