CHKLOVSKI, Dmitri, Sparse Coding, Janelia, Feb 26 2014
<p class="MsoNormal" >CHKLOVSKI, Dmitri, Sparse Coding, Janelia, Feb 26 2014</p>
<p>
Olshausen and Field 1996 key paper (prev cited) learning receptive fields)</p>
<p>
image is the data matrix X.</p>
<p>
weight matrix is W</p>
<p>
try to recompute X as WY.... Y is the firing rates of the neurons at first layer.</p>
<p>
mu = learning rate in gradient descent...</p>
<p>
neural computation as online matrix factorization</p>
<p>
</p>
<p>
the factorization problem</p>
<p>
neuron with multiple inputs and an output
input is vector X (firing rates of upstream neurons)
W are their weights (strengths of synapses) and y is the firing rate of the neuron's axon.</p>
<p>
but, a neuron sees streaming data...</p>
<p>
streaming data with time produces a data matrix X that is approx</p>
<p>
W * y transpose.</p>
<p>
can use PCA....</p>
<p>
Wy, gives a vector and then look at difference btwn Wyi and xt and find the PCA's (the projections of the data onto basis)</p>
<p>
but neuron cannot do this...</p>
<p>
neuron must respond in real time to input...</p>
<p>
want to optimize match:</p>
<p>
so, there is a continual optimization of weights...</p>
<p>
can you do this online (without seeing the entire stream)</p>
<p>
Oja...he shows model from Yang 1995;</p>
<p>
Why PCA is not enough</p>
<p>
neurons do not follow gaussian distrib.</p>
<p>
many more cases when you have high firing rates (heavy tail) and
lots of neurons that are silent a lot.</p>
<p>
an experiment: look at a pyram cell: and use a</p>
<p>
soft threshold function...</p>
<p>
an L0 norm is y or zero</p>
<p>
type II neurons eg fast spiking interneuron...</p>
<p>
up to threshold.... no firing then goes up linearly with input current (in type 1 cell)</p>
<p>
Q from B. Wandell: is (strict adherence to) dot product important...
A no... I'm certainly aware of dendritic nonlinearities.</p>
<p>
learning rate depends on age and activity:</p>
<p>
stimulate pre and pose synap neurons and see change in weights...</p>
<p>
Poo and Isaacson in 2007
shows % of LTP with age from age 5 down to age 30 etc.</p>
<p>
(from olfac bulb to pyriform cortex.... learning rate in sensory organs goes down)
but not in cortical to cortical connex (that continues to have high LTP)</p>
<p>
feed fwd connex in primary connections are learned early
and they do not change over time... (Once they have learned the statistics of the world.)</p>
<p>
that was topic one: age versus activity dependence of LTP
</p>
<p>
not just neurons, but also the synap weight distrib is non Gaussian...</p>
<p>
large peak at zero... not connected....</p>
<p>
Varshney, Sjostroma and chklovskii, in 2006</p>
<p>
comment why would someone expect it to be gaussian
A; that might follow from a PCA model (which no one beleives)
(it actually has a fatter tail)
need regularizer:</p>
<p>
what would you waste resources on a silent synapse...</p>
<p>
eg if presynap activity cannot overcome SoftThreshold...</p>
<p>
comment (EJ Chi.) why doesn't silent syanpse disappear completely...
A: cuz at some time in the past it was useful....
if you get rid of it, then correl. btwn pre and post neurons would disappear.</p>
<p>
can online algos perform as well as offline? (neurons cannot store all the data)</p>
<p>
(by offline, he means given all the data (as used in a PCA)</p>
<p>
online convex optimization by Hannah et al.</p>
<p>
*</p>
<p>
rank 1 matrix factorization (means, only one axon with one firing rate at any given time)</p>
<p>
doing higher rank matrix factorizaiton</p>
<p>
each neuron explains away part of the output Y
eg y1 explains away part and then y2 tries to explain the residual
and next y3... etc.</p>
<p>
Oja (pronouned Oy-uh): difft local learning rules...</p>
<p>
*</p>
<p>
next topic: temporal matrix factorization:</p>
<p>
eg parallel channels in early visual processing</p>
<p>
photoreceptors going to bipolar cells</p>
<p>
from Euler and Masland in 2000...</p>
<p>
why have both sustained and transient responses? (in difft channels in retina, eg)</p>
<p>
these are also optimal for time (just as gabors are optimal solution for space)...</p>
<p>
can do PCA over time... and he shows PC1 and PC2...</p>
<p>
tau is a temporal discounting factor...</p>
<p>
his theory produces learned features that look exaclty like the bipolar cell features..
ie On-sustained; On-transient; Off-sustained; and off transient;</p>
<p>
these were all trained together...</p>
<p>
trained with natural stim movie (a video) and looked at a single pixel.</p>
<p>
</p>
<p>
temporal learning : STDP and spike cross-correlation</p>
<p>
(every neuron must do both time and space learning)</p>
<p>
led us to STDP rule.</p>
<p>
our view: <b>STDP itself is an on-line algorithm</b>
that <b>itself must be learned!</b> (our view)</p>
<p>
synapses represent cross-correl btwn spatial and temporal channels...</p>
<p>
the famous STPD curve (as in Zhang 1998) emerges from learning spike cross correlations</p>
<p>
*</p>
<p>
summary of talK;</p>
<p>
olshausen and fields model is right and must be done online (in real time)...
can reproduce a lot of bio facts....</p>
<p>
*</p>
<p>
jay: BCM apprroach: please contrast your approach to theirs.</p>
<p>
A: BCM learning rule: can be thought of gradient descent...</p>
<p>
we have tried to stay with convex cost functions...
want provable convergence to a global minimum...</p>
<p>
Jay: don't you assume stationarity... A: yes , but...
can have multiple (internal ) agents that learn at difft rates.</p>
<p>
Bill N: was barlow right or wrong? (given his retraction)</p>
<p>
A: it is a good model for retina (cuz need to do lots of compression)
but (barlow says) perhaps not univ applicable....</p>
<p>
bruno: want as many neurons as possible, but let's use them in the best way possible.</p>
<p>
BW: have you ever looked at a really bad stimulus (highly non-natural)..
David D. answers: white noise fires everything... that's why you want to threshold it out...</p>
<p>
</p>