LE QUOC DEEP LEARNING MBC Oct 29 2012
<p class="MsoNormal" >LE QUOC DEEP LEARNING MBC Oct 29 2012</p>
<p>
(at this time, Quoc was a PhD student of Andrew Ng; now at Google (> 2013))
This PhD work became famous (as the Google cat detector.) See NYT article.</p>
<p>
machine learning; deep nets;</p>
<p>
this is at 5:15PM Monday Oct 29 to MBC grad students in Math 380C room bldg 380;</p>
<p>
why is computer vision hard?</p>
<p>
want to recognize Einstein eg... <b>you </b>see Einstein but what computer sees is the intensity matrix..</p>
<p>
Feature reps... learning algo...</p>
<p>
want to separate faces from nonfaces...</p>
<p>
how about MOUTH vs EYE... easier to plot these in a hyperplane with a separating plane...</p>
<p>
feature extraction... do not want to have to do this by hand... want to do it automat.</p>
<p>
we don't have an eye or mouth detector...</p>
<p>
hand-crafted images... HOG or SIFT... for vision</p>
<p>
audition: spectrum...</p>
<p>
what we are not good at is the feature repres...</p>
<p>
input is motorcycle eg... what are the best features...</p>
<p>
images to vison features go to detection/ id</p>
<p>
and similar with audio stream or text stream...</p>
<p>
sl: one-learning algo hypothesis in the brain...</p>
<p>
Roe 1992</p>
<p>
rewire vision to auditory cortex learns to see (actually in salamader... he says mice... no)</p>
<p>
sensor reps in brain</p>
<p>
seeing with your tongue... blind climber..</p>
<p>
human echolocation</p>
<p>
key ideas in this talk</p>
<p>
use unlabeled data to invent features</p>
<p>
scale up to learn high level abstract features...</p>
<p>
we have a lot of data at google... from cloud...</p>
<p>
USE UNLABELED DATA to invent FEATURES.. key idea...</p>
<p>
TICA topographic independent component analysis..</p>
<p>
combine features in a local area...</p>
<p>
take dot product of image and filter... say filter is 3 by 3 = 9 blocks...</p>
<p>
take a weighted sum...</p>
<p>
learning principle...</p>
<p>
take sum across all positions and across all examples...</p>
<p>
why did you do 2d topological graph...</p>
<p>
want to run across positions to get position invariance...</p>
<p>
want to get translation invariance and scale invariance...</p>
<p>
Learned invariances from unlabeled data...</p>
<p>
see Le et al convol neural nets NIPS 2010...</p>
<p>
eg by sweeping an edge across difft positions... the feature does note change...</p>
<p>
better chance to work over all modalities...</p>
<p>
CVPR 2011 Le et al learning hierarch spatio-temp features for action recog with indep subspace analysis..</p>
<p>
his learned features beat all competitors in KTH and Hollywood2 and UCF and YouTube...</p>
<p>
theirs 76%; hand engr was abt 70% accuracy...</p>
<p>
</p>
<p>
Is it poss to learn abstract features... eg Jennifer Aniston neurons...</p>
<p>
Desimone 1984 neurons selective for faces and hands</p>
<p>
Quiroga 2004 Jennifer Aniston neurons...</p>
<p>
want to build high level detectors for car, people.... that can cross modality...</p>
<p>
scale is important <b>largest artif network has abt 107 connex</b>...
brain has perhaps 1015 (or less)</p>
<p>
It's better to have more features...</p>
<p>
<b># of features always improves accuracy</b>..</p>
<p>
Coates, et al Ana ala of singler natl and ....</p>
<p>
Banko and Brill Scaling to Very Large Corpora for Natl Lang Disambig 2001</p>
<p>
MORE DATA is better for training...</p>
<p>
they compared Naive-bayes to perceptron to Winnom... they all improve with mroe data...</p>
<p>
Most features are local...</p>
<p>
RICA features... initially every features connects to every pixel but later cut it down...</p>
<p>
first get a local receptive field network...</p>
<p>
have machine 1 look at left side of image and machine 2 look at right side...</p>
<p>
challenges with 1000s of machines (must make it fault tolerant)</p>
<p>
send paramaters to a server which updates its belief...</p>
<p>
parameter server is in the middle...</p>
<p>
Q why does an avg make any sense... if manifold is nonconvexr and nonlinear...</p>
<p>
A might still be locally convex and locally linear...</p>
<p>
practially it works well</p>
<p>
Le et al building high-levell features using large-scale unsupervised learning ICML 2012...
(he describe parameter server in that paper)</p>
<p>
summary of scaling up</p>
<p>
he describes an asynch SGDs</p>
<p>
looked at 10M 200 by 200 images...</p>
<p>
get these from google...</p>
<p>
network has 1B parameters...</p>
<p>
unlabeled images from YouTube/WWW</p>
<p>
train on 2000 machines = 16,000 cores for 1 week.</p>
<p>
train 1.15 B parameters</p>
<p>
this is small compared to visual cortex but 100X larger than prev reproted...</p>
<p>
have visualization to show optimal stimulus by numerical optimizaiton</p>
<p>
top stimuli from the test set...
he shows abt 50 faces that are the top stimuli..</p>
<p>
shows ghostly face that is optimal stimulus...</p>
<p>
</p>
<p>
The CAT neuron...</p>
<p>
he shows optimal stim for a CAT </p>
<p>
cute pic of cat with a lime on its head...</p>
<p>
see NY Times July 25 2012 with Andrew Ng... cat detection!</p>
<p>
best stimuli</p>
<p>
feature 1... speckles
feature 7 purple flowers</p>
<p>
feature 8 yellow flowers on green backgrund</p>
<p>
feature 10 zebra
feature 11 keyboard
feature 12 wine bottle</p>
<p>
feature 13 pizza...</p>
<p>
key paper is Le building high-level features using large-scale unserpvised learning ICML 2012...</p>
<p>
feature 6 flatware</p>
<p>
Q what are low level features for yellow flowers</p>
<p>
for faces its eyes and mouth...</p>
<p>
feature 9 ducks in a pond, seagulls etc.</p>
<p>
Q how many neurons in highest layer A = 50,000...</p>
<p>
Q how many are learning interesting things like cows, etc</p>
<p>
A only a few hundred
most features are just low level..</p>
<p>
ImageNet classif</p>
<p>
22,000 categories
14M iamges
hand engr features SIFT HOG LBP
spatial pyramid sparcoding/ comrpession..</p>
<p>
eg roughtail stingray vs mantaray...</p>
<p>
he shows just the rays, sharks...</p>
<p>
random guess = 0.005% in this task</p>
<p>
state of the art is Westona and Bengio 2011 9.5% accuracy</p>
<p>
we get 19.4% accuracy !!!! feature learning from raw pixels...</p>
<p>
using only 1000 categories (smaller dataset our method yields > 50% accur (but still below SOTA)</p>
<p>
....</p>
<p>
so this dramatical improves sppech recog; machine translat
it is benefiting many google products...</p>
<p>
ImageNet is organ in a hierarchy</p>
<p>
there is a tradeoff btwn sensitivity and selectivity...</p>
<p>
his network has 3 layers</p>
<p>
the first layers are edge detectors...</p>
<p>
the classifer at the top is just one layer...</p>
<p>
he shows barn, spoonbill, planetarium...</p>
<p>
Q i ask abt faces turned upside down...
A he did not do it...</p>
<p>
Q i get upset with 1.15 B parameters... it's too many...
A we did an expt where we made it smaller
but bigger was better...</p>
<p>
Q do u know if the network has converged?</p>
<p>
Q are u concerned abt overfitting and if not why not?</p>
<p>
A prob not overfit cuz training set it so large...</p>
<p>
google.. so easy to get unlabeled data...</p>
<p>
Asynch Stochastic Gradient Descent... Asynch <b>Stoch Grad Desc SGD...</b></p>
<p>
SGD is v noisy... don't want master to wait for slaves...</p>
<p>
Q deep network vs wide?</p>
<p>
Q perhaps 2% of top level features are recognizable (eg faces or cats)</p>
<p>
uses backprop algo
</p>