HINTON Geoff NEURAL NETS for Machine Learning COURSERA 2012
<p>Oct 08, 2014 09:42 PM to sign in; click this link; then email and pswd...</p>
<p>
Sep 25, 2014 10:12 PM 2nd time thru this; first time was 2 years ago...</p>
<p>
Nov 28, 2012 07:59 PM</p>
<p>
I sign up today... audit only.</p>
<p>
1a <b>why we need machine learning</b>:
recog 3d objects in a <b>cluttered scene</b>; novel viewpoint in novel lighting...</p>
<p>
fraudulent credit card; changing rule; fraud is a moving target;
people change their tricks;</p>
<p>
specify correct output for a given input...
program will contain millions of numbers... should work for new cases;</p>
<p>
ML approach: produces program that does the job...
retrain it, if data changes...</p>
<p>
produce task-specific program by employing lots of data...</p>
<p>
best things done by learning algos:
objects in real scenes; facial ids and expressions; spoken words</p>
<p>
recog anomalies:
unusual patterns of sensor readings at nuclear power plant
(want to recog abnormalities without having it blow up)</p>
<p>
future stock prices; movies preferences;</p>
<p>
MNIST hand-written digits... like fruit fly db.</p>
<p>
genetics of fruit flies; (breed fast) MNIST DB of hand-written digits;
MNIST is our standard task.</p>
<p>
not very confident: numbers will not overlay; templates will not work...</p>
<p>
<b>ImageNet TASK</b> 1000 difft object classes in 1.3 M high res training images
2010 best system got 47% error for 1st choice</p>
<p>
VERY COOL MISTAKES in these images...</p>
<p>
Speech Reco: extract a vector every 10 ms; whichphoneme is being spoken;
must place bets on alternatives; </p>
<p>
George Dahl and Abdel Mohamed... are doing better than prev systems.</p>
<p>
Phone rec on the TIMIT benchmark.</p>
<p>
Li Deng at MSR realized</p>
<p>
see Hinton IEEE Signal Proce 2012...</p>
<p>
<b>Lecture 1b real neurons</b>... source of inspiration</p>
<p>
one axon that branches...</p>
<p>
he uses axon hillock for initial segment
he uses 1011 for neurons and 104 for synapses = 1015 synapses.
or 1014 for 103 synapses.</p>
<p>
synapse strengths: # of vesicles and # of receptors.</p>
<p>
MNIST each image has 784 pixels;
connect to 1000 neurons connect to 10 output targets.</p>
<p>
baby ferrets: reroute vision to auditory cortex... will learn to deal with vision in audit cortex;
flexibility.. can be repurposed. like an FPGA.
standard parallel hardware and then it adapts thru experience.</p>
<p>
lecture 1c: simple models of neurons:</p>
<p>
models of idealized neurons: are atoms... (remove the non-essentials)</p>
<p>
Linear neurons:</p>
<p>
y = (b = bias) + Sigma xiwi... different from below: linear if above threshold;</p>
<p>
<b>binary THRESHOLD neuron.</b>..</p>
<p>
mcCulloch-Pitts 1943 ... if sum> threshold... then send out a spike...
(logic was the main inspiration)</p>
<p>
(two difft ways to write it: theta = -b....
ie z must be > theta to produce an output spike)</p>
<p>
if z > theta (threshold) then 1 else 0.</p>
<p>
rectified linear neuron... </p>
<p>
sloped response if z > 0</p>
<p>
sigmoid neuron... use logistic function...</p>
<p>
y = 1/ 1 + e-z smooth derivatives....</p>
<p>
stochastic binary neurons... same as sigmoid but returns a 1 or 0
by rolling die with probabiliyt p.</p>
<p>
lecture 1d <b> simple example of learning:</b></p>
<p>
he shows a template matching algo... too rigid (from MNIST task... 5 by 5 digits);
(this is an instructive example; it WORKS but its rigid and limited);</p>
<p>
the winner is the template that has the biggest overlap with the ink.</p>
<p>
Lecture 1e 3 types of learning:</p>
<p>
<b>supervised learning: predict an output given an input vector.</b>..</p>
<p>
RL: select an action to maximize payoff. (rewards)</p>
<p>
reinforcement learning not covered in this course...</p>
<p>
unsupervised: discover a good internal repres of the input.</p>
<p>
Regression: predict real number output
classification: example: label 1 and 0.</p>
<p>
supervised learning y = f (x; W)</p>
<p>
x is an input vector; W a set of weights (or parameters)</p>
<p>
learning usu means adjusting the params to reduce error.</p>
<p>
minimize the squared diff.</p>
<p>
RL: maximize expected sum of future rewards;
usu discount factor; rewards are delayed;
scalar reward (not much info); will not cover RL in this course.</p>
<p>
Unsupervised learning: ignored by ML community;
clustering was the focusing...</p>
<p>
learn disparity btwn two images (not based on stubbing your toe 1000 times)</p>
<p>
Hi d inputs typicaly live near low d manifolds.</p>
<p>
PCA is one limted form of this... assume a plane in high d space.</p>
<p>
it is economical if we can reduce down to 2 bits.</p>
<p>
or big vector in which more components are zerol</p>
<p>
or v sparse code in which only feature is non-zero.</p>
<p>
</p>
<p>
Lecture 2a: feedfoward is most common;
but recurrent can remember... but more complex.</p>
<p>
feedforward vs recurrent (difficult to train)
feed-forward neural nets:
if more than one hidden layer, we call them "deep" NNs. (if more than one layer of hidden)</p>
<p>
at each stage eg in speech reco, want same thing said by difft speakers to be more similar
and difft things said byone speak to become less similar...</p>
<p>
recurrent nets: complex dynamics;</p>
<p>
</p>
<p>
he shows recurrent netw modeling sequences; arrows going from hidden satte at t0 to t1.
Ilya Sutskever 2011 used rec NN to predict new text in Wiki !!! lecture 2: part a architec.</p>
<p>
used 86 chars including punctuation...</p>
<p>
pick next char... one char at a time... most likely character.</p>
<p>
he shows example paragraph: amazingly correct.</p>
<p>
symmetr connected networks...easier to model...</p>
<p>
John Hopfield cannot model cycles....
*
Lecture 2b Perceptrons: early 1960s....
fell into disfavor cuz of Minsky and Papert:</p>
<p>
start Frank Rosenblatt...<b> prin of neurodynamics</b> early 1960s
v big book;
1) convert raw input vector into a vector of hand-picked feature activs.
2) learn how to weight each feature to get a single scalar.
add all the weighted features.</p>
<p>
</p>
<p>
lots of grand claims.... false claim: tanks and trucks... tanks on sunny day vs
trucks on cloudy day (just used total intensity of image)</p>
<p>
1969: Minsky and Papert: Perceptrons: showed limitations; their critique;
genl impression was that they debunked NNs... but not true... wildly generalized
that claim. </p>
<p>
google uses percetprons...</p>
<p>
binary threhold neurons. McCulloch and Pitts neurons:
0 or 1 if S xiwi > theta or b (bias);
to learn bias... just add an extra neuron.</p>
<p>
PERCEPTRONS: LECT 2C:
WEIGHT SPACE: look at his diagram: figure for st space:
scalar prod of wt vector and input vector;</p>
<p>
convex learning: combo of 2 solns is itself a soln.</p>
<p>
Lect 2e what preceptronss can't do:
if u make a huge # of features for eevery discrim then...</p>
<p>
cannot learn: exclusive OR: 1,1 => 1 and 0,0 => 1 else 0.
not linearly separable... he shows figure in data space;</p>
<p>
cannot learn if transformastionf form a group;
translat with wrap-around form a group.</p>
<p>
NN must learn the feature detectors themselves.</p>
<p>
we need multiple layers of adaptive, non -linear hidden usints.</p>
<p>
*
lecture 3a learning the weights of a linear neuron.... like a perceptron.</p>
<p>
reduced sq error of output... Sum wixi these are linear filter neurons.</p>
<p>
*
Oct 8 2014: perceptrons always get better to a good set of weights;</p>
<p>
multilayer NNs;
outputs get closer to the desired output.</p>
<p>
linear neuron with sq error...: Sum of x dot x. = y.</p>
<p>
want to use an iterative method;</p>
<p>
only order ish, chips and ketchup several portions each; cashier "total price of meal"</p>
<p>
w = (w fish,,w chips,w ketchup)</p>
<p>
150 for fish
chips for 50
and 100 for a portion of ketch</p>
<p>
target value is 850; and we buy 2, 5, 3</p>
<p>
start with 50, 50, 50</p>
<p>
resid error is 350...</p>
<p>
delta wi = epsilon * xi (t-y)</p>
<p>
say eps = 1/35</p>
<p>
</p>
<p>
</p>
<p>
</p>
<p>
</p>
<p>
</p>
<p>
</p>
<p>
</p>
<p>
</p>
<p>
</p>
<p>
</p>
<p>
</p>
<p>
</p>
<p>
</p>
<p>
</p>
<p>
</p>
<p>
</p>
<p>
</p>
<p>
</p>
<p>
</p>
<p>
</p>
<p>
</p>