Part 07 - Overfitting, Testing, and Regularization
313K subscribers
SUBSCRIBED
We've built and trained our neural network, but before we celebrate, we must be sure that our model is representative of the real world. Supporting Code: https://github.com/stephencwelch/Neur...Nate Silver's Book: http://www.amazon.com/Signal-Noise-Ma...Caltech Machine Learning Course: https://work.caltech.edu/telecourse.htmlAnd the lecture shown:
In this series, we will build and train a complete Artificial Neural Network in python. New videos every other friday. Part 1: Data + Architecture Part 2: Forward Propagation Part 3: Gradient Descent Part 4: Backpropagation Part 5: Numerical Gradient Checking Part 6: Training Part 7: Overfitting, Testing, and Regularization @stephencwelch welchlabs.com
00:00
last time we trained our neural network
00:02
and it made suspiciously good
00:03
predictions of your score on a test
00:05
based on how many hours you slept and
00:07
how many hours you've studied the night
00:09
before before we celebrate and begin
00:11
changing our sleep and study habits we
00:13
need some way to ensure that our model
00:15
reflects the real world to do this let's
00:18
first spend some time thinking about
00:19
data like a lot of data our input and
00:22
output values come from real world
00:24
observations the assumption here is that
00:27
there is some underlying process and our
00:29
observations give us insight into that
00:31
process but our observations are not the
00:34
same thing as the process they are just
00:36
a sample our observation says that when
00:40
we sleep for three hours and study for
00:42
five the grade we earned was a 75 but
00:45
does this mean that every time you sleep
00:46
for three hours and study for five you
00:48
will earn a 75 of course not because
00:50
there are many other variables that
00:52
matter here such as the difficulty of
00:54
the test or whether you've been paying
00:56
attention in lectures we could quantify
00:58
these variables to build a better model
01:01
but even if we did there would still be
01:03
an element of uncertainty that we could
01:05
never explicitly model for example maybe
01:08
the test was multiple-choice and you
01:10
guessed on a few problems one way to
01:12
think about this is that observations
01:15
are composed of signal and noise Nate
01:17
silver the guy who correctly predicted
01:19
the u.s. election results for 50 out of
01:21
50 US states in 2012 wrote a great book
01:25
on exactly this the idea is that were
01:28
interested in an underlying process the
01:30
signal but in real data our signal will
01:32
always be obscured by some level of
01:35
noise an interesting example of this
01:38
shows up when comparing the SAT scores
01:40
of students who take the SAT both junior
01:42
and senior year right on the College
01:45
Board's website it says the higher of
01:48
students scores as a junior the more
01:50
likely that students subsequent scores
01:52
will drop why would this be it seems
01:55
like students who did well junior year
01:57
would also do well senior year we can
01:59
make sense of this by considering that
02:01
SAT scores are composed of a signal and
02:03
a noise component
02:05
the signal being the underlying aptitude
02:07
of a student and the noise being other
02:09
factors that affect test scores
02:10
basically if the student had a good day
02:12
or not of the students who did well the
02:15
first time we expect a disproportionate
02:17
number to have had a good day and since
02:20
having a good day is random when the
02:22
subset of students have a regular or bad
02:24
test day on their next test their scores
02:26
on average will go down so if we can
02:30
convince our model to fit the signal and
02:32
not the noise we should be able to avoid
02:34
overfitting first we'll work on
02:37
diagnosing overfitting and then we'll
02:39
work on fix again last time we showed
02:41
our model predictions across the input
02:43
space for various combinations of our
02:45
sleeping in our studying will change our
02:48
data a bit to make overfitting more
02:50
obvious and retrain our model on the new
02:52
data set if we re-examine our
02:55
predictions across our sample space we
02:57
see some strange behavior neural
02:59
networks are really powerful models and
03:02
we see here that all that power has been
03:04
used to fit our data really closely
03:06
which creates a problem our model no
03:08
longer is reflective of the real world
03:10
according to our model in some cases
03:13
studying more will actually push your
03:14
score down which seems unlikely
03:16
hopefully studying more will not
03:18
decrease your score so it appears our
03:21
model is overfitting but how do we know
03:23
for sure a widely accepted method is to
03:26
split our data into two portions
03:28
training and testing we won't touch our
03:30
testing data while training the model
03:32
and only use it to see how we're doing
03:34
our testing data is a simulation of the
03:37
real world we can plot the error on our
03:39
training and testing stats as we train
03:41
our model and identify the exact point
03:44
at which overfitting begins we can also
03:46
plot testing and training error as a
03:48
function of model complexity and see
03:50
similar babe so now we know overfitting
03:53
is a problem but how do we fix it one
03:55
way is to throw more data at the problem
03:57
a simple rule of thumb as presented by
03:59
Yasser Abu Mustafa and his excellent
04:02
machine learning course available from
04:03
Caltech is that you should have at least
04:05
ten times as many examples
04:07
as the degrees of freedom in your model
04:09
for us since we have nine weights that
04:12
can change we would need 90 observations
04:15
which we certainly don't have another
04:18
popular and effective way to mitigate
04:19
overfitting is to use a technique called
04:22
regularization one way to implement
04:24
regularization is to add a term to our
04:26
cost function that penalizes overly
04:29
complex models a simple but effective
04:32
way to do this is to add together the
04:34
square of the weight to our cost
04:36
function this way models with larger
04:38
magnitudes of weights cost more we'll
04:41
need to normalize the other part of our
04:43
cost function to ensure that our ratio
04:45
of the two error terms does not change
04:47
with respect to the number of examples
04:50
will introduce a regularization hyper
04:53
parameter lambda that allows us to tune
04:55
the relative cost higher values of
04:58
lambda will impose bigger penalties for
05:00
high model complexity if we train our
05:03
model now we see the fit is still good
05:05
but our model is no longer interested in
05:08
exactly fitting our data further our
05:10
training and testing errors are much
05:12
closer and we've successfully reduced
05:14
overfitting on this data set to further
05:17
reduce overfitting we could increase
05:18
lambda and that's it we've trained and
05:22
evaluated an artificial neural network
05:24
to predict your score on a test based on
05:26
how many hours you slept and studied the
05:28
night before
05:29
I've really enjoyed making these videos
05:32
and I wanted to say a big thank you to
05:34
everyone who's watched and commented I
05:36
want to point out that supporting
05:38
ipython notebooks are linked below and
05:40
available at welsh labs comm /blog to be
05:44
kept up to date on future videos
05:45
you can subscribe below and follow me on
05:47
twitter thanks
English (auto-generated)
###