Part 05 - Numerical Gradient Checking
313K subscribers
SUBSCRIBED
When building complex systems like neural networks, checking portions of your work can save hours of headache. Here we'll check our gradient computations. Supporting code: https://github.com/stephencwelch/Neur...Link to excellent Stanford tutorial: http://ufldl.stanford.edu/wiki/index....In this series, we will build and train a complete Artificial Neural Network in python. New videos every other friday. Part 1: Data + Architecture Part 2: Forward Propagation Part 3: Gradient Descent Part 4: Backpropagation Part 5: Numerical Gradient Checking Part 6: Training Part 7: Overfitting, Testing, and Regularization @stephencwelch
00:00
Last time, we did a bunch of calculus to find the rate of change of our cost, J,
00:05
with respect to our parameters, W.
00:07
Although each calculus step was pretty straight forward, it’s still relatively easy to make mistakes.
00:12
What’s worse, is that our network doesn’t have a good way to tell us that it’s broken
00:17
– code with incorrectly implemented gradients may appear to be functioning just fine.
00:22
This is the most nefarious kind of error when building complex systems.
00:26
Big, in-your-face errors suck initially
00:29
but it’s clear that you must fix this error for your work to succeed.
00:33
More subtle errors can be more troublesome because they hide in your code and steal hours of your time,
00:39
slowly degrading performance, while you wonder what the problem is.
00:42
A good solution here is to test the gradient computation part of our code,
00:47
just as developer would unit test new portions of their code.
00:51
We’ll combine a simple understanding of the derivative with some mild cleverness to perform numerical gradient checking.
00:57
If our code passes this test,
00:59
we can be quite confident that we have computed and coded up our gradients correctly.
01:04
To get started, let’s quickly review derivatives.
01:06
Derivatives tell us the slope, or how steep a function is.
01:10
Once you’re familiar with calculus,
01:12
it’s easy to take for granted the inner workings of the derivative
01:16
we just accept that the derivative of x2 is 2x by the power rule.
01:20
However, depending on how mean your calculus teacher was,
01:24
you may have spent months not being taught the power rule, and instead required to compute derivatives using the definition.
01:31
Taking derivatives this way is a bit tedious,
01:34
but still important
01:35
it provides us a deeper understanding of what a derivative is,
01:38
and it’s going to help us solve our current problem.
01:41
The definition of the derivative is really a glorified slope formula.
01:46
The numerator gives us the change in y values,
01:49
while the denominator is convenient way to express the change in x values.
01:53
By including the limit, we are applying the slope formula across an infinitely small region
01:59
it’s like zooming in on our function, until it becomes linear.
02:02
The definition tells us to zoom in until our x distance is infinitely small,
02:07
but computers can’t really handle infinitely small numbers,
02:11
especially when they’re in the bottom parts of fractions
02:13
if we try to plug in something too small,
02:16
we will quickly lose precision.
02:18
The good news here is that if we plug in something reasonable small,
02:22
we can still get surprisingly good numerical estimates of the derivative.
02:26
We’ll modify our approach slightly by picking a point in the middle of the interval we would like to test,
02:32
and call the distance we move in each direction epsilon.
02:35
Let’s test our method with a simple function, x squared.
02:39
We’ll choose a reasonable small value for epsilon,
02:42
and compute the slope of x2 at a given point by finding the function value just above and just below our test point.
02:49
We can then compare our result to our symbolic derivative 2x,
02:53
at the test point
02:55
If the numbers match, we’re in business!
02:58
We can use the same approach to numerically evaluate the gradient of our neural network.
03:03
It’s a little more complicated this time, since we have 9 gradient values,
03:07
and we’re interested in the gradient of our cost function.
03:10
We’ll make things simpler by testing one gradient at a time.
03:14
We’ll “perturb” each weight
03:16
adding epsilon to the current value and computing the cost function,
03:19
subtracting epsilon from the current value and computing the cost function,
03:23
and then computing the slope between these two values.
03:27
We’ll repeat this process across all our weights,
03:31
and when we’re done we’ll have a numerical gradient vector,
03:34
with the same number of values as we have weights.
03:37
It’s this vector we would like to compare to our official gradient calculation.
03:42
We see that our vectors appear very similar,
03:45
which is a good sign, but we need to quantify just how similar they are.
03:49
A nice way to do this is to divide the norm of the difference
03:53
by the norm of the sum of the vectors we would like to compare.
03:56
Typical results should be on the order of 10-8 or less if you’ve computed your gradient correctly.
04:03
And that’s it, we can now check our computations and eliminate gradient errors before they become a problem.
04:09
Next time we’ll train our Neural Network.
English (United States)
###