Part 04 - Backpropagation
313K subscribers
SUBSCRIBED
Backpropagation as simple as possible, but no simpler. Perhaps the most misunderstood part of neural networks, Backpropagation of errors is the key step that allows ANNs to learn. In this video, I give the derivation and thought processes behind backpropagation using high school level calculus. Supporting Code and Equations: https://github.com/stephencwelch/Neur...In this series, we will build and train a complete Artificial Neural Network in python. New videos every other friday. Part 1: Data + Architecture Part 2: Forward Propagation Part 3: Gradient Descent Part 4: Backpropagation Part 5: Numerical Gradient Checking Part 6: Training Part 7: Overfitting, Testing, and Regularization @stephencwelch
SHOW LESS
00:00
Last time we decided to use gradient descent to train our Neural Network
00:04
so it could make better predictions of your score on a test
00:06
based on how many hours you slept and how many hours you studied the night before.
00:11
To perform gradient descent we need an equation and some code for our gradient ∂J/∂W.
00:18
Our weights W are spread across two matrices.
00:21
W(1) and W(2)
00:23
We'll separate our ∂J/∂W computation in the same way, by computing ∂J/∂W(1) and ∂J/∂W(2)
00:32
independently.
00:33
We should have just as many gradient values as weight values, so when we're done our matrices
00:39
∂J/∂W(1) and ∂J/∂W(2) will be the same size as W(1) and W(2)
00:47
Let's work on ∂J/∂W(2) first
00:50
The sum in our cost function adds the error from each example to create an overall cost.
00:56
We'll take advantage of the sum rule in differentiation which says that the derivative of the sums equals the sum of the derivatives.
01:04
We can move our sigma (Σ) outside and just worry about the derivative of the inside expression first.
01:11
To keep things simple, we'll temporarily forget about our summation.
01:15
Once we've computed ∂J/∂W for a single example,
01:18
we'll go back and add all our individual derivative terms together
01:23
We can now evaluate our first derivative. The power rule tells us to bring down our exponent 2 and multiply.
01:30
To finish our derivative we need to apply the chain rule.
01:33
The chain rule tells us how to take the derivative of a function inside of a function
01:38
and generally says that we take the derivative of the outside function and multiply it by the derivative of the inside function
01:45
One way to express the chain rule is as the product of derivatives this will come in very handy as we progress through backpropagation
01:53
In fact a better name for backpropagation might be: Don't stop doing the chain rule, ever.
02:00
We've taken the derivative of the outside of our cost function. Now we need to multiply it by the derivative of the inside.
02:07
y is just our test scores which won't change so the derivative of y, a constant, with respect to W is zero
02:15
y-hat (ŷ), on the other hand, does change with respect to W(2)
02:19
So we'll apply the chain rule and multiply our results by minus ∂ŷ/∂W(2)
02:25
We now need to think about the derivative of ŷ with respect to W(2)
02:30
Equation (4) tells us that ŷ is our activation function of z(3) so it will be helpful to apply the chain rule again
02:38
to break ∂ŷ/∂W(2) into
02:41
∂ŷ/∂z(3) times ∂z(3)/∂W(2)
02:45
to find the rate of change of ŷ with respect to z(3) we need to differentiate our sigmoid activation function
02:52
with respect to z
02:54
Now is a good time to add a new Python method for our derivative of our sigmoid function, sigmoidPrime
03:00
Our derivative should be largest where sigmoid function is the steepest, at the value z=0
03:07
we can now replace ∂ŷ/∂z(3) with f-prime of z(3)
03:13
Our final piece of the puzzle is ∂z(3)/∂W(2)
03:17
This term represents the change of z, our third layer activity, with respect to the weights in the second layer.
03:24
z(3) is the matrix product of our activities a(2) and our weights W(2).
03:29
The activities from layer 2 are multiplied by their corresponding weights and added together to yield z(3)
03:36
If we focus on a single synapse for a moment, we see a simple linear relationship between W and z where a is the slope.
03:44
So for each synapse, ∂z/∂W(2) is just the activation a, on that synapse
03:50
Another way to think about what the calculus is doing here is that it is backpropagating the error to each weight.
03:57
By multiplying by the activity on each synapse, the weights that contribute more to the overall error will have larger activations,
04:05
yield larger ∂J/∂W(2) values,
04:07
and will be changed more when we perform gradient descent.
04:11
We need to be careful with our dimensionality here, and if we're clever, we can take care of that summation we got rid of earlier.
04:18
The first part of our equation, y-ŷ, is of the same dimension of their output data, 3x1.
04:25
f-prime of z 3 is of the same size and our first operation is a scalar multiplication.
04:31
Our resulting 3x1 matrix is referred to as the backpropagating error, δ(3)
04:37
We determined that ∂z(3)/∂W(2) is equal to the activity of each synapse.
04:42
Each value in δ(3) needs to be multiplied by each activity.
04:47
We can achieve this by transposing a(2) and matrix multiplying by δ(3)
04:52
What's cool here Is that the matrix multiplication also takes care of our earlier omission.
04:57
It adds up the ∂J/∂W terms across all our examples
05:02
Another way to think about what's happening here is that each example our algorithm sees has a certain cost and a certain gradient.
05:10
The gradient with respect to each example pulls our gradient descent algorithm in a certain direction.
05:15
It's like every example gets a vote on which way is downhill and when we perform batch gradient descent,
05:22
we just add together everyone's vote, call it downhill, and move in that direction.
05:27
We'll code up our gradients in Python, in a new method, costFunctionPrime
05:32
Numpy's .multiply() method performs element-wise multiplication and the .dot() method performs matrix multiplication
05:40
We now have one final term to compute, ∂J/∂W(1)
05:44
The derivation begins the same way as before, by computing the derivative through our final layer,
05:50
first ∂J/∂ŷ, then ∂ŷ/∂z(3).
05:54
We now take the derivative across our synapses,
05:56
which is a little different from our job last time, which was computing the derivative with respect to the weights on our synapses.
06:03
There's still a nice linear relationship along each synapse,
06:06
but now we're interested in the rate of change of z(3) with respect to a(2)
06:11
Now the slope is just equal to the weight value for that synapse.
06:15
We can achieve this mathematically by multiplying by W(2) transposed
06:20
Our next term to work on is ∂a(2)/∂z(2).
06:24
This step is just like the derivative across our layer 3 neurons, so we can just multiply by f-prime of z(2).
06:32
Our final computation here is ∂z(2)/∂W(1)
06:36
This is very similar to our ∂z(3)/∂W(2) computation. There is a simple linear
06:41
relationship on the synapses between z and W(1). In this case though, the slope is the input value, x.
06:48
We can use the same technique as last time and multiplied by x transposed,
06:53
effectively applying the derivative and adding our ∂J/∂W(1)s together
06:57
across all our examples. All that's left is to code this equation in Python.
07:02
What's cool here is that if we want to make a deeper neural network, we could just stack a bunch of these operations together.
07:09
So how should we change our Ws to decrease our cost?
07:13
We can now compute ∂J/∂W, which tells us which way is uphill in our 9-dimensional optimization space.
07:21
If we move this way by adding a scalar times our derivative to all of our weights,
07:26
our cost will increase. And if we do the opposite,
07:29
subtract our gradient from our weights, we will move downhill and reduce our cost.
07:34
This simple step downhill is the core of gradient descent
07:38
and a key part of how even very sophisticated learning algorithms are trained.
07:43
Next time we'll perform numerical gradient checking to make sure our math is correct.
English
###