Part 03 - Gradient Descent
313K subscribers
SUBSCRIBED
Neural Networks Demystified @stephencwelch Supporting Code: https://github.com/stephencwelch/Neur...Link to Yann's Talk:http://videolectures.net/eml07_lecun_...In this short series, we will build and train a complete Artificial Neural Network in python. New videos every other friday. Part 1: Data + Architecture Part 2: Forward Propagation Part 3: Gradient Descent Part 4: Backpropagation Part 5: Numerical Gradient Checking Part 6: Training Part 7: Overfitting, Testing, and Regularization
00:01
Last time we built a neural network in python that made really bad predictions of your score on a test
00:06
Based on how many hours you slept and how many hours you studied the night before?
00:11
This time we'll focus on the theory of making those predictions better
00:15
we can initialize the network we built last time and pass in our normalized Data x
00:20
Using our forward method and have a look at our estimate of y y hat
00:25
Right now our predictions are pretty inaccurate to improve our model
00:29
we first need to quantify exactly how wrong our predictions are well do this with a cost function a
00:35
Cost function allows us to Express exactly how wrong or costly our model is given our examples
00:42
One way to compute an overall cost is to take each error value square it and add these values together
00:50
Multiplying by one half will make things simpler down the road
00:53
now that we have a cost our job is to minimize it and
00:57
someone says they're training a network what they really mean is that they're minimizing a cost function
01:02
Our cost is a function of two things
01:05
Our examples, and the weights on our synapses
01:07
We don't have much control over our data. So will minimize our cost by changing the weights
01:13
Conceptually this is a pretty simple concept, we have a collection of nine individual weights.
01:18
And we're saying that there is some combination of w's that will make our cost, j, as small as possible.
01:24
When I first saw this problem in Machine learning
01:26
I thought I'll just try all the weights until I find the best one after all I have a computer
01:33
enter the curse of dimensionality
01:35
Here's the problem let's pretend for a second that we only have one weight instead of nine
01:40
To find the ideal value for our weight that will minimize our cost
01:44
We need to try a bunch of values for w let's say we test a thousand values
01:50
That doesn't seem so bad; After all, my computer is pretty fast
01:54
It takes about point zero four seconds to check a thousand different weight values for our neural Network
02:00
Since we've competed the cost for a wide range of values of w we can just pick the one with the smallest cost
02:06
Let that be our weight, and now we've trained our network
02:09
So you may be thinking that point zero four seconds the trainer network is not so bad
02:13
And we haven't even optimized anything yet. Plus, there are other way faster languages than python out there
02:20
Before we optimize though, let's consider the full complexity of the problem
02:25
Remember the point zero four seconds required is only for one weight, and we have nine total
02:31
Let's next consider two weights for a moment, to maintain the same precision
02:35
We now need to check 1,000 times a thousand or a million values. This is a lot of work even for a fast computer.
02:43
After our million evaluations we found our solution
02:46
But it took an agonizing 40 seconds the real curse of dimensionality kicks in as we continue to add dimensions
02:53
Searching through three weights would take a billion evaluations or 11 hours searching through all nine weights
02:59
We need for our simple Neural Network would take one quadrillion
03:03
268 Trillion 391 billion
03:06
679 million three hundred and fifty thousand five hundred and eighty three and a half years
03:11
For that reason that just try everything or brute Force optimization method is clearly not going to work
03:18
Let's return to the one-dimensional case and see if we can be more clever
03:22
Let's evaluate our cost function for a specific value of w if w is 1.1 for example
03:29
We can run our cost function and see that J is 2.8
03:33
We haven't learned much yet, but let's try to add a little more information to what we [already] know
03:39
What if we could figure out which way was downhill if we could we would know whether to make w smaller or larger to?
03:46
Decrease the cost
03:47
We could test the cost function immediately to the left and to the right of our test point and see which is smaller
03:53
This is called numerical estimation and is sometimes a good approach but for us there is a better way
03:59
Let's look at our equation so far
04:02
we have five equations, but we could really think of them as one big approach and
04:07
since we have one big equation that uniquely Determines our cost J from x y w1 and W2
04:13
We can [use] our good friend calculus to find exactly what looking for
04:18
We want to know which way is downhill that is what is the rate of change of J with respect to w?
04:24
Also known as the derivative and in this case since we're just considering one weight at a time. This is a partial derivative
04:32
We can derive an expression for DJ 2w
04:35
That will give us the rate of change of J with respect to w for any value of w if DJ
04:41
Dw is positive then the cost function is going uphill if Dj. Dw is negative and the cost function is going Downhill now
04:48
We can really speed things up since we [know] in which direction the cost decreases
04:52
We can save all the time that we would have spent searching in the wrong direction
04:57
We can save you even more computational time by iteratively taking steps downhill and stopping when the cost stopped getting smaller
05:05
This method is known as gradient descent and although it may not seem so impressive in one dimension
05:10
it is capable of incredible speed ups in higher dimensions in
05:14
Fact in our final video will show that what would have taken 10 to the 27th function evaluations with our brute Force method
05:21
Will take less than a hundred evaluations with gradient descent?
05:25
Gradient descent allows us to find needles in very very very large haystacks
05:31
Now before we celebrate too much here. There is a restriction
05:34
What if our cost function doesn't always go in the same direction? What if it goes up and then back down?
05:40
The mathematical name for this is non convex
05:43
And it could really throw off our gradient descent algorithm by getting it stuck in a local minimum instead of our ideal Global Minima
05:51
One of the reasons we chose our cost function to be the sum of squared errors was to exploit [the] convex Nature of quadratic equations
05:59
We know that the graph of y equals x squared is a nice convex parabola and it turns out that higher dimensional versions are to
06:07
another piece of the puzzle
06:08
here is that depending on how we use our data it might not matter if our function is convex or not if
06:14
We use our examples one at a time instead of all at once
06:18
Sometimes it won't matter our function is convex. We will still find a good solution
06:23
this is called stochastic gradient descent
06:26
So maybe we shouldn't be afraid of non convex loss functions as neural Network Wizard Iyanla Kuhn says in his excellent
06:32
Talk who is afraid of non convex Loss functions?
06:36
The details of gradient descent are a deep topic for another day for now
06:40
We're going to do our gradient descent batch style
06:43
Where we use all our examples at once and the way we've set up our cost function will keep things nice and convex
06:49
Next time we'll compute and code up our gradients
English
###