Part 02 - Forward Propagation
313K subscribers
SUBSCRIBED
Neural Networks Demystified @stephencwelch Supporting Code: https://github.com/stephencwelch/Neur...In this short series, we will build and train a complete Artificial Neural Network in python. New videos every other friday. Part 1: Data + Architecture Part 2: Forward Propagation Part 3: Gradient Descent Part 4: Backpropagation Part 5: Numerical Gradient Checking Part 6: Training Part 7: Overfitting, Testing, and Regularization
00:00
Last time we set up our neural network on paper. This time we'll implement it
00:04
the programming language Python.
00:05
We'll build our network as a Python class. In our init method we'll take care of
00:10
instantiating important constants and variables.
00:12
We'll make these values accessible to the whole class by placing a "self."
00:17
in front each variable name. Our network has two inputs,
00:21
three hidden units, and one output. These are examples of hyperparameters.
00:25
Hyperparameters are constants that establish the structure and behaviour
00:29
of our network. But are not
00:31
updated as we train the network. Our learning algorithm is not capable of
00:35
for example
00:36
deciding that it needs another hidden unit.
00:38
This is something that we must decide on before training.
00:41
What neural networks do learn on the parameters, specifically in the weight
00:45
on the synapses. We'll take care of moving our data through our network and the method called
00:51
forward. Rather than pass inputs through the network one at a time
00:54
we're going to use matrices to pass through multiple inputs at once.
00:58
Doing this allows for big computational speed ups, especially when using tools
01:02
like MATLAB or NumPy.
01:04
Our input data matrix X is a dimension 3 by 2
01:08
because we have three two-dimensional examples. Our corresponding output data
01:12
Y is a dimension 3 by 1.
01:17
Each input value or element in matrix X needs to be multiplied by a
01:20
corresponding weight
01:22
and then added together with all the other results for each neuron.
01:26
This is a complicated operation, but if we take the three outputs we're looking for
01:30
as a single row of the matrix
01:32
and place all our individual weights into a matrix of weights
01:35
we can create the exact behavior we need by multiplying our
01:39
input data matrix by our weight matrix. Using matrix multiplication allows us to pass
01:44
multiple inputs through at once
01:46
by simply adding rows to the matrix X. From here on out will refer to these matrices
01:52
X, W1, and Z2
01:54
where Z2 is the activity of our second layer.
01:58
Notice that each entry in Z is a sum of weighted inputs to each neuron.
02:02
Z is a size of three by three: one row for each example
02:06
and one column for each hidden unit. We now have our first official
02:10
formula: Z2 = X * W1.
02:14
Matrix notation is really nice here because it allows us to express the
02:17
complex underlying process
02:19
in a single line. Now that we have the activities of our second layer, Z2,
02:24
we need to apply the activation function. We'll independently apply the
02:29
function to each
02:29
entry in matrix Z using a Python method for this called sigmoid
02:33
because we using a sigmoid as our activation function. Using NumPy is
02:37
really nice here
02:38
because we can pass in a scalar, vector or matrix. NumPy will apply the
02:42
activation function
02:43
element-wise and returns a result and the same dimensions as it was given.
02:48
We now have our second formula for forward propagation
02:52
Using f to denote our activation function,
02:54
We can write that a2 (our second layer activity)
02:57
is equal to f(Z2).
02:59
a2 will be a matrix the same size as Z2.
03:03
in this case 3x3
03:06
To finish forward propagation
03:07
we need to propagate a2 all the way to our output y-hat.
03:11
We've already done the heavy lifting in our previous layer
03:13
So all we have to do now is to multiply a2
03:15
by our second layer weights W2
03:18
and apply one more activation function.
03:21
W2 is of size 3x1
03:23
one weight for each synapse
03:25
multiplying a2 (a 3x3)
03:27
by W2, a 3x1
03:29
results in a 3x1 matrix Z3
03:32
(the activity of our third layer).
03:34
Z3 has 3 activity values,
03:36
one for each example.
03:38
Last but not least we'll apply our activation function to Z3
03:42
yeilding our official estimate of test scores:
03:44
ŷ
03:46
We need to implement our forward propagation formulas in Python.
03:49
First we'll initialize our weight matrices within our _init_ method.
03:53
For starting values, we'll use random numbers.
03:56
We'll implement forward propagation in our Forward method
03:58
using numpy built-in dot method for matrix multiplication and our own sigmoid method.
04:04
And there you have it.
04:05
A Python class capable of estimating your test score
04:08
given how many hours you sleep and how many hours you study.
04:11
We can pass in our own input data and get real output,
04:14
now you may be noticing that our estimates are quite terrible.
04:18
That's because we have not yet trained our network
04:20
and that's what we're going to work on next time.
English
###