Eigenvectors and eigenvalues
<p>00:19
"Eigenvectors and eigenvalues" is one of those topics that a lot of students find particularly unintuitive.
00:26
Questions like "why are we doing this" and "what does this actually mean"
00:29
are too often left just floating away in an unanswered sea of computations.
00:33
And as I put out the videos of the series,
00:35
a lot of you have commented about looking forward to visualizing this topic in particular.
00:40
I suspect that
00:41
the reason for this is not so much that eigen-things are particularly complicated or poorly explained.
00:46
In fact, it's comparatively straightforward
00:48
and I think most books do a fine job explaining it.
00:51
The issue is that
00:52
it only really make sense if you have a solid visual understanding for many of the topics that precede it.
00:59
Most important here is that you know how to think about matrices as linear transformations,
01:04
but you also need to be comfortable with things
01:06
like determinants, linear systems of equations and change of basis.
01:10
Confusion about eigen stuffs usually has more to do with a shaky foundation in one of these topics
01:16
than it does with eigenvectors and eigenvalues themselves.
01:20
To start, consider some linear transformation in two dimensions,
01:23
like the one shown here.
01:25
It moves the basis vector i-hat to the coordinates (3, 0) and j-hat to (1, 2),
01:31
so it's represented with a matrix, whose columns are (3, 0) and (1, 2).
01:36
Focus in on what it does to one particular vector
01:39
and think about the span of that vector, the line passing through its origin and its tip.
01:45
Most vectors are going to get knocked off their span during the transformation.
01:48
I mean, it would seem pretty coincidental
01:51
if the place where the vector landed also happens to be somewhere on that line.
01:57
But some special vectors do remain on their own span,
02:00
meaning the effect that the matrix has on such a vector is just to stretch it or squish it, like a scalar.
02:09
For this specific example, the basis vector i-hat is one such special vector.
02:15
The span of i-hat is the x-axis,
02:17
and from the first column of the matrix,
02:19
we can see that i-hat moves over to 3 times itself, still on that x axis.
02:26
What's more, because of the way linear transformations work,
02:29
any other vector on the x-axis is also just stretched by a factor of 3
02:34
and hence remains on its own span.
02:38
A slightly sneakier vector that remains on its own span during this transformation is (-1, 1),
02:44
it ends up getting stretched by a factor of 2.
02:49
And again, linearity is going to imply that
02:51
any other vector on the diagonal line spanned by this guy
02:55
is just going to get stretched out by a factor of 2.
02:59
And for this transformation,
03:01
those are all the vectors with this special property of staying on their span.
03:05
Those on the x-axis getting stretched out by a factor of 3
03:08
and those on this diagonal line getting stretched by a factor of 2.
03:12
Any other vector is going to get rotated somewhat during the transformation,
03:16
knocked off the line that it spans.
03:22
As you might have guessed by now,
03:24
these special vectors are called the "eigenvectors" of the transformation,
03:28
and each eigenvector has associated with it, what's called an "eigenvalue",
03:33
which is just the factor by which it stretched or squashed during the transformation.
03:40
Of course, there's nothing special about stretching vs. squishing
03:43
or the fact that these eigenvalues happen to be positive.
03:46
In another example, you could have an eigenvector with eigenvalue -1/2,
03:51
meaning that the vector gets flipped and squished by a factor of 1/2.
03:57
But the important part here is that it stays on the line that it spans out without getting rotated off of it.
04:04
For a glimpse of why this might be a useful thing to think about,
04:07
consider some three-dimensional rotation.
04:11
If you can find an eigenvector for that rotation,
04:15
a vector that remains on its own span,
04:17
what you have found is the axis of rotation.
04:22
And it's much easier to think about a 3-D rotation in terms of some axis of rotation and an angle by which is rotating,
04:30
rather than thinking about the full 3-by-3 matrix associated with that transformation.
04:37
In this case, by the way, the corresponding eigenvalue would have to be 1,
04:41
since rotations never stretch or squish anything,
04:43
so the length of the vector would remain the same.
04:48
This pattern shows up a lot in linear algebra.
04:50
With any linear transformation described by a matrix, you could understand what it's doing
04:55
by reading off the columns of this matrix as the landing spots for basis vectors.
05:00
But often a better way to get at the heart of what the linear transformation actually does,
05:05
less dependent on your particular coordinate system,
05:08
is to find the eigenvectors and eigenvalues.
05:15
I won't cover the full details on methods for computing eigenvectors and eigenvalues here,
05:20
but I'll try to give an overview of the computational ideas
05:23
that are most important for a conceptual understanding.
05:27
Symbolically, here's what the idea of an eigenvector looks like.
05:30
A is the matrix representing some transformation,
05:34
with v as the eigenvector,
05:36
and λ is a number, namely the corresponding eigenvalue.
05:40
What this expression is saying is that the matrix-vector product - A times v
05:45
gives the same result as just scaling the eigenvector v by some value λ.
05:51
So finding the eigenvectors and their eigenvalues of a matrix A
05:55
comes down to finding the values of v and λ that make this expression true.
06:02
It's a little awkward to work with at first,
06:03
because that left hand side represents matrix-vector multiplication,
06:07
but the right hand side here is scalar-vector multiplication.
06:11
So let's start by rewriting that right hand side as some kind of matrix-vector multiplication,
06:16
using a matrix, which has the effect of scaling any vector by a factor of λ.
06:21
The columns of such a matrix will represent what happens to each basis vector,
06:25
and each basis vector is simply times λ,
06:29
so this matrix will have the number λ down the diagonal with 0's everywhere else.
06:36
The common way to write this guy is to factor that λ out and write it as λ times I,
06:41
where I is the identity matrix with 1's down the diagonal.
06:45
With both sides looking like matrix-vector multiplication,
06:48
we can subtract off that right hand side and factor out the v.
06:54
So what we now have is a new matrix - A minus λ times the identity,
06:58
and we're looking for a vector v, such that this new matrix times v gives the zero vector.
07:06
Now this will always be true if v itself is the zero vector,
07:10
but that's boring.
07:11
What we want is a non-zero eigenvector.
07:14
And if you watched Chapters 5 and 6,
07:16
you'll know that the only way it's possible for the product of a matrix with a non-zero vector to become zero
07:22
is if the transformation associated with that matrix squishes space into a lower dimension.
07:29
And that squishification corresponds to a zero determinant for the matrix.
07:35
To be concrete, let's say your matrix a has columns (2, 1) and (2, 3),
07:40
and think about subtracting off a variable amount λ from each diagonal entry.
07:46
Now imagine tweaking λ, turning a knob to change its value.
07:51
As that value of λ changes,
07:53
the matrix itself changes, and so the determinant of the matrix changes.
07:58
The goal here is to find a value of λ that will make this determinant zero,
08:03
meaning the tweaked transformation squishes space into a lower dimension.
08:08
In this case, the sweet spot comes when λ equals 1.
08:12
Of course, if we have chosen some other matrix,
08:14
the eigenvalue might not necessarily be 1, the sweet spot might be hit some other value of λ.
08:20
So this is kind of a lot, but let's unravel what this is saying.
08:23
When λ equals 1, the matrix A minus λ times the identity squishes space onto a line.
08:30
That means there's a non-zero vector v,
08:33
such that A minus λ times the identity times v equals the zero vector.
08:40
And remember, the reason we care about that is because it means A times v equals λ times v,
08:49
which you can read off as saying that the vector v is an eigenvector of A,
08:53
staying on its own span during the transformation A.
08:58
In this example, the corresponding eigenvalue is 1, so v would actually just a fixed in place.
09:06
Pause and ponder if you need to make sure that line of reasoning feels good.
09:13
This is the kind of thing I mentioned in the introduction,
09:16
if you didn't have a solid grasp of determinants
09:18
and why they relate to linear systems of equations having non-zero solutions,
09:23
an expression like this would feel completely out of the blue.
09:28
To see this in action, let's revisit the example from the start
09:31
with the matrix whose columns are (3, 0) and (1, 2).
09:35
To find if a value λ is an eigenvalue,
09:38
subtracted from the diagonals of this matrix and compute the determinant.
09:51
Doing this, we get a certain quadratic polynomial in λ, (3-λ)(2-λ).
09:57
Since λ can only be an eigenvalue if this determinant happens to be zero,
10:02
you can conclude that the only possible eigenvalues are λ equals 2 and λ equals 3.
10:10
To figure out what the eigenvectors are that actually have one of these eigenvalues, say λ equals 2,
10:15
plug in that value of λ to the matrix
10:19
and then solve for which vectors this diagonally altered matrix sends to 0.
10:25
If you computed this the way you would any other linear system,
10:28
you'd see that the solutions are all the vectors on the diagonal line spanned by (-1, 1).
10:35
This corresponds to the fact that the unaltered matrix [(3, 0), (1, 2)]
10:39
has the effect of stretching all those vectors by a factor of 2.
10:46
Now, a 2-D transformation doesn't have to have eigenvectors.
10:50
For example, consider a rotation by 90 degrees.
10:53
This doesn't have any eigenvectors, since it rotates every vector off of its own span.
11:01
If you actually try computing the eigenvalues of a rotation like this, notice what happens.
11:06
Its matrix has columns (0, 1) and (-1, 0),
11:11
subtract off λ from the diagonal elements and look for when the determinant is 0.
11:18
In this case, you get the polynomial λ2+1,
11:22
the only roots of that polynomial are the imaginary numbers i and -i.
11:28
The fact that there are no real number solutions indicates that there are no eigenvectors.
11:35
Another pretty interesting example worth holding in the back of your mind is a shear.
11:40
This fixes i-hat in place and moves j-hat one over,
11:44
so its matrix has columns (1, 0) and (1, 1).
11:48
All of the vectors on the x-axis are eigenvectors with eigenvalue 1, since they remain fixed in place.
11:55
In fact, these are the only eigenvectors.
11:58
When you subtract off λ from the diagonals and compute the determinant,
12:03
what you get is (1-λ)2,
12:09
and the only root of this expression is λ equals 1.
12:14
This lines up with what we see geometrically that all of the eigenvectors have eigenvalue 1.
12:21
Keep in mind though,
12:21
it's also possible to have just one eigenvalue, but with more than just a line full of eigenvectors.
12:29
A simple example is a matrix that scales everything by 2,
12:33
the only eigenvalue is 2, but every vector in the plane gets to be an eigenvector with that eigenvalue.
12:42
Now is another good time to pause and ponder some of this
12:44
before I move on to the last topic.
13:03
I want to finish off here with the idea of an eigenbasis,
13:07
which relies heavily on ideas from the last video.
13:11
Take a look at what happens if our basis vectors just so happened to be eigenvectors.
13:17
For example, maybe i-hat is scaled by -1 and j-hat is scaled by 2.
13:23
Writing their new coordinates as the columns of a matrix,
13:26
notice that those scalar multiples -1 and 2, which are the eigenvalues of i-hat and j-hat,
13:32
sit on the diagonal of our matrix and every other entry is a 0.
13:38
Anytime a matrix has 0's everywhere other than the diagonal,
13:42
it's called, reasonably enough, a diagonal matrix.
13:45
And the way to interpret this is that all the basis vectors are eigenvectors,
13:50
with the diagonal entries of this matrix being their eigenvalues.
13:57
There are a lot of things that make diagonal matrices much nicer to work with.
14:01
One big one is that
14:03
it's easier to compute what will happen if you multiply this matrix by itself a whole bunch of times.
14:09
Since all one of these matrices does is scale each basis vector by some eigenvalue,
14:14
applying that matrix many times, say 100 times,
14:18
is just going to correspond to scaling each basis vector by the 100-th power of the corresponding eigenvalue.
14:25
In contrast, try computing the 100-th power of a non-diagonal matrix.
14:30
Really, try it for a moment, it's a nightmare.
14:36
Of course, you will rarely be so lucky as to have your basis vectors also be eigenvectors,
14:42
but if your transformation has a lot of eigenvectors, like the one from the start of this video,
14:46
enough so that you can choose a set that spans the full space,
14:51
then you could change your coordinate system so that these eigenvectors are your basis vectors.
14:57
I talked about change of basis last video,
14:59
but I'll go through a super quick reminder here
15:01
of how to express a transformation currently written in our coordinate system into a different system.
15:08
Take the coordinates of the vectors that you want to use as a new basis,
15:11
which, in this case, means are two eigenvectors,
15:14
that make those coordinates the columns of a matrix, known as the change of basis matrix.
15:20
When you sandwich the original transformation
15:22
putting the change of basis matrix on it's right
15:25
and the inverse of the change of basis matrix on its left,
15:28
the result will be a matrix representing that same transformation,
15:32
but from the perspective of the new basis vectors coordinate system.
15:37
The whole point of doing this with eigenvectors is that
15:40
this new matrix is guaranteed to be diagonal with its corresponding eigenvalues down that diagonal.
15:47
This is because it represents working in a coordinate system
15:50
where what happens to the basis vectors is that they get scaled during the transformation.
15:55
A set of basis vectors, which are also eigenvectors,
15:58
is called, again, reasonably enough, an "eigenbasis".
16:02
So if, for example, you needed to compute the 100-th power of this matrix,
16:07
it would be much easier to change to an eigenbasis,
16:10
compute the 100-th power in that system,
16:13
then convert back to our standard system.
16:16
You can't do this with all transformations.
16:18
A shear, for example, doesn't have enough eigenvectors to span the full space.
16:23
But if you can find an eigenbasis, it makes matrix operations really lovely.
16:29
For those of you willing to work through a pretty neat puzzle to see what this looks like in action
16:32
and how it can be used to produce some surprising results, I'll leave up a prompt here on the screen.
16:37
It takes a bit of work, but I think you'll enjoy it.
16:40
The next and final video of this series is going to be on abstract vector spaces.
16:45
See you then!</p>