03 - Orthogonal Vectors
2.7M subscribers
SUBSCRIBED
MIT A 2020 Vision of Linear Algebra, Spring 2020 Instructor: Gilbert Strang View the complete course: https://ocw.mit.edu/2020-visionYouTube Playlist: https://www.youtube.com/playlist?list...Professor Strang describes in detail orthogonal vectors and matrices and subspaces. He explains Gram-Schmidt orthogonalization, as well as the Least Squares method for line fitting and non-square matrices. License: Creative Commons BY-NC-SA More information at https://ocw.mit.edu/termsMore courses at https://ocw.mit.edu
00:00
00:14
GILBERT STRANG: OK, ready for part three of this vision
00:21
of linear algebra.
00:23
So the key word in part three is orthogonal, which
00:26
again means perpendicular.
00:29
So we have perpendicular vectors.
00:32
We can imagine those.
00:34
We have something called orthogonal matrices.
00:37
That's when-- I've got one here.
00:41
An orthogonal matrix is when we have these columns.
00:46
I'm always going to use the letter
00:47
Q for an orthogonal matrix.
00:50
And I look at its columns, and every column
00:52
is perpendicular to every other column.
00:56
So I don't just have two perpendicular vectors
00:59
going like this.
01:00
I have n of them because I'm in n dimensions.
01:03
And you just imagine xyz axes or xyzw axes,
01:09
go up to 4D for relativity, go up to 8D
01:13
for string theory, 8 dimensions.
01:17
We just have vectors.
01:18
After all, it's just this row of numbers or a column of numbers.
01:23
And we can decide when things are perpendicular by that test.
01:28
Like say the test for Q1 to be perpendicular to Qn
01:34
is that row times that column.
01:38
When I say times, I mean dot product, multiply every pair.
01:43
Q1 transpose Qn gives that 0 up there.
01:47
So the columns are perpendicular.
01:50
And those matrices are the best to compute with.
01:52
And again, they're called Q.
01:55
And one way to, a quick matrix way,
01:58
because there's always a matrix way to explain something,
02:03
and you'll see how quick it is here.
02:06
This business of having columns that
02:09
are perpendicular to each other, and actually, I'm
02:11
going to make those lengths of all those column vectors all 1,
02:15
just to sort of normalize it.
02:18
Then all that's expressed by, if I multiply Q transpose by Q,
02:25
I'm taking all those dot products,
02:28
and I'm getting 1s when it's Q against itself.
02:33
And I'm getting 0s when it's one 1 Q versus another Q.
02:37
And again, just think of three perpendicular axes.
02:41
Those directions are the Q1, Q2, Q3.
02:45
OK?
02:46
So we really want to compute with those.
02:49
Here's an example.
02:51
Well, that has just two perpendicular axes.
02:54
I didn't have space for the third one.
02:56
So do you see that those two columns are perpendicular?
03:03
Again, what does that mean?
03:04
I take the dot product.
03:06
Minus 1 times 2, 2.
03:08
2 times minus 1, another minus 2.
03:10
So I'm up to minus 4 at this point.
03:12
And then 2 times 2 gives a plus 4.
03:14
So it all washes out to 0.
03:17
And why is that 1/3 there?
03:20
Why is that?
03:21
That's so that these vectors will have length 1.
03:25
There will be unit vectors.
03:27
Yeah, and how do I figure, the length of a vector,
03:30
just while we're at it?
03:33
I take 1 squared or minus 1 squared gives me 1.
03:37
2 squared and 2 squared, I take the dot product with itself.
03:42
So minus 1 squared, 2 squared, and 2
03:44
squared, that adds up to 9.
03:46
The square root of 9 is the length.
03:48
I'm just doing Pythagoras here.
03:51
There is one side of a triangle.
03:53
Here is a second side of a triangle.
03:56
It's a right triangle because that vector
03:58
is perpendicular to that one.
04:00
It's in 3D because they have three components.
04:04
And I didn't write a third direction.
04:07
And their length one vectors because just that's
04:14
how when I compute the length and remember
04:16
about the 1/3, which is put in there to give a length 1.
04:22
So OK.
04:23
So these matrices are with Q transpose times Q equal I.
04:28
That again, that's the matrix shorthand for all
04:31
I've just said.
04:33
And those matrices are the best because they don't
04:39
change the length of anything.
04:40
You don't have blow up.
04:42
You don't have going to 0.
04:43
You can multiply together 1,000 matrices,
04:46
and you'll still have another orthogonal matrix.
04:50
Yes, a little family of beautiful matrices.
04:53
OK, and very, very useful.
04:57
OK, and there was a good example.
04:59
Oh, I think the way I got that example,
05:01
I just added a third row.
05:04
The third column, sorry.
05:05
The third column.
05:06
So 2 squared plus 2 squared plus minus 1 squared.
05:09
That adds up to 9.
05:11
When I take the square root, I get 3.
05:13
So that has length 3.
05:15
I divided by 3.
05:17
So it would have length 1.
05:19
We always want to see 1s, like we do there.
05:23
And if I-- here's a simple fact.
05:26
But great.
05:28
Then if I have two of these matrices
05:30
or 50 of these matrices, I could multiply them together.
05:33
And I would still have length of 1.
05:35
I'd still have orthogonal matrices.
05:38
1 times 1 times 1 forever is 1.
05:41
OK, so there's probably something hiding here.
05:45
Oh, yeah.
05:46
Oh, yeah, to understand why these matrices are important,
05:51
this one, this line is telling me that, if I have a vector x,
05:55
and I multiply by Q, it doesn't change the length.
06:00
This is a symbol for length squared.
06:03
And that's equal to the original length squared.
06:06
Length it is preserved by these Qs.
06:08
Everything is preserved.
06:09
You're multiplying effectively by the matrix versions
06:13
of 1 and minus 1.
06:16
And a rotation is a very significant very valuable
06:22
orthogonal matrix, which just has cosines and signs.
06:25
And everybody's remembering that cosine squared plus sine
06:29
squared is 1 from trig.
06:33
So that's an orthogonal matrix.
06:38
Oh, it's also orthogonal because the dot
06:41
product between that one and that one,
06:44
you're OK for the dot product.
06:45
That product gives me minus sine cosine, plus sine cosine, 0.
06:51
So the column 1 is orthogonal to column 2.
06:55
That's good.
06:56
OK.
06:58
These lambdas that you see here are
07:03
something called eigenvalues.
07:04
That's not allowed until the next lecture.
07:08
OK, all right, now, here's something.
07:11
Here's a computing thing.
07:13
If we have a bunch of columns, not orthogonal, not length 1,
07:20
then, often, we would like to convert them
07:24
to, so we call those, A1 to AN.
07:27
Nothing special about those columns.
07:30
We would like to convert them to orthogonal columns
07:33
because they're the beautiful ones, Q1 up to Qn.
07:36
And two guys called Graham and Schmidt figured out
07:41
a way to do that.
07:42
And a century later, we're still using their idea.
07:46
Well, I don't know whose idea it was actually.
07:48
I think Graham had the idea.
07:49
And I'm not really sure what Schmidt, how he got into it.
07:54
Well, he may have repeated the idea.
07:56
So OK, so I won't go all the details.
08:01
But here's what the point is the point
08:04
is, if I have a bunch of columns that are independent,
08:10
they go in different directions, but they're not
08:12
90 degree directions.
08:14
Then I can convert it to a 90 degree one
08:17
to perpendicular axes with a matrix R,
08:22
happens to be triangular, that did the moving around, did
08:29
take that combinations.
08:30
So A equal QR is one of the fundamental steps
08:35
of linear algebra and computational linear algebra.
08:39
Very, very often, we're given a matrix
08:42
A. We want a nice matrix Q, so we do this Graham Schmidt step
08:48
to make the columns orthogonal.
08:51
And oh, here's a first step of Graham Schmidt.
08:56
But you'll need practice to see all the steps.
09:03
Maybe not.
09:04
OK, so here, what's the advantage
09:09
of perpendicular vectors?
09:11
Suppose I have a triangle.
09:13
And one side is perpendicular to the second side.
09:17
How does that help?
09:19
Well, that's a right triangle then.
09:22
Side A perpendicular to side B. And of course, Pythagoras, now
09:27
we're really going back, Pythagoras said,
09:30
a squared plus b squared is c squared.
09:33
So we have beautiful formulas when things are perpendicular.
09:36
If the angles are not 90 degrees when the cosine of 90 degrees
09:42
is 1 or maybe the sine of 90 degrees is 1,
09:46
yeah, sine of 90 degrees is 1.
09:50
For those perfect angles, 0 and 90 degrees,
09:56
we can do everything.
09:58
And here is a place that Q fits.
10:03
10:05
This is like the first big application of linear algebra.
10:10
So let me just say what it is.
10:13
And it uses these cubes.
10:17
So what's the application that's called least squares?
10:20
And you start with equations, Ax equal b.
10:26
You always think of that as a matrix
10:28
times the unknown vector, being known,
10:30
right hand side b Ax equal b.
10:33
So suppose we have too many equations.
10:38
That often happens.
10:39
If you take too many measurements,
10:40
you want to get an exact x.
10:43
So you do more and more measurements to b.
10:46
You're pasting more and more conditions on x.
10:48
And you're not going to find an exact x because you've
10:52
got too many equations. m is bigger than n.
10:55
We might have 2,000 measurements,
10:57
say, from medical things or from satellites.
11:02
And we might have only two unknowns,
11:04
fitting a straight line with only two variables.
11:07
So how am I going to solve 2,000 equations with two
11:10
unknowns Well, I'm not.
11:12
But I look for the best solution.
11:16
How close can I come?
11:18
And that's what least squares is about.
11:20
You get Ax as close as possible to b.
11:24
And probably, this will show how the--
11:29
yeah.
11:29
Yeah, here's the right equation.
11:31
When you-- here's my message.
11:34
When you can't solve Ax equal b, multiply both sides
11:39
by A transpose.
11:41
Then you can solve this equation.
11:43
That's the right equation.
11:44
So I put a little hat on that x to show
11:48
that it doesn't solve the original equation, Ax equal b,
11:52
but it comes the closest.
11:54
It's the closest solution I could find.
11:57
And it's discovered by multiplying both sides
12:01
by this A transpose matrix.
12:03
So A transpose A is a terrifically important matrix.
12:06
It's a square matrix.
12:09
See, A didn't have to be square.
12:11
I could have lots of measurements there,
12:13
many, many equations, long, thin matrix for A.
12:17
But A transpose A always comes out square and also symmetric.
12:23
And it's just a great matrix for theory.
12:28
And this QR business makes it work in practice.
12:33
Let me see if there's more.
12:34
So this is, oh, yeah.
12:36
This is the geometry.
12:39
So I start with a matrix A. It's only got a few columns, maybe
12:45
even only two columns.
12:47
So its column space is just a plane, not the whole space.
12:52
But my right hand side b is somewhere else in whole space.
12:56
You see this situation.
12:58
I can only solve Ax equal b when b is
13:02
a combination of the columns.
13:04
And here, it's not.
13:06
The measurements weren't perfect.
13:07
I'm off somewhere.
13:09
So how do you deal with that?
13:12
Geometry tells you.
13:15
You can't deal with b.
13:17
You can't solve Ax equal b.
13:20
So you drop a perpendicular.
13:22
You find the closest point, the projection that
13:25
in the space where you can solve.
13:28
So then you solve Ax equal p.
13:32
That's what least squares is all about, fitting the best
13:35
straight line, the best parabola,
13:37
whatever, is all linear algebra of perpendicular
13:42
things and orthogonal matrices.
13:45
OK, I think that's what I can say about orthogonal.
13:50
Well, it'll come in again.
13:51
Orthogonal matrices, perpendicular columns
13:55
is so beautiful, but next is coming eigenvectors.
13:59
And that's another chapter.
14:00
So I'll stop here.
14:02
Good.
14:02
Thanks.
14:04
English - CC (English)
###