CHRISTINE BREINER: Welcome back to recitation.
In this video, what I'd like you to do is use least squares
to fit a line to the following data, which includes three
points: the point 0, 1, the point 2, 1, and
the point 3, 4.
And I've drawn a rough picture where these points are on a
graph, and I'll be talking a little bit about that after
you try this problem.
So why don't you try to solve the problem based on the
technique you learned in class, and then bring the
video back up.
I'll show you again what you're actually doing, and
then I will also solve the problem and you can see, you
can compare your answer to mine.
OK, welcome back.
So the first thing I want to do when we're talking about
least squares to fit a line to these data, what I'd like to
do is I'm going to draw a line on here, and we're going to
talk about what the least squares actually
is trying to do.
And so I'm going to draw this line, say,
something like this.
That's my first guess on what might be the actual least
squares line for these data.
And if you recall, what you're actually trying to do is
you're trying to minimize a certain quantity, and the
quantity you're trying to minimize is the difference
between the actual value you get and the expected value you
get, the square of that.
So this is one thing we're going to square.
So even pictorially, we could say we have something that
contributes some area, OK, and then this is another thing
that's the difference between the expected value and y and
the actual value you get.
So let me try and make that squared, because we square
that value.
And then this last one is pretty small.
The difference between the expected value and the actual
value is very small over in this case from
the picture I drew.
And so what you're trying to do is you're trying to find
the line that minimizes the area of the sum of these
three, in this case, three squares.
So if I, you know, if I tip the line down further here,
but I kept it fixed there, the area of this square would get
bigger, because this distance would get bigger, but, of
course, the area of this square would get smaller.
And so you're trying to figure out a way to optimize, in this
case, minimize, the area of these--
sorry, the sum of the areas of these squares by figuring out
where to put the line.
Now, I mentioned if I could fix this here and move it up
and down, then that would change the
slope but fix the intercept.
But I could also move the intercept up and down, and I
could slide the line up and down, and that would fix the
slope but vary the intercept, or I could do both.
And ultimately, that's what you're doing.
You're trying to find the best y-intercept and the best slope
at the same time, and that's why you learned this process
in multivariable calculus, because you're trying to
optimize something that has two quantities
that you can change.
You can change slope and you can change the y-intercept, so
that's why you learned it now instead of in
single-variable calculus.
Now, if we come over here, I wrote down ahead of time what
the equations are you get when you optimize, when you're
trying to optimize the least squares line.
So if you write down, you know, the function as a
function of a and b, and you take the derivative with
respect to a and you set it equal to zero, you get the
first equation.
When you take the derivative with respect to b and you set
it equal to zero, you get the second equation.
And so I'm just going to fill in the details and then tell
you what the solution is, and then I'm going to mention a
little more sophisticated thing we can do, or I guess
maybe the next level of least squares we can do, that's not
a linear problem.
So I'll do that last.
So what I want to do is I put all the values I need over
here on the right.
So all the values we need are on the right.
We're going to need the sum of all the xi's and that's 5.
The sum of all the yi's is 6.
The sum of the xi squareds is 13, and the sum of the xi yi
product is 14.
Now, where are these coming from?
Remember, what is xi and what is yi?
If we come back over here, this is x1, y1,
x2, y2, x3, y3, OK?
So that's sort of how we get-- if we switch the whole pairs
around, it doesn't matter.
Obviously, if we switched x- and y-values together, it does
matter, and that you see actually from the equation
does matter.
Right here you have a product of them.
So let me write this out.
And again, we're solving for a and b, so in this case, a is
the slope and b is the intercept, right?
So let me write everything in.
So we actually get that we want to solve the following
system: 13a plus 5b is equal to 14.
Make sure I put in everything correctly.
And then the second one is 5a plus--
n in this case is 3, because there were three points--
plus 3b is equal to 6.
OK, this is a system of equations.
It has two equations and two unknowns, and
if you solve this--
I should look at my notes, because I didn't memorize it
and I'm not going to do all the algebra.
If you go to solve this, what you actually get is that a is
equal to 6/7 and b is equal to 4/7.
So you get that the line--
let me write it here.
You get a is equal to 6/7 and b is equal to 4/7, OK?
And so what that tells you is if you come back over here,
the line we actually want--
I'll write the solution right here-- the line we actually
want is the line y equals 6/7x plus 4/7.
That is our least squares line that is our solution.
OK?
So that is actually--
solving for a line to fit these data, you get the
following solution.
But I want to point out, and what we're going to do, after
this video there will be a challenge problem to have you
actually finish this.
I want to point out that you can actually also fit a
higher-degree polynomial to these data.
For instance, you could try and use the technique of least
squares to fit a parabola to these data.
And let me point out what the function would look
like in this case.
So this is for the challenge problem.
For the challenge problem, it now will be a function of
three variables, so it will look something like this.
We'll say a comma b comma c, and the point that we'll be
wanting to minimize, or the function we'll be wanting to
minimize, we're summing over i, again from 1 to 3--
I didn't write that down above, but it was fairly
obvious that that was what we were summing over--
the following thing.
We have yi minus this big quantity: axi squared plus bxi
plus c and we square the whole thing.
And so what this is actually giving you is, just as in the
picture before, this is giving you the difference between the
expected value and the actual value.
So this is your actual y-value, and based on the
parabola you're looking for, what's in parentheses is your
expected y-value, right?
So this would actually be my parabola.
When I evaluate it at xi, I get a number.
I get a value, and that value will be on the parabola.
This will be the y-value associated to the
x-value from the data.
So this is the actual value.
This is the expected value.
We take the difference and square it.
And because we want to minimize how we're going to
solve this problem and this is what you'll have to do is,
just like with the line situation, you're going to
take the derivative of this function with respect to each
of these three variables separately.
So you'll take f sub a, and then you'll set
that equal to zero.
And then you'll take f sub b, you'll set that equal to zero.
And then you'll take f sub c, and you'll set
that equal to zero.
Again, we set them all equal to zero, because this is
really an optimization problem.
We want to minimize something.
We want to minimize this quantity.
So you take each of those three derivatives, partial
derivatives, set them equal to zero, and you have a system of
three equations with three variables.
And you know how to solve such a system, by using matrices,
for instance.
That would be one way to solve such a system.
And so then you can actually come up with a formula for the
sort of parabola of best fit, you could think of it as.
And so I'm going to stop there, because I think from
here, you guys will do it.
But I just want to point out this is where you're going to
be starting this next sort of challenge problem to find the
quadratic of best fit for these three data points.
So that's where I'll end it.