WEBVTT 00:00:00.000 --> 00:00:00.040 align:middle line:90% 00:00:00.040 --> 00:00:02.460 align:middle line:84% The following content is provided under a Creative 00:00:02.460 --> 00:00:03.870 align:middle line:90% Commons license. 00:00:03.870 --> 00:00:06.910 align:middle line:84% Your support will help MIT OpenCourseWare continue to 00:00:06.910 --> 00:00:10.560 align:middle line:84% offer high quality educational resources for free. 00:00:10.560 --> 00:00:13.460 align:middle line:84% To make a donation or view additional materials from 00:00:13.460 --> 00:00:19.290 align:middle line:84% hundreds of MIT courses, visit MIT OpenCourseWare at 00:00:19.290 --> 00:00:21.732 align:middle line:90% ocw.mit.edu. 00:00:21.732 --> 00:00:24.170 align:middle line:84% JOHN TSITSIKLIS: And we're going to continue today with 00:00:24.170 --> 00:00:26.820 align:middle line:84% our discussion of classical statistics. 00:00:26.820 --> 00:00:29.290 align:middle line:84% We'll start with a quick review of what we discussed 00:00:29.290 --> 00:00:34.680 align:middle line:84% last time, and then talk about two topics that cover a lot of 00:00:34.680 --> 00:00:37.740 align:middle line:84% statistics that are happening in the real world. 00:00:37.740 --> 00:00:39.510 align:middle line:90% So two basic methods. 00:00:39.510 --> 00:00:43.730 align:middle line:84% One is the method of linear regression, and the other one 00:00:43.730 --> 00:00:46.500 align:middle line:84% is the basic methods and tools for how to 00:00:46.500 --> 00:00:49.540 align:middle line:90% do hypothesis testing. 00:00:49.540 --> 00:00:53.970 align:middle line:84% OK, so these two are topics that any scientifically 00:00:53.970 --> 00:00:57.170 align:middle line:84% literate person should know something about. 00:00:57.170 --> 00:00:59.570 align:middle line:84% So we're going to introduce the basic ideas 00:00:59.570 --> 00:01:01.860 align:middle line:90% and concepts involved. 00:01:01.860 --> 00:01:07.580 align:middle line:84% So in classical statistics we basically have essentially a 00:01:07.580 --> 00:01:11.250 align:middle line:84% family of possible models about the world. 00:01:11.250 --> 00:01:15.190 align:middle line:84% So the world is the random variable that we observe, and 00:01:15.190 --> 00:01:19.370 align:middle line:84% we have a model for it, but actually not just one model, 00:01:19.370 --> 00:01:20.960 align:middle line:90% several candidate models. 00:01:20.960 --> 00:01:24.380 align:middle line:84% And each candidate model corresponds to a different 00:01:24.380 --> 00:01:28.070 align:middle line:84% value of a parameter theta that we do not know. 00:01:28.070 --> 00:01:32.275 align:middle line:84% So in contrast to Bayesian statistics, this theta is 00:01:32.275 --> 00:01:35.540 align:middle line:84% assumed to be a constant that we do not know. 00:01:35.540 --> 00:01:38.190 align:middle line:84% It is not modeled as a random variable, there's no 00:01:38.190 --> 00:01:40.480 align:middle line:84% probabilities associated with theta. 00:01:40.480 --> 00:01:43.380 align:middle line:84% We only have probabilities about the X's. 00:01:43.380 --> 00:01:47.320 align:middle line:84% So in this context what is a reasonable way of choosing a 00:01:47.320 --> 00:01:49.350 align:middle line:90% value for the parameter? 00:01:49.350 --> 00:01:53.470 align:middle line:84% One general approach is the maximum likelihood approach, 00:01:53.470 --> 00:01:56.090 align:middle line:84% which chooses the theta for which 00:01:56.090 --> 00:01:58.630 align:middle line:90% this quantity is largest. 00:01:58.630 --> 00:02:00.690 align:middle line:84% So what does that mean intuitively? 00:02:00.690 --> 00:02:04.550 align:middle line:84% I'm trying to find the value of theta under which the data 00:02:04.550 --> 00:02:08.970 align:middle line:84% that I observe are most likely to have occurred. 00:02:08.970 --> 00:02:11.470 align:middle line:84% So is the thinking is essentially as follows. 00:02:11.470 --> 00:02:13.970 align:middle line:84% Let's say I have to choose between two choices of theta. 00:02:13.970 --> 00:02:16.520 align:middle line:84% Under this theta the X that I observed 00:02:16.520 --> 00:02:17.940 align:middle line:90% would be very unlikely. 00:02:17.940 --> 00:02:21.350 align:middle line:84% Under that theta the X that I observed would have a decent 00:02:21.350 --> 00:02:22.830 align:middle line:90% probability of occurring. 00:02:22.830 --> 00:02:28.340 align:middle line:84% So I chose the latter as my estimate of theta. 00:02:28.340 --> 00:02:31.200 align:middle line:84% It's interesting to do the comparison with the Bayesian 00:02:31.200 --> 00:02:34.110 align:middle line:84% approach which we did discuss last time, in the Bayesian 00:02:34.110 --> 00:02:38.430 align:middle line:84% approach we also maximize over theta, but we maximize a 00:02:38.430 --> 00:02:43.220 align:middle line:84% quantity in which the relation between X's and thetas run the 00:02:43.220 --> 00:02:44.520 align:middle line:90% opposite way. 00:02:44.520 --> 00:02:47.500 align:middle line:84% Here in the Bayesian world, Theta is a random variable. 00:02:47.500 --> 00:02:48.980 align:middle line:90% So it has a distribution. 00:02:48.980 --> 00:02:53.030 align:middle line:84% Once we observe the data, it has a posterior distribution, 00:02:53.030 --> 00:02:56.480 align:middle line:84% and we find the value of Theta, which is most likely 00:02:56.480 --> 00:02:59.250 align:middle line:84% under the posterior distribution. 00:02:59.250 --> 00:03:03.090 align:middle line:84% As we discussed last time when you do this maximization now 00:03:03.090 --> 00:03:05.750 align:middle line:84% the posterior distribution is given by this expression. 00:03:05.750 --> 00:03:09.760 align:middle line:84% The denominator doesn't matter, and if you were to 00:03:09.760 --> 00:03:12.790 align:middle line:90% take a prior, which is flat-- 00:03:12.790 --> 00:03:16.210 align:middle line:84% that is a constant independent of Theta, then that 00:03:16.210 --> 00:03:17.640 align:middle line:90% term would go away. 00:03:17.640 --> 00:03:19.360 align:middle line:84% And syntactically, at least, the two 00:03:19.360 --> 00:03:21.970 align:middle line:90% approaches look the same. 00:03:21.970 --> 00:03:28.170 align:middle line:84% So syntactically, or formally, maximum likelihood estimation 00:03:28.170 --> 00:03:32.225 align:middle line:84% is the same as Bayesian estimation in which you assume 00:03:32.225 --> 00:03:36.090 align:middle line:84% a prior which is flat, so that all possible values of Theta 00:03:36.090 --> 00:03:37.570 align:middle line:90% are equally likely. 00:03:37.570 --> 00:03:40.790 align:middle line:84% Philosophically, however, they're very different things. 00:03:40.790 --> 00:03:44.150 align:middle line:84% Here I'm picking the most likely value of Theta. 00:03:44.150 --> 00:03:47.140 align:middle line:84% Here I'm picking the value of Theta under which the observed 00:03:47.140 --> 00:03:51.050 align:middle line:84% data would have been more likely to occur. 00:03:51.050 --> 00:03:53.590 align:middle line:84% So maximum likelihood estimation is a general 00:03:53.590 --> 00:03:57.820 align:middle line:84% purpose method, so it's applied all over the place in 00:03:57.820 --> 00:04:02.220 align:middle line:84% many, many different types of estimation problems. 00:04:02.220 --> 00:04:05.100 align:middle line:84% There is a special kind of estimation problem in which 00:04:05.100 --> 00:04:08.040 align:middle line:84% you may forget about maximum likelihood estimation, and 00:04:08.040 --> 00:04:12.700 align:middle line:84% come up with an estimate in a straightforward way. 00:04:12.700 --> 00:04:15.680 align:middle line:84% And this is the case where you're trying to estimate the 00:04:15.680 --> 00:04:22.390 align:middle line:84% mean of the distribution of X, where X is a random variable. 00:04:22.390 --> 00:04:25.140 align:middle line:84% You observe several independent identically 00:04:25.140 --> 00:04:30.020 align:middle line:84% distributed random variables X1 up to Xn. 00:04:30.020 --> 00:04:32.880 align:middle line:84% All of them have the same distribution as this X. 00:04:32.880 --> 00:04:34.600 align:middle line:90% So they have a common mean. 00:04:34.600 --> 00:04:37.020 align:middle line:84% We do not know the mean we want to estimate it. 00:04:37.020 --> 00:04:40.560 align:middle line:84% What is more natural than just taking the average of the 00:04:40.560 --> 00:04:42.470 align:middle line:90% values that we have observed? 00:04:42.470 --> 00:04:46.150 align:middle line:84% So you generate lots of X's, take the average of them, and 00:04:46.150 --> 00:04:50.560 align:middle line:84% you expect that this is going to be a reasonable estimate of 00:04:50.560 --> 00:04:53.420 align:middle line:84% the true mean of that random variable. 00:04:53.420 --> 00:04:56.290 align:middle line:84% And indeed we know from the weak law of large numbers that 00:04:56.290 --> 00:05:00.790 align:middle line:84% this estimate converges in probability to the true mean 00:05:00.790 --> 00:05:02.680 align:middle line:90% of the random variable. 00:05:02.680 --> 00:05:04.870 align:middle line:84% The other thing that we talked about last time is that 00:05:04.870 --> 00:05:07.770 align:middle line:84% besides giving a point estimate we may want to also 00:05:07.770 --> 00:05:13.530 align:middle line:84% give an interval that tells us something about where we might 00:05:13.530 --> 00:05:16.170 align:middle line:90% believe theta to lie. 00:05:16.170 --> 00:05:21.950 align:middle line:84% And 1-alpha confidence interval is in interval 00:05:21.950 --> 00:05:24.200 align:middle line:90% generated based on the data. 00:05:24.200 --> 00:05:26.860 align:middle line:84% So it's an interval from this value to that value. 00:05:26.860 --> 00:05:30.120 align:middle line:84% These values are written with capital letters because 00:05:30.120 --> 00:05:32.390 align:middle line:84% they're random, because they depend on the data 00:05:32.390 --> 00:05:33.870 align:middle line:90% that we have seen. 00:05:33.870 --> 00:05:36.740 align:middle line:84% And this gives us an interval, and we would like this 00:05:36.740 --> 00:05:40.600 align:middle line:84% interval to have the property that theta is inside that 00:05:40.600 --> 00:05:42.830 align:middle line:84% interval with high probability. 00:05:42.830 --> 00:05:46.805 align:middle line:84% So typically we would take 1-alpha to be a quantity such 00:05:46.805 --> 00:05:49.780 align:middle line:90% as 95% for example. 00:05:49.780 --> 00:05:54.340 align:middle line:84% In which case we have a 95% confidence interval. 00:05:54.340 --> 00:05:56.980 align:middle line:84% As we discussed last time it's important to have the right 00:05:56.980 --> 00:06:00.730 align:middle line:84% interpretation of what's 95% means. 00:06:00.730 --> 00:06:04.640 align:middle line:84% What it does not mean is the following-- 00:06:04.640 --> 00:06:09.800 align:middle line:84% the unknown value has 95% percent probability of being 00:06:09.800 --> 00:06:12.450 align:middle line:84% in the interval that we have generated. 00:06:12.450 --> 00:06:14.550 align:middle line:84% That's because the unknown value is not a random 00:06:14.550 --> 00:06:15.910 align:middle line:90% variable, it's a constant. 00:06:15.910 --> 00:06:18.930 align:middle line:84% Once we generate the interval either it's inside or it's 00:06:18.930 --> 00:06:22.500 align:middle line:84% outside, but there's no probabilities involved. 00:06:22.500 --> 00:06:26.415 align:middle line:84% Rather the probabilities are to be interpreted over the 00:06:26.415 --> 00:06:28.590 align:middle line:90% random interval itself. 00:06:28.590 --> 00:06:31.730 align:middle line:84% What a statement like this says is that if I have a 00:06:31.730 --> 00:06:37.060 align:middle line:84% procedure for generating 95% confidence intervals, then 00:06:37.060 --> 00:06:40.800 align:middle line:84% whenever I use that procedure I'm going to get a random 00:06:40.800 --> 00:06:44.260 align:middle line:84% interval, and it's going to have 95% probability of 00:06:44.260 --> 00:06:48.270 align:middle line:84% capturing the true value of theta. 00:06:48.270 --> 00:06:53.010 align:middle line:84% So most of the time when I use this particular procedure for 00:06:53.010 --> 00:06:56.170 align:middle line:84% generating confidence intervals the true theta will 00:06:56.170 --> 00:06:59.440 align:middle line:84% happen to lie inside that confidence interval with 00:06:59.440 --> 00:07:01.190 align:middle line:90% probability 95%. 00:07:01.190 --> 00:07:04.230 align:middle line:84% So the randomness in this statement is with respect to 00:07:04.230 --> 00:07:09.190 align:middle line:84% my confidence interval, it's not with respect to theta, 00:07:09.190 --> 00:07:11.880 align:middle line:90% because theta is not random. 00:07:11.880 --> 00:07:14.710 align:middle line:84% How does one construct confidence intervals? 00:07:14.710 --> 00:07:17.500 align:middle line:84% There's various ways of going about it, but in the case 00:07:17.500 --> 00:07:20.330 align:middle line:84% where we're dealing with the estimation of the mean of a 00:07:20.330 --> 00:07:23.790 align:middle line:84% random variable doing this is straightforward using the 00:07:23.790 --> 00:07:25.680 align:middle line:90% central limit theorem. 00:07:25.680 --> 00:07:31.440 align:middle line:84% Basically we take our estimated mean, that's the 00:07:31.440 --> 00:07:35.910 align:middle line:84% sample mean, and we take a symmetric interval to the left 00:07:35.910 --> 00:07:38.220 align:middle line:84% and to the right of the sample mean. 00:07:38.220 --> 00:07:42.340 align:middle line:84% And we choose the width of that interval by looking at 00:07:42.340 --> 00:07:43.680 align:middle line:90% the normal tables. 00:07:43.680 --> 00:07:50.180 align:middle line:84% So if this quantity, 1-alpha is 95% percent, we're going to 00:07:50.180 --> 00:07:55.790 align:middle line:84% look at the 97.5 percentile of the normal distribution. 00:07:55.790 --> 00:07:59.910 align:middle line:84% Find the constant number that corresponds to that value from 00:07:59.910 --> 00:08:02.790 align:middle line:84% the normal tables, and construct the confidence 00:08:02.790 --> 00:08:07.350 align:middle line:84% intervals according to this formula. 00:08:07.350 --> 00:08:10.810 align:middle line:84% So that gives you a pretty mechanical way of going about 00:08:10.810 --> 00:08:13.250 align:middle line:84% constructing confidence intervals when you're 00:08:13.250 --> 00:08:15.270 align:middle line:90% estimating the sample mean. 00:08:15.270 --> 00:08:18.650 align:middle line:84% So constructing confidence intervals in this way involves 00:08:18.650 --> 00:08:19.630 align:middle line:90% an approximation. 00:08:19.630 --> 00:08:22.230 align:middle line:84% The approximation is the central limit theorem. 00:08:22.230 --> 00:08:24.490 align:middle line:84% We are pretending that the sample mean is a 00:08:24.490 --> 00:08:26.400 align:middle line:90% normal random variable. 00:08:26.400 --> 00:08:30.110 align:middle line:84% Which is, more or less, right when n is large. 00:08:30.110 --> 00:08:32.780 align:middle line:84% That's what the central limit theorem tells us. 00:08:32.780 --> 00:08:36.429 align:middle line:84% And sometimes we may need to do some extra approximation 00:08:36.429 --> 00:08:39.480 align:middle line:84% work, because quite often we do not know the 00:08:39.480 --> 00:08:41.030 align:middle line:90% true value of sigma. 00:08:41.030 --> 00:08:43.559 align:middle line:84% So we need to do some work either to estimate 00:08:43.559 --> 00:08:45.360 align:middle line:90% sigma from the data. 00:08:45.360 --> 00:08:48.520 align:middle line:84% So sigma is, of course, the standard deviation of the X's. 00:08:48.520 --> 00:08:51.410 align:middle line:84% We may want to estimate it from the data, or we may have 00:08:51.410 --> 00:08:54.450 align:middle line:84% an upper bound on sigma, and we just use that upper bound. 00:08:54.450 --> 00:08:57.430 align:middle line:90% 00:08:57.430 --> 00:09:02.520 align:middle line:84% So now let's move on to a new topic. 00:09:02.520 --> 00:09:09.420 align:middle line:84% A lot of statistics in the real world are of the 00:09:09.420 --> 00:09:12.540 align:middle line:90% following flavor. 00:09:12.540 --> 00:09:16.820 align:middle line:84% So suppose that X is the SAT score of a student in high 00:09:16.820 --> 00:09:23.620 align:middle line:84% school, and Y is the MIT GPA of that same student. 00:09:23.620 --> 00:09:27.570 align:middle line:84% So you expect that there is a relation between these two. 00:09:27.570 --> 00:09:31.240 align:middle line:84% So you go and collect data for different students, and you 00:09:31.240 --> 00:09:35.470 align:middle line:84% record for a typical student this would be their SAT score, 00:09:35.470 --> 00:09:37.700 align:middle line:90% that could be their MIT GPA. 00:09:37.700 --> 00:09:43.720 align:middle line:84% And you plot all this data on an (X,Y) diagram. 00:09:43.720 --> 00:09:48.240 align:middle line:84% Now it's reasonable to believe that there is some systematic 00:09:48.240 --> 00:09:49.940 align:middle line:90% relation between the two. 00:09:49.940 --> 00:09:54.650 align:middle line:84% So people who had higher SAT scores in high school may have 00:09:54.650 --> 00:09:57.110 align:middle line:90% higher GPA in college. 00:09:57.110 --> 00:10:00.310 align:middle line:84% Well that may or may not be true. 00:10:00.310 --> 00:10:05.270 align:middle line:84% You want to construct a model of this kind, and see to what 00:10:05.270 --> 00:10:08.330 align:middle line:84% extent a relation of this type is true. 00:10:08.330 --> 00:10:15.560 align:middle line:84% So you might hypothesize that the real world is described by 00:10:15.560 --> 00:10:17.390 align:middle line:90% a model of this kind. 00:10:17.390 --> 00:10:22.730 align:middle line:84% That there is a linear relation between the SAT 00:10:22.730 --> 00:10:27.710 align:middle line:90% score, and the college GPA. 00:10:27.710 --> 00:10:30.560 align:middle line:84% So it's a linear relation with some parameters, theta0 and 00:10:30.560 --> 00:10:33.060 align:middle line:90% theta1 that we do not know. 00:10:33.060 --> 00:10:37.460 align:middle line:84% So we assume a linear relation for the data, and depending on 00:10:37.460 --> 00:10:41.690 align:middle line:84% the choices of theta0 and theta1 it could be a different 00:10:41.690 --> 00:10:43.530 align:middle line:90% line through those data. 00:10:43.530 --> 00:10:47.670 align:middle line:84% Now we would like to find the best model of this kind to 00:10:47.670 --> 00:10:49.230 align:middle line:90% explain the data. 00:10:49.230 --> 00:10:52.260 align:middle line:84% Of course there's going to be some randomness. 00:10:52.260 --> 00:10:55.370 align:middle line:84% So in general it's going to be impossible to find a line that 00:10:55.370 --> 00:10:57.780 align:middle line:84% goes through all of the data points. 00:10:57.780 --> 00:11:04.020 align:middle line:84% So let's try to find the best line that comes closest to 00:11:04.020 --> 00:11:05.810 align:middle line:90% explaining those data. 00:11:05.810 --> 00:11:08.520 align:middle line:90% And here's how we go about it. 00:11:08.520 --> 00:11:13.100 align:middle line:84% Suppose we try some particular values of theta0 and theta1. 00:11:13.100 --> 00:11:15.750 align:middle line:90% These give us a certain line. 00:11:15.750 --> 00:11:20.760 align:middle line:84% Given that line, we can make predictions. 00:11:20.760 --> 00:11:24.470 align:middle line:84% For a student who had this x, the model that we have would 00:11:24.470 --> 00:11:27.580 align:middle line:84% predict that y would be this value. 00:11:27.580 --> 00:11:32.150 align:middle line:84% The actual y is something else, and so this quantity is 00:11:32.150 --> 00:11:37.660 align:middle line:84% the error that our model would make in predicting the y of 00:11:37.660 --> 00:11:39.580 align:middle line:90% that particular student. 00:11:39.580 --> 00:11:43.350 align:middle line:84% We would like to choose a line for which the predictions are 00:11:43.350 --> 00:11:45.110 align:middle line:90% as good as possible. 00:11:45.110 --> 00:11:47.790 align:middle line:84% And what do we mean by as good as possible? 00:11:47.790 --> 00:11:51.150 align:middle line:84% As our criteria we're going to take the following. 00:11:51.150 --> 00:11:54.070 align:middle line:84% We are going to look at the prediction error that our 00:11:54.070 --> 00:11:56.310 align:middle line:84% model makes for each particular student. 00:11:56.310 --> 00:12:01.050 align:middle line:84% Take the square of that, and then add them up over all of 00:12:01.050 --> 00:12:02.580 align:middle line:90% our data points. 00:12:02.580 --> 00:12:06.140 align:middle line:84% So what we're looking at is the sum of this quantity 00:12:06.140 --> 00:12:08.270 align:middle line:84% squared, that quantity squared, that quantity 00:12:08.270 --> 00:12:09.570 align:middle line:90% squared, and so on. 00:12:09.570 --> 00:12:13.220 align:middle line:84% We add all of these squares, and we would like to find the 00:12:13.220 --> 00:12:17.500 align:middle line:84% line for which the sum of these squared prediction 00:12:17.500 --> 00:12:20.910 align:middle line:84% errors are as small as possible. 00:12:20.910 --> 00:12:23.950 align:middle line:90% So that's the procedure. 00:12:23.950 --> 00:12:27.100 align:middle line:84% We have our data, the X's and the Y's. 00:12:27.100 --> 00:12:31.340 align:middle line:84% And we're going to find theta's the best model of this 00:12:31.340 --> 00:12:35.580 align:middle line:84% type, the best possible model, by minimizing this sum of 00:12:35.580 --> 00:12:38.010 align:middle line:90% squared errors. 00:12:38.010 --> 00:12:41.020 align:middle line:84% So that's a method that one could pull out of the hat and 00:12:41.020 --> 00:12:44.120 align:middle line:84% say OK, that's how I'm going to build my model. 00:12:44.120 --> 00:12:46.730 align:middle line:84% And it sounds pretty reasonable. 00:12:46.730 --> 00:12:49.530 align:middle line:84% And it sounds pretty reasonable even if you don't 00:12:49.530 --> 00:12:51.660 align:middle line:84% know anything about probability. 00:12:51.660 --> 00:12:55.340 align:middle line:84% But does it have some probabilistic justification? 00:12:55.340 --> 00:12:59.280 align:middle line:84% It turns out that yes, you can motivate this method with 00:12:59.280 --> 00:13:03.100 align:middle line:84% probabilistic considerations under certain assumptions. 00:13:03.100 --> 00:13:07.360 align:middle line:84% So let's make a probabilistic model that's going to lead us 00:13:07.360 --> 00:13:10.600 align:middle line:84% to these particular way of estimating the parameters. 00:13:10.600 --> 00:13:12.920 align:middle line:84% So here's a probabilistic model. 00:13:12.920 --> 00:13:18.090 align:middle line:84% I pick a student who had a specific SAT score. 00:13:18.090 --> 00:13:21.190 align:middle line:84% And that could be done at random, but also could be done 00:13:21.190 --> 00:13:22.330 align:middle line:90% in a systematic way. 00:13:22.330 --> 00:13:25.240 align:middle line:84% That is, I pick a student who had an SAT of 600, a student 00:13:25.240 --> 00:13:33.170 align:middle line:84% of 610 all the way to 1,400 or 1,600, whatever the 00:13:33.170 --> 00:13:34.670 align:middle line:90% right number is. 00:13:34.670 --> 00:13:36.320 align:middle line:90% I pick all those students. 00:13:36.320 --> 00:13:40.370 align:middle line:84% And I assume that for a student of this kind there's a 00:13:40.370 --> 00:13:44.500 align:middle line:84% true model that tells me that their GPA is going to be a 00:13:44.500 --> 00:13:48.580 align:middle line:84% random variable, which is something predicted by their 00:13:48.580 --> 00:13:52.690 align:middle line:84% SAT score plus some randomness, some random noise. 00:13:52.690 --> 00:13:56.400 align:middle line:84% And I model that random noise by independent normal random 00:13:56.400 --> 00:14:00.710 align:middle line:84% variables with 0 mean and a certain variance. 00:14:00.710 --> 00:14:04.470 align:middle line:84% So this is a specific probabilistic model, and now I 00:14:04.470 --> 00:14:09.010 align:middle line:84% can think about doing maximum likelihood estimation for this 00:14:09.010 --> 00:14:10.980 align:middle line:90% particular model. 00:14:10.980 --> 00:14:14.490 align:middle line:84% So to do maximum likelihood estimation here I need to 00:14:14.490 --> 00:14:19.830 align:middle line:84% write down the likelihood of the y's that I have observed. 00:14:19.830 --> 00:14:23.380 align:middle line:84% What's the likelihood of the y's that I have observed? 00:14:23.380 --> 00:14:28.425 align:middle line:84% Well, a particular w has a likelihood of the form e to 00:14:28.425 --> 00:14:33.030 align:middle line:84% the minus w squared over (2 sigma-squared). 00:14:33.030 --> 00:14:37.070 align:middle line:84% That's the likelihood of a particular w. 00:14:37.070 --> 00:14:40.310 align:middle line:84% The probability, or the likelihood of observing a 00:14:40.310 --> 00:14:43.990 align:middle line:84% particular value of y, that's the same as the likelihood 00:14:43.990 --> 00:14:49.020 align:middle line:84% that w takes a value of y minus this, minus that. 00:14:49.020 --> 00:14:52.850 align:middle line:84% So the likelihood of the y's is of this form. 00:14:52.850 --> 00:14:57.360 align:middle line:84% Think of this as just being the w_i-squared. 00:14:57.360 --> 00:15:01.370 align:middle line:90% So this is the density -- 00:15:01.370 --> 00:15:06.060 align:middle line:84% and if we have multiple data you multiply the likelihoods 00:15:06.060 --> 00:15:07.660 align:middle line:90% of the different y's. 00:15:07.660 --> 00:15:12.090 align:middle line:84% So you have to write something like this. 00:15:12.090 --> 00:15:16.390 align:middle line:84% Since the w's are independent that means that the y's are 00:15:16.390 --> 00:15:17.910 align:middle line:90% also independent. 00:15:17.910 --> 00:15:21.410 align:middle line:84% The likelihood of a y vector is the product of the 00:15:21.410 --> 00:15:24.240 align:middle line:84% likelihoods of the individual y's. 00:15:24.240 --> 00:15:27.800 align:middle line:84% The likelihood of every individual y is of this form. 00:15:27.800 --> 00:15:33.050 align:middle line:84% Where w is y_i minus these two quantities. 00:15:33.050 --> 00:15:36.000 align:middle line:84% So this is the form that the likelihood function is going 00:15:36.000 --> 00:15:38.880 align:middle line:84% to take under this particular model. 00:15:38.880 --> 00:15:42.260 align:middle line:84% And under the maximum likelihood methodology we want 00:15:42.260 --> 00:15:49.170 align:middle line:84% to maximize this quantity with respect to theta0 and theta1. 00:15:49.170 --> 00:15:56.930 align:middle line:84% Now to do this maximization you might as well consider the 00:15:56.930 --> 00:16:00.990 align:middle line:84% logarithm and maximize the logarithm, which is just the 00:16:00.990 --> 00:16:02.730 align:middle line:90% exponent up here. 00:16:02.730 --> 00:16:05.750 align:middle line:84% Maximizing this exponent because we have a minus sign 00:16:05.750 --> 00:16:08.900 align:middle line:84% is the same as minimizing the exponent 00:16:08.900 --> 00:16:10.840 align:middle line:90% without the minus sign. 00:16:10.840 --> 00:16:12.840 align:middle line:90% Sigma squared is a constant. 00:16:12.840 --> 00:16:17.970 align:middle line:84% So what you end up doing is minimizing this quantity here, 00:16:17.970 --> 00:16:20.120 align:middle line:84% which is the same as what we had in our 00:16:20.120 --> 00:16:23.640 align:middle line:90% linear regression methods. 00:16:23.640 --> 00:16:29.400 align:middle line:84% So in conclusion you might choose to do linear regression 00:16:29.400 --> 00:16:34.490 align:middle line:84% in this particular way, just because it looks 00:16:34.490 --> 00:16:36.210 align:middle line:90% reasonable or plausible. 00:16:36.210 --> 00:16:41.050 align:middle line:84% Or you might interpret what you're doing as maximum 00:16:41.050 --> 00:16:45.220 align:middle line:84% likelihood estimation, in which you assume a model of 00:16:45.220 --> 00:16:49.520 align:middle line:84% this kind where the noise terms are normal random 00:16:49.520 --> 00:16:51.970 align:middle line:84% variables with the same distribution -- 00:16:51.970 --> 00:16:54.540 align:middle line:84% independent identically distributed. 00:16:54.540 --> 00:17:01.320 align:middle line:84% So linear regression implicitly makes an assumption 00:17:01.320 --> 00:17:02.840 align:middle line:90% of this kind. 00:17:02.840 --> 00:17:07.380 align:middle line:84% It's doing maximum likelihood estimation as if the world was 00:17:07.380 --> 00:17:11.000 align:middle line:84% really described by a model of this form, and with the W's 00:17:11.000 --> 00:17:12.560 align:middle line:90% being random variables. 00:17:12.560 --> 00:17:17.920 align:middle line:84% So this gives us at least some justification that this 00:17:17.920 --> 00:17:21.800 align:middle line:84% particular approach to fitting lines to data is not so 00:17:21.800 --> 00:17:25.579 align:middle line:84% arbitrary, but it has a sound footing. 00:17:25.579 --> 00:17:30.530 align:middle line:84% OK so then once you accept this formulation as being a 00:17:30.530 --> 00:17:32.920 align:middle line:84% reasonable one what's the next step? 00:17:32.920 --> 00:17:37.760 align:middle line:84% The next step is to see how to carry out this minimization. 00:17:37.760 --> 00:17:42.220 align:middle line:84% This is not a very difficult minimization to do. 00:17:42.220 --> 00:17:48.260 align:middle line:84% The way it's done is by setting the derivatives of 00:17:48.260 --> 00:17:50.930 align:middle line:90% this expression to 0. 00:17:50.930 --> 00:17:54.500 align:middle line:84% Now because this is a quadratic function of theta0 00:17:54.500 --> 00:17:55.410 align:middle line:90% and theta1-- 00:17:55.410 --> 00:17:57.270 align:middle line:84% when you take the derivatives with respect 00:17:57.270 --> 00:17:58.940 align:middle line:90% to theta0 and theta1-- 00:17:58.940 --> 00:18:03.250 align:middle line:84% you get linear functions of theta0 and theta1. 00:18:03.250 --> 00:18:08.010 align:middle line:84% And you end up solving a system of linear equations in 00:18:08.010 --> 00:18:09.630 align:middle line:90% theta0 and theta1. 00:18:09.630 --> 00:18:15.660 align:middle line:84% And it turns out that there's very nice and simple formulas 00:18:15.660 --> 00:18:18.950 align:middle line:84% for the optimal estimates of the parameters in 00:18:18.950 --> 00:18:20.510 align:middle line:90% terms of the data. 00:18:20.510 --> 00:18:23.910 align:middle line:84% And the formulas are these ones. 00:18:23.910 --> 00:18:28.130 align:middle line:84% I said that these are nice and simple formulas. 00:18:28.130 --> 00:18:29.800 align:middle line:90% Let's see why. 00:18:29.800 --> 00:18:31.270 align:middle line:90% How can we interpret them? 00:18:31.270 --> 00:18:34.050 align:middle line:90% 00:18:34.050 --> 00:18:42.250 align:middle line:84% So suppose that the world is described by a model of this 00:18:42.250 --> 00:18:48.990 align:middle line:84% kind, where the X's and Y's are random variables. 00:18:48.990 --> 00:18:53.920 align:middle line:84% And where W is a noise term that's independent of X. So 00:18:53.920 --> 00:18:57.750 align:middle line:84% we're assuming that a linear model is indeed true, but not 00:18:57.750 --> 00:18:58.530 align:middle line:90% exactly true. 00:18:58.530 --> 00:19:01.790 align:middle line:84% There's always some noise associated with any particular 00:19:01.790 --> 00:19:04.980 align:middle line:90% data point that we obtain. 00:19:04.980 --> 00:19:10.880 align:middle line:84% So if a model of this kind is true, and the W's have 0 mean 00:19:10.880 --> 00:19:15.370 align:middle line:84% then we have that the expected value of Y would be theta0 00:19:15.370 --> 00:19:23.570 align:middle line:84% plus theta1 expected value of X. And because W has 0 mean 00:19:23.570 --> 00:19:26.200 align:middle line:90% there's no extra term. 00:19:26.200 --> 00:19:31.660 align:middle line:84% So in particular, theta0 would be equal to expected value of 00:19:31.660 --> 00:19:37.380 align:middle line:84% Y minus theta1 expected value of X. 00:19:37.380 --> 00:19:40.660 align:middle line:84% So let's use this equation to try to come up with a 00:19:40.660 --> 00:19:44.060 align:middle line:90% reasonable estimate of theta0. 00:19:44.060 --> 00:19:47.220 align:middle line:84% I do not know the expected value of Y, but I 00:19:47.220 --> 00:19:48.430 align:middle line:90% can estimate it. 00:19:48.430 --> 00:19:49.820 align:middle line:90% How do I estimate it? 00:19:49.820 --> 00:19:53.460 align:middle line:84% I look at the average of all the y's that I have obtained. 00:19:53.460 --> 00:19:57.320 align:middle line:84% so I replace this, I estimate it with the average of the 00:19:57.320 --> 00:19:59.940 align:middle line:90% data I have seen. 00:19:59.940 --> 00:20:02.430 align:middle line:90% Here, similarly with the X's. 00:20:02.430 --> 00:20:06.820 align:middle line:84% I might not know the expected value of X's, but I have data 00:20:06.820 --> 00:20:08.520 align:middle line:90% points for the x's. 00:20:08.520 --> 00:20:13.070 align:middle line:84% I look at the average of all my data points, I come up with 00:20:13.070 --> 00:20:16.380 align:middle line:84% an estimate of this expectation. 00:20:16.380 --> 00:20:21.390 align:middle line:84% Now I don't know what theta1 is, but my procedure is going 00:20:21.390 --> 00:20:25.320 align:middle line:84% to generate an estimate of theta1 called theta1 hat. 00:20:25.320 --> 00:20:29.230 align:middle line:84% And once I have this estimate, then a reasonable person would 00:20:29.230 --> 00:20:33.400 align:middle line:84% estimate theta0 in this particular way. 00:20:33.400 --> 00:20:37.320 align:middle line:84% So that's how my estimate of theta0 is going to be 00:20:37.320 --> 00:20:38.490 align:middle line:90% constructed. 00:20:38.490 --> 00:20:41.420 align:middle line:90% It's this formula here. 00:20:41.420 --> 00:20:44.700 align:middle line:84% We have not yet addressed the harder question, which is how 00:20:44.700 --> 00:20:47.490 align:middle line:84% to estimate theta1 in the first place. 00:20:47.490 --> 00:20:50.830 align:middle line:84% So to estimate theta0 I assumed that I already had an 00:20:50.830 --> 00:20:52.180 align:middle line:90% estimate for a theta1. 00:20:52.180 --> 00:20:55.090 align:middle line:90% 00:20:55.090 --> 00:21:02.060 align:middle line:84% OK, the right formula for the estimate of theta1 happens to 00:21:02.060 --> 00:21:03.140 align:middle line:90% be this one. 00:21:03.140 --> 00:21:08.632 align:middle line:84% It looks messy, but let's try to interpret it. 00:21:08.632 --> 00:21:12.970 align:middle line:84% What I'm going to do is I'm going to take this model for 00:21:12.970 --> 00:21:18.340 align:middle line:84% simplicity let's assume that they're the random variables 00:21:18.340 --> 00:21:19.590 align:middle line:90% have 0 means. 00:21:19.590 --> 00:21:22.940 align:middle line:90% 00:21:22.940 --> 00:21:28.800 align:middle line:84% And see how we might estimate how we might 00:21:28.800 --> 00:21:30.960 align:middle line:90% try to estimate theta1. 00:21:30.960 --> 00:21:36.270 align:middle line:84% Let's multiply both sides of this equation by X. So we get 00:21:36.270 --> 00:21:48.470 align:middle line:84% Y times X equals theta0 plus theta0 times X plus theta1 00:21:48.470 --> 00:21:54.530 align:middle line:84% times X-squared, plus X times W. And now take expectations 00:21:54.530 --> 00:21:56.420 align:middle line:90% of both sides. 00:21:56.420 --> 00:22:00.160 align:middle line:84% If I have 0 mean random variables the expected value 00:22:00.160 --> 00:22:07.210 align:middle line:84% of Y times X is just the covariance of X with Y. 00:22:07.210 --> 00:22:10.640 align:middle line:84% I have assumed that my random variables have 0 means, so the 00:22:10.640 --> 00:22:13.680 align:middle line:90% expectation of this is 0. 00:22:13.680 --> 00:22:17.970 align:middle line:84% This one is going to be the variance of X, so I have 00:22:17.970 --> 00:22:23.260 align:middle line:84% theta1 times variance of X. And since I'm assuming that my 00:22:23.260 --> 00:22:26.990 align:middle line:84% random variables have 0 mean, and I'm also assuming that W 00:22:26.990 --> 00:22:32.250 align:middle line:84% is independent of X this last term also has 0 mean. 00:22:32.250 --> 00:22:39.280 align:middle line:84% So under such a probabilistic model this equation is true. 00:22:39.280 --> 00:22:43.620 align:middle line:84% If we knew the variance and the covariance then we would 00:22:43.620 --> 00:22:45.930 align:middle line:90% know the value of theta1. 00:22:45.930 --> 00:22:49.080 align:middle line:84% But we only have data, we do not necessarily know the 00:22:49.080 --> 00:22:53.070 align:middle line:84% variance and the covariance, but we can estimate it. 00:22:53.070 --> 00:22:55.885 align:middle line:84% What's a reasonable estimate of the variance? 00:22:55.885 --> 00:22:59.390 align:middle line:84% The reasonable estimate of the variance is this quantity here 00:22:59.390 --> 00:23:03.195 align:middle line:84% divided by n, and the reasonable estimate of the 00:23:03.195 --> 00:23:06.730 align:middle line:84% covariance is that numerator divided by n. 00:23:06.730 --> 00:23:09.410 align:middle line:90% 00:23:09.410 --> 00:23:11.510 align:middle line:84% So this is my estimate of the mean. 00:23:11.510 --> 00:23:15.390 align:middle line:84% I'm looking at the squared distances from the mean, and I 00:23:15.390 --> 00:23:18.740 align:middle line:84% average them over lots and lots of data. 00:23:18.740 --> 00:23:23.990 align:middle line:84% This is the most reasonable way of estimating the variance 00:23:23.990 --> 00:23:26.070 align:middle line:90% of our distribution. 00:23:26.070 --> 00:23:31.400 align:middle line:84% And similarly the expected value of this quantity is the 00:23:31.400 --> 00:23:35.020 align:middle line:84% covariance of X with Y, and then we have lots and lots of 00:23:35.020 --> 00:23:35.830 align:middle line:90% data points. 00:23:35.830 --> 00:23:38.895 align:middle line:84% This quantity here is going to be a very good estimate of the 00:23:38.895 --> 00:23:40.140 align:middle line:90% covariance. 00:23:40.140 --> 00:23:44.820 align:middle line:84% So basically what this formula does is-- 00:23:44.820 --> 00:23:46.520 align:middle line:90% one way of thinking about it-- 00:23:46.520 --> 00:23:50.870 align:middle line:84% is that it starts from this relation which is true 00:23:50.870 --> 00:23:57.230 align:middle line:84% exactly, but estimates the covariance and the variance on 00:23:57.230 --> 00:24:00.820 align:middle line:84% the basis of the data, and then using these estimates to 00:24:00.820 --> 00:24:05.770 align:middle line:84% come up with an estimate of theta1. 00:24:05.770 --> 00:24:09.890 align:middle line:84% So this gives us a probabilistic interpretation 00:24:09.890 --> 00:24:13.620 align:middle line:84% of the formulas that we have for the way that the estimates 00:24:13.620 --> 00:24:14.990 align:middle line:90% are constructed. 00:24:14.990 --> 00:24:19.560 align:middle line:84% If you're willing to assume that this is the true model of 00:24:19.560 --> 00:24:22.640 align:middle line:84% the world, the structure of the true model of the world, 00:24:22.640 --> 00:24:24.460 align:middle line:84% except that you do not know means and 00:24:24.460 --> 00:24:27.590 align:middle line:90% covariances, and variances. 00:24:27.590 --> 00:24:33.010 align:middle line:84% Then this is a natural way of estimating those unknown 00:24:33.010 --> 00:24:34.260 align:middle line:90% parameters. 00:24:34.260 --> 00:24:36.770 align:middle line:90% 00:24:36.770 --> 00:24:39.800 align:middle line:84% All right, so we have a closed-form formula, we can 00:24:39.800 --> 00:24:43.620 align:middle line:84% apply it whenever we have data. 00:24:43.620 --> 00:24:47.810 align:middle line:84% Now linear regression is a subject on which there are 00:24:47.810 --> 00:24:51.520 align:middle line:84% whole courses, and whole books that are given. 00:24:51.520 --> 00:24:54.560 align:middle line:84% And the reason for that is that there's a lot more that 00:24:54.560 --> 00:24:58.840 align:middle line:84% you can bring into the topic, and many ways that you can 00:24:58.840 --> 00:25:02.350 align:middle line:84% elaborate on the simple solution that we got for the 00:25:02.350 --> 00:25:05.880 align:middle line:84% case of two parameters and only two random variables. 00:25:05.880 --> 00:25:09.550 align:middle line:84% So let me give you a little bit of flavor of what are the 00:25:09.550 --> 00:25:12.950 align:middle line:84% topics that come up when you start looking into linear 00:25:12.950 --> 00:25:14.200 align:middle line:90% regression in more depth. 00:25:14.200 --> 00:25:16.840 align:middle line:90% 00:25:16.840 --> 00:25:24.390 align:middle line:84% So in our discussions so far we made the linear model in 00:25:24.390 --> 00:25:28.370 align:middle line:84% which we're trying to explain the values of one variable in 00:25:28.370 --> 00:25:30.860 align:middle line:84% terms of the values of another variable. 00:25:30.860 --> 00:25:35.010 align:middle line:84% We're trying to explain GPAs in terms of SAT scores, or 00:25:35.010 --> 00:25:39.640 align:middle line:84% we're trying to predict GPAs in terms of SAT scores. 00:25:39.640 --> 00:25:47.910 align:middle line:84% But maybe your GPA is affected by several factors. 00:25:47.910 --> 00:25:56.380 align:middle line:84% For example maybe your GPA is affected by your SAT score, 00:25:56.380 --> 00:26:01.820 align:middle line:84% also the income of your family, the years of education 00:26:01.820 --> 00:26:06.720 align:middle line:84% of your grandmother, and many other factors like that. 00:26:06.720 --> 00:26:11.970 align:middle line:84% So you might write down a model in which I believe that 00:26:11.970 --> 00:26:17.820 align:middle line:84% GPA has a relation, which is a linear function of all these 00:26:17.820 --> 00:26:20.520 align:middle line:84% other variables that I mentioned. 00:26:20.520 --> 00:26:24.350 align:middle line:84% So perhaps you have a theory of what determines performance 00:26:24.350 --> 00:26:29.540 align:middle line:84% at college, and you want to build a model of that type. 00:26:29.540 --> 00:26:31.460 align:middle line:84% How do we go about in this case? 00:26:31.460 --> 00:26:33.830 align:middle line:84% Well, again we collect the data points. 00:26:33.830 --> 00:26:37.980 align:middle line:84% We look at the i-th student, who has a college GPA. 00:26:37.980 --> 00:26:42.090 align:middle line:84% We record their SAT score, their family income, and 00:26:42.090 --> 00:26:45.010 align:middle line:84% grandmother's years of education. 00:26:45.010 --> 00:26:50.390 align:middle line:84% So this is one data point that is for one particular student. 00:26:50.390 --> 00:26:52.580 align:middle line:84% We postulate the model of this form. 00:26:52.580 --> 00:26:56.160 align:middle line:84% For the i-th student this would be the mistake that our 00:26:56.160 --> 00:26:59.940 align:middle line:84% model makes if we have chosen specific values for those 00:26:59.940 --> 00:27:01.070 align:middle line:90% parameters. 00:27:01.070 --> 00:27:05.450 align:middle line:84% And then we go and choose the parameters that are going to 00:27:05.450 --> 00:27:07.950 align:middle line:84% give us, again, the smallest possible 00:27:07.950 --> 00:27:10.000 align:middle line:90% sum of squared errors. 00:27:10.000 --> 00:27:12.360 align:middle line:84% So philosophically it's exactly the same as what we 00:27:12.360 --> 00:27:15.700 align:middle line:84% were discussing before, except that now we're including 00:27:15.700 --> 00:27:19.560 align:middle line:84% multiple explanatory variables in our model instead of a 00:27:19.560 --> 00:27:22.600 align:middle line:90% single explanatory variable. 00:27:22.600 --> 00:27:24.070 align:middle line:90% So that's the formulation. 00:27:24.070 --> 00:27:26.070 align:middle line:90% What do you do next? 00:27:26.070 --> 00:27:29.420 align:middle line:84% Well, to do this minimization you're going to take 00:27:29.420 --> 00:27:32.750 align:middle line:84% derivatives once you have your data, you have a function of 00:27:32.750 --> 00:27:34.310 align:middle line:90% these three parameters. 00:27:34.310 --> 00:27:37.190 align:middle line:84% You take the derivative with respect to the parameter, set 00:27:37.190 --> 00:27:39.170 align:middle line:84% the derivative equal to 0, you get the 00:27:39.170 --> 00:27:41.060 align:middle line:90% system of linear equations. 00:27:41.060 --> 00:27:43.450 align:middle line:84% You throw that system of linear equations to the 00:27:43.450 --> 00:27:46.260 align:middle line:84% computer, and you get numerical values for the 00:27:46.260 --> 00:27:48.060 align:middle line:90% optimal parameters. 00:27:48.060 --> 00:27:52.130 align:middle line:84% There are no nice closed-form formulas of the type that we 00:27:52.130 --> 00:27:54.510 align:middle line:84% had in the previous slide when you're dealing 00:27:54.510 --> 00:27:56.230 align:middle line:90% with multiple variables. 00:27:56.230 --> 00:28:02.240 align:middle line:84% Unless you're willing to go into matrix notation. 00:28:02.240 --> 00:28:04.760 align:middle line:84% In that case you can again write down closed-form 00:28:04.760 --> 00:28:07.290 align:middle line:84% formulas, but they will be a little less intuitive than 00:28:07.290 --> 00:28:09.210 align:middle line:90% what we had before. 00:28:09.210 --> 00:28:13.550 align:middle line:84% But the moral of the story is that numerically this is a 00:28:13.550 --> 00:28:16.480 align:middle line:90% procedure that's very easy. 00:28:16.480 --> 00:28:18.780 align:middle line:84% It's a problem, an optimization problem that the 00:28:18.780 --> 00:28:20.680 align:middle line:90% computer can solve for you. 00:28:20.680 --> 00:28:23.290 align:middle line:84% And it can solve it for you very quickly. 00:28:23.290 --> 00:28:25.470 align:middle line:84% Because all that it involves is solving a 00:28:25.470 --> 00:28:26.720 align:middle line:90% system of linear equations. 00:28:26.720 --> 00:28:29.590 align:middle line:90% 00:28:29.590 --> 00:28:34.270 align:middle line:84% Now when you choose your explanatory variables you may 00:28:34.270 --> 00:28:37.940 align:middle line:90% have some choices. 00:28:37.940 --> 00:28:43.550 align:middle line:84% One person may think that your GPA a has something to do with 00:28:43.550 --> 00:28:45.340 align:middle line:90% your SAT score. 00:28:45.340 --> 00:28:48.480 align:middle line:84% Some other person may think that your GPA has something to 00:28:48.480 --> 00:28:51.800 align:middle line:84% do with the square of your SAT score. 00:28:51.800 --> 00:28:55.380 align:middle line:84% And that other person may want to try to build a 00:28:55.380 --> 00:28:58.840 align:middle line:90% model of this kind. 00:28:58.840 --> 00:29:01.550 align:middle line:84% Now when would you want to do this? ? 00:29:01.550 --> 00:29:07.830 align:middle line:84% Suppose that the data that you have looks like this. 00:29:07.830 --> 00:29:12.177 align:middle line:90% 00:29:12.177 --> 00:29:15.740 align:middle line:84% If the data looks like this then you might be tempted to 00:29:15.740 --> 00:29:20.710 align:middle line:84% say well a linear model does not look right, but maybe a 00:29:20.710 --> 00:29:25.650 align:middle line:84% quadratic model will give me a better fit for the data. 00:29:25.650 --> 00:29:30.690 align:middle line:84% So if you want to fit a quadratic model to the data 00:29:30.690 --> 00:29:35.550 align:middle line:84% then what you do is you take X-squared as your explanatory 00:29:35.550 --> 00:29:42.520 align:middle line:84% variable instead of X, and you build a model of this kind. 00:29:42.520 --> 00:29:45.910 align:middle line:84% There's nothing really different in models of this 00:29:45.910 --> 00:29:48.830 align:middle line:84% kind compared to models of that kind. 00:29:48.830 --> 00:29:54.700 align:middle line:84% They are still linear models because we have theta's 00:29:54.700 --> 00:29:57.630 align:middle line:84% showing up in a linear fashion. 00:29:57.630 --> 00:30:00.460 align:middle line:84% What you take as your explanatory variables, whether 00:30:00.460 --> 00:30:02.870 align:middle line:84% it's X, whether it's X-squared, or whether it's 00:30:02.870 --> 00:30:05.390 align:middle line:84% some other function that you chose. 00:30:05.390 --> 00:30:09.590 align:middle line:84% Some general function h of X, doesn't make a difference. 00:30:09.590 --> 00:30:14.470 align:middle line:84% So think of you h of X as being your new X. So you can 00:30:14.470 --> 00:30:17.620 align:middle line:84% formulate the problem exactly the same way, except that 00:30:17.620 --> 00:30:21.035 align:middle line:84% instead of using X's you choose h of X's. 00:30:21.035 --> 00:30:23.610 align:middle line:90% 00:30:23.610 --> 00:30:26.540 align:middle line:84% So it's basically a question do I want to build a model 00:30:26.540 --> 00:30:31.390 align:middle line:84% that explains Y's based on the values of X, or do I want to 00:30:31.390 --> 00:30:35.190 align:middle line:84% build a model that explains Y's on the basis of the values 00:30:35.190 --> 00:30:38.970 align:middle line:84% of h of X. Which is the right value to use? 00:30:38.970 --> 00:30:42.160 align:middle line:84% And with this picture here, we see that it can make a 00:30:42.160 --> 00:30:43.160 align:middle line:90% difference. 00:30:43.160 --> 00:30:47.070 align:middle line:84% A linear model in X might be a poor fit, but a quadratic 00:30:47.070 --> 00:30:49.660 align:middle line:84% model might give us a better fit. 00:30:49.660 --> 00:30:55.450 align:middle line:84% So this brings to the topic of how to choose your functions h 00:30:55.450 --> 00:30:59.480 align:middle line:84% of X if you're dealing with a real world problem. 00:30:59.480 --> 00:31:03.080 align:middle line:84% So in a real world problem you're just given X's and Y's. 00:31:03.080 --> 00:31:05.990 align:middle line:84% And you have the freedom of building models of 00:31:05.990 --> 00:31:07.120 align:middle line:90% any kind you want. 00:31:07.120 --> 00:31:11.330 align:middle line:84% You have the freedom of choosing a function h of X of 00:31:11.330 --> 00:31:13.130 align:middle line:90% any type that you want. 00:31:13.130 --> 00:31:14.980 align:middle line:90% So this turns out to be a quite 00:31:14.980 --> 00:31:18.800 align:middle line:90% difficult and tricky topic. 00:31:18.800 --> 00:31:22.630 align:middle line:84% Because you may be tempted to overdo it. 00:31:22.630 --> 00:31:28.450 align:middle line:84% For example, I got my 10 data points, and I could say OK, 00:31:28.450 --> 00:31:35.660 align:middle line:84% I'm going to choose an h of X. I'm going to choose h of X and 00:31:35.660 --> 00:31:40.300 align:middle line:84% actually multiple h's of X to do a multiple linear 00:31:40.300 --> 00:31:45.030 align:middle line:84% regression in which I'm going to build a model that's uses a 00:31:45.030 --> 00:31:47.600 align:middle line:90% 10th degree polynomial. 00:31:47.600 --> 00:31:51.160 align:middle line:84% If I choose to fit my data with a 10th degree polynomial 00:31:51.160 --> 00:31:54.680 align:middle line:84% I'm going to fit my data perfectly, but I may obtain a 00:31:54.680 --> 00:31:58.530 align:middle line:84% model is does something like this, and goes through all my 00:31:58.530 --> 00:31:59.930 align:middle line:90% data points. 00:31:59.930 --> 00:32:03.830 align:middle line:84% So I can make my prediction errors extremely small if I 00:32:03.830 --> 00:32:08.820 align:middle line:84% use lots of parameters, and if I choose my h functions 00:32:08.820 --> 00:32:09.930 align:middle line:90% appropriately. 00:32:09.930 --> 00:32:11.800 align:middle line:84% But clearly this would be garbage. 00:32:11.800 --> 00:32:15.270 align:middle line:84% If you get those data points, and you say here's my model 00:32:15.270 --> 00:32:16.420 align:middle line:90% that explains them. 00:32:16.420 --> 00:32:21.320 align:middle line:84% That has a polynomial going up and down, then you're probably 00:32:21.320 --> 00:32:22.900 align:middle line:90% doing something wrong. 00:32:22.900 --> 00:32:26.180 align:middle line:84% So choosing how complicated those functions, 00:32:26.180 --> 00:32:27.900 align:middle line:90% the h's, should be. 00:32:27.900 --> 00:32:32.020 align:middle line:84% And how many explanatory variables to use is a very 00:32:32.020 --> 00:32:36.560 align:middle line:84% delicate and deep topic on which there's deep theory that 00:32:36.560 --> 00:32:39.910 align:middle line:84% tells you what you should do, and what you shouldn't do. 00:32:39.910 --> 00:32:43.830 align:middle line:84% But the main thing that one should avoid doing is having 00:32:43.830 --> 00:32:46.620 align:middle line:84% too many parameters in your model when you 00:32:46.620 --> 00:32:48.900 align:middle line:90% have too few data. 00:32:48.900 --> 00:32:52.350 align:middle line:84% So if you only have 10 data points, you shouldn't have 10 00:32:52.350 --> 00:32:53.350 align:middle line:90% free parameters. 00:32:53.350 --> 00:32:56.150 align:middle line:84% With 10 free parameters you will be able to fit your data 00:32:56.150 --> 00:33:00.760 align:middle line:84% perfectly, but you wouldn't be able to really rely on the 00:33:00.760 --> 00:33:02.010 align:middle line:90% results that you are seeing. 00:33:02.010 --> 00:33:06.050 align:middle line:90% 00:33:06.050 --> 00:33:12.630 align:middle line:84% OK, now in practice, when people run linear regressions 00:33:12.630 --> 00:33:15.410 align:middle line:84% they do not just give point estimates for 00:33:15.410 --> 00:33:17.370 align:middle line:90% the parameters theta. 00:33:17.370 --> 00:33:20.300 align:middle line:84% But similar to what we did for the case of estimating the 00:33:20.300 --> 00:33:23.790 align:middle line:84% mean of a random variable you might want to give confidence 00:33:23.790 --> 00:33:27.200 align:middle line:84% intervals that sort of tell you how much randomness there 00:33:27.200 --> 00:33:30.730 align:middle line:84% is when you estimate each one of the particular parameters. 00:33:30.730 --> 00:33:33.960 align:middle line:84% There are formulas for building confidence intervals 00:33:33.960 --> 00:33:36.230 align:middle line:84% for the estimates of the theta's. 00:33:36.230 --> 00:33:38.520 align:middle line:84% We're not going to look at them, it would 00:33:38.520 --> 00:33:39.990 align:middle line:90% take too much time. 00:33:39.990 --> 00:33:44.600 align:middle line:84% Also you might want to estimate the variance in the 00:33:44.600 --> 00:33:47.400 align:middle line:84% noise that you have in your model. 00:33:47.400 --> 00:33:52.540 align:middle line:84% That is if you are pretending that your true model is of the 00:33:52.540 --> 00:33:57.026 align:middle line:84% kind we were discussing before, namely Y equals theta1 00:33:57.026 --> 00:34:02.190 align:middle line:84% times X plus W, and W has a variance sigma squared. 00:34:02.190 --> 00:34:05.170 align:middle line:84% You might want to estimate this, because it tells you 00:34:05.170 --> 00:34:09.199 align:middle line:84% something about the model, and this is called standard error. 00:34:09.199 --> 00:34:11.929 align:middle line:84% It puts a limit on how good predictions 00:34:11.929 --> 00:34:14.730 align:middle line:90% your model can make. 00:34:14.730 --> 00:34:18.170 align:middle line:84% Even if you have the correct theta0 and theta1, and 00:34:18.170 --> 00:34:22.179 align:middle line:84% somebody tells you X you can make a prediction about Y, but 00:34:22.179 --> 00:34:24.710 align:middle line:84% that prediction will not be accurate. 00:34:24.710 --> 00:34:26.739 align:middle line:84% Because there's this additional randomness. 00:34:26.739 --> 00:34:29.699 align:middle line:84% And if that additional randomness is big, then your 00:34:29.699 --> 00:34:33.810 align:middle line:84% predictions will also have a substantial error in them. 00:34:33.810 --> 00:34:38.300 align:middle line:84% There's another quantity that gets reported usually. 00:34:38.300 --> 00:34:41.400 align:middle line:84% This is part of the computer output that you get when you 00:34:41.400 --> 00:34:45.500 align:middle line:84% use a statistical package which is called R-square. 00:34:45.500 --> 00:34:49.920 align:middle line:84% And its a measure of the explanatory power of the model 00:34:49.920 --> 00:34:52.469 align:middle line:84% that you have built linear regression. 00:34:52.469 --> 00:34:55.650 align:middle line:90% Using linear regression. 00:34:55.650 --> 00:35:01.030 align:middle line:84% Instead of defining R-square exactly, let me give you a 00:35:01.030 --> 00:35:05.170 align:middle line:84% sort of analogous quantity that's involved. 00:35:05.170 --> 00:35:08.030 align:middle line:84% After you do your linear regression you can look at the 00:35:08.030 --> 00:35:10.600 align:middle line:90% following quantity. 00:35:10.600 --> 00:35:15.720 align:middle line:84% You look at the variance of Y, which is something that you 00:35:15.720 --> 00:35:17.400 align:middle line:90% can estimate from data. 00:35:17.400 --> 00:35:23.370 align:middle line:84% This is how much randomness there is in Y. And compare it 00:35:23.370 --> 00:35:28.090 align:middle line:84% with the randomness that you have in Y, but conditioned on 00:35:28.090 --> 00:35:35.840 align:middle line:84% X. So this quantity tells me if I knew X how much 00:35:35.840 --> 00:35:39.820 align:middle line:84% randomness would there still be in my Y? 00:35:39.820 --> 00:35:43.650 align:middle line:84% So if I know X, I have more information, so Y is more 00:35:43.650 --> 00:35:44.390 align:middle line:90% constrained. 00:35:44.390 --> 00:35:48.640 align:middle line:84% There's less randomness in Y. This is the randomness in Y if 00:35:48.640 --> 00:35:50.790 align:middle line:90% I don't know anything about X. 00:35:50.790 --> 00:35:54.855 align:middle line:84% So naturally this quantity would be less than 1, and if 00:35:54.855 --> 00:35:58.830 align:middle line:84% this quantity is small it would mean that whenever I 00:35:58.830 --> 00:36:03.320 align:middle line:84% know X then Y is very well known. 00:36:03.320 --> 00:36:07.440 align:middle line:84% Which essentially tells me that knowing x allows me to 00:36:07.440 --> 00:36:12.370 align:middle line:84% make very good predictions about Y. Knowing X means that 00:36:12.370 --> 00:36:17.390 align:middle line:84% I'm explaining away most of the randomness in Y. 00:36:17.390 --> 00:36:22.590 align:middle line:84% So if you read a statistical study that uses linear 00:36:22.590 --> 00:36:29.730 align:middle line:84% regression you might encounter statements of the form 60% of 00:36:29.730 --> 00:36:36.140 align:middle line:84% a student's GPA is explained by the family income. 00:36:36.140 --> 00:36:40.600 align:middle line:84% If you read the statements of this kind it's really refers 00:36:40.600 --> 00:36:43.160 align:middle line:90% to quantities of this kind. 00:36:43.160 --> 00:36:47.820 align:middle line:84% Out of the total variance in Y, how much variance is left 00:36:47.820 --> 00:36:50.060 align:middle line:90% after we build our model? 00:36:50.060 --> 00:36:56.490 align:middle line:84% So if only 40% of the variance of Y is left after we build 00:36:56.490 --> 00:37:00.700 align:middle line:84% our model, that means that X explains 60% of the 00:37:00.700 --> 00:37:02.510 align:middle line:90% variations in Y's. 00:37:02.510 --> 00:37:06.570 align:middle line:84% So the idea is that randomness in Y is 00:37:06.570 --> 00:37:09.560 align:middle line:90% caused by multiple sources. 00:37:09.560 --> 00:37:12.025 align:middle line:84% Our explanatory variable and random noise. 00:37:12.025 --> 00:37:15.610 align:middle line:84% And we ask the question what percentage of the total 00:37:15.610 --> 00:37:19.940 align:middle line:90% randomness in Y is explained by 00:37:19.940 --> 00:37:23.030 align:middle line:90% variations in the X parameter? 00:37:23.030 --> 00:37:26.860 align:middle line:84% And how much of the total randomness in Y is attributed 00:37:26.860 --> 00:37:30.390 align:middle line:90% just to random effects? 00:37:30.390 --> 00:37:34.050 align:middle line:84% So if you have a model that explains most of the variation 00:37:34.050 --> 00:37:37.710 align:middle line:84% in Y then you can think that you have a good model that 00:37:37.710 --> 00:37:42.550 align:middle line:84% tells you something useful about the real world. 00:37:42.550 --> 00:37:45.990 align:middle line:84% Now there's lots of things that can go wrong when you use 00:37:45.990 --> 00:37:50.670 align:middle line:84% linear regression, and there's many pitfalls. 00:37:50.670 --> 00:37:56.440 align:middle line:84% One pitfall happens when you have this situation that's 00:37:56.440 --> 00:37:58.300 align:middle line:90% called heteroskedacisity. 00:37:58.300 --> 00:38:01.020 align:middle line:84% So suppose your data are of this kind. 00:38:01.020 --> 00:38:06.550 align:middle line:90% 00:38:06.550 --> 00:38:09.330 align:middle line:90% So what's happening here? 00:38:09.330 --> 00:38:17.640 align:middle line:84% You seem to have a linear model, but when X is small you 00:38:17.640 --> 00:38:19.200 align:middle line:90% have a very good model. 00:38:19.200 --> 00:38:23.830 align:middle line:84% So this means that W has a small variance when X is here. 00:38:23.830 --> 00:38:26.760 align:middle line:84% On the other hand, when X is there you have a lot of 00:38:26.760 --> 00:38:27.970 align:middle line:90% randomness. 00:38:27.970 --> 00:38:32.080 align:middle line:84% This would be a situation in which the W's are not 00:38:32.080 --> 00:38:35.840 align:middle line:84% identically distributed, but the variance of the W's, of 00:38:35.840 --> 00:38:40.360 align:middle line:84% the noise, has something to do with the X's. 00:38:40.360 --> 00:38:43.720 align:middle line:84% So with different regions of our x-space we have different 00:38:43.720 --> 00:38:45.260 align:middle line:90% amounts of noise. 00:38:45.260 --> 00:38:47.615 align:middle line:84% What will go wrong in this situation? 00:38:47.615 --> 00:38:51.290 align:middle line:84% Since we're trying to minimize sum of squared errors, we're 00:38:51.290 --> 00:38:54.080 align:middle line:84% really paying attention to the biggest errors. 00:38:54.080 --> 00:38:57.010 align:middle line:84% Which will mean that we are going to pay attention to 00:38:57.010 --> 00:38:59.690 align:middle line:84% these data points, because that's where the big errors 00:38:59.690 --> 00:39:01.130 align:middle line:90% are going to be. 00:39:01.130 --> 00:39:04.250 align:middle line:84% So the linear regression formulas will end up building 00:39:04.250 --> 00:39:09.110 align:middle line:84% a model based on these data, which are the most noisy ones. 00:39:09.110 --> 00:39:14.810 align:middle line:84% Instead of those data that are nicely stacked in order. 00:39:14.810 --> 00:39:17.410 align:middle line:84% Clearly that's not to the right thing to do. 00:39:17.410 --> 00:39:21.500 align:middle line:84% So you need to change something, and use the fact 00:39:21.500 --> 00:39:25.800 align:middle line:84% that the variance of W changes with the X's, and there are 00:39:25.800 --> 00:39:27.770 align:middle line:90% ways of dealing with it. 00:39:27.770 --> 00:39:31.280 align:middle line:84% It's something that one needs to be careful about. 00:39:31.280 --> 00:39:34.580 align:middle line:84% Another possibility of getting into trouble is if you're 00:39:34.580 --> 00:39:38.550 align:middle line:84% using multiple explanatory variables that are very 00:39:38.550 --> 00:39:41.330 align:middle line:90% closely related to each other. 00:39:41.330 --> 00:39:47.500 align:middle line:84% So for example, suppose that I tried to predict your GPA by 00:39:47.500 --> 00:39:54.100 align:middle line:84% looking at your SAT the first time that you took it plus 00:39:54.100 --> 00:39:58.290 align:middle line:84% your SAT the second time that you took your SATs. 00:39:58.290 --> 00:40:00.470 align:middle line:84% I'm assuming that almost everyone takes the 00:40:00.470 --> 00:40:02.450 align:middle line:90% SAT more than once. 00:40:02.450 --> 00:40:05.630 align:middle line:84% So suppose that you had a model of this kind. 00:40:05.630 --> 00:40:09.380 align:middle line:84% Well, SAT on your first try and SAT on your second try are 00:40:09.380 --> 00:40:12.480 align:middle line:84% very likely to be fairly close. 00:40:12.480 --> 00:40:17.570 align:middle line:84% And you could think of coming up with estimates in which 00:40:17.570 --> 00:40:19.390 align:middle line:90% this is ignored. 00:40:19.390 --> 00:40:22.780 align:middle line:84% And you build a model based on this, or an alternative model 00:40:22.780 --> 00:40:25.810 align:middle line:84% in which this term is ignored, and you make predictions based 00:40:25.810 --> 00:40:27.430 align:middle line:90% on the second SAT. 00:40:27.430 --> 00:40:31.840 align:middle line:84% And both models are likely to be essentially as good as the 00:40:31.840 --> 00:40:34.430 align:middle line:84% other one, because these two quantities are 00:40:34.430 --> 00:40:36.630 align:middle line:90% essentially the same. 00:40:36.630 --> 00:40:41.440 align:middle line:84% So in that case, your theta's that you estimate are going to 00:40:41.440 --> 00:40:44.880 align:middle line:84% be very sensitive to little details of the data. 00:40:44.880 --> 00:40:48.560 align:middle line:84% You change your data, you have your data, and your data tell 00:40:48.560 --> 00:40:52.170 align:middle line:84% you that this coefficient is big and that 00:40:52.170 --> 00:40:52.760 align:middle line:90% coefficient is small. 00:40:52.760 --> 00:40:56.060 align:middle line:84% You change your data just a tiny bit, and your theta's 00:40:56.060 --> 00:40:57.720 align:middle line:90% would drastically change. 00:40:57.720 --> 00:41:00.750 align:middle line:84% So this is a case in which you have multiple explanatory 00:41:00.750 --> 00:41:04.110 align:middle line:84% variables, but they're redundant in the sense that 00:41:04.110 --> 00:41:07.300 align:middle line:84% they're very closely related to each other, and perhaps 00:41:07.300 --> 00:41:08.830 align:middle line:90% with a linear relation. 00:41:08.830 --> 00:41:11.980 align:middle line:84% So one must be careful about the situation, and do special 00:41:11.980 --> 00:41:15.940 align:middle line:84% tests to make sure that this doesn't happen. 00:41:15.940 --> 00:41:20.900 align:middle line:84% Finally the biggest and most common blunder is that you run 00:41:20.900 --> 00:41:24.910 align:middle line:84% your linear regression, you get your linear model, and 00:41:24.910 --> 00:41:26.760 align:middle line:90% then you say oh, OK. 00:41:26.760 --> 00:41:33.340 align:middle line:84% Y is caused by X according to this particular formula. 00:41:33.340 --> 00:41:36.940 align:middle line:84% Well, all that we did was to identify a linear relation 00:41:36.940 --> 00:41:40.120 align:middle line:84% between X and Y. This doesn't tell us anything. 00:41:40.120 --> 00:41:44.130 align:middle line:84% Whether it's Y that causes X, or whether it's X that causes 00:41:44.130 --> 00:41:48.850 align:middle line:84% Y, or maybe both X and Y are caused by some other variable 00:41:48.850 --> 00:41:51.110 align:middle line:90% that we didn't think about. 00:41:51.110 --> 00:41:56.800 align:middle line:84% So building a good linear model that has small errors 00:41:56.800 --> 00:42:00.980 align:middle line:84% does not tell us anything about causal relations between 00:42:00.980 --> 00:42:02.320 align:middle line:90% the two variables. 00:42:02.320 --> 00:42:05.210 align:middle line:84% It only tells us that there's a close association between 00:42:05.210 --> 00:42:06.010 align:middle line:90% the two variables. 00:42:06.010 --> 00:42:10.370 align:middle line:84% If you know one you can make predictions about the other. 00:42:10.370 --> 00:42:13.370 align:middle line:84% But it doesn't tell you anything about the underlying 00:42:13.370 --> 00:42:18.120 align:middle line:84% physics, that there's some physical mechanism that 00:42:18.120 --> 00:42:22.310 align:middle line:84% introduces the relation between those variables. 00:42:22.310 --> 00:42:26.430 align:middle line:84% OK, that's it about linear regression. 00:42:26.430 --> 00:42:30.510 align:middle line:84% Let us start the next topic, which is hypothesis testing. 00:42:30.510 --> 00:42:35.140 align:middle line:84% And we're going to continue with it next time. 00:42:35.140 --> 00:42:37.780 align:middle line:84% So here, instead of trying to estimate continuous 00:42:37.780 --> 00:42:41.920 align:middle line:84% parameters, we have two alternative hypotheses about 00:42:41.920 --> 00:42:46.550 align:middle line:84% the distribution of the X random variable. 00:42:46.550 --> 00:42:53.620 align:middle line:84% So for example our random variable could be either 00:42:53.620 --> 00:42:58.480 align:middle line:84% distributed according to this distribution, under H0, or it 00:42:58.480 --> 00:43:02.930 align:middle line:84% might be distributed according to this distribution under H1. 00:43:02.930 --> 00:43:06.230 align:middle line:84% And we want to make a decision which distribution is the 00:43:06.230 --> 00:43:07.990 align:middle line:90% correct one? 00:43:07.990 --> 00:43:10.850 align:middle line:84% So we're given those two distributions, and some common 00:43:10.850 --> 00:43:14.290 align:middle line:84% terminologies that one of them is the null hypothesis-- 00:43:14.290 --> 00:43:16.600 align:middle line:84% sort of the default hypothesis, and we have some 00:43:16.600 --> 00:43:18.290 align:middle line:90% alternative hypotheses-- 00:43:18.290 --> 00:43:20.560 align:middle line:84% and we want to check whether this one is true, 00:43:20.560 --> 00:43:21.950 align:middle line:90% or that one is true. 00:43:21.950 --> 00:43:24.500 align:middle line:84% So you obtain a data point, and you 00:43:24.500 --> 00:43:26.060 align:middle line:90% want to make a decision. 00:43:26.060 --> 00:43:28.820 align:middle line:84% In this picture what would a reasonable person 00:43:28.820 --> 00:43:30.650 align:middle line:90% do to make a decision? 00:43:30.650 --> 00:43:35.500 align:middle line:84% They would probably choose a certain threshold, Xi, and 00:43:35.500 --> 00:43:43.540 align:middle line:84% decide that H1 is true if your data falls in this interval. 00:43:43.540 --> 00:43:49.590 align:middle line:84% And decide that H0 is true if you fall on the side. 00:43:49.590 --> 00:43:51.660 align:middle line:84% So that would be a reasonable way of 00:43:51.660 --> 00:43:54.100 align:middle line:90% approaching the problem. 00:43:54.100 --> 00:43:59.160 align:middle line:84% More generally you take the set of all possible X's, and 00:43:59.160 --> 00:44:03.050 align:middle line:84% you divide the set of possible X's into two regions. 00:44:03.050 --> 00:44:11.110 align:middle line:84% One is the rejection region, in which you decide H1, 00:44:11.110 --> 00:44:13.170 align:middle line:90% or you reject H0. 00:44:13.170 --> 00:44:15.760 align:middle line:90% 00:44:15.760 --> 00:44:21.640 align:middle line:84% And the complement of that region is where you decide H0. 00:44:21.640 --> 00:44:25.210 align:middle line:84% So this is the x-space of your data. 00:44:25.210 --> 00:44:28.350 align:middle line:84% In this example here, x was one-dimensional. 00:44:28.350 --> 00:44:31.770 align:middle line:84% But in general X is going to be a vector, where all the 00:44:31.770 --> 00:44:34.790 align:middle line:84% possible data vectors that you can get, they're 00:44:34.790 --> 00:44:36.600 align:middle line:90% divided into two types. 00:44:36.600 --> 00:44:40.400 align:middle line:84% If it falls in this set you'd make one decision. 00:44:40.400 --> 00:44:43.770 align:middle line:84% If it falls in that set, you make the other decision. 00:44:43.770 --> 00:44:47.380 align:middle line:84% OK, so how would you characterize the performance 00:44:47.380 --> 00:44:49.690 align:middle line:84% of the particular way of making a decision? 00:44:49.690 --> 00:44:53.000 align:middle line:90% Suppose I chose my threshold. 00:44:53.000 --> 00:44:57.960 align:middle line:84% I may make mistakes of two possible types. 00:44:57.960 --> 00:45:03.360 align:middle line:84% Perhaps H0 is true, but my data happens to fall here. 00:45:03.360 --> 00:45:07.560 align:middle line:84% In which case I make a mistake, and this would be a 00:45:07.560 --> 00:45:10.730 align:middle line:90% false rejection of H0. 00:45:10.730 --> 00:45:15.070 align:middle line:84% If my data falls here I reject H0. 00:45:15.070 --> 00:45:16.890 align:middle line:90% I decide H1. 00:45:16.890 --> 00:45:19.510 align:middle line:90% Whereas H0 was true. 00:45:19.510 --> 00:45:21.690 align:middle line:84% The probability of this happening? 00:45:21.690 --> 00:45:24.890 align:middle line:90% Let's call it alpha. 00:45:24.890 --> 00:45:28.040 align:middle line:84% But there's another kind of error that can be made. 00:45:28.040 --> 00:45:32.810 align:middle line:84% Suppose that H1 was true, but by accident my data happens to 00:45:32.810 --> 00:45:34.250 align:middle line:90% falls on that side. 00:45:34.250 --> 00:45:36.610 align:middle line:84% Then I'm going to make an error again. 00:45:36.610 --> 00:45:40.540 align:middle line:84% I'm going to decide H0 even though H1 was true. 00:45:40.540 --> 00:45:42.570 align:middle line:90% How likely is this to occur? 00:45:42.570 --> 00:45:46.420 align:middle line:84% This would be the area under this curve here. 00:45:46.420 --> 00:45:50.600 align:middle line:84% And that's the other type of error than can be made, and 00:45:50.600 --> 00:45:55.400 align:middle line:84% beta is the probability of this particular type of error. 00:45:55.400 --> 00:45:57.550 align:middle line:90% Both of these are errors. 00:45:57.550 --> 00:45:59.640 align:middle line:84% Alpha is the probability of error of one kind. 00:45:59.640 --> 00:46:02.110 align:middle line:84% Beta is the probability of an error of the other kind. 00:46:02.110 --> 00:46:03.510 align:middle line:84% You would like the probabilities 00:46:03.510 --> 00:46:05.050 align:middle line:90% of error to be small. 00:46:05.050 --> 00:46:07.550 align:middle line:84% So you would like to make both alpha and 00:46:07.550 --> 00:46:09.780 align:middle line:90% beta as small as possible. 00:46:09.780 --> 00:46:13.300 align:middle line:84% Unfortunately that's not possible, there's a trade-off. 00:46:13.300 --> 00:46:17.540 align:middle line:84% If I go to my threshold it this way, then alpha become 00:46:17.540 --> 00:46:20.760 align:middle line:84% smaller, but beta becomes bigger. 00:46:20.760 --> 00:46:22.770 align:middle line:90% So there's a trade-off. 00:46:22.770 --> 00:46:29.350 align:middle line:84% If I make my rejection region smaller one kind of error is 00:46:29.350 --> 00:46:31.880 align:middle line:84% less likely, but the other kind of error 00:46:31.880 --> 00:46:34.670 align:middle line:90% becomes more likely. 00:46:34.670 --> 00:46:38.050 align:middle line:90% So we got this trade-off. 00:46:38.050 --> 00:46:39.620 align:middle line:90% So what do we do about it? 00:46:39.620 --> 00:46:41.570 align:middle line:90% How do we move systematically? 00:46:41.570 --> 00:46:45.680 align:middle line:84% How do we come up with rejection regions? 00:46:45.680 --> 00:46:48.900 align:middle line:84% Well, what the theory basically tells you is it 00:46:48.900 --> 00:46:53.200 align:middle line:84% tells you how you should create those regions. 00:46:53.200 --> 00:46:57.860 align:middle line:84% But it doesn't tell you exactly how. 00:46:57.860 --> 00:47:00.970 align:middle line:84% It tells you the general shape of those regions. 00:47:00.970 --> 00:47:05.120 align:middle line:84% For example here, the theory who tells us that the right 00:47:05.120 --> 00:47:07.430 align:middle line:84% thing to do would be to put the threshold and make 00:47:07.430 --> 00:47:10.910 align:middle line:84% decisions one way to the right, one way to the left. 00:47:10.910 --> 00:47:12.830 align:middle line:84% But it might not necessarily tell us 00:47:12.830 --> 00:47:15.020 align:middle line:90% where to put the threshold. 00:47:15.020 --> 00:47:18.890 align:middle line:84% Still, it's useful enough to know that the way to make a 00:47:18.890 --> 00:47:20.960 align:middle line:84% good decision would be in terms of 00:47:20.960 --> 00:47:22.400 align:middle line:90% a particular threshold. 00:47:22.400 --> 00:47:24.770 align:middle line:84% Let me make this more specific. 00:47:24.770 --> 00:47:27.380 align:middle line:84% We can take our inspiration from the solution of the 00:47:27.380 --> 00:47:29.820 align:middle line:84% hypothesis testing problem that we had in 00:47:29.820 --> 00:47:31.370 align:middle line:90% the Bayesian case. 00:47:31.370 --> 00:47:34.130 align:middle line:84% In the Bayesian case we just pick the hypothesis which is 00:47:34.130 --> 00:47:37.480 align:middle line:90% more likely given the data. 00:47:37.480 --> 00:47:40.080 align:middle line:84% The produced posterior probabilities using Bayesian 00:47:40.080 --> 00:47:42.770 align:middle line:84% rule, they're written this way. 00:47:42.770 --> 00:47:45.240 align:middle line:84% And this term is the same as that term. 00:47:45.240 --> 00:47:49.500 align:middle line:84% They cancel out, then let me collect terms here and there. 00:47:49.500 --> 00:47:52.370 align:middle line:90% 00:47:52.370 --> 00:47:54.030 align:middle line:90% I get an expression here. 00:47:54.030 --> 00:47:56.090 align:middle line:84% I think the version you have in your handout 00:47:56.090 --> 00:47:57.340 align:middle line:90% is the correct one. 00:47:57.340 --> 00:47:59.810 align:middle line:90% 00:47:59.810 --> 00:48:02.082 align:middle line:84% The one on the slide was not the correct one, so 00:48:02.082 --> 00:48:03.730 align:middle line:90% I'm fixing it here. 00:48:03.730 --> 00:48:06.920 align:middle line:84% OK, so this is the form of how you make decisions in the 00:48:06.920 --> 00:48:08.720 align:middle line:90% Bayesian case. 00:48:08.720 --> 00:48:10.620 align:middle line:84% What you do in the Bayesian case, you 00:48:10.620 --> 00:48:13.270 align:middle line:90% calculate this ratio. 00:48:13.270 --> 00:48:17.110 align:middle line:84% Let's call it the likelihood ratio. 00:48:17.110 --> 00:48:20.770 align:middle line:84% And compare that ratio to a threshold. 00:48:20.770 --> 00:48:22.916 align:middle line:84% And the threshold that you should be using in the 00:48:22.916 --> 00:48:25.240 align:middle line:84% Bayesian case has something to do with the prior 00:48:25.240 --> 00:48:28.000 align:middle line:84% probabilities of the two hypotheses. 00:48:28.000 --> 00:48:31.840 align:middle line:84% In the non-Bayesian case we do not have prior probabilities, 00:48:31.840 --> 00:48:34.690 align:middle line:84% so we do not know how to set this threshold. 00:48:34.690 --> 00:48:38.350 align:middle line:84% But we're going to do is we're going to keep this particular 00:48:38.350 --> 00:48:42.690 align:middle line:84% structure anyway, and maybe use some other considerations 00:48:42.690 --> 00:48:44.480 align:middle line:90% to pick the threshold. 00:48:44.480 --> 00:48:51.030 align:middle line:84% So we're going to use a likelihood ratio test, that's 00:48:51.030 --> 00:48:54.260 align:middle line:84% how it's called in which we calculate a quantity of this 00:48:54.260 --> 00:48:56.830 align:middle line:84% kind that we call the likelihood, and compare it 00:48:56.830 --> 00:48:58.480 align:middle line:90% with a threshold. 00:48:58.480 --> 00:49:00.530 align:middle line:84% So what's the interpretation of this likelihood? 00:49:00.530 --> 00:49:03.140 align:middle line:90% 00:49:03.140 --> 00:49:04.290 align:middle line:90% We ask-- 00:49:04.290 --> 00:49:08.570 align:middle line:84% the X's that I have observed, how likely were they to occur 00:49:08.570 --> 00:49:10.460 align:middle line:90% if H1 was true? 00:49:10.460 --> 00:49:14.590 align:middle line:84% And how likely were they to occur if H0 was true? 00:49:14.590 --> 00:49:20.560 align:middle line:84% This ratio could be big if my data are plausible they might 00:49:20.560 --> 00:49:22.400 align:middle line:90% occur under H1. 00:49:22.400 --> 00:49:25.400 align:middle line:84% But they're very implausible, extremely unlikely 00:49:25.400 --> 00:49:27.380 align:middle line:90% to occur under H0. 00:49:27.380 --> 00:49:30.060 align:middle line:84% Then my thinking would be well the data that I saw are 00:49:30.060 --> 00:49:33.300 align:middle line:84% extremely unlikely to have occurred under H0. 00:49:33.300 --> 00:49:36.780 align:middle line:90% So H0 is probably not true. 00:49:36.780 --> 00:49:39.820 align:middle line:84% I'm going to go for H1 and choose H1. 00:49:39.820 --> 00:49:43.920 align:middle line:84% So when this ratio is big it tells us that the data that 00:49:43.920 --> 00:49:47.720 align:middle line:84% we're seeing are better explained if we assume H1 to 00:49:47.720 --> 00:49:50.620 align:middle line:84% be true rather than H0 to be true. 00:49:50.620 --> 00:49:53.970 align:middle line:84% So I calculate this quantity, compare it with a threshold, 00:49:53.970 --> 00:49:56.200 align:middle line:84% and that's how I make my decision. 00:49:56.200 --> 00:49:59.360 align:middle line:84% So in this particular picture, for example the way it would 00:49:59.360 --> 00:50:02.930 align:middle line:84% go would be the likelihood ratio in this picture goes 00:50:02.930 --> 00:50:07.230 align:middle line:84% monotonically with my X. So comparing the likelihood ratio 00:50:07.230 --> 00:50:10.150 align:middle line:84% to the threshold would be the same as comparing my x to the 00:50:10.150 --> 00:50:12.890 align:middle line:84% threshold, and we've got the question of how 00:50:12.890 --> 00:50:13.920 align:middle line:90% to choose the threshold. 00:50:13.920 --> 00:50:17.880 align:middle line:84% The way that the threshold is chosen is usually done by 00:50:17.880 --> 00:50:21.560 align:middle line:84% fixing one of the two probabilities of error. 00:50:21.560 --> 00:50:26.710 align:middle line:84% That is, I say, that I want my error of one particular type 00:50:26.710 --> 00:50:30.160 align:middle line:84% to be a given number, so I fix this alpha. 00:50:30.160 --> 00:50:33.160 align:middle line:84% And then I try to find where my threshold should be. 00:50:33.160 --> 00:50:36.095 align:middle line:84% So that this probability theta, probability out there, 00:50:36.095 --> 00:50:39.190 align:middle line:90% is just equal to alpha. 00:50:39.190 --> 00:50:42.050 align:middle line:84% And then the other probability of error, beta, will be 00:50:42.050 --> 00:50:44.190 align:middle line:90% whatever it turns out to be. 00:50:44.190 --> 00:50:48.140 align:middle line:84% So somebody picks alpha ahead of time. 00:50:48.140 --> 00:50:52.210 align:middle line:84% Based on the probability of a false rejection based on 00:50:52.210 --> 00:50:55.890 align:middle line:84% alpha, I find where my threshold is going to be. 00:50:55.890 --> 00:50:59.890 align:middle line:84% I choose my threshold, and that determines subsequently 00:50:59.890 --> 00:51:01.270 align:middle line:90% the value of beta. 00:51:01.270 --> 00:51:07.340 align:middle line:84% So we're going to continue with this story next time, and 00:51:07.340 --> 00:51:08.590 align:middle line:90% we'll stop here. 00:51:08.590 --> 00:51:49.120 align:middle line:90%