WEBVTT 00:00:00.000 --> 00:00:00.040 align:middle line:90% 00:00:00.040 --> 00:00:02.460 align:middle line:84% The following content is provided under a Creative 00:00:02.460 --> 00:00:03.870 align:middle line:90% Commons license. 00:00:03.870 --> 00:00:06.910 align:middle line:84% Your support will help MIT OpenCourseWare continue to 00:00:06.910 --> 00:00:10.560 align:middle line:84% offer high quality educational resources for free. 00:00:10.560 --> 00:00:13.460 align:middle line:84% To make a donation or view additional materials from 00:00:13.460 --> 00:00:19.290 align:middle line:84% hundreds of MIT courses, visit MIT OpenCourseWare at 00:00:19.290 --> 00:00:22.160 align:middle line:90% ocw.mit.edu 00:00:22.160 --> 00:00:26.640 align:middle line:84% PROFESSOR: OK, if you have not yet done it, please take a 00:00:26.640 --> 00:00:30.510 align:middle line:84% moment to go through the course evaluation website and 00:00:30.510 --> 00:00:32.880 align:middle line:84% enter your comments for the class. 00:00:32.880 --> 00:00:36.250 align:middle line:84% So what we're going to do today to wrap things up is 00:00:36.250 --> 00:00:39.070 align:middle line:84% we're going to go through a tour of the world of 00:00:39.070 --> 00:00:41.320 align:middle line:90% hypothesis testing. 00:00:41.320 --> 00:00:44.500 align:middle line:84% See a few examples of hypothesis tests, starting 00:00:44.500 --> 00:00:48.280 align:middle line:84% from simple ones such as the one the setting that we 00:00:48.280 --> 00:00:51.220 align:middle line:84% discussed last time in which you just have two hypotheses, 00:00:51.220 --> 00:00:53.130 align:middle line:84% you're trying to choose between them. 00:00:53.130 --> 00:00:56.040 align:middle line:84% But also look at more complicated situations in 00:00:56.040 --> 00:01:00.600 align:middle line:84% which you have one basic hypothesis. 00:01:00.600 --> 00:01:03.720 align:middle line:84% Let's say that you have a fair coin and you want to test it 00:01:03.720 --> 00:01:06.450 align:middle line:84% against the hypotheses that your coin is not fair, but 00:01:06.450 --> 00:01:09.770 align:middle line:84% that alternative hypothesis is really lots of different 00:01:09.770 --> 00:01:11.000 align:middle line:90% hypothesis. 00:01:11.000 --> 00:01:12.310 align:middle line:90% So is my coin fair? 00:01:12.310 --> 00:01:13.700 align:middle line:90% Is my die fair? 00:01:13.700 --> 00:01:15.980 align:middle line:84% Do I have the correct distribution for random 00:01:15.980 --> 00:01:17.510 align:middle line:90% variable, and so on. 00:01:17.510 --> 00:01:20.960 align:middle line:84% And I'm going to end up with a few general comments about 00:01:20.960 --> 00:01:23.190 align:middle line:90% this whole business. 00:01:23.190 --> 00:01:28.370 align:middle line:84% So the sad thing in simple hypothesis testing problems is 00:01:28.370 --> 00:01:28.990 align:middle line:90% the following-- 00:01:28.990 --> 00:01:33.610 align:middle line:84% we have two possible models, and this is the classical 00:01:33.610 --> 00:01:36.680 align:middle line:84% world so we do not have any prior probabilities on the two 00:01:36.680 --> 00:01:37.850 align:middle line:90% hypotheses. 00:01:37.850 --> 00:01:41.340 align:middle line:84% Usually we want to think of these hypotheses as not being 00:01:41.340 --> 00:01:44.730 align:middle line:84% completely symmetrical, but rather one is the default 00:01:44.730 --> 00:01:48.180 align:middle line:84% hypothesis, and usually it's referred to as the null 00:01:48.180 --> 00:01:49.630 align:middle line:90% hypothesis. 00:01:49.630 --> 00:01:53.400 align:middle line:84% And you want to check whether the null hypothesis is true, 00:01:53.400 --> 00:01:57.170 align:middle line:84% whether things are normal as you would have expected them 00:01:57.170 --> 00:02:00.900 align:middle line:84% to be, or whether it turns out to be false, in which case an 00:02:00.900 --> 00:02:03.750 align:middle line:84% alternative hypothesis would be correct. 00:02:03.750 --> 00:02:05.710 align:middle line:90% So how does one go about it? 00:02:05.710 --> 00:02:08.919 align:middle line:90% 00:02:08.919 --> 00:02:12.720 align:middle line:84% No matter what approach you use, in the end you're going 00:02:12.720 --> 00:02:14.220 align:middle line:90% to end up doing the following. 00:02:14.220 --> 00:02:17.620 align:middle line:84% You have the space of all simple observations that you 00:02:17.620 --> 00:02:18.980 align:middle line:90% may obtain. 00:02:18.980 --> 00:02:21.630 align:middle line:84% So when you do the experiment you're going to get an X 00:02:21.630 --> 00:02:25.050 align:middle line:84% vector, a vector of data that's somewhere. 00:02:25.050 --> 00:02:27.760 align:middle line:84% And for some vectors you're going to decide that you 00:02:27.760 --> 00:02:31.410 align:middle line:84% accept H. Note for some vectors that you reject H0 and 00:02:31.410 --> 00:02:33.160 align:middle line:90% you accept H1. 00:02:33.160 --> 00:02:37.100 align:middle line:84% So what you will end up doing is that you're going to have 00:02:37.100 --> 00:02:42.130 align:middle line:84% some division of the space of all X's into two parts, and 00:02:42.130 --> 00:02:45.660 align:middle line:84% one part is the rejection region, and one part is the 00:02:45.660 --> 00:02:47.050 align:middle line:90% acceptance region. 00:02:47.050 --> 00:02:50.440 align:middle line:84% So if you fall in here you accept H0, if you fall here 00:02:50.440 --> 00:02:53.240 align:middle line:90% you'd reject H0. 00:02:53.240 --> 00:02:57.750 align:middle line:84% So to design a hypothesis test basically you need to come up 00:02:57.750 --> 00:03:03.360 align:middle line:84% with a division of your X space into two pieces. 00:03:03.360 --> 00:03:08.770 align:middle line:84% So the figuring out how to do this involves two elements. 00:03:08.770 --> 00:03:12.640 align:middle line:84% One element is to decide what kind of shape so I want for my 00:03:12.640 --> 00:03:14.740 align:middle line:90% dividing curve? 00:03:14.740 --> 00:03:18.240 align:middle line:84% And having chosen the shape of the dividing curve, where 00:03:18.240 --> 00:03:20.540 align:middle line:90% exactly do I put it? 00:03:20.540 --> 00:03:23.980 align:middle line:84% So if you were to cut this space using, let's say, a 00:03:23.980 --> 00:03:27.360 align:middle line:84% straight cut you might put it here, or you might put it 00:03:27.360 --> 00:03:28.930 align:middle line:84% there, or you might put it there. 00:03:28.930 --> 00:03:31.730 align:middle line:84% Where exactly are you going to put it? 00:03:31.730 --> 00:03:33.530 align:middle line:84% So let's look at those two steps. 00:03:33.530 --> 00:03:38.700 align:middle line:84% The first issue is to decide the general shape of your 00:03:38.700 --> 00:03:43.440 align:middle line:84% rejection region, which is the structure of your test. 00:03:43.440 --> 00:03:47.420 align:middle line:84% And the way this is done for the case of two hypothesis is 00:03:47.420 --> 00:03:52.050 align:middle line:84% by writing down the likelihood ratio between the two 00:03:52.050 --> 00:03:52.840 align:middle line:90% hypothesis. 00:03:52.840 --> 00:03:56.860 align:middle line:84% So let's call that quantity l of X. It's something that you 00:03:56.860 --> 00:04:00.280 align:middle line:84% can compute given the data that you have. 00:04:00.280 --> 00:04:04.660 align:middle line:84% A high value of l of X basically means that this 00:04:04.660 --> 00:04:08.140 align:middle line:84% probability here tends to be bigger than this probability. 00:04:08.140 --> 00:04:12.150 align:middle line:84% It means that the data that you have seen are quite likely 00:04:12.150 --> 00:04:15.650 align:middle line:84% to have occurred under H1, but less likely to have 00:04:15.650 --> 00:04:18.399 align:middle line:90% occurred under H0. 00:04:18.399 --> 00:04:22.360 align:middle line:84% So if you see data that they are more plausible, can be 00:04:22.360 --> 00:04:26.630 align:middle line:84% better explained, under H1, then this ratio is big, and 00:04:26.630 --> 00:04:31.030 align:middle line:84% you're going to choose in favor of H1 or reject H0. 00:04:31.030 --> 00:04:32.950 align:middle line:84% That's what you do if you have discrete data. 00:04:32.950 --> 00:04:34.380 align:middle line:90% You use the PMFs. 00:04:34.380 --> 00:04:37.450 align:middle line:84% If you have densities, in the case of continues data, again 00:04:37.450 --> 00:04:42.740 align:middle line:84% you consider the ratio of the two densities. 00:04:42.740 --> 00:04:47.250 align:middle line:84% So a big l of X is evidence that your data are more 00:04:47.250 --> 00:04:51.570 align:middle line:84% compatible with H1 rather than H0. 00:04:51.570 --> 00:04:59.140 align:middle line:84% Once you accept this kind of structure then your decision 00:04:59.140 --> 00:05:02.920 align:middle line:84% is really made in terms of that single number. 00:05:02.920 --> 00:05:06.270 align:middle line:84% That is, you had your data that was some kind of vector, 00:05:06.270 --> 00:05:09.930 align:middle line:84% and you condense your data into a single number-- a 00:05:09.930 --> 00:05:12.080 align:middle line:90% statistic as it's called-- 00:05:12.080 --> 00:05:15.150 align:middle line:84% in this case the likelihood ratio, and you put the 00:05:15.150 --> 00:05:19.880 align:middle line:84% dividing point somewhere here call it Xi. 00:05:19.880 --> 00:05:22.600 align:middle line:84% And in this region you accept H1, in this 00:05:22.600 --> 00:05:25.940 align:middle line:90% region you accept H0. 00:05:25.940 --> 00:05:30.410 align:middle line:84% So by committing ourselves to using the likelihood ratio in 00:05:30.410 --> 00:05:33.650 align:middle line:84% order to carry out the test we have gone from this 00:05:33.650 --> 00:05:38.030 align:middle line:84% complicated picture of finding a dividing line in x-space, to 00:05:38.030 --> 00:05:42.860 align:middle line:84% a simpler problem of just finding a dividing point on 00:05:42.860 --> 00:05:45.280 align:middle line:90% the real line. 00:05:45.280 --> 00:05:46.960 align:middle line:90% OK, how are we going? 00:05:46.960 --> 00:05:51.290 align:middle line:84% So what's left to do is to choose this threshold, Xi. 00:05:51.290 --> 00:05:53.920 align:middle line:84% Or as it's called, the critical value, 00:05:53.920 --> 00:05:56.560 align:middle line:90% for making our decision. 00:05:56.560 --> 00:06:01.930 align:middle line:84% And you can place it anywhere, but one way of deciding where 00:06:01.930 --> 00:06:03.240 align:middle line:90% to place it is the following-- 00:06:03.240 --> 00:06:07.740 align:middle line:84% look at the distribution of this random variable, l of X. 00:06:07.740 --> 00:06:11.760 align:middle line:84% It's has a certain distribution under H0, and it 00:06:11.760 --> 00:06:16.210 align:middle line:84% has some other distribution under H1. 00:06:16.210 --> 00:06:19.650 align:middle line:84% If I put my threshold here, here's what's going to happen. 00:06:19.650 --> 00:06:24.360 align:middle line:84% When H0 is true, there is this much probability that I'm 00:06:24.360 --> 00:06:27.360 align:middle line:84% going to end up making an incorrect decision. 00:06:27.360 --> 00:06:31.000 align:middle line:84% If H0 is true there's still a probability that my likelihood 00:06:31.000 --> 00:06:35.100 align:middle line:84% ratio will be bigger than Xi, and that's the probability of 00:06:35.100 --> 00:06:38.590 align:middle line:84% making an incorrect decision of this particular type. 00:06:38.590 --> 00:06:42.720 align:middle line:84% That is of making a false rejection of H0. 00:06:42.720 --> 00:06:46.330 align:middle line:84% Usually one sets this probability to a certain 00:06:46.330 --> 00:06:48.230 align:middle line:90% number, alpha. 00:06:48.230 --> 00:06:51.770 align:middle line:90% For example alpha being 5 %. 00:06:51.770 --> 00:06:55.680 align:middle line:84% And once you decide that you want this to be 5 %, that 00:06:55.680 --> 00:07:00.630 align:middle line:84% determines where this number Psi(Xi) is going to be. 00:07:00.630 --> 00:07:07.340 align:middle line:84% So the idea here is that I'm going to reject H0 if the data 00:07:07.340 --> 00:07:12.350 align:middle line:84% that I have seen are quite incompatible with H0. 00:07:12.350 --> 00:07:16.860 align:middle line:84% if they're quite unlikely to have occurred under H0. 00:07:16.860 --> 00:07:19.690 align:middle line:90% And I take this level, 5%. 00:07:19.690 --> 00:07:25.670 align:middle line:84% So I see my data and then I say well if H0 was true, the 00:07:25.670 --> 00:07:29.380 align:middle line:84% probability that I would have seen data of this kind would 00:07:29.380 --> 00:07:31.390 align:middle line:90% be less than 5 %. 00:07:31.390 --> 00:07:35.390 align:middle line:84% Given that I saw those data, that suggests that H0 is not 00:07:35.390 --> 00:07:37.860 align:middle line:84% true, and I end up rejecting H0. 00:07:37.860 --> 00:07:40.770 align:middle line:90% 00:07:40.770 --> 00:07:44.150 align:middle line:84% Now of course there's the other type of error 00:07:44.150 --> 00:07:45.190 align:middle line:90% probability. 00:07:45.190 --> 00:07:50.550 align:middle line:84% If I put my threshold here, if H1 is true but my likelihood 00:07:50.550 --> 00:07:53.470 align:middle line:84% ratio falls here I'm going to make a mistake of 00:07:53.470 --> 00:07:55.250 align:middle line:90% the opposite kind. 00:07:55.250 --> 00:07:59.780 align:middle line:84% H1 is true, but my likelihood ratio turned out to be small, 00:07:59.780 --> 00:08:02.370 align:middle line:90% and I decided in favor of H0. 00:08:02.370 --> 00:08:05.680 align:middle line:84% This is an error of the other kind, this probability of 00:08:05.680 --> 00:08:08.030 align:middle line:90% error we call beta. 00:08:08.030 --> 00:08:10.070 align:middle line:84% And you can see that there's a trade-off 00:08:10.070 --> 00:08:12.300 align:middle line:90% between alpha and beta. 00:08:12.300 --> 00:08:15.710 align:middle line:84% If you move your threshold this way alpha become smaller, 00:08:15.710 --> 00:08:18.320 align:middle line:90% but beta becomes larger. 00:08:18.320 --> 00:08:22.120 align:middle line:84% And the general picture is, in your trade-off, depending on 00:08:22.120 --> 00:08:25.970 align:middle line:84% where you put your threshold is as follows-- 00:08:25.970 --> 00:08:31.370 align:middle line:84% you can make this beta to be 0 if you put your threshold out 00:08:31.370 --> 00:08:34.809 align:middle line:84% here, but in that case you are certain that you're going to 00:08:34.809 --> 00:08:37.000 align:middle line:84% make a mistake of the opposite kind. 00:08:37.000 --> 00:08:42.360 align:middle line:84% So beta equals 0, alpha equals 1 is one possibility. 00:08:42.360 --> 00:08:46.420 align:middle line:84% Beta equals 1 alpha equals 0 is the other possibility if 00:08:46.420 --> 00:08:49.620 align:middle line:84% you send your thresholds complete to the other side. 00:08:49.620 --> 00:08:51.950 align:middle line:84% And in general you're going to get a trade-off 00:08:51.950 --> 00:08:54.930 align:middle line:90% curve of some sort. 00:08:54.930 --> 00:08:58.720 align:middle line:84% And if you want to use a specific value of alpha, for 00:08:58.720 --> 00:09:04.030 align:middle line:84% example alpha being 0.05, then that's going to determine for 00:09:04.030 --> 00:09:07.820 align:middle line:90% you the probability for beta. 00:09:07.820 --> 00:09:11.410 align:middle line:84% Now there's a general, and quite important theorem in 00:09:11.410 --> 00:09:13.640 align:middle line:84% statistics, which were are not proving. 00:09:13.640 --> 00:09:17.500 align:middle line:84% And which tells us that when we use likelihood ratio tests 00:09:17.500 --> 00:09:21.670 align:middle line:84% we get the best possible trade-off curve. 00:09:21.670 --> 00:09:26.720 align:middle line:84% You could think of other ways of making your decisions. 00:09:26.720 --> 00:09:30.780 align:middle line:84% Other ways of cutting off your x-space into a rejection and 00:09:30.780 --> 00:09:32.090 align:middle line:90% acceptance region. 00:09:32.090 --> 00:09:36.050 align:middle line:84% But any other way that you do it is going to end up with 00:09:36.050 --> 00:09:39.900 align:middle line:84% some probabilities of error that are going to be above 00:09:39.900 --> 00:09:41.990 align:middle line:90% this particular curve. 00:09:41.990 --> 00:09:46.570 align:middle line:84% So the likelihood ratio test turns out to give you the best 00:09:46.570 --> 00:09:49.200 align:middle line:84% possible way of dealing with this trade-off 00:09:49.200 --> 00:09:50.750 align:middle line:90% between alpha and beta. 00:09:50.750 --> 00:09:54.090 align:middle line:84% We cannot minimize alpha and beta simultaneously, there's a 00:09:54.090 --> 00:09:56.280 align:middle line:90% trade-off between them. 00:09:56.280 --> 00:10:02.420 align:middle line:84% But at least we would like to have a test that deals with 00:10:02.420 --> 00:10:04.380 align:middle line:84% this trade-off in the best possible way. 00:10:04.380 --> 00:10:07.770 align:middle line:84% For a given value of alpha we want to have the smallest 00:10:07.770 --> 00:10:09.490 align:middle line:90% possible value of beta. 00:10:09.490 --> 00:10:13.900 align:middle line:84% And as the theorem is that the likelihood ratio tests do have 00:10:13.900 --> 00:10:15.240 align:middle line:90% this optimality property. 00:10:15.240 --> 00:10:18.270 align:middle line:84% For a given value of alpha they minimize the probability 00:10:18.270 --> 00:10:20.610 align:middle line:90% of error of a different kind. 00:10:20.610 --> 00:10:23.380 align:middle line:84% So let's make all these concrete and look at the 00:10:23.380 --> 00:10:24.680 align:middle line:90% simple example. 00:10:24.680 --> 00:10:27.980 align:middle line:84% We have two normal distributions 00:10:27.980 --> 00:10:29.610 align:middle line:90% with different means. 00:10:29.610 --> 00:10:32.850 align:middle line:84% So under H0 you have a mean of 0. 00:10:32.850 --> 00:10:36.790 align:middle line:90% Under H1 you have a mean of 1. 00:10:36.790 --> 00:10:40.810 align:middle line:84% You get your data, you actually get several data 00:10:40.810 --> 00:10:43.770 align:middle line:84% drawn from one of the two distributions. 00:10:43.770 --> 00:10:45.560 align:middle line:84% And you want to make a decision, which one 00:10:45.560 --> 00:10:47.050 align:middle line:90% of the two is true? 00:10:47.050 --> 00:10:50.400 align:middle line:84% So what you do is you write down the likelihood ratio. 00:10:50.400 --> 00:10:54.730 align:middle line:84% The density for a vector of data, if that vector was 00:10:54.730 --> 00:10:57.490 align:middle line:90% generated according to H0 -- 00:10:57.490 --> 00:11:00.470 align:middle line:84% which is this one, and the density if it was generated 00:11:00.470 --> 00:11:02.810 align:middle line:90% according to H1. 00:11:02.810 --> 00:11:06.510 align:middle line:84% Since we have multiple data the density of a vector is the 00:11:06.510 --> 00:11:09.830 align:middle line:84% product of the densities of the individual elements. 00:11:09.830 --> 00:11:11.800 align:middle line:84% Since we're dealing with normals we have those 00:11:11.800 --> 00:11:13.500 align:middle line:90% exponential factors. 00:11:13.500 --> 00:11:15.550 align:middle line:84% A product of exponentials gives us an 00:11:15.550 --> 00:11:17.340 align:middle line:90% exponential of the sum. 00:11:17.340 --> 00:11:20.170 align:middle line:84% I'll spare you the details, but this is the form of the 00:11:20.170 --> 00:11:21.230 align:middle line:90% likelihood ratio. 00:11:21.230 --> 00:11:23.960 align:middle line:84% The likelihood ratio test tells us that we should 00:11:23.960 --> 00:11:28.360 align:middle line:84% calculate this quantity after we get your data, and compare 00:11:28.360 --> 00:11:30.750 align:middle line:90% with a threshold. 00:11:30.750 --> 00:11:35.340 align:middle line:84% Now you can do some algebra here, and simplify. 00:11:35.340 --> 00:11:39.150 align:middle line:84% And by tracing down the inequalities you're taking 00:11:39.150 --> 00:11:41.840 align:middle line:84% logarithms of both sides, and so on. 00:11:41.840 --> 00:11:47.350 align:middle line:84% One comes to the conclusion that using a test that has a 00:11:47.350 --> 00:11:52.150 align:middle line:84% threshold on this ratio is equivalent to calculating this 00:11:52.150 --> 00:11:56.920 align:middle line:84% quantity, and comparing it with a threshold. 00:11:56.920 --> 00:12:01.220 align:middle line:84% Basically this quantity here is monotonic in that quantity. 00:12:01.220 --> 00:12:04.510 align:middle line:84% This being larger than the threshold is equivalent to 00:12:04.510 --> 00:12:07.400 align:middle line:84% this being larger than the threshold. 00:12:07.400 --> 00:12:10.310 align:middle line:84% So this tells us the general structure of the likelihood 00:12:10.310 --> 00:12:12.770 align:middle line:84% ratio test in this particular case. 00:12:12.770 --> 00:12:15.640 align:middle line:84% And it's nice because it tells us that we can make our 00:12:15.640 --> 00:12:20.340 align:middle line:84% decisions by looking at this simple summary of the data. 00:12:20.340 --> 00:12:23.810 align:middle line:84% This quantity, this summary of the data on the basis of which 00:12:23.810 --> 00:12:29.130 align:middle line:84% we make our decision is called a statistic. 00:12:29.130 --> 00:12:32.850 align:middle line:84% So you take your data, which is a multi-dimensional vector, 00:12:32.850 --> 00:12:37.850 align:middle line:84% and you condense it to a single number, and then you 00:12:37.850 --> 00:12:40.630 align:middle line:84% make a decision on the basis of that number. 00:12:40.630 --> 00:12:42.750 align:middle line:84% So this is the structure of the test. 00:12:42.750 --> 00:12:47.430 align:middle line:84% If I get a large sum of Xi's this is evidence in favor of 00:12:47.430 --> 00:12:50.430 align:middle line:84% H1 because here the mean is larger. 00:12:50.430 --> 00:12:54.990 align:middle line:84% And so I'm going to decide in favor of H1 or reject H0 if 00:12:54.990 --> 00:12:56.650 align:middle line:84% the sum is bigger than the threshold. 00:12:56.650 --> 00:12:58.750 align:middle line:90% How do I choose my threshold? 00:12:58.750 --> 00:13:01.080 align:middle line:84% Well I would like to choose my threshold so that the 00:13:01.080 --> 00:13:04.990 align:middle line:84% probability of an incorrect decision when H0 is true the 00:13:04.990 --> 00:13:09.980 align:middle line:84% probability of a false rejection equals 00:13:09.980 --> 00:13:10.890 align:middle line:90% to a certain number. 00:13:10.890 --> 00:13:14.400 align:middle line:84% Alpha, such as for example 5 %. 00:13:14.400 --> 00:13:19.210 align:middle line:84% So you're given here that this is 5 %. 00:13:19.210 --> 00:13:20.660 align:middle line:84% You know the distribution of this random 00:13:20.660 --> 00:13:22.240 align:middle line:90% variable, it's normal. 00:13:22.240 --> 00:13:24.980 align:middle line:84% And you want to find the threshold value that makes 00:13:24.980 --> 00:13:26.430 align:middle line:90% this to be true. 00:13:26.430 --> 00:13:28.300 align:middle line:84% So this is a type of problem that you have 00:13:28.300 --> 00:13:29.360 align:middle line:90% seen several times. 00:13:29.360 --> 00:13:32.910 align:middle line:84% You go to the normal tables, and you figure it out. 00:13:32.910 --> 00:13:35.790 align:middle line:90% So the sum of the Xi's has some 00:13:35.790 --> 00:13:38.160 align:middle line:90% distribution, it's normal. 00:13:38.160 --> 00:13:41.090 align:middle line:84% So that's the distribution of the sum of the Xi's. 00:13:41.090 --> 00:13:44.620 align:middle line:84% And you want this probability here to be alpha. 00:13:44.620 --> 00:13:49.520 align:middle line:84% For this to happen what is the threshold value that makes 00:13:49.520 --> 00:13:50.870 align:middle line:90% this to be true? 00:13:50.870 --> 00:13:55.570 align:middle line:84% So you know how to solve problems of this kind using 00:13:55.570 --> 00:13:58.420 align:middle line:90% the normal tables. 00:13:58.420 --> 00:14:02.730 align:middle line:84% A slightly different example is one in which you have two 00:14:02.730 --> 00:14:05.900 align:middle line:84% normal distributions that have the same mean -- 00:14:05.900 --> 00:14:07.580 align:middle line:90% let's take it to be 0 -- 00:14:07.580 --> 00:14:10.580 align:middle line:84% but they have a different variance. 00:14:10.580 --> 00:14:15.080 align:middle line:84% So it's sort of natural that here, if your X's that you see 00:14:15.080 --> 00:14:19.880 align:middle line:84% are kind of big on either side you would choose H1. 00:14:19.880 --> 00:14:23.500 align:middle line:84% If your X's are near 0 then that's evidence for the 00:14:23.500 --> 00:14:27.120 align:middle line:84% smaller variance you would choose H0. 00:14:27.120 --> 00:14:30.740 align:middle line:84% So to proceed formally you again write down to the form 00:14:30.740 --> 00:14:33.190 align:middle line:90% of the likelihood ratio. 00:14:33.190 --> 00:14:39.780 align:middle line:84% So again the density of an X vector under H0 is this one. 00:14:39.780 --> 00:14:41.680 align:middle line:84% It's the product of the densities of 00:14:41.680 --> 00:14:43.410 align:middle line:90% each one of the Xi's. 00:14:43.410 --> 00:14:47.030 align:middle line:84% Product of normal densities gives you a product of 00:14:47.030 --> 00:14:50.180 align:middle line:84% exponentials, which is exponential of the sum, and 00:14:50.180 --> 00:14:52.070 align:middle line:84% that's the expression that you get. 00:14:52.070 --> 00:14:54.560 align:middle line:84% Under the other hypothesis the only thing that 00:14:54.560 --> 00:14:56.530 align:middle line:90% changes is the variance. 00:14:56.530 --> 00:14:59.800 align:middle line:84% And the variance, in the normal distribution, shows up 00:14:59.800 --> 00:15:02.970 align:middle line:84% here in the denominator of the exponent. 00:15:02.970 --> 00:15:04.560 align:middle line:90% So you put it there. 00:15:04.560 --> 00:15:07.390 align:middle line:84% So this is the general structure of the likelihood 00:15:07.390 --> 00:15:08.650 align:middle line:90% ratio test. 00:15:08.650 --> 00:15:10.400 align:middle line:90% And now you do some algebra. 00:15:10.400 --> 00:15:14.110 align:middle line:84% These terms are constants comparing this ratio to a 00:15:14.110 --> 00:15:17.190 align:middle line:84% constant is the same as just comparing the ratio of the 00:15:17.190 --> 00:15:19.050 align:middle line:90% exponentials to a constant. 00:15:19.050 --> 00:15:23.710 align:middle line:84% Then you take logarithms, you want to compare the logarithm 00:15:23.710 --> 00:15:25.650 align:middle line:90% of this thing to a constant. 00:15:25.650 --> 00:15:28.210 align:middle line:84% You do a little bit of algebra, and in the end you 00:15:28.210 --> 00:15:32.180 align:middle line:84% find that the structure of the test is to reject H0 if the 00:15:32.180 --> 00:15:37.740 align:middle line:84% sum of the squares of the Xi's is bigger than the threshold. 00:15:37.740 --> 00:15:41.360 align:middle line:84% So by committing to a likelihood ratio test you are 00:15:41.360 --> 00:15:45.060 align:middle line:84% told that you should be making it your decision according to 00:15:45.060 --> 00:15:46.940 align:middle line:90% a rule of this type. 00:15:46.940 --> 00:15:51.450 align:middle line:84% So this fixes the shape or the structure of the decision 00:15:51.450 --> 00:15:53.670 align:middle line:84% region, of the rejection region. 00:15:53.670 --> 00:15:56.660 align:middle line:84% And the only thing that's left, once more, is to pick 00:15:56.660 --> 00:16:00.190 align:middle line:84% this threshold in order to have the property that the 00:16:00.190 --> 00:16:05.340 align:middle line:84% probability of a false rejection is equal to say 5 %. 00:16:05.340 --> 00:16:09.490 align:middle line:84% So that's the probability that H0 is true, but the sum of the 00:16:09.490 --> 00:16:11.450 align:middle line:84% squares accidentally happens to be 00:16:11.450 --> 00:16:13.080 align:middle line:90% bigger than my threshold. 00:16:13.080 --> 00:16:17.330 align:middle line:84% In which case I end up deciding H1. 00:16:17.330 --> 00:16:21.570 align:middle line:84% How do I find the value of Xi prime? 00:16:21.570 --> 00:16:25.150 align:middle line:84% Well what I need to do is to look at the picture, more or 00:16:25.150 --> 00:16:29.100 align:middle line:84% less of this kind, but now I need to look at the 00:16:29.100 --> 00:16:32.870 align:middle line:84% distribution of the sum of the Xi's squared. 00:16:32.870 --> 00:16:36.190 align:middle line:84% Actually the sum of the Xi's squared is a non-negative 00:16:36.190 --> 00:16:37.580 align:middle line:90% random variable. 00:16:37.580 --> 00:16:40.280 align:middle line:84% So it's going to have a distribution that's 00:16:40.280 --> 00:16:44.910 align:middle line:90% something like this. 00:16:44.910 --> 00:16:50.540 align:middle line:84% I look at that distribution, and once more I want this tail 00:16:50.540 --> 00:16:54.300 align:middle line:84% probability to be alpha, and that determines where my 00:16:54.300 --> 00:16:56.370 align:middle line:90% threshold is going to be. 00:16:56.370 --> 00:17:00.595 align:middle line:84% So that's again a simple exercise provided that you 00:17:00.595 --> 00:17:03.650 align:middle line:84% know the distribution of this quantity. 00:17:03.650 --> 00:17:05.540 align:middle line:90% Do you know it? 00:17:05.540 --> 00:17:08.980 align:middle line:84% Well we don't really know it, we have not dealt with this 00:17:08.980 --> 00:17:11.859 align:middle line:84% particular distribution in this class. 00:17:11.859 --> 00:17:15.730 align:middle line:84% But in principle you should be able to find what it is. 00:17:15.730 --> 00:17:18.459 align:middle line:84% It's a derived distribution problem. 00:17:18.459 --> 00:17:22.920 align:middle line:84% You know the distribution of Xi, it's normal. 00:17:22.920 --> 00:17:26.410 align:middle line:84% Therefore, by solving a derived distribution problem 00:17:26.410 --> 00:17:30.400 align:middle line:84% you can find the distribution of Xi squared. 00:17:30.400 --> 00:17:34.180 align:middle line:84% And the Xi squared's are independent of each other, 00:17:34.180 --> 00:17:36.400 align:middle line:84% because the Xi's are independent. 00:17:36.400 --> 00:17:39.190 align:middle line:84% So you want to find the distribution of the sum of 00:17:39.190 --> 00:17:41.750 align:middle line:84% random variables with known distributions. 00:17:41.750 --> 00:17:44.410 align:middle line:84% And since they're independent, in principle, you can do this 00:17:44.410 --> 00:17:46.470 align:middle line:90% using the convolution formula. 00:17:46.470 --> 00:17:49.720 align:middle line:84% So in principle, and if you're patient enough, you will be 00:17:49.720 --> 00:17:52.830 align:middle line:84% able to find the distribution of this random variable. 00:17:52.830 --> 00:17:57.430 align:middle line:84% And then you plot it or tabulate it, and find where 00:17:57.430 --> 00:18:02.870 align:middle line:84% exactly is the 95th percentile of that distribution, and that 00:18:02.870 --> 00:18:05.290 align:middle line:90% determines your threshold. 00:18:05.290 --> 00:18:08.310 align:middle line:84% So this distribution actually turns out to have a nice and 00:18:08.310 --> 00:18:11.000 align:middle line:90% simple closed-form formula. 00:18:11.000 --> 00:18:13.740 align:middle line:84% Because this is a pretty common test, people have 00:18:13.740 --> 00:18:15.220 align:middle line:90% tabulated that distribution. 00:18:15.220 --> 00:18:17.370 align:middle line:84% It's called the chi-square distribution. 00:18:17.370 --> 00:18:19.512 align:middle line:84% There's tables available for it. 00:18:19.512 --> 00:18:23.390 align:middle line:84% And you look up in the tables, you find the 95th percentile 00:18:23.390 --> 00:18:25.900 align:middle line:84% of the distribution, and this way you 00:18:25.900 --> 00:18:28.280 align:middle line:90% determine your threshold. 00:18:28.280 --> 00:18:31.140 align:middle line:84% So what's the moral of the story? 00:18:31.140 --> 00:18:34.800 align:middle line:84% The structure of the likelihood ratio test tells 00:18:34.800 --> 00:18:40.470 align:middle line:84% you what kind of decision region you're going to have. 00:18:40.470 --> 00:18:42.880 align:middle line:84% It tells you that for this particular test you should be 00:18:42.880 --> 00:18:46.360 align:middle line:84% using the sum of the Xi squared's as your statistic, 00:18:46.360 --> 00:18:48.460 align:middle line:84% as the basis for making your decision. 00:18:48.460 --> 00:18:51.840 align:middle line:84% And then you need to solve a derived distribution problem 00:18:51.840 --> 00:18:53.110 align:middle line:90% to find the probability 00:18:53.110 --> 00:18:55.500 align:middle line:90% distribution of your statistic. 00:18:55.500 --> 00:19:00.290 align:middle line:84% Find the distribution of this quantity under H0, and 00:19:00.290 --> 00:19:03.000 align:middle line:84% finally, based on that distribution, after you have 00:19:03.000 --> 00:19:05.330 align:middle line:84% derived it, then determine your threshold. 00:19:05.330 --> 00:19:08.240 align:middle line:90% 00:19:08.240 --> 00:19:10.360 align:middle line:84% So now let's move on to a somewhat 00:19:10.360 --> 00:19:13.090 align:middle line:90% more complicated situation. 00:19:13.090 --> 00:19:18.090 align:middle line:84% You have a coin, and you are told that I tried 00:19:18.090 --> 00:19:21.040 align:middle line:90% to make a fair coin. 00:19:21.040 --> 00:19:22.450 align:middle line:90% Is it fair? 00:19:22.450 --> 00:19:25.200 align:middle line:84% So you have the hypothesis, which is the default-- 00:19:25.200 --> 00:19:26.320 align:middle line:90% the null hypothesis-- 00:19:26.320 --> 00:19:27.890 align:middle line:90% that the coin is fair. 00:19:27.890 --> 00:19:29.690 align:middle line:90% But maybe it isn't. 00:19:29.690 --> 00:19:31.880 align:middle line:84% So you have the alternative hypothesis that 00:19:31.880 --> 00:19:34.030 align:middle line:90% your coin is not fair. 00:19:34.030 --> 00:19:36.690 align:middle line:84% Now what's different in this context is that your 00:19:36.690 --> 00:19:41.830 align:middle line:84% alternative hypothesis is not just one specific hypothesis. 00:19:41.830 --> 00:19:45.990 align:middle line:84% Your alternative hypothesis consists of many alternatives. 00:19:45.990 --> 00:19:49.270 align:middle line:84% It includes the hypothesis that p is 0.6. 00:19:49.270 --> 00:19:53.930 align:middle line:84% It includes the hypothesis that p is 0.51. 00:19:53.930 --> 00:19:58.850 align:middle line:84% It includes the hypothesis that p is 0.48, and so on. 00:19:58.850 --> 00:20:05.030 align:middle line:84% So you're testing this hypothesis versus all this 00:20:05.030 --> 00:20:08.070 align:middle line:84% family of alternative hypothesis. 00:20:08.070 --> 00:20:11.080 align:middle line:84% What you will end up doing is essentially the following-- 00:20:11.080 --> 00:20:12.480 align:middle line:90% you get some data. 00:20:12.480 --> 00:20:15.080 align:middle line:84% That is, you flip the coin a number of times. 00:20:15.080 --> 00:20:17.640 align:middle line:84% Let's say you flip it 1,000 times. 00:20:17.640 --> 00:20:20.290 align:middle line:90% You observe some outcome. 00:20:20.290 --> 00:20:24.580 align:middle line:90% Let's say you saw 472 heads. 00:20:24.580 --> 00:20:31.650 align:middle line:84% And you ask the question if this hypothesis is true is 00:20:31.650 --> 00:20:35.790 align:middle line:84% this value really possible under that hypothesis? 00:20:35.790 --> 00:20:39.450 align:middle line:84% Or would it be very much of an outlier? 00:20:39.450 --> 00:20:44.220 align:middle line:84% If it looks like an extreme outlier under this hypothesis 00:20:44.220 --> 00:20:47.780 align:middle line:84% then I reject it, and I accept the alternative. 00:20:47.780 --> 00:20:50.800 align:middle line:84% If this number turns out to be something within the range 00:20:50.800 --> 00:20:56.690 align:middle line:84% that you would have expected then you keep, or accept your 00:20:56.690 --> 00:20:59.080 align:middle line:90% null hypothesis. 00:20:59.080 --> 00:21:03.200 align:middle line:84% OK so what does it mean to be an outlier or not? 00:21:03.200 --> 00:21:05.430 align:middle line:84% First you take your data, and you condense 00:21:05.430 --> 00:21:07.220 align:middle line:90% them to a single number. 00:21:07.220 --> 00:21:10.240 align:middle line:84% So your detailed data actually would have been a sequence of 00:21:10.240 --> 00:21:12.440 align:middle line:84% heads/tails, heads/tails and all that. 00:21:12.440 --> 00:21:16.370 align:middle line:84% Any reasonable person would tell you that you shouldn't 00:21:16.370 --> 00:21:19.430 align:middle line:84% really care about the exact sequence of heads and tails. 00:21:19.430 --> 00:21:22.570 align:middle line:84% Let's just base our decision on the number of heads that we 00:21:22.570 --> 00:21:24.380 align:middle line:90% have observed. 00:21:24.380 --> 00:21:28.870 align:middle line:84% So using some kind of reasoning which could be 00:21:28.870 --> 00:21:33.650 align:middle line:84% mathematical, or intuitive, or involving artistry-- 00:21:33.650 --> 00:21:38.400 align:middle line:84% you pick a one-dimensional, or scalar summary of the data 00:21:38.400 --> 00:21:39.450 align:middle line:90% that you have seen. 00:21:39.450 --> 00:21:42.250 align:middle line:84% In this case, the summary of the data is just the number of 00:21:42.250 --> 00:21:44.330 align:middle line:84% heads that's a quite reasonable one. 00:21:44.330 --> 00:21:47.880 align:middle line:84% And so you commit yourself to make a decision on the basis 00:21:47.880 --> 00:21:49.080 align:middle line:90% of this quantity. 00:21:49.080 --> 00:21:52.670 align:middle line:84% And you ask the quantity that I'm seeing does it look like 00:21:52.670 --> 00:21:53.680 align:middle line:90% an outlier? 00:21:53.680 --> 00:21:57.710 align:middle line:84% Or does it look more or less OK? 00:21:57.710 --> 00:22:00.540 align:middle line:84% OK, what does it mean to be an outlier? 00:22:00.540 --> 00:22:04.900 align:middle line:84% You want to choose the shape of this rejection region, but 00:22:04.900 --> 00:22:08.750 align:middle line:84% on the basis of that single number s. 00:22:08.750 --> 00:22:11.240 align:middle line:84% And again, the reasonable thing to do in this context 00:22:11.240 --> 00:22:15.170 align:middle line:84% would be to argue as follows-- if my coin is fair I expect to 00:22:15.170 --> 00:22:16.850 align:middle line:90% see n over 2 heads. 00:22:16.850 --> 00:22:18.540 align:middle line:90% That's the expected value. 00:22:18.540 --> 00:22:23.330 align:middle line:84% If the number of heads I see is far from the expected 00:22:23.330 --> 00:22:26.030 align:middle line:90% number of heads then I consider 00:22:26.030 --> 00:22:27.750 align:middle line:90% this to be an outlier. 00:22:27.750 --> 00:22:30.470 align:middle line:84% So if this number is bigger than some threshold Xi. 00:22:30.470 --> 00:22:33.600 align:middle line:84% I consider it to be an outlier, and then I'm going to 00:22:33.600 --> 00:22:36.100 align:middle line:90% reject my hypothesis. 00:22:36.100 --> 00:22:38.930 align:middle line:90% So we picked our statistic. 00:22:38.930 --> 00:22:44.990 align:middle line:84% We picked the general form of how we're going to make our 00:22:44.990 --> 00:22:50.000 align:middle line:84% decision, and then we pick a certain significance, or 00:22:50.000 --> 00:22:51.690 align:middle line:90% confidence level that we want. 00:22:51.690 --> 00:22:54.470 align:middle line:90% Again, this famous 5% number. 00:22:54.470 --> 00:22:58.310 align:middle line:84% And we're going to declare something to be an outlier if 00:22:58.310 --> 00:23:01.380 align:middle line:84% it lies in the region that has 5% or less 00:23:01.380 --> 00:23:03.270 align:middle line:90% probability of occurring. 00:23:03.270 --> 00:23:07.560 align:middle line:84% That is I'm picking my rejection region so that if H0 00:23:07.560 --> 00:23:11.870 align:middle line:84% is true under the default, or null hypothesis, there's only 00:23:11.870 --> 00:23:17.380 align:middle line:84% 5% chance that by accident I fall there, and the thing 00:23:17.380 --> 00:23:21.540 align:middle line:84% makes me think that H1 is going to be true. 00:23:21.540 --> 00:23:25.690 align:middle line:90% 00:23:25.690 --> 00:23:28.770 align:middle line:84% So now what's left to do is to pick the 00:23:28.770 --> 00:23:30.920 align:middle line:90% value of this threshold. 00:23:30.920 --> 00:23:34.410 align:middle line:84% This is a calculation of the usual kind. 00:23:34.410 --> 00:23:39.580 align:middle line:84% I want to pick my threshold, my Xi number so that the 00:23:39.580 --> 00:23:44.150 align:middle line:84% probability that s is further from the mean by an amount of 00:23:44.150 --> 00:23:47.200 align:middle line:90% Xi is less than 5%. 00:23:47.200 --> 00:23:50.630 align:middle line:84% Or that the probability of being inside 00:23:50.630 --> 00:23:52.300 align:middle line:90% the acceptance region-- 00:23:52.300 --> 00:23:55.240 align:middle line:84% so that the distance from the default is 00:23:55.240 --> 00:23:56.380 align:middle line:90% less than my threshold. 00:23:56.380 --> 00:23:59.880 align:middle line:90% I want that to be 95%. 00:23:59.880 --> 00:24:04.380 align:middle line:84% So this is an equality that you can get using the central 00:24:04.380 --> 00:24:06.760 align:middle line:84% limit theorem and the normal tables. 00:24:06.760 --> 00:24:10.230 align:middle line:84% There's 95% probability that the number of heads is going 00:24:10.230 --> 00:24:14.920 align:middle line:84% to be within 31 from the correct mean. 00:24:14.920 --> 00:24:17.910 align:middle line:84% So the way the exercise is done of course, is that we 00:24:17.910 --> 00:24:20.640 align:middle line:90% start with this number, 5%. 00:24:20.640 --> 00:24:24.410 align:middle line:84% Which translates to this number 95%. 00:24:24.410 --> 00:24:27.960 align:middle line:84% And once we have fixed that number then you ask the 00:24:27.960 --> 00:24:34.370 align:middle line:84% question what number should we have here to make this 00:24:34.370 --> 00:24:36.500 align:middle line:90% equality to be true? 00:24:36.500 --> 00:24:39.360 align:middle line:84% It's again a problem of this kind. 00:24:39.360 --> 00:24:42.820 align:middle line:84% You have a quantity whose distribution you know. 00:24:42.820 --> 00:24:43.950 align:middle line:90% Why do you know it? 00:24:43.950 --> 00:24:46.390 align:middle line:84% The number of heads by the central limit theorem is 00:24:46.390 --> 00:24:47.970 align:middle line:90% approximately normal. 00:24:47.970 --> 00:24:51.560 align:middle line:84% So this here talks about the normal distribution. 00:24:51.560 --> 00:24:56.330 align:middle line:84% You set your alpha to be 5%, and you ask where should I put 00:24:56.330 --> 00:24:59.690 align:middle line:84% my threshold so that this probability of being out there 00:24:59.690 --> 00:25:01.530 align:middle line:90% is only 5%? 00:25:01.530 --> 00:25:03.750 align:middle line:84% Now in our particular example the threshold 00:25:03.750 --> 00:25:05.970 align:middle line:90% turned out to be 31. 00:25:05.970 --> 00:25:09.170 align:middle line:84% This number turned out was just 28 away 00:25:09.170 --> 00:25:10.960 align:middle line:90% from the correct mean. 00:25:10.960 --> 00:25:14.150 align:middle line:84% So these distance was less than the threshold. 00:25:14.150 --> 00:25:17.280 align:middle line:90% So we end up not rejecting H0. 00:25:17.280 --> 00:25:20.430 align:middle line:90% 00:25:20.430 --> 00:25:23.820 align:middle line:84% So we have our rejection region. 00:25:23.820 --> 00:25:28.900 align:middle line:84% The way we designed it is that when H0 is true there's only a 00:25:28.900 --> 00:25:32.960 align:middle line:84% small chance, 5%, that we get to data out of there. 00:25:32.960 --> 00:25:35.510 align:middle line:84% Data that we would call an outlier. 00:25:35.510 --> 00:25:39.330 align:middle line:84% If we see such an outlier we reject H0. 00:25:39.330 --> 00:25:43.930 align:middle line:84% If what we see is not an outlier as in this case, where 00:25:43.930 --> 00:25:47.090 align:middle line:84% that distance turned out to be kind of small, then we 00:25:47.090 --> 00:25:50.980 align:middle line:90% do not reject H0. 00:25:50.980 --> 00:25:54.700 align:middle line:84% An interesting little piece of language here, people 00:25:54.700 --> 00:25:57.490 align:middle line:84% generally prefer to use this terminology-- 00:25:57.490 --> 00:26:01.820 align:middle line:84% to say that H0 is not rejected by the data. 00:26:01.820 --> 00:26:06.490 align:middle line:84% Instead of saying that H0 is accepted. 00:26:06.490 --> 00:26:09.260 align:middle line:84% In some sense they're both saying the same thing, but the 00:26:09.260 --> 00:26:11.940 align:middle line:90% difference is sort of subtle. 00:26:11.940 --> 00:26:17.240 align:middle line:84% When I say not rejected what I mean is that I got some data 00:26:17.240 --> 00:26:20.560 align:middle line:84% that are compatible with my hypothesis. 00:26:20.560 --> 00:26:26.470 align:middle line:84% That is the data that I got do not falsify the hypothesis 00:26:26.470 --> 00:26:29.520 align:middle line:84% that I had, my null hypothesis. 00:26:29.520 --> 00:26:34.500 align:middle line:84% So my null hypothesis is still alive, and may be true. 00:26:34.500 --> 00:26:38.700 align:middle line:84% But from data you can never really prove that the 00:26:38.700 --> 00:26:41.360 align:middle line:90% hypothesis is correct. 00:26:41.360 --> 00:26:46.190 align:middle line:84% Perhaps my coin is not fair in some other complicated way. 00:26:46.190 --> 00:26:51.660 align:middle line:90% 00:26:51.660 --> 00:26:55.980 align:middle line:84% Perhaps I was just lucky, and even though my coin is not 00:26:55.980 --> 00:26:58.930 align:middle line:84% fair I ended up with an outcome that 00:26:58.930 --> 00:27:01.270 align:middle line:90% suggests that it's fair. 00:27:01.270 --> 00:27:04.600 align:middle line:84% Perhaps my coin flips are not independent as I 00:27:04.600 --> 00:27:06.020 align:middle line:90% assumed in my model. 00:27:06.020 --> 00:27:11.860 align:middle line:84% So there's many ways that my null hypothesis could be 00:27:11.860 --> 00:27:15.010 align:middle line:84% wrong, and still I got data that tells me that my 00:27:15.010 --> 00:27:16.970 align:middle line:90% hypothesis is OK. 00:27:16.970 --> 00:27:20.980 align:middle line:84% So this is the general way that things work in science. 00:27:20.980 --> 00:27:24.340 align:middle line:84% One comes up with a model or a theory. 00:27:24.340 --> 00:27:28.480 align:middle line:84% This is the default theory, and we work with that theory 00:27:28.480 --> 00:27:31.100 align:middle line:84% trying to find whether there are examples 00:27:31.100 --> 00:27:32.450 align:middle line:90% that violate the theory. 00:27:32.450 --> 00:27:35.550 align:middle line:84% If you find data and examples that violate the theory your 00:27:35.550 --> 00:27:38.560 align:middle line:84% theory is falsified, and you need to look for a new one. 00:27:38.560 --> 00:27:43.090 align:middle line:84% But when you have your theory, really no amount of data can 00:27:43.090 --> 00:27:45.810 align:middle line:84% prove that your theory is correct. 00:27:45.810 --> 00:27:49.950 align:middle line:84% So we have the default theory that the speed of light is 00:27:49.950 --> 00:27:54.620 align:middle line:84% constant as long as we do not find any data that runs 00:27:54.620 --> 00:27:56.210 align:middle line:90% counter to it. 00:27:56.210 --> 00:27:59.650 align:middle line:84% We stay with that theory, but there's no way of really 00:27:59.650 --> 00:28:03.710 align:middle line:84% proving this, no matter how many experiments we do. 00:28:03.710 --> 00:28:06.590 align:middle line:84% But there could be experiments that falsify that theory, in 00:28:06.590 --> 00:28:10.580 align:middle line:84% which case we need to do look for a new one. 00:28:10.580 --> 00:28:14.450 align:middle line:84% So there's a bit of an asymmetry here in how we treat 00:28:14.450 --> 00:28:16.510 align:middle line:90% the alternative hypothesis. 00:28:16.510 --> 00:28:22.900 align:middle line:84% H0 is the default which we'll accept until we see some 00:28:22.900 --> 00:28:25.350 align:middle line:90% evidence to the contrary. 00:28:25.350 --> 00:28:30.170 align:middle line:84% And if we see some evidence to the contrary we reject it. 00:28:30.170 --> 00:28:33.580 align:middle line:84% As long as we do not see evidence to the contrary then 00:28:33.580 --> 00:28:35.940 align:middle line:84% we keep working with it, but always take it 00:28:35.940 --> 00:28:38.200 align:middle line:90% with a grain of salt. 00:28:38.200 --> 00:28:42.210 align:middle line:84% You can never really prove that a coin has a bias exactly 00:28:42.210 --> 00:28:43.860 align:middle line:90% equal to 1/2. 00:28:43.860 --> 00:28:50.360 align:middle line:84% Maybe the bias is equal to 0.50001, so 00:28:50.360 --> 00:28:52.440 align:middle line:90% the bias is not 1/2. 00:28:52.440 --> 00:28:56.180 align:middle line:84% But with an experiment with 1,000 coin tosses you wouldn't 00:28:56.180 --> 00:28:59.200 align:middle line:90% be able to see this effect. 00:28:59.200 --> 00:29:03.750 align:middle line:90% 00:29:03.750 --> 00:29:07.870 align:middle line:84% OK, so that's how you go about testing about whether your 00:29:07.870 --> 00:29:09.120 align:middle line:90% coin is fair. 00:29:09.120 --> 00:29:13.150 align:middle line:84% You can also think about testing whether a die is fair. 00:29:13.150 --> 00:29:17.130 align:middle line:84% So for a die the null hypothesis would be that every 00:29:17.130 --> 00:29:21.830 align:middle line:84% possible result when you roll the die has equal probability 00:29:21.830 --> 00:29:23.860 align:middle line:90% and equal to 1/6. 00:29:23.860 --> 00:29:27.720 align:middle line:84% And you also make the hypothesis that your die rolls 00:29:27.720 --> 00:29:30.900 align:middle line:84% are statistically independent from each other. 00:29:30.900 --> 00:29:36.050 align:middle line:84% So I take my die, I roll it a number of times, little n, and 00:29:36.050 --> 00:29:40.240 align:middle line:84% I count how many 1's I got, how many 2's I got, how many 00:29:40.240 --> 00:29:43.430 align:middle line:84% 3's I got, and these are my data. 00:29:43.430 --> 00:29:48.400 align:middle line:84% I count how many times I observed a specific result in 00:29:48.400 --> 00:29:51.660 align:middle line:84% my die roll that was equal to sum i. 00:29:51.660 --> 00:29:53.410 align:middle line:90% And now I ask the question-- 00:29:53.410 --> 00:29:58.050 align:middle line:84% the Ni's that I observed, are they compatible with my 00:29:58.050 --> 00:30:01.000 align:middle line:90% hypothesis or not? 00:30:01.000 --> 00:30:05.560 align:middle line:84% What does compatible to my hypothesis mean? 00:30:05.560 --> 00:30:12.570 align:middle line:84% Under the null hypothesis Ni should be approximately equal, 00:30:12.570 --> 00:30:17.750 align:middle line:84% or is equal in expectation to N times little Pi. 00:30:17.750 --> 00:30:23.170 align:middle line:84% And in our example this little Pi is of course 1/6. 00:30:23.170 --> 00:30:28.210 align:middle line:84% So if my die is fair the number of ones I expect to see 00:30:28.210 --> 00:30:31.110 align:middle line:84% is equal to the number of rolls times 1/6. 00:30:31.110 --> 00:30:35.070 align:middle line:84% The number of 2's I expect to see is again that same number. 00:30:35.070 --> 00:30:37.970 align:middle line:84% Of course there's randomness, so I do not expect to get 00:30:37.970 --> 00:30:39.420 align:middle line:90% exactly that number. 00:30:39.420 --> 00:30:42.420 align:middle line:84% But I can ask how far away from the 00:30:42.420 --> 00:30:45.380 align:middle line:90% expected values was i? 00:30:45.380 --> 00:30:51.470 align:middle line:84% If my capital Ni's turn to be very different from N/6 this 00:30:51.470 --> 00:30:55.110 align:middle line:84% is evidence that my die is not fair. 00:30:55.110 --> 00:31:01.000 align:middle line:84% If those numbers turn out to be close to N times 1/6 then 00:31:01.000 --> 00:31:05.180 align:middle line:84% I'm going to say there's no evidence that would lead me to 00:31:05.180 --> 00:31:06.870 align:middle line:90% reject this hypothesis. 00:31:06.870 --> 00:31:10.850 align:middle line:84% So this hypothesis remains alive. 00:31:10.850 --> 00:31:16.390 align:middle line:84% So someone has come up with this thought that maybe the 00:31:16.390 --> 00:31:20.730 align:middle line:84% right statistic to use, or the right way of quantifying how 00:31:20.730 --> 00:31:23.910 align:middle line:84% far away are the Ni's from their mean is to 00:31:23.910 --> 00:31:25.590 align:middle line:90% look at this quantity. 00:31:25.590 --> 00:31:29.520 align:middle line:84% So I'm looking at the expected value of Ni under the null 00:31:29.520 --> 00:31:30.700 align:middle line:90% hypothesis. 00:31:30.700 --> 00:31:34.760 align:middle line:84% See what I got, take the square of this, and add it 00:31:34.760 --> 00:31:36.040 align:middle line:90% over all i's. 00:31:36.040 --> 00:31:40.930 align:middle line:84% But also throw in these terms in the denominator. 00:31:40.930 --> 00:31:46.010 align:middle line:84% And why that term is there, that's a longer story. 00:31:46.010 --> 00:31:49.740 align:middle line:84% One can write down certain likelihood ratios, do certain 00:31:49.740 --> 00:31:53.010 align:middle line:84% Taylor Series approximations, and there's a Heuristic 00:31:53.010 --> 00:31:58.120 align:middle line:84% argument that justifies why this would be a good form for 00:31:58.120 --> 00:31:59.810 align:middle line:90% the test to use. 00:31:59.810 --> 00:32:02.660 align:middle line:84% So there's a certain art that's involved in this step 00:32:02.660 --> 00:32:06.370 align:middle line:84% that some people somehow decided that it's a reasonable 00:32:06.370 --> 00:32:08.730 align:middle line:90% thing to do is to calcelate. 00:32:08.730 --> 00:32:12.300 align:middle line:84% Once you get your results to calculate this one-dimensional 00:32:12.300 --> 00:32:16.740 align:middle line:84% summary of your result, this is going to be your statistic, 00:32:16.740 --> 00:32:19.550 align:middle line:84% and compare that statistic to a threshold. 00:32:19.550 --> 00:32:21.680 align:middle line:84% And that's how you make your decision. 00:32:21.680 --> 00:32:27.310 align:middle line:84% So by this point we have fixed the type of the rejection 00:32:27.310 --> 00:32:29.740 align:middle line:84% region that we're going to have. 00:32:29.740 --> 00:32:32.780 align:middle line:84% So we've chosen the qualitative structure of our 00:32:32.780 --> 00:32:36.230 align:middle line:84% test, and the only thing that's now left is to choose 00:32:36.230 --> 00:32:38.820 align:middle line:84% the particular threshold we're going to use. 00:32:38.820 --> 00:32:41.550 align:middle line:84% And the recipe, once more, is the same. 00:32:41.550 --> 00:32:44.840 align:middle line:84% We want to set our threshold so that the probability of a 00:32:44.840 --> 00:32:47.320 align:middle line:90% false rejection is 5%. 00:32:47.320 --> 00:32:52.040 align:middle line:84% We want the probability that our data fall in here is only 00:32:52.040 --> 00:32:55.990 align:middle line:84% 5% when the null hypothesis is true. 00:32:55.990 --> 00:33:01.040 align:middle line:84% So that's the same as setting our threshold Xi so that the 00:33:01.040 --> 00:33:03.940 align:middle line:84% probability that our test statistic is 00:33:03.940 --> 00:33:05.960 align:middle line:90% bigger than that threshold. 00:33:05.960 --> 00:33:11.470 align:middle line:84% We want that probability to be only 0.05. 00:33:11.470 --> 00:33:15.140 align:middle line:84% So to solve a problem of this kind what is it 00:33:15.140 --> 00:33:16.820 align:middle line:90% that you need to do? 00:33:16.820 --> 00:33:19.490 align:middle line:84% You need to find the probability distribution of 00:33:19.490 --> 00:33:23.810 align:middle line:84% capital T. So once more it's the same picture. 00:33:23.810 --> 00:33:26.370 align:middle line:90% 00:33:26.370 --> 00:33:32.200 align:middle line:84% You need to do some calculations of some sort, and 00:33:32.200 --> 00:33:36.550 align:middle line:84% come up with the distribution of the random variable T, 00:33:36.550 --> 00:33:39.060 align:middle line:90% where T is defined this way. 00:33:39.060 --> 00:33:41.400 align:middle line:84% You want to find this distribution 00:33:41.400 --> 00:33:43.190 align:middle line:90% under hypothesis H0. 00:33:43.190 --> 00:33:48.820 align:middle line:90% 00:33:48.820 --> 00:33:53.780 align:middle line:84% Once you find what that distribution is then you can 00:33:53.780 --> 00:33:55.480 align:middle line:90% solve this usual problem. 00:33:55.480 --> 00:33:58.470 align:middle line:84% I want this probability here to be 5%. 00:33:58.470 --> 00:34:01.860 align:middle line:90% What should my threshold be? 00:34:01.860 --> 00:34:03.930 align:middle line:84% So what does this boil down to? 00:34:03.930 --> 00:34:08.510 align:middle line:84% Finding the distribution of capital T is in some sense a 00:34:08.510 --> 00:34:13.350 align:middle line:84% messy, difficult, derived distribution problem. 00:34:13.350 --> 00:34:16.239 align:middle line:84% From this model we know the distribution 00:34:16.239 --> 00:34:17.489 align:middle line:90% of the capital Ni's. 00:34:17.489 --> 00:34:20.290 align:middle line:90% 00:34:20.290 --> 00:34:23.800 align:middle line:84% And actually we can even write down the joint distribution of 00:34:23.800 --> 00:34:26.840 align:middle line:90% the capital Ni's. 00:34:26.840 --> 00:34:29.690 align:middle line:84% In fact we can make an approximation here. 00:34:29.690 --> 00:34:33.219 align:middle line:84% Capital Ni is a binomial random variable. 00:34:33.219 --> 00:34:39.790 align:middle line:84% Let's say the number of 1's that I got in little N rolls 00:34:39.790 --> 00:34:41.090 align:middle line:90% off my die. 00:34:41.090 --> 00:34:43.300 align:middle line:84% So that's a binomial random variable. 00:34:43.300 --> 00:34:45.860 align:middle line:84% When little n is big this is going to be 00:34:45.860 --> 00:34:48.040 align:middle line:90% approximately normal. 00:34:48.040 --> 00:34:52.060 align:middle line:84% So we have normal random variables, or approximately 00:34:52.060 --> 00:34:54.260 align:middle line:90% normal minus a constant. 00:34:54.260 --> 00:34:55.770 align:middle line:84% They're still approximately normal. 00:34:55.770 --> 00:35:01.070 align:middle line:84% We take the squares of these, scale them so you can solve a 00:35:01.070 --> 00:35:03.730 align:middle line:84% derived distribution problem to find the distribution of 00:35:03.730 --> 00:35:04.930 align:middle line:90% this quantity. 00:35:04.930 --> 00:35:08.550 align:middle line:84% You can do more work, more derived distribution work, and 00:35:08.550 --> 00:35:12.080 align:middle line:84% find the distribution of capital T. So this is a 00:35:12.080 --> 00:35:17.500 align:middle line:84% tedious matter, but because this test is used quite often, 00:35:17.500 --> 00:35:20.080 align:middle line:84% again people have done those calculations. 00:35:20.080 --> 00:35:23.600 align:middle line:84% They have found the distribution of capital T, and 00:35:23.600 --> 00:35:25.250 align:middle line:90% it's available in tables. 00:35:25.250 --> 00:35:29.090 align:middle line:84% And you go to those tables, and you find the appropriate 00:35:29.090 --> 00:35:31.370 align:middle line:84% threshold for making a decision of this type. 00:35:31.370 --> 00:35:36.160 align:middle line:90% 00:35:36.160 --> 00:35:40.720 align:middle line:84% Now to give you a sense of how complicated hypothesis one 00:35:40.720 --> 00:35:47.190 align:middle line:84% might have to deal with let's make things one level more 00:35:47.190 --> 00:35:48.370 align:middle line:90% complicated. 00:35:48.370 --> 00:35:55.200 align:middle line:84% So here you can think this X is a discrete random variable. 00:35:55.200 --> 00:35:57.770 align:middle line:84% This is the outcome of my roll. 00:35:57.770 --> 00:36:02.760 align:middle line:84% And I had a model in which the possible values of my discrete 00:36:02.760 --> 00:36:06.030 align:middle line:84% random variables they have probabilities 00:36:06.030 --> 00:36:07.870 align:middle line:90% all equal to 1/6. 00:36:07.870 --> 00:36:13.280 align:middle line:84% So my null hypothesis here was a particular PMF for the 00:36:13.280 --> 00:36:17.810 align:middle line:84% random variable capital X. So another way of phrasing what 00:36:17.810 --> 00:36:19.950 align:middle line:84% happened in this problem was the 00:36:19.950 --> 00:36:24.700 align:middle line:90% question is my PMF correct? 00:36:24.700 --> 00:36:30.580 align:middle line:84% So this is the PMF of the result of one die roll. 00:36:30.580 --> 00:36:33.950 align:middle line:84% You're asking the question is my PMF correct? 00:36:33.950 --> 00:36:36.740 align:middle line:90% Make it more complicated. 00:36:36.740 --> 00:36:41.510 align:middle line:84% How about the question of the type is my PDF correct when I 00:36:41.510 --> 00:36:45.220 align:middle line:90% have continuous data? 00:36:45.220 --> 00:36:50.900 align:middle line:84% So I have hypothesized that's the probability distribution 00:36:50.900 --> 00:36:54.780 align:middle line:84% that I have is let's say a particular normal. 00:36:54.780 --> 00:36:58.990 align:middle line:84% I get lots of results from that random variable. 00:36:58.990 --> 00:37:04.450 align:middle line:84% Can I tell whether my results look like normal or not? 00:37:04.450 --> 00:37:06.650 align:middle line:84% What are some ways of going about it? 00:37:06.650 --> 00:37:09.450 align:middle line:84% Well, we saw in the previous slide that there is a 00:37:09.450 --> 00:37:13.110 align:middle line:84% methodology for deciding if your PMF is correct. 00:37:13.110 --> 00:37:19.090 align:middle line:84% So you could take your normal results, the data that you got 00:37:19.090 --> 00:37:23.200 align:middle line:84% from your experiment, and discretize them, and so now 00:37:23.200 --> 00:37:25.500 align:middle line:84% you're dealing with discrete data. 00:37:25.500 --> 00:37:31.200 align:middle line:84% And sort of used in previous methodology to solve a 00:37:31.200 --> 00:37:34.900 align:middle line:84% discrete problem of the type is my PDF correct? 00:37:34.900 --> 00:37:41.320 align:middle line:84% So in practice the way this is done is that you get all your 00:37:41.320 --> 00:37:49.920 align:middle line:84% data, let's say data points of this kind. 00:37:49.920 --> 00:37:56.400 align:middle line:84% You split your space into bins, and you count how many 00:37:56.400 --> 00:38:00.190 align:middle line:90% you have in each bin. 00:38:00.190 --> 00:38:07.180 align:middle line:84% So you get this, and that, and that, and nothing. 00:38:07.180 --> 00:38:10.020 align:middle line:84% So that's a histogram that you get from the 00:38:10.020 --> 00:38:11.020 align:middle line:90% data that you have. 00:38:11.020 --> 00:38:14.670 align:middle line:84% Like the very familiar histograms that you see after 00:38:14.670 --> 00:38:16.860 align:middle line:90% each one of our quizzes. 00:38:16.860 --> 00:38:21.760 align:middle line:84% So if you look at these histogram, and you ask does it 00:38:21.760 --> 00:38:24.060 align:middle line:90% look like normal? 00:38:24.060 --> 00:38:27.700 align:middle line:84% OK, we need a systematic way of going about it. 00:38:27.700 --> 00:38:33.140 align:middle line:84% If it were normal you can calculate the probability of 00:38:33.140 --> 00:38:36.760 align:middle line:90% falling in this interval. 00:38:36.760 --> 00:38:39.120 align:middle line:84% The probability of falling in that interval, probability of 00:38:39.120 --> 00:38:40.890 align:middle line:90% falling into that interval. 00:38:40.890 --> 00:38:45.480 align:middle line:84% So you would have expected values of how many results, or 00:38:45.480 --> 00:38:48.210 align:middle line:84% data points, you would have in this interval. 00:38:48.210 --> 00:38:52.170 align:middle line:84% And compare these expected values for each interval with 00:38:52.170 --> 00:38:54.830 align:middle line:84% the actual ones that you observed. 00:38:54.830 --> 00:38:58.290 align:middle line:84% And then take the sum of squares, and so on, exactly as 00:38:58.290 --> 00:38:59.700 align:middle line:90% in the previous slide. 00:38:59.700 --> 00:39:03.010 align:middle line:84% And this gives you a way of going about it. 00:39:03.010 --> 00:39:07.060 align:middle line:90% 00:39:07.060 --> 00:39:09.710 align:middle line:90% This is a little messy. 00:39:09.710 --> 00:39:14.530 align:middle line:84% It gets hard to do because you have the difficult decision of 00:39:14.530 --> 00:39:19.180 align:middle line:84% how do you choose the bin size? 00:39:19.180 --> 00:39:22.430 align:middle line:84% If you take your bins to be very narrow you would get lots 00:39:22.430 --> 00:39:25.680 align:middle line:84% of bins with 0's, and a few bins that only have one 00:39:25.680 --> 00:39:26.840 align:middle line:90% outcome in them. 00:39:26.840 --> 00:39:29.120 align:middle line:84% It probably wouldn't feel right. 00:39:29.120 --> 00:39:32.110 align:middle line:84% If you choose your bins to be very wide then you're losing a 00:39:32.110 --> 00:39:33.680 align:middle line:90% lot of information. 00:39:33.680 --> 00:39:39.240 align:middle line:84% Is there some way of making a test without creating bins? 00:39:39.240 --> 00:39:43.330 align:middle line:84% This is just to illustrate the clever ideas of what 00:39:43.330 --> 00:39:45.640 align:middle line:84% statisticians have thought about. 00:39:45.640 --> 00:39:51.960 align:middle line:84% And here's a really cute way of going about a test, whether 00:39:51.960 --> 00:39:53.750 align:middle line:84% my distribution is correct or not. 00:39:53.750 --> 00:39:56.980 align:middle line:90% 00:39:56.980 --> 00:40:00.790 align:middle line:84% Here we're essentially plotting a PMF, or an 00:40:00.790 --> 00:40:02.630 align:middle line:90% approximation of a PDF. 00:40:02.630 --> 00:40:06.040 align:middle line:84% And we ask does it look like the PDF we assumed? 00:40:06.040 --> 00:40:09.930 align:middle line:84% Instead of working with PDFs let's work with cumulative 00:40:09.930 --> 00:40:11.800 align:middle line:90% distribution functions. 00:40:11.800 --> 00:40:13.840 align:middle line:90% So how does this go? 00:40:13.840 --> 00:40:20.160 align:middle line:84% The true normal distribution that I have hypothesized, the 00:40:20.160 --> 00:40:22.310 align:middle line:84% density that I'm hypothesizing-- my null 00:40:22.310 --> 00:40:23.350 align:middle line:90% hypothesis-- 00:40:23.350 --> 00:40:26.950 align:middle line:84% has a certain CDF that I can plot. 00:40:26.950 --> 00:40:36.820 align:middle line:84% So supposed that my hypothesis H0 is that the X's are normal 00:40:36.820 --> 00:40:42.630 align:middle line:84% with our standard normals, and I plot the CDF of the standard 00:40:42.630 --> 00:40:46.360 align:middle line:84% normal, which is the sort of continuous looking curve here. 00:40:46.360 --> 00:40:53.310 align:middle line:84% Now I get my data, and I plot the empirical CDF. 00:40:53.310 --> 00:40:54.930 align:middle line:90% What's the empirical CDF? 00:40:54.930 --> 00:40:59.830 align:middle line:84% In the empirical CDF you ask the question what fraction of 00:40:59.830 --> 00:41:02.940 align:middle line:90% the data fell below 0? 00:41:02.940 --> 00:41:04.450 align:middle line:90% You get a number. 00:41:04.450 --> 00:41:07.920 align:middle line:84% What fraction of my data fell below 1? 00:41:07.920 --> 00:41:08.730 align:middle line:90% I get a number. 00:41:08.730 --> 00:41:12.590 align:middle line:84% What fraction of my data fell below 2, and so on. 00:41:12.590 --> 00:41:15.780 align:middle line:84% So you're talking about fractions of the data that 00:41:15.780 --> 00:41:18.760 align:middle line:84% fell below each particular number. 00:41:18.760 --> 00:41:21.640 align:middle line:84% And by plotting those fractions as a function of 00:41:21.640 --> 00:41:26.740 align:middle line:84% this number you get something that looks like a CDF. 00:41:26.740 --> 00:41:31.670 align:middle line:84% And it's the CDF suggested by the data. 00:41:31.670 --> 00:41:35.800 align:middle line:84% Now the fraction of the data that fall below 0 in my 00:41:35.800 --> 00:41:38.530 align:middle line:90% experiment is-- 00:41:38.530 --> 00:41:43.280 align:middle line:90% if my hypothesis were true-- 00:41:43.280 --> 00:41:46.470 align:middle line:90% expected to be 1/2. 00:41:46.470 --> 00:41:49.280 align:middle line:84% 1/2 is the value of the true CDF. 00:41:49.280 --> 00:41:51.730 align:middle line:84% I look at the fraction that I got, it's 00:41:51.730 --> 00:41:54.470 align:middle line:90% expected to be that number. 00:41:54.470 --> 00:41:56.800 align:middle line:84% But there's randomness, so it's might be a little 00:41:56.800 --> 00:41:58.300 align:middle line:90% different than that. 00:41:58.300 --> 00:42:03.490 align:middle line:84% For any particular value, the fraction that I got below a 00:42:03.490 --> 00:42:04.350 align:middle line:90% certain number-- 00:42:04.350 --> 00:42:09.970 align:middle line:84% the fraction of data that we're below, 2, its 00:42:09.970 --> 00:42:15.310 align:middle line:84% expectation is the probability of falling below 2, which is 00:42:15.310 --> 00:42:16.740 align:middle line:90% the correct CDF. 00:42:16.740 --> 00:42:21.060 align:middle line:84% So if my hypothesis is true the empirical CDF that I get 00:42:21.060 --> 00:42:24.900 align:middle line:84% based on data should, when n is large, be very 00:42:24.900 --> 00:42:27.100 align:middle line:90% close to the true CDF. 00:42:27.100 --> 00:42:31.350 align:middle line:84% So a way of judging whether my model is correct or not is to 00:42:31.350 --> 00:42:38.300 align:middle line:84% look at the assumed CDF, the CDF under hypothesis H0. 00:42:38.300 --> 00:42:41.880 align:middle line:84% Look at the CDF that I constructed based on the data, 00:42:41.880 --> 00:42:45.440 align:middle line:84% and see whether they're close enough or not. 00:42:45.440 --> 00:42:48.150 align:middle line:84% And by close enough, I mean I'm going to look at all the 00:42:48.150 --> 00:42:52.000 align:middle line:84% possible X's, and look at the maximum distance between those 00:42:52.000 --> 00:42:53.300 align:middle line:90% two curves. 00:42:53.300 --> 00:42:59.140 align:middle line:84% And I'm going to have a test that decides in favor of H0 if 00:42:59.140 --> 00:43:03.550 align:middle line:84% this distance is small, and in favor of H1 if 00:43:03.550 --> 00:43:06.110 align:middle line:90% this distance is large. 00:43:06.110 --> 00:43:07.790 align:middle line:84% That still leaves me the problem of 00:43:07.790 --> 00:43:09.570 align:middle line:90% coming up with a threshold. 00:43:09.570 --> 00:43:13.180 align:middle line:84% Where exactly do I put my threshold? 00:43:13.180 --> 00:43:17.230 align:middle line:84% Because this test is important enough, and is used frequently 00:43:17.230 --> 00:43:20.990 align:middle line:84% people have made the effort to try to understand the 00:43:20.990 --> 00:43:23.240 align:middle line:84% probability distribution of this quite 00:43:23.240 --> 00:43:25.280 align:middle line:90% difficult random variable. 00:43:25.280 --> 00:43:28.220 align:middle line:84% One needs to do lots of approximations and clever 00:43:28.220 --> 00:43:32.550 align:middle line:84% calculations, but these have led to values and tabulated 00:43:32.550 --> 00:43:34.570 align:middle line:84% values for the probability distribution 00:43:34.570 --> 00:43:36.210 align:middle line:90% of this random variable. 00:43:36.210 --> 00:43:39.340 align:middle line:84% And, for example, those tabulated values tell us that 00:43:39.340 --> 00:43:45.030 align:middle line:84% if we want 5% false rejection probability, then our 00:43:45.030 --> 00:43:48.860 align:middle line:84% threshold should be 1.36 divided by the 00:43:48.860 --> 00:43:50.570 align:middle line:90% square root of n. 00:43:50.570 --> 00:43:53.870 align:middle line:84% So we know where to put our threshold for 00:43:53.870 --> 00:43:55.280 align:middle line:90% this particular value. 00:43:55.280 --> 00:43:59.680 align:middle line:84% If we want this particular error or error 00:43:59.680 --> 00:44:02.380 align:middle line:90% probability to occur. 00:44:02.380 --> 00:44:06.320 align:middle line:84% So that's about as hard and sophisticated classical 00:44:06.320 --> 00:44:08.070 align:middle line:90% statistics get. 00:44:08.070 --> 00:44:12.920 align:middle line:84% You want to have tests for hypotheses that are not so 00:44:12.920 --> 00:44:15.910 align:middle line:90% easy to handle. 00:44:15.910 --> 00:44:21.260 align:middle line:84% People somehow think of clever ways of doing 00:44:21.260 --> 00:44:22.500 align:middle line:90% tests of this kind. 00:44:22.500 --> 00:44:26.970 align:middle line:84% How to compare the theoretical predictions with the observed 00:44:26.970 --> 00:44:29.650 align:middle line:84% predictions with the observed data. 00:44:29.650 --> 00:44:34.430 align:middle line:84% Come up with some measure of the difference between theory 00:44:34.430 --> 00:44:38.270 align:middle line:84% and data, and if that difference is big, than you 00:44:38.270 --> 00:44:39.520 align:middle line:90% reject your hypothesis. 00:44:39.520 --> 00:44:42.340 align:middle line:90% 00:44:42.340 --> 00:44:45.640 align:middle line:84% OK, of course that's not the end of the field of 00:44:45.640 --> 00:44:49.000 align:middle line:84% statistics, there's a lot more. 00:44:49.000 --> 00:44:52.000 align:middle line:84% In some ways, as we kept moving through today's 00:44:52.000 --> 00:44:55.240 align:middle line:84% lecture, the way that we constructed those rejection 00:44:55.240 --> 00:44:57.680 align:middle line:84% regions was more and more ad hoc. 00:44:57.680 --> 00:45:02.220 align:middle line:84% I pulled out of a hat a particular measure of fit 00:45:02.220 --> 00:45:04.980 align:middle line:90% between data and the model. 00:45:04.980 --> 00:45:09.470 align:middle line:84% And I said let's just use a test based on this. 00:45:09.470 --> 00:45:13.890 align:middle line:84% There are attempts at more or less systematic ways of coming 00:45:13.890 --> 00:45:17.350 align:middle line:84% up with the general shape of rejection regions that have at 00:45:17.350 --> 00:45:20.540 align:middle line:84% least some desirable or favorable theoretical 00:45:20.540 --> 00:45:21.790 align:middle line:90% properties. 00:45:21.790 --> 00:45:24.620 align:middle line:90% 00:45:24.620 --> 00:45:28.300 align:middle line:84% Some more specific problems that people study-- 00:45:28.300 --> 00:45:31.690 align:middle line:84% instead of having a test, is this the correct PDF? 00:45:31.690 --> 00:45:33.140 align:middle line:90% Yes or no. 00:45:33.140 --> 00:45:37.670 align:middle line:84% I just give you data, and I ask you tell me, give me a 00:45:37.670 --> 00:45:41.270 align:middle line:90% model or a PDF for those data. 00:45:41.270 --> 00:45:45.000 align:middle line:84% OK, my thoughts of this kind are of many types. 00:45:45.000 --> 00:45:50.640 align:middle line:84% One general method is you form a histogram, and then you take 00:45:50.640 --> 00:45:54.570 align:middle line:84% your histogram and plot a smooth line, that kind of fits 00:45:54.570 --> 00:45:55.680 align:middle line:90% the histogram. 00:45:55.680 --> 00:45:59.140 align:middle line:84% This still leaves the question of how do you choose the bins? 00:45:59.140 --> 00:46:00.780 align:middle line:84% The bin size in your histograms. 00:46:00.780 --> 00:46:02.620 align:middle line:90% How narrow do you take them? 00:46:02.620 --> 00:46:05.920 align:middle line:84% And that depends on how many data you have, and there's a 00:46:05.920 --> 00:46:09.190 align:middle line:84% lot of theory that tells you about the best way of choosing 00:46:09.190 --> 00:46:12.890 align:middle line:84% the bin sizes, and the best ways of smoothing the data 00:46:12.890 --> 00:46:14.640 align:middle line:90% that you have. 00:46:14.640 --> 00:46:18.090 align:middle line:84% A completely different topic is in signal processing -- 00:46:18.090 --> 00:46:20.200 align:middle line:90% you want to do your inference. 00:46:20.200 --> 00:46:22.810 align:middle line:84% Not only you want it to be good, but you also want it to 00:46:22.810 --> 00:46:25.520 align:middle line:84% be fast in a computational way. 00:46:25.520 --> 00:46:28.010 align:middle line:84% You get data in real time, lots of data. 00:46:28.010 --> 00:46:31.330 align:middle line:84% You want to keep processing and revising your estimates 00:46:31.330 --> 00:46:35.220 align:middle line:84% and your decisions as they come and go. 00:46:35.220 --> 00:46:38.950 align:middle line:84% Another topic that was briefly touched upon the last couple 00:46:38.950 --> 00:46:43.010 align:middle line:84% of lectures is that when you set up a model, like a linear 00:46:43.010 --> 00:46:46.540 align:middle line:84% regression model, you choose some explanatory variables, 00:46:46.540 --> 00:46:50.230 align:middle line:84% and you try to predict y from your X, these variables. 00:46:50.230 --> 00:46:52.720 align:middle line:84% You have a choice of what to take as 00:46:52.720 --> 00:46:55.440 align:middle line:90% your explanatory variables. 00:46:55.440 --> 00:47:02.560 align:middle line:84% Are there systematic ways of picking the right X variables 00:47:02.560 --> 00:47:04.520 align:middle line:90% to try to estimate a Y. 00:47:04.520 --> 00:47:08.360 align:middle line:84% For example should I try to estimate Y on the basis of X? 00:47:08.360 --> 00:47:10.320 align:middle line:90% Or on the basis of X-squared? 00:47:10.320 --> 00:47:12.960 align:middle line:84% How do I decide between the two? 00:47:12.960 --> 00:47:17.000 align:middle line:84% Finally, the rage these days has to do with anything big, 00:47:17.000 --> 00:47:18.490 align:middle line:90% high-demensional. 00:47:18.490 --> 00:47:23.410 align:middle line:84% Complicated models of complicated things, and tons 00:47:23.410 --> 00:47:24.650 align:middle line:90% and tons of data. 00:47:24.650 --> 00:47:27.430 align:middle line:84% So these days data are generated everywhere. 00:47:27.430 --> 00:47:30.230 align:middle line:84% The amounts of data are humongous. 00:47:30.230 --> 00:47:33.120 align:middle line:84% Also, the problems that people are interested in tend to be 00:47:33.120 --> 00:47:35.500 align:middle line:84% very complicated with lots of parameters. 00:47:35.500 --> 00:47:39.800 align:middle line:84% So I need specially tailored methods that can give you good 00:47:39.800 --> 00:47:44.220 align:middle line:84% results, or decent results even in the face of these huge 00:47:44.220 --> 00:47:47.290 align:middle line:84% amounts of data, and possibly with computational 00:47:47.290 --> 00:47:48.310 align:middle line:90% constraints. 00:47:48.310 --> 00:47:50.720 align:middle line:84% So with huge amounts of data you want methods that are 00:47:50.720 --> 00:47:56.460 align:middle line:84% simple, but still can deliver for you meaningful answers. 00:47:56.460 --> 00:48:00.170 align:middle line:84% Now as I mentioned some time ago, this whole field of 00:48:00.170 --> 00:48:03.960 align:middle line:84% statistics is very different from the field of probability. 00:48:03.960 --> 00:48:06.530 align:middle line:84% In some sense all that we're doing in statistics is 00:48:06.530 --> 00:48:08.100 align:middle line:90% probabilistic calculations. 00:48:08.100 --> 00:48:10.360 align:middle line:84% That's what the theory kind of does. 00:48:10.360 --> 00:48:12.870 align:middle line:84% But there's a big element of art. 00:48:12.870 --> 00:48:16.550 align:middle line:84% You saw that we chose the shape of some decision regions 00:48:16.550 --> 00:48:19.840 align:middle line:84% or rejection regions in a somewhat ad hoc way. 00:48:19.840 --> 00:48:21.660 align:middle line:84% There's even more basic things. 00:48:21.660 --> 00:48:23.260 align:middle line:90% How do you organize your data? 00:48:23.260 --> 00:48:26.690 align:middle line:84% How do you think about which hypotheses you would like to 00:48:26.690 --> 00:48:28.300 align:middle line:90% test, and so on. 00:48:28.300 --> 00:48:31.710 align:middle line:84% There's a lot of art that's involved here, and there's a 00:48:31.710 --> 00:48:33.510 align:middle line:90% lot that can go wrong. 00:48:33.510 --> 00:48:36.630 align:middle line:84% So I'm going to close with a note that you can take either 00:48:36.630 --> 00:48:39.050 align:middle line:90% as pessimistic or optimistic. 00:48:39.050 --> 00:48:42.880 align:middle line:84% There is a famous paper that came out a few years ago and 00:48:42.880 --> 00:48:46.440 align:middle line:84% has been cited about a 1,000 times or so. 00:48:46.440 --> 00:48:50.110 align:middle line:84% And the title of the paper is Why Most Published Research 00:48:50.110 --> 00:48:51.850 align:middle line:90% Findings Are False. 00:48:51.850 --> 00:48:56.080 align:middle line:84% And it's actually a very good argument why, in fields like 00:48:56.080 --> 00:48:59.900 align:middle line:84% psychology or the medical science and all that a lot of 00:48:59.900 --> 00:49:01.160 align:middle line:90% what you see published-- 00:49:01.160 --> 00:49:03.410 align:middle line:84% that yes, this drug has an effect on 00:49:03.410 --> 00:49:05.000 align:middle line:90% that particular disease-- 00:49:05.000 --> 00:49:08.030 align:middle line:84% is actually false, because people do not do their 00:49:08.030 --> 00:49:09.780 align:middle line:90% statistics correctly. 00:49:09.780 --> 00:49:12.130 align:middle line:84% There's lots of biases in what people do. 00:49:12.130 --> 00:49:16.300 align:middle line:84% I mean an obvious bias is that you only published a result 00:49:16.300 --> 00:49:19.190 align:middle line:90% when you see something. 00:49:19.190 --> 00:49:22.770 align:middle line:84% So the null hypothesis is that the drug doesn't work. 00:49:22.770 --> 00:49:26.820 align:middle line:84% You do your tests, the drug didn't work, OK, you just go 00:49:26.820 --> 00:49:27.960 align:middle line:90% home and cry. 00:49:27.960 --> 00:49:33.380 align:middle line:84% But if by accident that 5% happens, and even though the 00:49:33.380 --> 00:49:37.320 align:middle line:84% drug doesn't work, you got some outlier data, and it 00:49:37.320 --> 00:49:38.760 align:middle line:90% seemed to be working. 00:49:38.760 --> 00:49:40.990 align:middle line:84% Then you're excited, you publish it. 00:49:40.990 --> 00:49:42.760 align:middle line:90% So that's clearly a bias. 00:49:42.760 --> 00:49:46.980 align:middle line:84% That gets results to be published, even though they do 00:49:46.980 --> 00:49:50.330 align:middle line:84% not have a solid foundation behind them. 00:49:50.330 --> 00:49:53.050 align:middle line:84% Then there's another thing, OK? 00:49:53.050 --> 00:49:55.440 align:middle line:90% I'm picking my 5%. 00:49:55.440 --> 00:49:59.940 align:middle line:84% So H0 is true there's a small probability that the data will 00:49:59.940 --> 00:50:04.160 align:middle line:84% look like an outlier, and in that case I 00:50:04.160 --> 00:50:06.270 align:middle line:90% published my result. 00:50:06.270 --> 00:50:08.160 align:middle line:90% OK it's only 5% -- 00:50:08.160 --> 00:50:10.300 align:middle line:84% it's not going to happen too often. 00:50:10.300 --> 00:50:15.200 align:middle line:84% But suppose that I go and do a 1,000 different tests? 00:50:15.200 --> 00:50:18.540 align:middle line:84% Test H0 against this hypothesis, test H0 against 00:50:18.540 --> 00:50:22.000 align:middle line:84% that hypothesis , test H0 against that hypothesis. 00:50:22.000 --> 00:50:26.230 align:middle line:84% Some of these tests, just by accident might turn out to be 00:50:26.230 --> 00:50:29.350 align:middle line:84% in favor of H1, and again these are 00:50:29.350 --> 00:50:31.170 align:middle line:90% selected to be published. 00:50:31.170 --> 00:50:35.720 align:middle line:84% So if you do lots and lots of tests and in each one you have 00:50:35.720 --> 00:50:38.980 align:middle line:84% a 5% probability of error, when you consider the 00:50:38.980 --> 00:50:41.980 align:middle line:84% collection of all those tests, actually the probability of 00:50:41.980 --> 00:50:46.940 align:middle line:84% making incorrect inferences is a lot more than 5%. 00:50:46.940 --> 00:50:51.400 align:middle line:84% One basic principle in being systematic about such studies 00:50:51.400 --> 00:50:55.950 align:middle line:84% is that you should first pick your hypothesis that you're 00:50:55.950 --> 00:50:59.230 align:middle line:84% going to test, then get your data, and do 00:50:59.230 --> 00:51:00.880 align:middle line:90% your hypothesis testing. 00:51:00.880 --> 00:51:05.640 align:middle line:84% What would be wrong is to get your data, look at them, and 00:51:05.640 --> 00:51:08.890 align:middle line:84% say OK I'm going now to test for these 100 different 00:51:08.890 --> 00:51:13.060 align:middle line:84% hypotheses, and I'm going to choose my hypothesis to be for 00:51:13.060 --> 00:51:16.580 align:middle line:84% features that look abnormal in my data. 00:51:16.580 --> 00:51:19.520 align:middle line:84% Well, given enough data, you can always find some 00:51:19.520 --> 00:51:21.650 align:middle line:90% abnormalities just by chance. 00:51:21.650 --> 00:51:24.380 align:middle line:84% And if you choose to make a statistical test-- 00:51:24.380 --> 00:51:26.710 align:middle line:90% is this abnormality present? 00:51:26.710 --> 00:51:28.090 align:middle line:90% Yes, it will be present. 00:51:28.090 --> 00:51:31.020 align:middle line:84% Because you first found the abnormality, and then you 00:51:31.020 --> 00:51:32.130 align:middle line:90% tested for it. 00:51:32.130 --> 00:51:35.210 align:middle line:84% So that's another way that things can go wrong. 00:51:35.210 --> 00:51:37.520 align:middle line:84% So the moral of this story is that while the world of 00:51:37.520 --> 00:51:40.200 align:middle line:84% probability is really beautiful and solid, you have 00:51:40.200 --> 00:51:40.960 align:middle line:90% your axioms. 00:51:40.960 --> 00:51:44.630 align:middle line:84% Every question has a unique answer that by now you can, 00:51:44.630 --> 00:51:48.250 align:middle line:84% all of you, find in a very reliable way. 00:51:48.250 --> 00:51:50.740 align:middle line:84% Statistics is a dirty and difficult business. 00:51:50.740 --> 00:51:53.010 align:middle line:84% And that's why the subject is not over. 00:51:53.010 --> 00:51:55.430 align:middle line:84% And if you're interested in it, it's worth taking 00:51:55.430 --> 00:51:58.920 align:middle line:84% follow-on courses in that direction. 00:51:58.920 --> 00:52:03.950 align:middle line:84% OK so have good luck in the final, do well, and have a 00:52:03.950 --> 00:52:05.200 align:middle line:90% nice vacation afterwards. 00:52:05.200 --> 00:52:06.260 align:middle line:90%