WEBVTT 00:00:00.790 --> 00:00:03.130 The following content is provided under a Creative 00:00:03.130 --> 00:00:04.550 Commons license. 00:00:04.550 --> 00:00:06.760 Your support will help MIT OpenCourseWare 00:00:06.760 --> 00:00:10.850 continue to offer high quality educational resources for free. 00:00:10.850 --> 00:00:13.390 To make a donation or to view additional materials 00:00:13.390 --> 00:00:17.320 from hundreds of MIT courses, visit MIT OpenCourseWare 00:00:17.320 --> 00:00:18.570 at ocw.mit.edu. 00:00:29.170 --> 00:00:31.300 JOHN GUTTAG: Welcome to Lecture 6. 00:00:31.300 --> 00:00:34.900 As usual, I want to start by posting some relevant reading. 00:00:37.650 --> 00:00:39.950 For those who don't know, this lovely picture 00:00:39.950 --> 00:00:46.300 is of the Casino at Monte Carlo, and shortly you'll 00:00:46.300 --> 00:00:51.820 see why we're talking about casinos and gambling today. 00:00:51.820 --> 00:00:54.550 Not because I want to encourage you to gamble your life 00:00:54.550 --> 00:00:56.500 savings away. 00:00:56.500 --> 00:00:59.390 A little history about Monte Carlo simulation, 00:00:59.390 --> 00:01:02.060 which is the topic of today's lecture. 00:01:02.060 --> 00:01:06.750 The concept was invented by the Polish American mathematician, 00:01:06.750 --> 00:01:07.940 Stanislaw Ulam. 00:01:11.380 --> 00:01:15.250 Probably more well known for his work on thermonuclear weapons 00:01:15.250 --> 00:01:17.770 than on mathematics, but he did do 00:01:17.770 --> 00:01:19.660 a lot of very important mathematics 00:01:19.660 --> 00:01:22.240 earlier in his life. 00:01:22.240 --> 00:01:25.930 The story here starts that he was ill, 00:01:25.930 --> 00:01:28.000 recovering from some serious illness, 00:01:28.000 --> 00:01:30.160 and was home and was bored and was 00:01:30.160 --> 00:01:33.820 playing a lot of games of solitaire, a game I 00:01:33.820 --> 00:01:36.100 suspect you've all played. 00:01:36.100 --> 00:01:38.950 Being a mathematician, he naturally wondered, 00:01:38.950 --> 00:01:41.950 what's the probability of my winning this stupid game which 00:01:41.950 --> 00:01:43.930 I keep losing? 00:01:43.930 --> 00:01:46.480 And so he actually spent quite a lot of time 00:01:46.480 --> 00:01:49.510 trying to work out the combinatorics, 00:01:49.510 --> 00:01:52.540 so that he could actually compute the probability. 00:01:52.540 --> 00:01:55.490 And despite being a really amazing mathematician, 00:01:55.490 --> 00:01:56.710 he failed. 00:01:56.710 --> 00:01:59.750 The combinatorics were just too complicated. 00:01:59.750 --> 00:02:03.870 So he thought, well suppose I just play lots of hands 00:02:03.870 --> 00:02:06.280 and count the number I win, divide by the number 00:02:06.280 --> 00:02:07.955 of hands I played. 00:02:07.955 --> 00:02:09.580 Well then he thought about it and said, 00:02:09.580 --> 00:02:13.110 well, I've already played a lot of hands and I haven't won yet. 00:02:13.110 --> 00:02:15.010 So it probably will take me years 00:02:15.010 --> 00:02:18.670 to play enough hands to actually get a good estimate, 00:02:18.670 --> 00:02:21.050 and I don't want to do that. 00:02:21.050 --> 00:02:24.370 So he said, well, suppose instead of playing the game, 00:02:24.370 --> 00:02:27.780 I just simulate the game on a computer. 00:02:27.780 --> 00:02:30.720 He had no idea how to use a computer, 00:02:30.720 --> 00:02:33.210 but he had friends in high places. 00:02:33.210 --> 00:02:36.990 And actually talked to John von Neumann, 00:02:36.990 --> 00:02:41.610 who is often viewed as the inventor of the stored program 00:02:41.610 --> 00:02:43.140 computer. 00:02:43.140 --> 00:02:46.470 And said, John, could you do this on your fancy new ENIAC 00:02:46.470 --> 00:02:47.670 machine? 00:02:47.670 --> 00:02:49.440 And on the lower right here, you'll 00:02:49.440 --> 00:02:51.930 see a picture of the ENIAC. 00:02:51.930 --> 00:02:54.140 It was a very large machine. 00:02:54.140 --> 00:02:55.190 It filled a room. 00:02:57.710 --> 00:02:59.690 And von Neumann said, sure, we could probably 00:02:59.690 --> 00:03:04.060 do it in only a few hours of computation. 00:03:04.060 --> 00:03:07.600 Today we would think of a few microseconds, 00:03:07.600 --> 00:03:09.850 but those machines were slow. 00:03:09.850 --> 00:03:13.170 Hence was born Monte Carlo simulation, 00:03:13.170 --> 00:03:16.410 and then they actually used it in the design of the hydrogen 00:03:16.410 --> 00:03:17.920 bomb. 00:03:17.920 --> 00:03:23.470 So it turned out to be not just useful for cards. 00:03:23.470 --> 00:03:26.500 So what is Monte Carlo simulation? 00:03:26.500 --> 00:03:29.500 It's a method of estimating the values 00:03:29.500 --> 00:03:32.350 of an unknown quantity using what is 00:03:32.350 --> 00:03:35.530 called inferential statistics. 00:03:35.530 --> 00:03:37.620 And we've been using inferential statistics 00:03:37.620 --> 00:03:40.700 for the last several lectures. 00:03:40.700 --> 00:03:43.580 The key concepts-- and I want to be careful about these things 00:03:43.580 --> 00:03:45.660 will be coming back to them-- 00:03:45.660 --> 00:03:48.240 are the population. 00:03:48.240 --> 00:03:51.270 So think of the population as the universe 00:03:51.270 --> 00:03:53.860 of possible examples. 00:03:53.860 --> 00:03:55.560 So in the case of solitaire, it's 00:03:55.560 --> 00:03:59.100 a universe of all possible games of solitaire 00:03:59.100 --> 00:04:01.170 that you could possibly play. 00:04:01.170 --> 00:04:06.060 I have no idea how big that is, but it's really big, 00:04:06.060 --> 00:04:08.810 Then we take that universe, that population, 00:04:08.810 --> 00:04:13.760 and we sample it by drawing a proper subset. 00:04:13.760 --> 00:04:16.760 Proper means not the whole thing. 00:04:16.760 --> 00:04:19.880 Usually more than one sample to be useful. 00:04:19.880 --> 00:04:22.310 Certainly more than 0. 00:04:22.310 --> 00:04:25.850 And then we make an inference about the population 00:04:25.850 --> 00:04:32.150 based upon some set of statistics we do on the sample. 00:04:32.150 --> 00:04:36.500 So the population is typically a very large set of examples, 00:04:36.500 --> 00:04:40.390 and the sample is a smaller set of examples. 00:04:40.390 --> 00:04:43.080 And the key fact that makes them work 00:04:43.080 --> 00:04:46.950 is that if we choose the sample at random, 00:04:46.950 --> 00:04:51.120 the sample will tend to exhibit the same properties 00:04:51.120 --> 00:04:55.950 as the population from which it is drawn. 00:04:55.950 --> 00:04:59.460 And that's exactly what we did with the random walk, right? 00:04:59.460 --> 00:05:03.300 There were a very large number of different random walks 00:05:03.300 --> 00:05:07.070 you could take of say, 10,000 steps. 00:05:07.070 --> 00:05:12.230 We didn't look at all possible random walks of 10,000 steps. 00:05:12.230 --> 00:05:16.400 We drew a small sample of, say 100 such walks, 00:05:16.400 --> 00:05:20.330 computed the mean of those 100, and said, 00:05:20.330 --> 00:05:25.520 we think that's probably a good expectation 00:05:25.520 --> 00:05:29.745 of what the mean would be of all the possible walks of 10,000 00:05:29.745 --> 00:05:30.245 steps. 00:05:33.200 --> 00:05:36.980 So we were depending upon this principle. 00:05:36.980 --> 00:05:40.390 And of course the key fact here is that the sample 00:05:40.390 --> 00:05:43.540 has to be random. 00:05:43.540 --> 00:05:47.810 If you start drawing the sample and it's not random, 00:05:47.810 --> 00:05:49.750 then there's no reason to expect it 00:05:49.750 --> 00:05:54.030 to have the same properties as that of the population. 00:05:54.030 --> 00:05:55.590 And we'll go on throughout the term, 00:05:55.590 --> 00:05:59.160 and talk about the various ways you can get fooled and think 00:05:59.160 --> 00:06:03.660 of a random sample when exactly you don't. 00:06:03.660 --> 00:06:05.770 All right, let's look at a very simple example. 00:06:05.770 --> 00:06:10.425 People like to use flipping coins because coins are easy. 00:06:13.120 --> 00:06:17.140 So let's assume we have some coin. 00:06:17.140 --> 00:06:21.820 All right, so I bought two coins slightly larger 00:06:21.820 --> 00:06:23.830 than the usual coin. 00:06:23.830 --> 00:06:27.100 And I can flip it. 00:06:27.100 --> 00:06:32.140 Flip it once, and let's consider one flip, 00:06:32.140 --> 00:06:35.050 and let's assume it came out heads. 00:06:35.050 --> 00:06:38.440 I have to say the coin I flipped is not actually a $20 gold 00:06:38.440 --> 00:06:42.670 piece, in case any of you were thinking of stealing it. 00:06:42.670 --> 00:06:48.590 All right, so we've got one flip, and it came up heads. 00:06:48.590 --> 00:06:51.310 And now I can ask you the question-- 00:06:51.310 --> 00:06:56.680 if I were to flip the same coin an infinite number of times, 00:06:56.680 --> 00:06:59.740 how confident would you be about answering 00:06:59.740 --> 00:07:04.080 that all infinite flips would be heads? 00:07:04.080 --> 00:07:05.910 Or even if I were to flip it once more, 00:07:05.910 --> 00:07:08.940 how confident would you be that the next flip would be heads? 00:07:08.940 --> 00:07:10.710 And the answer is not very. 00:07:13.550 --> 00:07:17.150 Well, suppose I flip the coin twice, 00:07:17.150 --> 00:07:20.370 and both times it came up heads. 00:07:20.370 --> 00:07:22.080 And I'll ask you the same question-- 00:07:22.080 --> 00:07:26.146 do you think that the next flip is likely to be heads? 00:07:26.146 --> 00:07:30.670 Well, maybe you would be more inclined to say yes 00:07:30.670 --> 00:07:34.060 and having only seen one flip, but you wouldn't really 00:07:34.060 --> 00:07:36.550 jump to say, sure. 00:07:36.550 --> 00:07:41.680 On the other hand, if I flipped it 100 times and all 100 flips 00:07:41.680 --> 00:07:46.420 came up heads, well, you might be suspicious 00:07:46.420 --> 00:07:51.650 that my coin only has a head on both sides, for example. 00:07:51.650 --> 00:07:55.450 Or is weighted in some funny way that it mostly comes up heads. 00:07:55.450 --> 00:07:59.050 And so a lot of people, maybe even me, if you said, 00:07:59.050 --> 00:08:01.090 I flipped it 100 times and it came up heads. 00:08:01.090 --> 00:08:03.570 What do you think the next one will be? 00:08:03.570 --> 00:08:06.035 My best guess would be probably heads. 00:08:10.100 --> 00:08:11.870 How about this one? 00:08:11.870 --> 00:08:15.230 So here I've simulated 100 flips, 00:08:15.230 --> 00:08:22.545 and we have 50 heads here, two heads here, And 48 tails. 00:08:26.150 --> 00:08:28.730 And now if I said, do you think that the probability 00:08:28.730 --> 00:08:32.179 of the next flip coming up heads-- 00:08:32.179 --> 00:08:33.679 is it 52 out of 100? 00:08:37.860 --> 00:08:45.720 Well, if you had to guess, that should be the guess you make. 00:08:45.720 --> 00:08:48.960 Based upon the available evidence, 00:08:48.960 --> 00:08:52.760 that's the best guess you should probably make. 00:08:52.760 --> 00:08:55.055 You have no reason to believe it's a fair coin. 00:08:55.055 --> 00:08:58.470 It could well be weighted. 00:08:58.470 --> 00:09:00.720 We don't see it with coins, but we see weighted dice 00:09:00.720 --> 00:09:02.760 all the time. 00:09:02.760 --> 00:09:04.830 We shouldn't, but they exist. 00:09:04.830 --> 00:09:06.210 You can buy them on the internet. 00:09:10.450 --> 00:09:16.930 So typically our best guess is what we've seen, 00:09:16.930 --> 00:09:20.110 but we really shouldn't have very much confidence 00:09:20.110 --> 00:09:22.050 in that guess. 00:09:22.050 --> 00:09:26.990 Because well, could've just been an accident. 00:09:26.990 --> 00:09:29.270 Highly unlikely even if the coin is fair 00:09:29.270 --> 00:09:30.860 that you'd get 50-50, right? 00:09:34.770 --> 00:09:40.330 So why when we see 100 samples and they all come up heads 00:09:40.330 --> 00:09:44.950 do we feel better about guessing heads for the 101st 00:09:44.950 --> 00:09:47.725 than we did when we saw two samples? 00:09:50.310 --> 00:09:56.390 And why don't we feel so good about guessing 52 out of 100 00:09:56.390 --> 00:10:00.595 when we've seen a hundred flips that came out 52 and 48? 00:10:03.230 --> 00:10:05.645 And the answer is something called variance. 00:10:08.900 --> 00:10:14.630 When I had all heads, there was no variability in my answer. 00:10:14.630 --> 00:10:18.110 I got the same answer all the time. 00:10:18.110 --> 00:10:21.980 And so there was no variability, and that intuitively-- 00:10:21.980 --> 00:10:25.280 and in fact, mathematically-- should make us feel confident 00:10:25.280 --> 00:10:28.390 that, OK, maybe that's really the way the world is. 00:10:31.300 --> 00:10:35.140 On the other hand, when almost half are heads and almost half 00:10:35.140 --> 00:10:39.360 are tails, there's a lot of variance. 00:10:39.360 --> 00:10:42.500 Right, it's hard to predict what the next one will be. 00:10:42.500 --> 00:10:46.860 And so we should have very little confidence 00:10:46.860 --> 00:10:48.810 that it isn't an accident that it happened 00:10:48.810 --> 00:10:53.650 to be 52-48 in one direction. 00:10:53.650 --> 00:10:57.360 So as the variance grows, we need larger samples 00:10:57.360 --> 00:10:59.460 to have the same amount of confidence. 00:11:02.930 --> 00:11:06.230 All right, let's look at that with a detailed example. 00:11:06.230 --> 00:11:10.685 We'll look at roulette in keeping with the theme of Monte 00:11:10.685 --> 00:11:12.660 Carlo simulation. 00:11:12.660 --> 00:11:17.450 This is a roulette wheel that could well be at Monte Carlo. 00:11:17.450 --> 00:11:19.700 There's no need to simulate roulette, by the way. 00:11:19.700 --> 00:11:24.860 It's a very simple game, but as we've 00:11:24.860 --> 00:11:26.780 seen with our earlier examples, it's 00:11:26.780 --> 00:11:30.560 nice when we're learning about simulations to simulate things 00:11:30.560 --> 00:11:34.340 where we actually can know what the actual answer is 00:11:34.340 --> 00:11:38.577 so that we can then understand our simulation better. 00:11:38.577 --> 00:11:40.910 For those of you who don't know how roulette is played-- 00:11:40.910 --> 00:11:44.270 is there anyone here who doesn't know how roulette is played? 00:11:44.270 --> 00:11:45.320 Good for you. 00:11:45.320 --> 00:11:47.150 You grew up virtuous. 00:11:47.150 --> 00:11:49.715 All right, so-- well all right. 00:11:49.715 --> 00:11:51.530 Maybe I won't go there. 00:11:51.530 --> 00:11:55.850 So you have a wheel that spins around, 00:11:55.850 --> 00:11:58.250 and in the middle are a bunch of pockets. 00:11:58.250 --> 00:11:59.975 Each pocket has a number and a color. 00:12:02.970 --> 00:12:05.950 You bet in advance on what number 00:12:05.950 --> 00:12:07.950 you think is going to come up, or what color you 00:12:07.950 --> 00:12:09.600 think is going to come up. 00:12:09.600 --> 00:12:13.680 Then somebody drops a ball in that wheel, gives it a spin. 00:12:13.680 --> 00:12:15.960 And through centrifugal force, the ball 00:12:15.960 --> 00:12:18.520 stays on the outside for a while. 00:12:18.520 --> 00:12:21.810 But as the wheel slows down and heads towards the middle, 00:12:21.810 --> 00:12:24.900 and eventually settles in one of those pockets. 00:12:24.900 --> 00:12:27.940 And you win or you lose. 00:12:27.940 --> 00:12:32.720 Now you can bet on it, and so let's look 00:12:32.720 --> 00:12:33.830 at an example of that. 00:12:33.830 --> 00:12:37.430 So here is a roulette game. 00:12:37.430 --> 00:12:39.620 I've called it fair roulette, because it's 00:12:39.620 --> 00:12:44.120 set up in such a way that in principle, if you bet, 00:12:44.120 --> 00:12:46.544 your expected value should be 0. 00:12:46.544 --> 00:12:47.960 You'll win some, you'll lose some, 00:12:47.960 --> 00:12:50.810 but it's fair in the sense that it's not either 00:12:50.810 --> 00:12:54.720 a negative or positive sum game. 00:12:54.720 --> 00:12:57.555 So as always, we have an underbar underbar in it. 00:13:01.400 --> 00:13:06.530 Well we're setting up the wheel with 36 pockets on it, 00:13:06.530 --> 00:13:09.026 so you can bet on the numbers 1 through 36. 00:13:11.690 --> 00:13:14.660 That's way range work, you'll recall. 00:13:14.660 --> 00:13:16.750 Initially, we don't know where the ball is, 00:13:16.750 --> 00:13:18.740 so we'll say it's none. 00:13:18.740 --> 00:13:23.900 And here's the key thing is, if you make a bet, 00:13:23.900 --> 00:13:27.170 this tells you what your odds are. 00:13:27.170 --> 00:13:30.350 That if you bet on a pocket and you win, 00:13:30.350 --> 00:13:34.930 you get [? len ?] of pockets minus 1. 00:13:34.930 --> 00:13:39.960 So This is why it's a fair game, right? 00:13:39.960 --> 00:13:40.820 You bet $1. 00:13:40.820 --> 00:13:47.210 If you win, you get $36, your dollar plus $35 back. 00:13:47.210 --> 00:13:49.970 If you lose, you lose. 00:13:49.970 --> 00:13:52.220 All right, self dot spin will be random dot 00:13:52.220 --> 00:13:55.100 choice among the pockets. 00:13:55.100 --> 00:13:58.870 And then there is simply bet, where you just 00:13:58.870 --> 00:14:02.350 can choose an amount to bet and the pocket you want to bet on. 00:14:02.350 --> 00:14:03.360 I've simplified it. 00:14:03.360 --> 00:14:05.480 I'm not allowing you to bet here on colors. 00:14:09.870 --> 00:14:11.460 All right, so then we can play it. 00:14:11.460 --> 00:14:12.540 So here is play roulette. 00:14:15.580 --> 00:14:18.100 I've made game the class a parameter, 00:14:18.100 --> 00:14:22.600 because later we'll look at other kinds of roulette games. 00:14:22.600 --> 00:14:25.120 You tell it how many spins. 00:14:25.120 --> 00:14:26.590 What pocket you want to bet on. 00:14:26.590 --> 00:14:28.840 For simplicity, I'm going to bet on this same pocket 00:14:28.840 --> 00:14:30.520 all the time. 00:14:30.520 --> 00:14:34.540 Pick your favorite lucky number and how much you want to bet, 00:14:34.540 --> 00:14:36.730 and then we'll have a simulation just like the ones 00:14:36.730 --> 00:14:40.010 we've already looked at. 00:14:40.010 --> 00:14:43.690 So the number you get right starts at 0. 00:14:43.690 --> 00:14:49.120 For I and range number of spins, we'll do a spin. 00:14:49.120 --> 00:14:54.100 And then tote pocket plus equal game dot that pocket. 00:14:54.100 --> 00:14:56.960 And it will come back either 0 if you've lost, 00:14:56.960 --> 00:15:00.990 or 35 if you've won. 00:15:00.990 --> 00:15:03.490 And then we'll just print the results. 00:15:03.490 --> 00:15:06.084 So we can do it. 00:15:06.084 --> 00:15:07.000 In fact, let's run it. 00:15:20.950 --> 00:15:23.430 So here it is. 00:15:23.430 --> 00:15:26.400 I guess I'm doing a million games here, so quite a few. 00:15:29.210 --> 00:15:31.280 Actually I'm going to do two. 00:15:31.280 --> 00:15:33.200 What happens when you spin it 100 times? 00:15:33.200 --> 00:15:36.050 What happens when you spin it a million times? 00:15:36.050 --> 00:15:37.520 And we'll see what we get. 00:15:48.840 --> 00:15:55.300 So what we see here is that we do 100 spins. 00:15:55.300 --> 00:16:01.390 The first time I did it my expected return was minus 100%. 00:16:01.390 --> 00:16:03.760 I lost everything I bet. 00:16:03.760 --> 00:16:06.010 Not so unlikely, given that the odds 00:16:06.010 --> 00:16:10.900 are pretty long that you could do 100 times without winning. 00:16:10.900 --> 00:16:17.760 Next time I did a 100, my return was a positive 44%, and then 00:16:17.760 --> 00:16:20.450 a positive 28%. 00:16:20.450 --> 00:16:24.560 So you can see, for 100 spins it's highly variable what 00:16:24.560 --> 00:16:27.820 the expected return is. 00:16:27.820 --> 00:16:29.740 That's one of the things that makes 00:16:29.740 --> 00:16:31.280 gambling attractive to people. 00:16:34.010 --> 00:16:37.690 If you go to a casino, 100 spins would be a pretty long night 00:16:37.690 --> 00:16:39.300 at the table. 00:16:39.300 --> 00:16:42.267 And maybe you'd won 44%, and you'd 00:16:42.267 --> 00:16:43.350 feel pretty good about it. 00:16:46.250 --> 00:16:48.080 What about a million spins? 00:16:48.080 --> 00:16:51.920 Well people aren't interested in that, but the casino is, right? 00:16:51.920 --> 00:16:54.380 They don't really care what happens with 100 spins. 00:16:54.380 --> 00:16:56.930 They care what happens with a million spins. 00:16:56.930 --> 00:17:00.760 What happens when everybody comes every night to play. 00:17:00.760 --> 00:17:04.119 And there what we see is-- 00:17:04.119 --> 00:17:07.510 you'll notice much less variance. 00:17:07.510 --> 00:17:15.390 Happens to be minus 0.04 plus 0.6 plus 0.79. 00:17:15.390 --> 00:17:18.540 So it's still not 0, but it's certainly, 00:17:18.540 --> 00:17:23.569 these are all closer to 0 than any of these are. 00:17:23.569 --> 00:17:25.380 We know it should be 0, but it doesn't 00:17:25.380 --> 00:17:28.200 happen to be in these examples. 00:17:28.200 --> 00:17:33.600 But not only are they closer to 0, they're closer together. 00:17:33.600 --> 00:17:37.980 There is much less variance in the results, right? 00:17:37.980 --> 00:17:40.560 So here I show you these three numbers, 00:17:40.560 --> 00:17:42.690 and ask what do you expect to happen? 00:17:42.690 --> 00:17:45.040 You have no clue, right? 00:17:45.040 --> 00:17:47.010 So I don't know, maybe I'll win a lot. 00:17:47.010 --> 00:17:48.794 Maybe I'll lose everything. 00:17:48.794 --> 00:17:51.210 I show you these three numbers, you're going to look at it 00:17:51.210 --> 00:17:53.190 and say, well you know, I'm going 00:17:53.190 --> 00:17:56.910 to be somewhere between around 0 and maybe 1%. 00:17:56.910 --> 00:17:58.490 But you're never going to guess it's 00:17:58.490 --> 00:18:00.240 going to be radically different from that. 00:18:03.140 --> 00:18:06.830 And if I were to change this number to be even higher, 00:18:06.830 --> 00:18:09.850 it would go even closer to 0. 00:18:09.850 --> 00:18:10.780 But we won't bother. 00:18:20.790 --> 00:18:24.930 OK, so these are the numbers we just 00:18:24.930 --> 00:18:29.940 looked at, because I said the seed to be the same. 00:18:29.940 --> 00:18:31.710 So what's going on here is something 00:18:31.710 --> 00:18:37.650 called the law of large numbers, or sometimes Bernoulli's law. 00:18:37.650 --> 00:18:42.200 This is a picture of Bernoulli on the stamp. 00:18:42.200 --> 00:18:45.400 It's one of the two most important theorems in all 00:18:45.400 --> 00:18:49.390 of statistics, and we'll come to the second most important 00:18:49.390 --> 00:18:52.000 theorem in the next lecture. 00:18:52.000 --> 00:18:55.330 Here it says, "in repeated independent tests 00:18:55.330 --> 00:19:00.760 with the same actual probability, the chance 00:19:00.760 --> 00:19:03.310 that the fraction of times the outcome differs 00:19:03.310 --> 00:19:07.000 from p converges to 0 as the number of trials 00:19:07.000 --> 00:19:09.470 goes to infinity." 00:19:09.470 --> 00:19:12.700 So this says if I were to spin this fair roulette 00:19:12.700 --> 00:19:16.450 wheel an infinite number of times, 00:19:16.450 --> 00:19:20.620 the expected-- the return would be 0. 00:19:20.620 --> 00:19:23.950 The real true probability from the mathematics. 00:19:23.950 --> 00:19:27.040 Well, infinite is a lot, but a million 00:19:27.040 --> 00:19:28.390 is getting closer to infinite. 00:19:28.390 --> 00:19:31.570 And what this says is the closer I get to infinite, 00:19:31.570 --> 00:19:35.140 the closer it will be to the true probability. 00:19:35.140 --> 00:19:39.990 So that's why we did better with a million than with a hundred. 00:19:39.990 --> 00:19:42.210 And if I did a 100 million, we'd do way better 00:19:42.210 --> 00:19:43.891 than I did with a million. 00:19:47.820 --> 00:19:52.080 I want to take a minute to talk about a way this law is 00:19:52.080 --> 00:19:54.820 often misunderstood. 00:19:54.820 --> 00:19:59.030 This is something called the gambler's fallacy. 00:19:59.030 --> 00:20:01.250 And all you have to do is say, let's 00:20:01.250 --> 00:20:03.650 go watch a sporting event. 00:20:03.650 --> 00:20:05.450 And you'll watch a batter strike out 00:20:05.450 --> 00:20:07.539 for the sixth consecutive time. 00:20:07.539 --> 00:20:09.080 The next time they come to the plate, 00:20:09.080 --> 00:20:12.121 the idiot announcer says, well he struck out six times 00:20:12.121 --> 00:20:12.620 in a row. 00:20:12.620 --> 00:20:17.030 He's due for a hit this time, because he's usually 00:20:17.030 --> 00:20:18.380 a pretty good hitter. 00:20:18.380 --> 00:20:20.320 Well that's nonsense. 00:20:20.320 --> 00:20:24.300 It says, people somehow believe that if deviations 00:20:24.300 --> 00:20:30.250 from expected occur, they'll be evened out in the future. 00:20:30.250 --> 00:20:33.790 And we'll see something similar to this that is true, 00:20:33.790 --> 00:20:36.380 but this is not true. 00:20:36.380 --> 00:20:38.420 And there is a great story about it. 00:20:38.420 --> 00:20:41.920 This is told in a book by [INAUDIBLE] and [INAUDIBLE]. 00:20:41.920 --> 00:20:46.290 And this truly happened in Monte Carlo, with Roulette. 00:20:46.290 --> 00:20:49.380 And you could either bet on black or red. 00:20:49.380 --> 00:20:53.620 Black came up 26 times in a row. 00:20:53.620 --> 00:20:55.230 Highly unlikely, right? 00:20:55.230 --> 00:20:59.390 2 to the 26th is a giant number. 00:20:59.390 --> 00:21:03.830 And what happened is, word got out on the casino floor 00:21:03.830 --> 00:21:07.100 that black had kept coming up way too often. 00:21:07.100 --> 00:21:09.860 And people more or less panicked to rush to the table 00:21:09.860 --> 00:21:14.160 to bet on red, saying, well it can't keep coming up black. 00:21:14.160 --> 00:21:16.350 Surely the next one will be red. 00:21:16.350 --> 00:21:20.060 And as it happened when the casino totaled up its winnings, 00:21:20.060 --> 00:21:23.980 it was a record night for the casino. 00:21:23.980 --> 00:21:26.590 Millions of francs got bet, because people were 00:21:26.590 --> 00:21:29.640 sure it would have to even out. 00:21:29.640 --> 00:21:32.580 Well if we think about it, probability 00:21:32.580 --> 00:21:38.230 of 26 consecutive reds is that. 00:21:38.230 --> 00:21:41.578 A pretty small number. 00:21:41.578 --> 00:21:46.760 But the probability of 26 consecutive reds 00:21:46.760 --> 00:21:49.670 when the previous 25 rolls were red is what? 00:21:54.071 --> 00:21:56.027 No, that. 00:21:56.027 --> 00:21:58.970 AUDIENCE: Oh, I thought you meant [INAUDIBLE]. 00:21:58.970 --> 00:22:02.380 JOHN GUTTAG: No, if you had 25 reds and then 00:22:02.380 --> 00:22:05.560 you spun the wheel once more, the probability 00:22:05.560 --> 00:22:10.270 of it having 26 reds is now 0.5, because these 00:22:10.270 --> 00:22:12.370 are independent events. 00:22:12.370 --> 00:22:15.190 Unless of course the wheel is rigged, and we're assuming 00:22:15.190 --> 00:22:18.010 it's not. 00:22:18.010 --> 00:22:20.050 People have a hard time accepting this, 00:22:20.050 --> 00:22:21.850 and I know it seems funny. 00:22:21.850 --> 00:22:24.820 But I guarantee there will be some point in the next month 00:22:24.820 --> 00:22:28.720 or so when you will find yourself thinking this way, 00:22:28.720 --> 00:22:30.580 that something has to even out. 00:22:30.580 --> 00:22:32.680 I did so badly on the midterm, I will 00:22:32.680 --> 00:22:34.090 have to do better on the final. 00:22:38.440 --> 00:22:41.380 That was mean, I'm sorry. 00:22:41.380 --> 00:22:43.090 All right, speaking of means-- 00:22:47.300 --> 00:22:47.800 see? 00:22:47.800 --> 00:22:49.466 Professor [? Grimm's ?] not the only one 00:22:49.466 --> 00:22:52.450 who can make bad jokes. 00:22:52.450 --> 00:22:54.940 There is something-- it's not the gambler's fallacy-- 00:22:54.940 --> 00:22:56.680 that's often confused with it, and that's 00:22:56.680 --> 00:22:59.810 called regression to the mean. 00:22:59.810 --> 00:23:05.530 This term was coined in 1885 by Francis Galton 00:23:05.530 --> 00:23:09.970 in a paper, of which I've shown you a page from it here. 00:23:09.970 --> 00:23:13.990 And the basic conclusion here was-- 00:23:13.990 --> 00:23:21.060 what this table says is if somebody's parents are 00:23:21.060 --> 00:23:25.480 both taller than average, it's likely 00:23:25.480 --> 00:23:27.625 that the child will be smaller than the parents. 00:23:31.260 --> 00:23:34.470 Conversely, if the parents are shorter than average, 00:23:34.470 --> 00:23:39.170 it's likely that the child will be taller than average. 00:23:39.170 --> 00:23:42.250 Now you can think about this in terms of genetics and stuff. 00:23:42.250 --> 00:23:43.720 That's not what he did. 00:23:43.720 --> 00:23:47.270 He just looked at a bunch of data, 00:23:47.270 --> 00:23:51.430 and the data actually supported this. 00:23:51.430 --> 00:23:56.160 And this led him to this notion of regression to the mean. 00:23:56.160 --> 00:23:57.570 And here's what it is, and here's 00:23:57.570 --> 00:24:00.450 the way in which it is subtly different from the gambler's 00:24:00.450 --> 00:24:02.430 fallacy. 00:24:02.430 --> 00:24:06.580 What he said here is, following an extreme event-- 00:24:06.580 --> 00:24:09.190 parents being unusually tall-- 00:24:09.190 --> 00:24:13.570 the next random event is likely to be less extreme. 00:24:13.570 --> 00:24:15.280 He didn't know much about genetics, 00:24:15.280 --> 00:24:18.550 and he kind of assumed the height of people were random. 00:24:18.550 --> 00:24:20.790 But we'll ignore that. 00:24:20.790 --> 00:24:25.000 OK, but the idea is here that it will be less extreme. 00:24:25.000 --> 00:24:28.220 So let's look at it in roulette. 00:24:28.220 --> 00:24:33.940 If I spin a fair roulette wheel 10 times and get 10 reds, 00:24:33.940 --> 00:24:36.030 that's an extreme event. 00:24:36.030 --> 00:24:42.940 Right, here's a probability of basically 1.1024. 00:24:42.940 --> 00:24:45.330 Now the gambler's fallacy says, if I 00:24:45.330 --> 00:24:48.360 were to spin it another 10 times, 00:24:48.360 --> 00:24:50.490 it would need to even out. 00:24:50.490 --> 00:24:54.660 As in I should get more blacks than you would usually 00:24:54.660 --> 00:24:57.465 get to make up for these excess reds. 00:25:00.600 --> 00:25:04.795 What regression to the mean says is different. 00:25:04.795 --> 00:25:08.640 It says, it's likely that in the next 10 spins, 00:25:08.640 --> 00:25:11.030 you will get fewer than 10 reds. 00:25:11.030 --> 00:25:14.870 You will get a less extreme event. 00:25:14.870 --> 00:25:16.490 Now it doesn't have to be 10. 00:25:16.490 --> 00:25:21.620 If I'd gotten 7 reds instead of 5, you'd consider that extreme, 00:25:21.620 --> 00:25:27.650 and you would bet that the next 10 would have fewer than 7. 00:25:27.650 --> 00:25:31.100 But you wouldn't bet that it would have fewer than 5. 00:25:37.160 --> 00:25:42.260 Because of this, if you now look at the average of the 20 spins, 00:25:42.260 --> 00:25:46.790 it will be closer to the mean of 50% reds 00:25:46.790 --> 00:25:50.810 than you got from the extreme first spins. 00:25:50.810 --> 00:25:53.810 So that's why it's called regression to the mean. 00:25:53.810 --> 00:25:57.620 The more samples you take, the more likely 00:25:57.620 --> 00:26:00.332 you'll get to the mean. 00:26:00.332 --> 00:26:01.814 Yes? 00:26:01.814 --> 00:26:03.790 AUDIENCE: So, roulette wheel spins 00:26:03.790 --> 00:26:05.272 are supposed to be independent. 00:26:05.272 --> 00:26:05.981 JOHN GUTTAG: Yes. 00:26:05.981 --> 00:26:07.730 AUDIENCE: So it seems like the second 10-- 00:26:07.730 --> 00:26:08.596 JOHN GUTTAG: Pardon? 00:26:08.596 --> 00:26:10.532 AUDIENCE: It seems like the second 10 times 00:26:10.532 --> 00:26:11.750 that you spin it. 00:26:11.750 --> 00:26:13.420 that shouldn't have to [INAUDIBLE]. 00:26:13.420 --> 00:26:15.503 JOHN GUTTAG: Has nothing to do with the first one. 00:26:15.503 --> 00:26:18.320 AUDIENCE: But you said it's likely [INAUDIBLE]. 00:26:18.320 --> 00:26:22.690 JOHN GUTTAG: Right, because you have an extreme event, which 00:26:22.690 --> 00:26:25.310 was unlikely. 00:26:25.310 --> 00:26:27.530 And now if you have another event, 00:26:27.530 --> 00:26:31.310 it's likely to be closer to the average 00:26:31.310 --> 00:26:33.691 than the extreme was to the average. 00:26:38.330 --> 00:26:42.090 Precisely because it is independent. 00:26:42.090 --> 00:26:44.020 That makes sense to everybody? 00:26:44.020 --> 00:26:44.520 Yeah? 00:26:44.520 --> 00:26:47.310 AUDIENCE: Isn't that the same as the gambler's fallacy, then? 00:26:47.310 --> 00:26:49.741 By saying that, because this was super unlikely, 00:26:49.741 --> 00:26:52.000 the next one [INAUDIBLE]. 00:26:52.000 --> 00:26:55.140 JOHN GUTTAG: No, the gambler's fallacy here-- 00:26:55.140 --> 00:26:59.250 and it's a good question, and indeed people often 00:26:59.250 --> 00:27:02.330 do get these things confused. 00:27:02.330 --> 00:27:06.350 The gambler's fallacy would say that the second 10 00:27:06.350 --> 00:27:09.020 spins would-- 00:27:09.020 --> 00:27:12.470 we would expect to have fewer than 5 reds, 00:27:12.470 --> 00:27:15.710 because you're trying to even out the unusual number of reds 00:27:15.710 --> 00:27:18.920 in the first Spin 00:27:18.920 --> 00:27:22.150 Whereas here we're not saying we would have fewer than 5. 00:27:22.150 --> 00:27:25.210 We're saying we'd probably have fewer than 10. 00:27:25.210 --> 00:27:27.580 That it'll be closer to the mean, 00:27:27.580 --> 00:27:29.750 not that it would be below the mean. 00:27:29.750 --> 00:27:31.630 Whereas the gambler's fallacy would say 00:27:31.630 --> 00:27:35.385 it should be below that mean to quote, even out, the first 10. 00:27:35.385 --> 00:27:38.870 Does that makes sense? 00:27:38.870 --> 00:27:40.080 OK, great questions. 00:27:40.080 --> 00:27:40.580 Thank you. 00:27:43.300 --> 00:27:45.560 All right, now you may not know this, 00:27:45.560 --> 00:27:47.830 but casinos are not in the business of being fair. 00:27:51.650 --> 00:27:55.700 And the way they don't do that is in Europe, 00:27:55.700 --> 00:27:57.250 they're not all red and black. 00:27:57.250 --> 00:28:01.060 They sneak in one green. 00:28:01.060 --> 00:28:05.080 And so now if you bet red, well sometimes 00:28:05.080 --> 00:28:06.880 it isn't always red or black. 00:28:06.880 --> 00:28:09.840 And furthermore, there is this 0. 00:28:09.840 --> 00:28:13.390 They index from 0 rather than from one, and so 00:28:13.390 --> 00:28:17.040 you don't get a full payoff. 00:28:17.040 --> 00:28:22.570 In American roulette, they manage to sneak in two greens. 00:28:22.570 --> 00:28:26.330 They have a 0 in a double 0. 00:28:26.330 --> 00:28:30.990 Tilting the odds even more in favor of the casino. 00:28:30.990 --> 00:28:34.220 So we can do that in our simulation. 00:28:34.220 --> 00:28:39.410 We'll look at European roulette as a subclass of fair roulette. 00:28:39.410 --> 00:28:43.490 I've just added this extra pocket, 0. 00:28:43.490 --> 00:28:46.730 And notice I have not changed the odds. 00:28:46.730 --> 00:28:49.670 So what you get if you get your number is no higher, 00:28:49.670 --> 00:28:52.230 but you're a little bit less likely to get it 00:28:52.230 --> 00:28:54.440 because we snuck in that 0. 00:28:54.440 --> 00:28:57.290 Than American roulette is a subclass of European roulette 00:28:57.290 --> 00:28:59.921 in which I add yet another pocket. 00:29:04.160 --> 00:29:06.510 All right, we can simulate those. 00:29:06.510 --> 00:29:08.540 Again, nice thing about simulations, 00:29:08.540 --> 00:29:11.300 we can play these games. 00:29:11.300 --> 00:29:16.910 So I've simulated 20 trials of 1,000 spins, 10,000 spins, 00:29:16.910 --> 00:29:20.970 100,000, and a million. 00:29:20.970 --> 00:29:24.710 And what do we see as we look at this? 00:29:24.710 --> 00:29:33.890 Well, right away we can see that fair roulette is usually 00:29:33.890 --> 00:29:36.710 a much better bet than either of the other two. 00:29:36.710 --> 00:29:41.090 That even with only 1,000 spins the return is negative. 00:29:44.930 --> 00:29:47.660 And as we get more and more as I got to a million, 00:29:47.660 --> 00:29:50.840 it starts to look much more like closer to 0. 00:29:50.840 --> 00:29:53.510 And these, we have reason to believe at least, 00:29:53.510 --> 00:29:57.320 are much closer to true expectation 00:29:57.320 --> 00:30:00.650 saying that, while you break even in fair roulette, 00:30:00.650 --> 00:30:09.350 you'll lose 2.7% in Europe and over 5% in Las Vegas, 00:30:09.350 --> 00:30:13.490 or soon in Massachusetts. 00:30:13.490 --> 00:30:18.787 All right, we're sampling, right? 00:30:18.787 --> 00:30:20.245 That's why the results will change, 00:30:20.245 --> 00:30:22.050 and if I ran a different simulation 00:30:22.050 --> 00:30:25.380 with a different seed I'd get different numbers. 00:30:25.380 --> 00:30:29.610 Whenever you're sampling, you can't be guaranteed 00:30:29.610 --> 00:30:32.100 to get perfect accuracy. 00:30:32.100 --> 00:30:34.630 It's always possible you get a weird sample. 00:30:37.280 --> 00:30:41.360 That's not to say that you won't get exactly the right answer. 00:30:41.360 --> 00:30:45.500 I might have spun the wheel twice 00:30:45.500 --> 00:30:50.060 and happened to get the exact right answer of the return. 00:30:54.090 --> 00:30:56.310 Actually not twice, because the math 00:30:56.310 --> 00:30:59.100 doesn't work out, but 35 times and gotten 00:30:59.100 --> 00:31:01.390 exactly the right answer. 00:31:01.390 --> 00:31:04.830 But that's not the point. 00:31:04.830 --> 00:31:06.810 We need to be able to differentiate 00:31:06.810 --> 00:31:11.070 between what happens to be true and what we actually know, 00:31:11.070 --> 00:31:13.810 in a rigorous sense, is true. 00:31:13.810 --> 00:31:17.350 Or maybe don't know it, but have real good reason 00:31:17.350 --> 00:31:19.450 to believe it's true. 00:31:19.450 --> 00:31:23.460 So it's not just a question of faith. 00:31:23.460 --> 00:31:25.380 And that gets us to what's in some sense 00:31:25.380 --> 00:31:30.360 the fundamental question of all computational statistics, 00:31:30.360 --> 00:31:32.490 is how many samples do we need to look 00:31:32.490 --> 00:31:38.100 at before we can have real, justifiable confidence 00:31:38.100 --> 00:31:41.140 in our answer? 00:31:41.140 --> 00:31:43.610 As we've just seen-- 00:31:43.610 --> 00:31:45.740 not just, a few minutes ago-- with the coins, 00:31:45.740 --> 00:31:48.260 our intuition tells us that it depends 00:31:48.260 --> 00:31:53.210 upon the variability in the underlying possibilities. 00:31:53.210 --> 00:31:56.360 So let's look at that more carefully. 00:31:56.360 --> 00:31:58.460 We have to look at the variation in the data. 00:32:02.680 --> 00:32:07.930 So let's look at first something called variance. 00:32:07.930 --> 00:32:09.540 So this is variance of x. 00:32:09.540 --> 00:32:14.680 Think of x as just a list of data examples, data items. 00:32:14.680 --> 00:32:19.100 And the variance is we first compute the average 00:32:19.100 --> 00:32:23.150 of value, that's mu. 00:32:23.150 --> 00:32:25.160 So mu is for the mean. 00:32:31.890 --> 00:32:37.290 For each little x and big X, we compare the difference 00:32:37.290 --> 00:32:38.460 of that and the mean. 00:32:38.460 --> 00:32:41.380 How far is it from the mean? 00:32:41.380 --> 00:32:45.280 And square of the difference, and then we just sum them. 00:32:45.280 --> 00:32:47.770 So this takes, how far is everything from the mean? 00:32:47.770 --> 00:32:49.170 We just add them all up. 00:32:52.470 --> 00:32:57.910 And then we end up dividing by the size of the set, 00:32:57.910 --> 00:33:00.430 the number of examples. 00:33:00.430 --> 00:33:02.830 Why do we have to do this division? 00:33:02.830 --> 00:33:05.830 Well, because we don't want to say something has high variance 00:33:05.830 --> 00:33:09.280 just because it has many members, right? 00:33:09.280 --> 00:33:12.700 So this sort of normalizes is by the number of members, 00:33:12.700 --> 00:33:18.980 and this just sums how different the members are from the mean. 00:33:18.980 --> 00:33:20.740 So if everything is the same value, 00:33:20.740 --> 00:33:22.230 what's the variance going to be? 00:33:22.230 --> 00:33:25.951 If I have a set of 1,000 6's, what's the variance? 00:33:25.951 --> 00:33:26.450 Yes? 00:33:26.450 --> 00:33:27.300 AUDIENCE: 0. 00:33:27.300 --> 00:33:27.925 JOHN GUTTAG: 0. 00:33:31.600 --> 00:33:35.869 You think this is going to be hard, but I came prepared. 00:33:35.869 --> 00:33:37.160 I was hoping this would happen. 00:33:41.560 --> 00:33:43.850 Look out, I don't know where this is going to go. 00:33:49.421 --> 00:33:50.129 [FIRES SLINGSHOT] 00:33:50.129 --> 00:33:53.510 AUDIENCE: [LAUGHTER] 00:33:53.510 --> 00:33:57.700 JOHN GUTTAG: All right, maybe it isn't the best technology. 00:33:57.700 --> 00:33:58.855 I'll go home and practice. 00:34:01.775 --> 00:34:03.400 And then the thing you're more familiar 00:34:03.400 --> 00:34:06.946 with is the standard deviation. 00:34:06.946 --> 00:34:08.820 And if you look at the standard deviation is, 00:34:08.820 --> 00:34:10.653 it's simply the square root of the variance. 00:34:13.710 --> 00:34:18.179 Now, let's understand this a little bit 00:34:18.179 --> 00:34:21.830 and first ask, why am I squaring this here, 00:34:21.830 --> 00:34:23.580 especially because later on I'm just going 00:34:23.580 --> 00:34:26.389 to take a square root anyway? 00:34:26.389 --> 00:34:29.730 Well squaring it has one virtue, which 00:34:29.730 --> 00:34:32.070 is that it means I don't care whether the difference is 00:34:32.070 --> 00:34:35.440 positive or negative. 00:34:35.440 --> 00:34:36.969 And I shouldn't, right? 00:34:36.969 --> 00:34:38.949 I don't care which side of the mean it's on, 00:34:38.949 --> 00:34:42.030 I just care it's not near the mean. 00:34:42.030 --> 00:34:43.889 But if that's all I wanted to do I 00:34:43.889 --> 00:34:45.320 could take the absolute value. 00:34:48.929 --> 00:34:50.989 The other thing we see with squaring 00:34:50.989 --> 00:34:56.810 is it gives the outliers extra emphasis, because I'm 00:34:56.810 --> 00:34:59.560 squaring that distance. 00:34:59.560 --> 00:35:02.470 Now you can think that's good or bad, 00:35:02.470 --> 00:35:04.105 but it's worth knowing it's a fact. 00:35:06.860 --> 00:35:09.710 The more important thing to think about 00:35:09.710 --> 00:35:17.160 is standard deviation all by itself is a meaningless number. 00:35:17.160 --> 00:35:21.880 You always have to think about it in the context of the mean. 00:35:21.880 --> 00:35:27.710 If I tell you the standard deviation is 100, 00:35:27.710 --> 00:35:30.410 you then say, well-- and I ask you whether it's big or small, 00:35:30.410 --> 00:35:32.600 you have no idea. 00:35:32.600 --> 00:35:35.630 If the mean is 100 and the standard deviation is 100, 00:35:35.630 --> 00:35:37.730 it's pretty big. 00:35:37.730 --> 00:35:40.670 If the mean is a billion and the standard deviation is 100, 00:35:40.670 --> 00:35:42.580 it's pretty small. 00:35:42.580 --> 00:35:49.670 So you should never want to look at just the standard deviation. 00:35:49.670 --> 00:35:51.220 All right, here is just some code 00:35:51.220 --> 00:35:54.720 to compute those, easy enough. 00:35:54.720 --> 00:35:56.760 Why am I doing this? 00:35:56.760 --> 00:36:01.370 Because we're now getting to the punch line. 00:36:01.370 --> 00:36:07.040 We often try and estimate values just by giving the mean. 00:36:07.040 --> 00:36:10.580 So we might report on an exam that the mean grade was 80. 00:36:15.940 --> 00:36:19.440 It's better instead of trying to describe 00:36:19.440 --> 00:36:22.240 an unknown value by it-- 00:36:22.240 --> 00:36:25.090 an unknown parameter by a single value, 00:36:25.090 --> 00:36:29.680 say the expected return on betting a roulette wheel, 00:36:29.680 --> 00:36:34.120 to provide a confidence interval. 00:36:34.120 --> 00:36:36.000 So what a confidence interval is is 00:36:36.000 --> 00:36:41.360 a range that's likely to contain the unknown value, 00:36:41.360 --> 00:36:44.990 and a confidence that the unknown value is 00:36:44.990 --> 00:36:46.190 within that range. 00:36:48.970 --> 00:36:50.910 So I might say on a fair roulette 00:36:50.910 --> 00:37:00.790 wheel I expect that your return will be between minus 1% 00:37:00.790 --> 00:37:07.390 and plus 1%, and I expect that to be true 95% of the time 00:37:07.390 --> 00:37:12.500 you play the game if you play 100 rolls, spins. 00:37:12.500 --> 00:37:15.160 If you take 100 spins of the roulette wheel, 00:37:15.160 --> 00:37:18.070 I expect that 95% of the time your return 00:37:18.070 --> 00:37:19.610 will be between this and that. 00:37:23.930 --> 00:37:28.020 So here, we're saying the return on betting a pocket 10 times, 00:37:28.020 --> 00:37:34.590 10,000 times in European roulette is minus 3.3%. 00:37:34.590 --> 00:37:37.170 I think that was the number we just saw. 00:37:37.170 --> 00:37:41.320 And now I'm going to add to that this margin of error, 00:37:41.320 --> 00:37:46.670 which is plus or minus 3.5% with a 95% level of confidence. 00:37:49.850 --> 00:37:52.990 What does this mean? 00:37:52.990 --> 00:37:57.000 If I were to conduct an infinite number of trials 00:37:57.000 --> 00:38:02.540 of 10,000 bets each, my expected average return 00:38:02.540 --> 00:38:06.830 would indeed be minus 3.3%, and it 00:38:06.830 --> 00:38:12.048 would be between these values 95% of the time. 00:38:15.040 --> 00:38:20.980 I've just subtracted and added this 3.5, 00:38:20.980 --> 00:38:23.050 saying nothing about what would happen 00:38:23.050 --> 00:38:25.150 in the other 5% of the time. 00:38:25.150 --> 00:38:27.100 How far away I might be from this, 00:38:27.100 --> 00:38:28.820 this is totally silent on that subject. 00:38:28.820 --> 00:38:29.320 Yes? 00:38:29.320 --> 00:38:33.304 AUDIENCE: I think you want 0.2 not 9.2. 00:38:33.304 --> 00:38:37.804 JOHN GUTTAG: Oh, let's see. 00:38:37.804 --> 00:38:38.690 Yep, I do. 00:38:38.690 --> 00:38:39.662 Thank you. 00:38:44.530 --> 00:38:46.596 We'll fix it on the spot. 00:38:46.596 --> 00:38:48.220 This is why you have to come to lecture 00:38:48.220 --> 00:38:49.720 rather than just reading the slides, 00:38:49.720 --> 00:38:52.204 because I make mistakes. 00:38:52.204 --> 00:38:52.870 Thank you, Eric. 00:39:01.010 --> 00:39:05.610 All right, so it's telling me that, and that's all it means. 00:39:05.610 --> 00:39:09.600 And it's amazing how often people don't quite 00:39:09.600 --> 00:39:10.560 know what this means. 00:39:10.560 --> 00:39:13.830 For example, when they look at a political pole 00:39:13.830 --> 00:39:17.610 and they see how many votes somebody is expected to get. 00:39:17.610 --> 00:39:19.830 And they see this confidence interval and say, 00:39:19.830 --> 00:39:21.450 what does that really mean? 00:39:21.450 --> 00:39:23.760 Most people don't know. 00:39:23.760 --> 00:39:26.870 But it does have a very precise meaning, and this is it. 00:39:29.820 --> 00:39:33.380 How do we compute confidence intervals? 00:39:33.380 --> 00:39:36.020 Most of the time we compute them using something 00:39:36.020 --> 00:39:37.320 called the empirical rule. 00:39:40.350 --> 00:39:44.100 Under some assumptions, which I'll get to a little bit later, 00:39:44.100 --> 00:39:50.340 the empirical rule says that if I take the data, find the mean, 00:39:50.340 --> 00:39:53.850 compute the standard deviation as we've just seen, 00:39:53.850 --> 00:39:59.250 68% of the data will be within one standard deviation in front 00:39:59.250 --> 00:40:01.720 of or behind the mean. 00:40:01.720 --> 00:40:04.210 Within one standard deviation of the mean. 00:40:04.210 --> 00:40:11.060 95% will be within 1.96 standard deviations. 00:40:11.060 --> 00:40:13.000 And that's what people usually use. 00:40:13.000 --> 00:40:16.220 Usually when people talk about confidence intervals, 00:40:16.220 --> 00:40:19.690 they're talking about the 95% confidence interval. 00:40:19.690 --> 00:40:23.170 And they use this 1.6 number. 00:40:23.170 --> 00:40:27.220 And 99.7% of the data will be within three 00:40:27.220 --> 00:40:29.870 standard deviations. 00:40:29.870 --> 00:40:32.360 So you can see if you are outside the third standard 00:40:32.360 --> 00:40:35.180 deviation, you are a pretty rare bird, 00:40:35.180 --> 00:40:37.460 for better or worse depending upon which side. 00:40:41.360 --> 00:40:44.890 All right, so let's apply the empirical rule 00:40:44.890 --> 00:40:48.440 to our roulette game. 00:40:48.440 --> 00:40:52.610 So I've got my three roulette games as before. 00:40:52.610 --> 00:40:54.420 I'm going to run a simple simulation. 00:40:57.100 --> 00:41:02.690 And the key thing to notice is really 00:41:02.690 --> 00:41:03.880 this print statement here. 00:41:07.090 --> 00:41:13.230 Right, that I'll print the mean, which I'm rounding. 00:41:13.230 --> 00:41:17.580 And then I'm going to give the confidence intervals, 00:41:17.580 --> 00:41:22.910 plus or minus, and I'll just take the standard deviation 00:41:22.910 --> 00:41:26.390 times 1.6 times 100, y times 100, 00:41:26.390 --> 00:41:28.380 because I'm showing you percentages. 00:41:31.140 --> 00:41:35.160 All right so again, very straightforward code. 00:41:35.160 --> 00:41:37.920 Just simulation, just like the ones we've been looking at. 00:41:40.720 --> 00:41:42.127 And well, I'm just going-- 00:41:42.127 --> 00:41:43.960 I don't think I'll bother running it for you 00:41:43.960 --> 00:41:45.910 in the interest of time. 00:41:45.910 --> 00:41:47.660 You can run it yourself. 00:41:47.660 --> 00:41:51.080 But here's what I got when I ran it. 00:41:51.080 --> 00:41:56.500 So when I simulated betting a pocket for 20 trials, 00:41:56.500 --> 00:41:58.170 we see that the-- 00:41:58.170 --> 00:42:01.870 of 1,000 spins each, for 1,000 spins 00:42:01.870 --> 00:42:06.520 the expected return for fair roulette happened to be 3.68%. 00:42:06.520 --> 00:42:08.190 A bit high. 00:42:08.190 --> 00:42:11.000 But you'll notice the confidence interval plus or minus 00:42:11.000 --> 00:42:15.440 27 includes the actual answer, which is 0. 00:42:20.190 --> 00:42:22.350 And we have very large confidence intervals 00:42:22.350 --> 00:42:24.560 for the other two games. 00:42:24.560 --> 00:42:28.730 If you go way down to the bottom where I've spun, spun the wheel 00:42:28.730 --> 00:42:36.920 many more times, what we'll see is 00:42:36.920 --> 00:42:43.410 that my expected return for fair roulette is much closer to 0 00:42:43.410 --> 00:42:45.480 than it was here. 00:42:45.480 --> 00:42:47.700 But more importantly, my confidence interval 00:42:47.700 --> 00:42:53.390 is much smaller, 0.8. 00:42:53.390 --> 00:42:58.300 So now I really have constrained it pretty well. 00:42:58.300 --> 00:43:02.870 Similarly, for the other two games you will see-- 00:43:02.870 --> 00:43:05.300 maybe it's more accurate, maybe it's less accurate, 00:43:05.300 --> 00:43:10.050 but importantly the confidence interval is smaller. 00:43:10.050 --> 00:43:15.950 So I have good reason to believe that the mean I'm computing 00:43:15.950 --> 00:43:20.240 is close to the true mean, because my confidence 00:43:20.240 --> 00:43:23.030 interval has shrunk. 00:43:23.030 --> 00:43:26.030 So that's the really important concept here, 00:43:26.030 --> 00:43:28.430 is that we don't just guess-- 00:43:28.430 --> 00:43:30.890 compute the value in the simulation. 00:43:30.890 --> 00:43:33.650 We use, in this case, the empirical rule 00:43:33.650 --> 00:43:39.440 to tell us how much faith we should have in that value. 00:43:39.440 --> 00:43:43.660 All right, the empirical rule doesn't always work. 00:43:43.660 --> 00:43:46.780 There are a couple of assumptions. 00:43:46.780 --> 00:43:51.410 One is that the mean estimation error is 0. 00:43:51.410 --> 00:43:52.310 What is that saying? 00:43:52.310 --> 00:43:57.290 That I'm just as likely to guess high as gas low. 00:43:57.290 --> 00:44:01.160 In most experiments of this sort, most simulations, 00:44:01.160 --> 00:44:04.700 that's a very fair assumption. 00:44:04.700 --> 00:44:07.370 There's no reason to guess I'd be systematically off 00:44:07.370 --> 00:44:10.230 in one direction or another. 00:44:10.230 --> 00:44:14.540 It's different when you use this in a laboratory experiment, 00:44:14.540 --> 00:44:17.570 where in fact, depending upon your laboratory technique, 00:44:17.570 --> 00:44:22.590 there may be a bias in your results in one direction. 00:44:22.590 --> 00:44:25.555 So we have to assume that there's no bias in our errors. 00:44:28.310 --> 00:44:31.370 And we have to assume that the distribution of errors 00:44:31.370 --> 00:44:34.450 is normal. 00:44:34.450 --> 00:44:36.410 And we'll come back to this in just a second. 00:44:36.410 --> 00:44:37.990 But this is a normal distribution, 00:44:37.990 --> 00:44:38.823 called the Gaussian. 00:44:41.990 --> 00:44:45.870 Under those two assumptions the empirical rule 00:44:45.870 --> 00:44:48.890 will always hold. 00:44:48.890 --> 00:44:51.050 All right, let's talk about distributions, 00:44:51.050 --> 00:44:54.630 since I just introduced one. 00:44:54.630 --> 00:44:57.750 We've been using a probability distribution. 00:44:57.750 --> 00:45:01.440 And this captures the notion of the relative frequency 00:45:01.440 --> 00:45:04.995 with which some random variable takes on different values. 00:45:07.510 --> 00:45:11.320 There are two kinds. , Discrete and these when the values are 00:45:11.320 --> 00:45:14.330 drawn from a finite set of values. 00:45:14.330 --> 00:45:17.020 So when I flip these coins, there 00:45:17.020 --> 00:45:20.630 are only two possible values, head or tails. 00:45:20.630 --> 00:45:23.720 And so if we look at the distribution of heads 00:45:23.720 --> 00:45:27.830 and tails, it's pretty simple. 00:45:27.830 --> 00:45:30.580 We just list the probability of heads. 00:45:30.580 --> 00:45:33.400 We list the probability of tails. 00:45:33.400 --> 00:45:37.390 We know that those two probabilities must add up to 1, 00:45:37.390 --> 00:45:42.660 and that fully describes our distribution. 00:45:42.660 --> 00:45:46.740 Continuous random variables are a bit trickier. 00:45:46.740 --> 00:45:51.972 They're drawn from a set of reals between two numbers. 00:45:51.972 --> 00:45:53.430 For the sake of argument, let's say 00:45:53.430 --> 00:45:57.130 those two numbers are 0 and 1. 00:45:57.130 --> 00:46:00.520 Well, we can't just enumerate the probability 00:46:00.520 --> 00:46:03.640 for each number. 00:46:03.640 --> 00:46:08.830 How many real numbers are there between 0 and 1? 00:46:08.830 --> 00:46:11.290 An infinite number, right? 00:46:11.290 --> 00:46:14.290 And so I can't say, for each of these infinite numbers, what's 00:46:14.290 --> 00:46:16.420 the probability of it occurring? 00:46:16.420 --> 00:46:20.770 Actually the probability is close to 0 for each of them. 00:46:20.770 --> 00:46:23.350 Is 0, if they're truly infinite. 00:46:23.350 --> 00:46:26.140 So I need to do something else, and what 00:46:26.140 --> 00:46:30.340 I do that is what's called the probability density function. 00:46:30.340 --> 00:46:35.330 This is a different kind of PDF than the one Adobe sells. 00:46:35.330 --> 00:46:37.820 So there, we don't give the probability 00:46:37.820 --> 00:46:42.060 of the random variable taking on a specific value. 00:46:42.060 --> 00:46:44.220 We give the probability of it lying 00:46:44.220 --> 00:46:45.730 somewhere between two values. 00:46:49.970 --> 00:46:54.455 And then we define a curve, which shows how it works. 00:46:54.455 --> 00:46:55.990 So let's look at an example. 00:46:58.520 --> 00:47:01.970 So we'll go back to normal distributions. 00:47:01.970 --> 00:47:05.932 This is-- for the continuous normal distribution, 00:47:05.932 --> 00:47:07.265 it's described by this function. 00:47:09.900 --> 00:47:13.020 And for those of you who don't know about the magic number e, 00:47:13.020 --> 00:47:16.670 this is one of many ways to define it. 00:47:16.670 --> 00:47:20.634 But I really don't care whether you remember this. 00:47:20.634 --> 00:47:22.300 I don't care whether you know what e is. 00:47:22.300 --> 00:47:24.310 I don't care if you know what this is. 00:47:24.310 --> 00:47:27.350 What we really want to say is, it looks like this. 00:47:31.150 --> 00:47:33.400 In this case, the mean is 0. 00:47:33.400 --> 00:47:34.555 It doesn't have to be 0. 00:47:37.880 --> 00:47:41.140 I've [INAUDIBLE] a mean of 0 and a standard deviation of 1. 00:47:41.140 --> 00:47:45.120 This is called the so-called standard normal distribution. 00:47:45.120 --> 00:47:49.440 But it's symmetric around the mean. 00:47:49.440 --> 00:47:51.950 And that gets back to, it's equally likely 00:47:51.950 --> 00:47:54.920 that our errors are in either direction, right? 00:47:54.920 --> 00:47:57.410 So it peaks at the mean. 00:47:57.410 --> 00:47:59.450 The peak is always at the mean. 00:47:59.450 --> 00:48:01.250 That's the most probable value, and it's 00:48:01.250 --> 00:48:03.460 symmetric about the mean. 00:48:05.970 --> 00:48:09.630 So if we look at it, for example, and I say, 00:48:09.630 --> 00:48:17.630 what's the probability of the number being between 0 and 1? 00:48:17.630 --> 00:48:19.460 I can look at it here and say, all right, 00:48:19.460 --> 00:48:24.930 let's draw a line here, and a line here. 00:48:24.930 --> 00:48:29.590 And then I can integrate the curve under here. 00:48:29.590 --> 00:48:31.800 And that tells me the probability 00:48:31.800 --> 00:48:35.760 of this random variable being between 0 and 1. 00:48:40.000 --> 00:48:44.120 If I want to know between minus 1 and 1. 00:48:44.120 --> 00:48:46.680 I just do this and then I integrate over that area. 00:48:49.190 --> 00:48:52.260 All right, so the area under the curve in this case 00:48:52.260 --> 00:48:55.560 defines the likelihood. 00:48:55.560 --> 00:48:57.930 Now I have to divide and normalize to actually get 00:48:57.930 --> 00:49:00.216 the answer between 0 and 1. 00:49:00.216 --> 00:49:01.590 So the question is, what fraction 00:49:01.590 --> 00:49:06.800 of the area under the curve is between minus 1 and 1? 00:49:06.800 --> 00:49:11.110 And that will tell me the probability. 00:49:11.110 --> 00:49:13.350 So what does the empirical rule tell us? 00:49:13.350 --> 00:49:16.300 What fraction is between minus 1 and 1, roughly? 00:49:19.210 --> 00:49:20.670 Yeah? 00:49:20.670 --> 00:49:22.740 68%, right? 00:49:22.740 --> 00:49:27.240 So that tells me 68% of the area under this curve 00:49:27.240 --> 00:49:30.840 is between minus 1 and 1, because my standard deviation 00:49:30.840 --> 00:49:34.450 is 1, roughly 68%. 00:49:34.450 --> 00:49:36.520 And maybe your eyes will convince you 00:49:36.520 --> 00:49:40.130 that's a reasonable guess. 00:49:40.130 --> 00:49:43.560 OK, we'll come back and look at this in a bit more detail 00:49:43.560 --> 00:49:45.910 on Monday of next week. 00:49:45.910 --> 00:49:48.210 And also look at the question of, 00:49:48.210 --> 00:49:51.930 why does this work in so many cases 00:49:51.930 --> 00:49:54.270 where we don't actually have a normal distribution 00:49:54.270 --> 00:49:56.240 to start with?