WEBVTT 00:00:00.050 --> 00:00:02.500 The following content is provided under a Creative 00:00:02.500 --> 00:00:04.019 Commons license. 00:00:04.019 --> 00:00:06.360 Your support will help MIT OpenCourseWare 00:00:06.360 --> 00:00:10.730 continue to offer high quality educational resources for free. 00:00:10.730 --> 00:00:13.340 To make a donation or view additional materials 00:00:13.340 --> 00:00:17.229 from hundreds of MIT courses, visit MIT OpenCourseWare 00:00:17.229 --> 00:00:17.854 at ocw.mit.edu. 00:00:21.738 --> 00:00:25.090 PROFESSOR: Good morning, everyone. 00:00:25.090 --> 00:00:28.820 Today, we're going to talk about boosting. 00:00:28.820 --> 00:00:33.900 Boosting is pretty awesome, and it's not as hard 00:00:33.900 --> 00:00:34.630 as it might seem. 00:00:34.630 --> 00:00:39.050 It's actually pretty easy, as long as you do it right. 00:00:39.050 --> 00:00:42.610 So let's take a look at this boosting problem. 00:00:42.610 --> 00:00:44.680 Just like with ID trees, you may have 00:00:44.680 --> 00:00:47.950 noticed there's the ID tree problem where there's a graph, 00:00:47.950 --> 00:00:52.200 and all of the tests are x and y-axis tests on that graph. 00:00:52.200 --> 00:00:54.150 And then there's the ID tree problem 00:00:54.150 --> 00:00:57.950 where there are a lot of crazy different classifiers 00:00:57.950 --> 00:01:00.480 about characteristics that are discrete. 00:01:00.480 --> 00:01:07.530 And then all of the sort of ID tree stumps, if you will, 00:01:07.530 --> 00:01:09.730 are built out of those discrete qualities. 00:01:09.730 --> 00:01:12.300 This is an example of a boosting problem 00:01:12.300 --> 00:01:15.510 of the second type with a bunch of discrete qualities, 00:01:15.510 --> 00:01:20.040 like evil, emo, transforms, sparkly. 00:01:20.040 --> 00:01:24.820 And it has an numerical quality number of romantic interests. 00:01:24.820 --> 00:01:26.680 So it's one of basically the two kinds 00:01:26.680 --> 00:01:28.346 of boosting problems that you might see. 00:01:28.346 --> 00:01:32.850 And now, there is the two-axis Cartesian problem. 00:01:32.850 --> 00:01:34.990 A good example of that is the hours 00:01:34.990 --> 00:01:41.290 of sleep versus coffee problem, which I, for one, am planning 00:01:41.290 --> 00:01:46.980 on doing in tutorial on Monday so that you guys sort of get 00:01:46.980 --> 00:01:49.390 a sense of both types of problems. 00:01:49.390 --> 00:01:50.990 This one looks big. 00:01:50.990 --> 00:01:54.190 I think it looks-- I mean, it has ten different vampires 00:01:54.190 --> 00:01:56.880 or non-vampires to classify. 00:01:56.880 --> 00:01:59.920 It has a whole bunch of possible classifiers. 00:01:59.920 --> 00:02:04.340 But if you do it right, you can do it fast. 00:02:04.340 --> 00:02:07.290 So here's the prompt for the problem. 00:02:07.290 --> 00:02:08.550 Let's see. 00:02:08.550 --> 00:02:10.840 After graduating MIT, you get a job 00:02:10.840 --> 00:02:12.620 working for Van Helsing and Sommers, 00:02:12.620 --> 00:02:16.160 a famous vampire-hunting consulting agency. 00:02:16.160 --> 00:02:18.080 Gabriel Van Helsing, one of the two founders, 00:02:18.080 --> 00:02:20.511 once attended several 6.034 lectures as a guest, 00:02:20.511 --> 00:02:22.010 and he remembers Professor Winston's 00:02:22.010 --> 00:02:24.262 vampire identification tree lecture. 00:02:24.262 --> 00:02:25.720 He assigns you the task of creating 00:02:25.720 --> 00:02:27.700 a superior classifier for vampires 00:02:27.700 --> 00:02:30.740 by using boosting on the following data. 00:02:30.740 --> 00:02:34.650 So we've got the ID number, which 00:02:34.650 --> 00:02:37.170 we can use to just write things out in shorthand. 00:02:37.170 --> 00:02:40.980 We've got the name of several vampires and non-vampires. 00:02:40.980 --> 00:02:43.502 Then you see whether they're a vampire or not. 00:02:43.502 --> 00:02:44.960 That's whether their classification 00:02:44.960 --> 00:02:48.590 is a plus or minus for the quality of vampire. 00:02:48.590 --> 00:02:54.260 After that, there's a bunch of possible ways 00:02:54.260 --> 00:02:56.170 to classify whether they're a vampire or not. 00:02:56.170 --> 00:02:58.800 There's whether or not they're evil, whether or not 00:02:58.800 --> 00:03:01.740 they're emo, whether or not they transform, 00:03:01.740 --> 00:03:04.540 and whether or not they're sparkly, as well 00:03:04.540 --> 00:03:08.260 as the number of romantic interests that they have. 00:03:08.260 --> 00:03:10.080 So for instance, on the one hand, 00:03:10.080 --> 00:03:13.160 you have Dracula, who's evil, but he's not emo. 00:03:13.160 --> 00:03:19.250 He can transform into a bat or a cloud of mist. 00:03:19.250 --> 00:03:22.390 He does not sparkle, and he has five romantic interests-- 00:03:22.390 --> 00:03:25.300 those three vampire chicks at the beginning, Wilhelmina 00:03:25.300 --> 00:03:28.200 Murray, and Lucy Westenra. 00:03:28.200 --> 00:03:34.300 So on the other hand, you have Squall Leonhart, 00:03:34.300 --> 00:03:37.510 who's the protagonist of Final Fantasy VII, is extremely emo 00:03:37.510 --> 00:03:41.350 and doesn't have any of the other characteristics. 00:03:41.350 --> 00:03:43.790 And he's not a vampire. 00:03:43.790 --> 00:03:45.280 However, he's a nice counterexample 00:03:45.280 --> 00:03:47.750 for a possible rule that all emo people 00:03:47.750 --> 00:03:49.875 are vampires because he's very, very emo, 00:03:49.875 --> 00:03:51.390 and he's not a vampire. 00:03:51.390 --> 00:03:54.852 So how will we go about tackling this problem with boosting? 00:03:54.852 --> 00:03:57.060 Well, there's a whole bunch of different classifiers. 00:03:57.060 --> 00:03:59.520 And if you think this is all of them-- 00:03:59.520 --> 00:04:01.640 like evil, emo, transforms, sparkly. 00:04:01.640 --> 00:04:05.510 romantics interests, and true-- it's actually only half 00:04:05.510 --> 00:04:06.380 of them. 00:04:06.380 --> 00:04:08.810 The other half are the opposite versions, 00:04:08.810 --> 00:04:10.060 but we'll ignore them for now. 00:04:13.300 --> 00:04:15.280 So if you look at these, you can probably 00:04:15.280 --> 00:04:17.630 figure out what they mean-- evil equals 00:04:17.630 --> 00:04:20.810 yes means vampire is what we're saying here-- 00:04:20.810 --> 00:04:21.990 except maybe true. 00:04:21.990 --> 00:04:24.730 You might be saying, why is there one that just says true? 00:04:24.730 --> 00:04:28.500 The one that just says true says that everybody is a vampire. 00:04:28.500 --> 00:04:29.880 You might think, oh, that sucks. 00:04:29.880 --> 00:04:32.250 But it's not that bad since seven of the 10 samples 00:04:32.250 --> 00:04:33.530 are vampires. 00:04:33.530 --> 00:04:38.690 The key, crucial thing about boosting 00:04:38.690 --> 00:04:42.520 is that for any possible classifier, 00:04:42.520 --> 00:04:46.000 like classifying on the evil dimension-- which actually 00:04:46.000 --> 00:04:47.650 sounds like some kind of weird place 00:04:47.650 --> 00:04:48.899 that you'd go in a comic book. 00:04:48.899 --> 00:04:51.690 But classifying on the emo dimension or whatever, 00:04:51.690 --> 00:04:56.970 as long as it's not a 50-50 split of the data, 00:04:56.970 --> 00:05:00.990 you're guaranteed to be able to use it 00:05:00.990 --> 00:05:03.630 in some way for boosting. 00:05:03.630 --> 00:05:07.220 If there is a 50-50 split, it's like flipping a coin, 00:05:07.220 --> 00:05:08.670 so it's useless. 00:05:08.670 --> 00:05:11.490 Because if you had some other thing, 00:05:11.490 --> 00:05:13.210 like gender equals male or female-- 00:05:13.210 --> 00:05:14.580 and let's say that was 50-50. 00:05:14.580 --> 00:05:15.700 It's not. 00:05:15.700 --> 00:05:17.450 But let's say it was 50-50 between vampire 00:05:17.450 --> 00:05:21.390 and non-vampire-- it's a useless classifier 00:05:21.390 --> 00:05:25.495 because it would be just the same as flipping a coin. 00:05:25.495 --> 00:05:26.690 You'd get no information. 00:05:26.690 --> 00:05:28.210 Now, you might say, wait a minute. 00:05:28.210 --> 00:05:31.960 What about classifiers that get worse than 50-50? 00:05:31.960 --> 00:05:33.470 What about them? 00:05:33.470 --> 00:05:36.430 Might not they be even worse than a 50-50 classifier? 00:05:36.430 --> 00:05:39.325 I claim a classifier that gets less than 50-50 00:05:39.325 --> 00:05:41.760 is still better than a classifier that 00:05:41.760 --> 00:05:44.242 gets exactly a 50-50 split. 00:05:44.242 --> 00:05:45.113 Is there a question? 00:05:45.113 --> 00:05:45.738 AUDIENCE: Yeah. 00:05:45.738 --> 00:05:49.224 In the ID tree example, somebody said you used 50-50 classifiers 00:05:49.224 --> 00:05:51.714 and played around. 00:05:51.714 --> 00:05:54.536 And after you already produced the elements per set, 00:05:54.536 --> 00:05:57.190 then you only used 50-50 classifiers [INAUDIBLE] 00:05:57.190 --> 00:05:57.690 per set. 00:05:57.690 --> 00:06:00.350 PROFESSOR: So the question is, in the ID tree example, 00:06:00.350 --> 00:06:03.900 you might use 50-50 classifiers in later rounds 00:06:03.900 --> 00:06:08.670 if, for instance, there's a 50-50 classifier 00:06:08.670 --> 00:06:11.420 except for that most of the things off of that side 00:06:11.420 --> 00:06:13.410 have been already removed. 00:06:13.410 --> 00:06:15.750 Let's say there's 20 data points, 00:06:15.750 --> 00:06:19.750 and there's a classifier that splits it 10 and 10. 00:06:19.750 --> 00:06:24.650 And it gets half plus half minus on both sides. 00:06:24.650 --> 00:06:27.209 But all the pluses from the right side 00:06:27.209 --> 00:06:29.000 have been removed by some other classifier. 00:06:29.000 --> 00:06:29.708 You might use it. 00:06:29.708 --> 00:06:30.360 That's true. 00:06:30.360 --> 00:06:32.235 But in boosting, you will never use something 00:06:32.235 --> 00:06:34.410 that's a 50-50 classifier. 00:06:34.410 --> 00:06:38.970 You never use something that has exactly a 50-50 chance 00:06:38.970 --> 00:06:40.690 of being correct. 00:06:40.690 --> 00:06:44.620 Because if it has a 50-50 chance of being correct, it's useless. 00:06:44.620 --> 00:06:51.080 And if it has a 50-50 chance of-- no, sorry. 00:06:51.080 --> 00:06:53.630 Let me specify again. 00:06:53.630 --> 00:06:58.945 You'll never use something that has 00:06:58.945 --> 00:07:03.480 a 50-50 chance of giving you the right answer given the weights. 00:07:03.480 --> 00:07:04.870 That's very, very important. 00:07:04.870 --> 00:07:08.940 And that may be what your question was getting at. 00:07:08.940 --> 00:07:12.070 As I'm about to show you and as Patrick told you 00:07:12.070 --> 00:07:14.910 in the lecture, in later rounds of boosting, 00:07:14.910 --> 00:07:18.430 you change the weights of each of the 10 data points. 00:07:18.430 --> 00:07:22.120 At first, you start with all weights being 1/10. 00:07:22.120 --> 00:07:23.680 The weights have add up to 1. 00:07:23.680 --> 00:07:25.580 In this case, you never, ever choose 00:07:25.580 --> 00:07:28.640 a classifier that gets five of them right and five of them 00:07:28.640 --> 00:07:29.540 wrong. 00:07:29.540 --> 00:07:31.530 In the later rounds, you'll never, ever 00:07:31.530 --> 00:07:36.750 choose a classifier that gets half of the weight 00:07:36.750 --> 00:07:40.430 wrong-- exactly half of the weight wrong. 00:07:40.430 --> 00:07:44.590 But half of the weight may not be half of the data points. 00:07:44.590 --> 00:07:46.740 So it's possible to choose a classifier that 00:07:46.740 --> 00:07:49.130 gets half of the data points wrong if it doesn't 00:07:49.130 --> 00:07:50.630 get half of the weight wrong. 00:07:50.630 --> 00:07:53.920 And that's similar to an ID tree when you've already 00:07:53.920 --> 00:07:55.330 gotten things right before. 00:07:55.330 --> 00:07:57.371 Because you'll see that the weight is going to go 00:07:57.371 --> 00:07:58.730 to the ones you got wrong. 00:07:58.730 --> 00:08:01.760 So I'm not saying that you should throw out 00:08:01.760 --> 00:08:04.400 right away anything that gets five of the points wrong. 00:08:04.400 --> 00:08:06.900 Hell, you shouldn't even throw out right away something that 00:08:06.900 --> 00:08:08.191 gets seven of the points wrong. 00:08:08.191 --> 00:08:11.390 It's possible-- possible-- that you can get seven of the points 00:08:11.390 --> 00:08:13.590 wrong, or getting less than half of the weight 00:08:13.590 --> 00:08:15.980 wrong if those other three points are really, really 00:08:15.980 --> 00:08:18.050 annoying to get right. 00:08:18.050 --> 00:08:19.930 And we'll see that later on. 00:08:19.930 --> 00:08:24.660 But for insight, at every step along the way, 00:08:24.660 --> 00:08:26.620 we're willing to choose any classifier that 00:08:26.620 --> 00:08:29.380 doesn't get 50-50. 00:08:29.380 --> 00:08:34.590 However, we want to choose the classifier that gets 00:08:34.590 --> 00:08:38.010 the most of the weight right. 00:08:38.010 --> 00:08:39.610 By most of the weight, at first, we 00:08:39.610 --> 00:08:41.030 mean most of the points right. 00:08:41.030 --> 00:08:44.810 Later, we will mean exactly what I said-- most of the weight. 00:08:44.810 --> 00:08:48.110 And if you don't understand that, it's sometimes 00:08:48.110 --> 00:08:51.304 hard to get it right away when Patrick just 00:08:51.304 --> 00:08:53.220 lectures through it, introduces a new concept. 00:08:53.220 --> 00:08:54.719 If you don't understand that, you'll 00:08:54.719 --> 00:08:58.040 see what I mean when we go through, all right? 00:08:58.040 --> 00:09:01.100 So my point I was making before is, what about things 00:09:01.100 --> 00:09:04.007 that get less than half of the weight right? 00:09:04.007 --> 00:09:06.340 Well, those are always OK because you can just flip them 00:09:06.340 --> 00:09:09.900 around, use their inverse, and that gets more than half 00:09:09.900 --> 00:09:12.010 of the weight right. 00:09:12.010 --> 00:09:15.570 It's sort of like-- yeah, it's sort 00:09:15.570 --> 00:09:18.270 of like my girlfriend always tells me 00:09:18.270 --> 00:09:21.790 that she is more than 50% likely to choose the wrong direction 00:09:21.790 --> 00:09:24.740 when you're trying to go between two places, which 00:09:24.740 --> 00:09:25.920 I'm kind of skeptical of. 00:09:25.920 --> 00:09:27.910 But I said, if that's really true, 00:09:27.910 --> 00:09:31.200 then we can just go wherever you didn't say to go, 00:09:31.200 --> 00:09:33.280 and we'll be more likely to go the right way. 00:09:33.280 --> 00:09:35.909 So you're actually really good at finding the place 00:09:35.909 --> 00:09:36.700 that we want to go. 00:09:36.700 --> 00:09:37.950 And then she's like, no, that won't work. 00:09:37.950 --> 00:09:40.270 Because then I'll know that you're going to do that, 00:09:40.270 --> 00:09:43.115 and I'll double say the wrong way. 00:09:43.115 --> 00:09:44.740 And then you'll go the wrong way again. 00:09:44.740 --> 00:09:47.670 But that not withstanding, you can see you 00:09:47.670 --> 00:09:50.250 can apply the same concept to boosting. 00:09:50.250 --> 00:09:52.270 And that's why underneath of this, 00:09:52.270 --> 00:09:55.450 I have all the opposite versions of all these tests. 00:09:55.450 --> 00:10:00.520 So what should we be doing to solve this problem more 00:10:00.520 --> 00:10:01.270 quickly? 00:10:01.270 --> 00:10:04.900 First, let's figure out which data points are misclassified 00:10:04.900 --> 00:10:06.980 by each of these classifiers. 00:10:06.980 --> 00:10:10.460 In other words, if we say all the evil things are vampires 00:10:10.460 --> 00:10:13.250 and all the non-evil things are not vampires, 00:10:13.250 --> 00:10:14.850 what do we get wrong? 00:10:14.850 --> 00:10:17.740 And if we do that for every classifier, 00:10:17.740 --> 00:10:19.830 then that'll make it faster later on. 00:10:19.830 --> 00:10:22.630 Because later on, we're going to go through classifiers, 00:10:22.630 --> 00:10:25.450 and we're going to have to add up the ones they got wrong. 00:10:25.450 --> 00:10:29.670 So this chart over here is going to be extremely useful. 00:10:29.670 --> 00:10:31.420 And on the test that this appeared in, 00:10:31.420 --> 00:10:34.020 they even made you fill it in to help yourself out. 00:10:34.020 --> 00:10:35.440 So let's see. 00:10:35.440 --> 00:10:38.560 If we said that all the evil equals yes are vampires 00:10:38.560 --> 00:10:43.130 and all the evil equals no are not vampires, 00:10:43.130 --> 00:10:47.400 then-- I'll do the first one for you. 00:10:47.400 --> 00:10:50.940 So we get all of the non-vampires 00:10:50.940 --> 00:10:53.390 correct because they are all evil equals no. 00:10:53.390 --> 00:10:59.770 But unfortunately, we get Angel, Edward Cullen, Saya Otonashi, 00:10:59.770 --> 00:11:06.880 and Lestat de Lioncourt wrong because they are vampires, 00:11:06.880 --> 00:11:08.275 and they're evil equals no. 00:11:11.420 --> 00:11:12.840 Apparently, Lestat is iffy. 00:11:12.840 --> 00:11:16.530 But I never read those books, and the Wikipedia article 00:11:16.530 --> 00:11:18.540 said that in the end, he wasn't that evil. 00:11:18.540 --> 00:11:20.310 So there we go. 00:11:20.310 --> 00:11:24.060 Evil equals yes misclassifies 2, 3, 4, and 5. 00:11:24.060 --> 00:11:28.310 All right, so let's try emo equals yes. 00:11:28.310 --> 00:11:29.950 I'll have someone else do it. 00:11:29.950 --> 00:11:32.750 So let's see if you guys got it. 00:11:32.750 --> 00:11:35.485 So if we say that all the emo people are vampires 00:11:35.485 --> 00:11:37.930 and all the non-emo people are not vampires, 00:11:37.930 --> 00:11:39.175 what do we get wrong? 00:11:39.175 --> 00:11:40.360 AUDIENCE: 1, 6, 7, 9. 00:11:40.360 --> 00:11:42.810 PROFESSOR: 1, 6, 7, 9-- that's exactly right, and fast. 00:11:42.810 --> 00:11:44.350 Good. 00:11:44.350 --> 00:11:46.230 We get 1, 6, 7, and 9 wrong. 00:11:46.230 --> 00:11:49.365 1, 6, and 7 are wrong because they are not emo, 00:11:49.365 --> 00:11:50.240 but they're vampires. 00:11:50.240 --> 00:11:53.065 9 is wrong because Squall is emo, and he is not a vampire. 00:11:56.210 --> 00:11:57.040 Good. 00:11:57.040 --> 00:12:02.150 OK, what if we said that exactly the transforming characters are 00:12:02.150 --> 00:12:05.502 vampires and the ones that do not transform are not vampires? 00:12:05.502 --> 00:12:06.710 Which ones will we get wrong? 00:12:11.690 --> 00:12:14.180 AUDIENCE: [INAUDIBLE]. 00:12:14.180 --> 00:12:16.670 PROFESSOR: Transforms is the next one over. 00:12:16.670 --> 00:12:18.662 AUDIENCE: All of the ones [INAUDIBLE]. 00:12:18.662 --> 00:12:19.408 PROFESSOR: So which ones would we 00:12:19.408 --> 00:12:21.866 get wrong if we said that said transforms yes were vampires 00:12:21.866 --> 00:12:23.642 and transforms no were not vampires? 00:12:23.642 --> 00:12:25.620 AUDIENCE: We'd get 8 wrong. 00:12:25.620 --> 00:12:27.036 PROFESSOR: We'd definitely 8 wrong 00:12:27.036 --> 00:12:30.008 because she's not a vampire. 00:12:30.008 --> 00:12:30.924 AUDIENCE: [INAUDIBLE]. 00:12:33.840 --> 00:12:36.170 PROFESSOR: Well, no. 00:12:36.170 --> 00:12:37.582 It's actually on there. 00:12:37.582 --> 00:12:43.095 AUDIENCE: And 3 and 4 we'd also get wrong. 00:12:43.095 --> 00:12:43.720 PROFESSOR: Yep. 00:12:43.720 --> 00:12:44.710 AUDIENCE: As well as 5. 00:12:44.710 --> 00:12:45.710 PROFESSOR: Yes, exactly. 00:12:45.710 --> 00:12:46.230 OK. 00:12:46.230 --> 00:12:46.700 Oh, man. 00:12:46.700 --> 00:12:47.741 You didn't see the chart. 00:12:47.741 --> 00:12:50.580 You were just like, hmmm. 00:12:50.580 --> 00:12:51.470 You saw the left. 00:12:51.470 --> 00:12:53.033 You just said, hm, which one of these 00:12:53.033 --> 00:12:53.850 are the transforming characters? 00:12:53.850 --> 00:12:55.320 OK, that's pretty hardcore. 00:12:55.320 --> 00:12:56.195 AUDIENCE: [LAUGHTER]. 00:12:59.240 --> 00:13:01.140 PROFESSOR: But yeah, 3, 4, 5, and 8. 00:13:01.140 --> 00:13:02.780 No, no, it's definitely given to you. 00:13:02.780 --> 00:13:04.210 That would be like the worst test 00:13:04.210 --> 00:13:06.920 ever for international students. 00:13:06.920 --> 00:13:10.320 Ah, if you don't know these ten characters as vampires, 00:13:10.320 --> 00:13:12.270 you lose. 00:13:12.270 --> 00:13:15.390 All right, so what about that sparkly 00:13:15.390 --> 00:13:19.160 equals yes is a vampire, and if it's not sparkly, 00:13:19.160 --> 00:13:20.634 it's not a vampire? 00:13:20.634 --> 00:13:22.050 This is guaranteed not to go well. 00:13:22.050 --> 00:13:24.875 What do you think it's going to get wrong? 00:13:24.875 --> 00:13:26.330 AUDIENCE: For sparkly? 00:13:26.330 --> 00:13:29.039 PROFESSOR: Yeah, sparkly equals yes are the only vampires. 00:13:29.039 --> 00:13:30.205 AUDIENCE: [INAUDIBLE] wrong. 00:13:30.205 --> 00:13:34.133 Angel's going to be wrong. 00:13:34.133 --> 00:13:43.709 Saya-- so 1, 2, 4, 5, 6, 7, and 8. 00:13:43.709 --> 00:13:45.250 PROFESSOR: And 8-- yes, that's right. 00:13:45.250 --> 00:13:47.950 It gets 1, 2, 4, 5, 6, 7, and 8 wrong. 00:13:47.950 --> 00:13:50.980 That's pretty awful. 00:13:50.980 --> 00:13:53.760 But dammit, it gets Edward Cullen right. 00:13:53.760 --> 00:13:57.230 And he's hard to get correct due to the fact 00:13:57.230 --> 00:13:58.860 that he's not very much like a vampire. 00:13:58.860 --> 00:14:02.100 He's more of a superhero who says he's a vampire. 00:14:02.100 --> 00:14:06.620 OK, so next, number of romantic interest greater than two. 00:14:06.620 --> 00:14:09.580 So if they have more than two romantic interest, 00:14:09.580 --> 00:14:10.510 they're a vampire. 00:14:10.510 --> 00:14:12.520 And otherwise, they're not a vampire. 00:14:12.520 --> 00:14:15.352 So which ones would that get wrong? 00:14:15.352 --> 00:14:15.852 Hm? 00:14:15.852 --> 00:14:16.643 AUDIENCE: 3 and 10. 00:14:16.643 --> 00:14:18.740 PROFESSOR: Just 3 and 10, that's right. 00:14:18.740 --> 00:14:22.850 Because Circe had Odysseus. 00:14:22.850 --> 00:14:24.360 She had Telemachus. 00:14:24.360 --> 00:14:27.360 Actually, she had that guy she turned into a woodpecker. 00:14:27.360 --> 00:14:31.510 She had that other guy who was a sea god who 00:14:31.510 --> 00:14:33.800 caused her to turn Scylla into the nine-headed thing, 00:14:33.800 --> 00:14:36.130 and probably at least one other person. 00:14:36.130 --> 00:14:37.770 So Circe it gets wrong. 00:14:37.770 --> 00:14:41.530 And it also gets Edward Cullen wrong because he only has one. 00:14:41.530 --> 00:14:45.240 So 3 and 10. 00:14:45.240 --> 00:14:47.150 You can tell I thought about this problem 00:14:47.150 --> 00:14:48.430 when I was writing it up. 00:14:48.430 --> 00:14:49.580 I wrote this one. 00:14:49.580 --> 00:14:53.400 All right, number of romantic interest greater than four. 00:14:53.400 --> 00:14:55.460 So it's a little bit different this time. 00:14:55.460 --> 00:14:58.482 Now you have to have at least four romantic interests-- 00:14:58.482 --> 00:15:00.190 or actually, greater than four, but there 00:15:00.190 --> 00:15:03.100 are none that are exactly four-- to be classified as a vampire. 00:15:03.100 --> 00:15:05.100 Which ones do you think it's going to get wrong? 00:15:05.100 --> 00:15:09.460 AUDIENCE: 3, 4, 10. 00:15:09.460 --> 00:15:16.510 PROFESSOR: Yup, it is going to get 3, 4, and 10 wrong. 00:15:16.510 --> 00:15:19.330 Because now, you run into the fact 00:15:19.330 --> 00:15:27.250 that Saya has that blond [INAUDIBLE] guy, Haji, and Kai. 00:15:27.250 --> 00:15:31.874 So the last of the positive ones-- because I claim I 00:15:31.874 --> 00:15:34.040 can derive all the negative ones if you guys give me 00:15:34.040 --> 00:15:35.220 the positive ones. 00:15:35.220 --> 00:15:37.511 The last of the positive ones is everybody's a vampire. 00:15:37.511 --> 00:15:39.393 Who does that get wrong? 00:15:39.393 --> 00:15:40.836 AUDIENCE: 8, 9, 10. 00:15:40.836 --> 00:15:42.625 PROFESSOR: Yes, OK. 00:15:46.210 --> 00:15:48.450 Now, I can derive all the negative ones 00:15:48.450 --> 00:15:51.750 from this without a sweat. 00:15:51.750 --> 00:16:00.462 Evil equals no-- well, it's 1, 6, 7, 8, 9, 10 00:16:00.462 --> 00:16:01.670 without looking at the chart. 00:16:05.720 --> 00:16:08.680 Raise your hand if you see why it's 1, 6, 7, 8, 9, 10 00:16:08.680 --> 00:16:11.390 without looking at the chart. 00:16:11.390 --> 00:16:13.780 Raise your hand if you don't. 00:16:13.780 --> 00:16:15.910 Nobody, OK-- wait. 00:16:15.910 --> 00:16:16.920 One hand. 00:16:16.920 --> 00:16:18.730 OK, I saw another one back there too later. 00:16:18.730 --> 00:16:19.980 They were just more tentative. 00:16:19.980 --> 00:16:23.410 OK, it's the complement of A because A is 00:16:23.410 --> 00:16:25.200 evil equals yes is a vampire. 00:16:25.200 --> 00:16:26.690 It gets 2, 3, 4, and 5 wrong. 00:16:26.690 --> 00:16:29.470 So therefore, evil equals no is a vampire is guaranteed 00:16:29.470 --> 00:16:30.810 to get all the opposite ones. 00:16:30.810 --> 00:16:32.630 AUDIENCE: Oh, we could have looked at that too? 00:16:32.630 --> 00:16:33.510 PROFESSOR: Yeah, we're looking here, 00:16:33.510 --> 00:16:35.370 but we're not looking at the big chart there. 00:16:35.370 --> 00:16:36.190 AUDIENCE: You can look at any? 00:16:36.190 --> 00:16:36.710 PROFESSOR: Oh, yeah. 00:16:36.710 --> 00:16:38.460 If you can't look at anything, then you're 00:16:38.460 --> 00:16:43.250 screwed-- unless not only are you as hardcore as this guy, 00:16:43.250 --> 00:16:46.700 but you've also memorized the numbers. 00:16:46.700 --> 00:16:56.470 All right, so emo equals no is going to be 2, 3, 4, 5, 8, 10. 00:16:56.470 --> 00:17:03.450 Transforms equals no is 1, 2, 6, 7, 9, 10. 00:17:03.450 --> 00:17:11.490 Sparkle equals no is 3, 9, 10. 00:17:11.490 --> 00:17:14.440 Romantic interest less than two is everything except 3 and 10-- 00:17:14.440 --> 00:17:20.805 1, 2, 4, 5, 6, 7, 8, 9. 00:17:20.805 --> 00:17:27.140 1, 2, 5, 6, 7, 8, 9. 00:17:27.140 --> 00:17:34.754 And then finally, everything but 8, 9, and 10, so 1, 2, 3, 4, 5, 00:17:34.754 --> 00:17:36.240 6, 7. 00:17:36.240 --> 00:17:37.860 All right, so when we started off, 00:17:37.860 --> 00:17:39.370 we know what everything gets wrong. 00:17:39.370 --> 00:17:41.740 I then make a bold claim, because there 00:17:41.740 --> 00:17:44.560 are n of these, which is 14. 00:17:44.560 --> 00:17:46.700 I make the claim that there are only six 00:17:46.700 --> 00:17:50.370 that, in your wildest dreams, you would ever possibly 00:17:50.370 --> 00:17:53.820 even consider using ever. 00:17:53.820 --> 00:17:56.071 And the rest, you would never, ever use. 00:17:56.071 --> 00:17:56.570 Question? 00:17:56.570 --> 00:17:58.861 AUDIENCE: Yeah, I just have a question about the number 00:17:58.861 --> 00:17:59.913 of romantic interests. 00:17:59.913 --> 00:18:00.538 PROFESSOR: Yes. 00:18:00.538 --> 00:18:03.018 AUDIENCE: You negated it without an equals on either side. 00:18:03.018 --> 00:18:04.010 PROFESSOR: That's true. 00:18:04.010 --> 00:18:06.884 That works only because there are [INAUDIBLE] 2 or 4. 00:18:06.884 --> 00:18:09.300 But it should have been negated with a less than or equal. 00:18:09.300 --> 00:18:11.250 I'm copying off of the quiz. 00:18:11.250 --> 00:18:12.780 But yes. 00:18:12.780 --> 00:18:15.334 I noticed that this morning when I was putting myself 00:18:15.334 --> 00:18:16.000 through my pace. 00:18:16.000 --> 00:18:17.530 I'm like, there's not a less than or equal to. 00:18:17.530 --> 00:18:18.113 Wait a minute. 00:18:18.113 --> 00:18:19.750 And then, oh, wait. 00:18:19.750 --> 00:18:21.429 It doesn't have any 2's or 4's. 00:18:21.429 --> 00:18:23.220 Actually, I don't remember writing them all 00:18:23.220 --> 00:18:24.280 as 5's and 3's. 00:18:24.280 --> 00:18:27.690 It's possible that somebody else in the post-editing process 00:18:27.690 --> 00:18:30.410 changed them all to be about the same number, 00:18:30.410 --> 00:18:32.960 and then changed the less than or equal tos 00:18:32.960 --> 00:18:33.840 to be less confusing. 00:18:33.840 --> 00:18:36.410 It's possible I had Circe at 4, and there was an equal 00:18:36.410 --> 00:18:37.270 to somewhere. 00:18:37.270 --> 00:18:38.980 And they were like, forget it. 00:18:38.980 --> 00:18:41.146 Because I can't think of the fifth romantic interest 00:18:41.146 --> 00:18:42.390 for her. 00:18:42.390 --> 00:18:44.820 So yes, normally, you would have to negate it 00:18:44.820 --> 00:18:46.700 with an equal to sign, but there happened 00:18:46.700 --> 00:18:49.750 to not be any things that are equal to 4 or 2 here. 00:18:49.750 --> 00:18:53.640 So they get away with it this time. 00:18:53.640 --> 00:18:56.819 But it's good practice. 00:18:56.819 --> 00:18:58.610 So I'm claiming that in our wildest dreams, 00:18:58.610 --> 00:19:01.800 we'd only ever want to use six of these ever. 00:19:01.800 --> 00:19:04.190 And the other eight, forget it. 00:19:04.190 --> 00:19:07.860 So let's see. 00:19:07.860 --> 00:19:11.445 I will call on people at random to-- the first people obviously 00:19:11.445 --> 00:19:13.620 are getting it really easy-- to tell me 00:19:13.620 --> 00:19:16.610 which of these you think that you might ever want to use. 00:19:16.610 --> 00:19:18.733 Give me one you might ever want to use. 00:19:18.733 --> 00:19:19.680 AUDIENCE: E. 00:19:19.680 --> 00:19:20.490 PROFESSOR: Why, E? 00:19:20.490 --> 00:19:21.310 Of course. 00:19:21.310 --> 00:19:22.740 That's the best one. 00:19:22.740 --> 00:19:25.180 Yes, that's one that you might ever want to use. 00:19:25.180 --> 00:19:28.780 I'll circle the ones that you might ever want to use. 00:19:28.780 --> 00:19:30.840 E-- it only gets 3 and 10 wrong. 00:19:30.840 --> 00:19:31.920 That's amazing. 00:19:31.920 --> 00:19:34.220 It's like the best class of [INAUDIBLE]. 00:19:34.220 --> 00:19:37.934 OK, so give me another one that you might ever want to us. 00:19:37.934 --> 00:19:38.433 AUDIENCE: F. 00:19:38.433 --> 00:19:40.620 PROFESSOR: F. Let's see. 00:19:40.620 --> 00:19:41.930 F-- F is great. 00:19:41.930 --> 00:19:43.820 It only gets three wrong. 00:19:43.820 --> 00:19:46.180 Do people agree that you would ever want to use F? 00:19:46.180 --> 00:19:46.810 AUDIENCE: No. 00:19:46.810 --> 00:19:48.143 PROFESSOR: Everyone's saying no. 00:19:48.143 --> 00:19:50.165 Why not? 00:19:50.165 --> 00:19:51.990 AUDIENCE: It's like E, except worse. 00:19:51.990 --> 00:19:54.000 PROFESSOR: It's like E, except worse. 00:19:54.000 --> 00:19:56.500 It's guaranteed at every step, no matter 00:19:56.500 --> 00:19:59.900 what the weights are, to have a worse accuracy than E. 00:19:59.900 --> 00:20:01.940 It is definitely good. 00:20:01.940 --> 00:20:04.530 If E wasn't around, it would be one of our best classifiers 00:20:04.530 --> 00:20:05.580 of all. 00:20:05.580 --> 00:20:08.830 But actually, F is not one of the six. 00:20:08.830 --> 00:20:11.670 This is why I had them write on the test that there were six, 00:20:11.670 --> 00:20:13.870 because people might not have found all six. 00:20:13.870 --> 00:20:16.130 Because people who did figure out not to include F 00:20:16.130 --> 00:20:18.380 might not have figured out to include some of the ones 00:20:18.380 --> 00:20:19.470 you want to include. 00:20:19.470 --> 00:20:20.030 So-- 00:20:20.030 --> 00:20:22.071 AUDIENCE: I don't understand why you can't use F. 00:20:22.071 --> 00:20:23.690 PROFESSOR: Why you can't use F, OK. 00:20:23.690 --> 00:20:28.200 So we start off with 1/10 weight for all our data points. 00:20:28.200 --> 00:20:31.190 But let's say during our time of boosting 00:20:31.190 --> 00:20:34.470 that all ten of the data points have now different weights. 00:20:34.470 --> 00:20:36.300 So we'll call whatever the weight of 3 00:20:36.300 --> 00:20:38.510 is, which you're going to get wrong-- you want 00:20:38.510 --> 00:20:40.200 to minimize the error, right? 00:20:40.200 --> 00:20:44.080 So that weight of 3, which goes into the error of the E, is x. 00:20:44.080 --> 00:20:45.600 The weight of 10 can be y. 00:20:45.600 --> 00:20:47.954 So if you're thinking of choosing 3, 00:20:47.954 --> 00:20:49.120 you know you're-- oh, sorry. 00:20:49.120 --> 00:20:52.035 If you're thinking of choosing E, your error is x plus y. 00:20:52.035 --> 00:20:52.660 AUDIENCE: Sure. 00:20:52.660 --> 00:20:54.493 PROFESSOR: If you're thinking of choosing F, 00:20:54.493 --> 00:20:57.920 your error is x plus y plus z, where z is the error of 4. 00:20:57.920 --> 00:21:00.280 And since you're never going to have a negative weight, 00:21:00.280 --> 00:21:02.462 x plus y plus z is always greater than x plus y. 00:21:02.462 --> 00:21:03.962 AUDIENCE: You can't choose something 00:21:03.962 --> 00:21:06.503 without the 3 and the 10 anymore because you already chose E. 00:21:06.503 --> 00:21:10.417 PROFESSOR: That's-- yes, you would probably choose something 00:21:10.417 --> 00:21:12.250 that didn't get the three and the ten wrong. 00:21:12.250 --> 00:21:15.340 But you would certainly never choose F ever 00:21:15.340 --> 00:21:18.600 because it's always worse than E. In fact, this is exactly 00:21:18.600 --> 00:21:21.600 the process that will allow you to find the correct six. 00:21:21.600 --> 00:21:23.930 And by "will," I mean "can." 00:21:23.930 --> 00:21:26.210 And by "can," I mean, let's see if you guys get it. 00:21:26.210 --> 00:21:28.580 Give me another one of the six that you might keep. 00:21:28.580 --> 00:21:29.590 AUDIENCE: K. 00:21:29.590 --> 00:21:32.520 PROFESSOR: K is the claim-- sparkly. 00:21:32.520 --> 00:21:36.560 K, I'm going to say will lose for the same reason 00:21:36.560 --> 00:21:40.120 as F. It's 3, 9, and 10. 00:21:40.120 --> 00:21:43.970 It's essentially similar to 3, 4, and 10. 00:21:43.970 --> 00:21:45.620 So-- oh, by the way, we should not 00:21:45.620 --> 00:21:50.380 be only going for the ones with the fewest incorrect. 00:21:50.380 --> 00:21:53.630 You need to be going for ones that do not have something 00:21:53.630 --> 00:21:55.030 that is strictly better. 00:21:55.030 --> 00:21:57.070 In this case, 3 and 10 wrong is strictly 00:21:57.070 --> 00:21:59.491 better than getting 3, 9, and 10 wrong. 00:21:59.491 --> 00:21:59.990 Question? 00:21:59.990 --> 00:22:02.230 AUDIENCE: Oh, I was going to say transform. 00:22:02.230 --> 00:22:04.021 PROFESSOR: You were going to say transform. 00:22:04.021 --> 00:22:06.460 You are going to be correct. 00:22:06.460 --> 00:22:09.775 Transforms is one of the ones we need, C. 3, 4, 5 00:22:09.775 --> 00:22:13.280 and 8-- there's nothing down here that gets fewer 00:22:13.280 --> 00:22:15.720 than those wrong. 00:22:15.720 --> 00:22:18.120 There's nothing that gets us 3, 4, 5 wrong, for instance. 00:22:20.800 --> 00:22:25.950 Yeah, there's no way to get 3, 4 wrong without getting either 00:22:25.950 --> 00:22:28.028 10 wrong, or 5 and 8 wrong. 00:22:28.028 --> 00:22:30.594 AUDIENCE: So why is the 8 like that? 00:22:30.594 --> 00:22:31.260 PROFESSOR: What? 00:22:31.260 --> 00:22:32.410 AUDIENCE: Why not G? 00:22:32.410 --> 00:22:34.030 PROFESSOR: Why not G? 00:22:34.030 --> 00:22:34.931 Why not? 00:22:34.931 --> 00:22:35.430 Why not G? 00:22:35.430 --> 00:22:38.078 Let's include G, too. 00:22:38.078 --> 00:22:38.619 AUDIENCE: Oh. 00:22:38.619 --> 00:22:40.070 We don't have to do it all? 00:22:40.070 --> 00:22:41.261 PROFESSOR: We need six. 00:22:41.261 --> 00:22:43.260 No, I just said give me any, and someone gave me 00:22:43.260 --> 00:22:45.310 the easiest one, E. Question? 00:22:45.310 --> 00:22:45.995 AUDIENCE: B 00:22:45.995 --> 00:22:46.870 PROFESSOR: Why not B? 00:22:46.870 --> 00:22:47.920 B looks great. 00:22:47.920 --> 00:22:52.674 I love B. Let's include B. Does someone else want 00:22:52.674 --> 00:22:54.590 to give another one that they want to include? 00:22:54.590 --> 00:22:55.650 AUDIENCE: A. 00:22:55.650 --> 00:22:57.321 PROFESSOR: A. Why not A? 00:22:57.321 --> 00:22:57.820 Sure. 00:22:57.820 --> 00:22:59.360 I mean, it's hard to see down here 00:22:59.360 --> 00:23:01.610 because there might be something better on the bottom. 00:23:01.610 --> 00:23:03.600 But yeah, there's not. 00:23:03.600 --> 00:23:05.090 So let's include A. Why not A? 00:23:05.090 --> 00:23:08.370 I love A. A's great. 00:23:08.370 --> 00:23:10.280 OK. 00:23:10.280 --> 00:23:18.330 So that is now five. 00:23:18.330 --> 00:23:20.500 There's one more that we need. 00:23:20.500 --> 00:23:25.360 It's by far the hardest one to find. 00:23:25.360 --> 00:23:27.880 Find me one more that there's nothing better than it. 00:23:27.880 --> 00:23:30.810 There's nothing that has a strict subset 00:23:30.810 --> 00:23:33.270 of the same ones wrong. 00:23:33.270 --> 00:23:34.090 AUDIENCE: Question. 00:23:34.090 --> 00:23:34.520 PROFESSOR: What? 00:23:34.520 --> 00:23:35.934 AUDIENCE: Sorry, can we quickly-- 00:23:35.934 --> 00:23:38.711 why would you choose A before we chose C? 00:23:38.711 --> 00:23:40.252 PROFESSOR: OK, why would you choose A 00:23:40.252 --> 00:23:41.580 before you've chosen C? 00:23:41.580 --> 00:23:45.490 Let's say 8 was a real problem for you. 00:23:45.490 --> 00:23:47.970 And you were just getting-- let's say that 3, 4 and 5, 00:23:47.970 --> 00:23:49.324 they weren't that bad. 00:23:49.324 --> 00:23:50.240 They weren't that bad. 00:23:50.240 --> 00:23:50.750 They weren't that bad. 00:23:50.750 --> 00:23:52.680 OK, you got them wrong here with transforms. 00:23:52.680 --> 00:23:58.600 You chose C. But sometime later, 8 was just by far your issue. 00:23:58.600 --> 00:23:59.770 All right? 00:23:59.770 --> 00:24:03.441 3, 4, and 5, and 8-- 3, 4 and 5 are much smaller weights 00:24:03.441 --> 00:24:03.940 than 8. 00:24:03.940 --> 00:24:07.790 And then after you've got 3, 4, 5, and 8 wrong, 3, 4, and 5 00:24:07.790 --> 00:24:08.840 were still not that bad. 00:24:08.840 --> 00:24:11.200 And 8 still was a high number. 00:24:11.200 --> 00:24:14.640 And then sometime later down the line, you're looking at things. 00:24:14.640 --> 00:24:16.092 And you're saying, you know what? 00:24:16.092 --> 00:24:17.800 I really don't want to get 8 wrong again, 00:24:17.800 --> 00:24:19.689 but I don't mind if I get 3, 4 and 5 wrong. 00:24:19.689 --> 00:24:21.230 Maybe I'll get it wrong with 2, which 00:24:21.230 --> 00:24:22.664 I've never gone wrong yet. 00:24:22.664 --> 00:24:25.080 Actually, none of the ones we've circled here get 2 wrong, 00:24:25.080 --> 00:24:27.950 so it's probably not that bad to get 2 wrong. 00:24:27.950 --> 00:24:31.540 So that's why-- because it doesn't get 8 wrong. 00:24:31.540 --> 00:24:35.405 If A was 2, 3, 4, 5, 8, you'd never take it. 00:24:35.405 --> 00:24:36.445 Do you see what I mean? 00:24:36.445 --> 00:24:37.820 Oh, did someone raise their hand? 00:24:37.820 --> 00:24:38.920 Did someone find it? 00:24:38.920 --> 00:24:40.378 AUDIENCE: I just have a question. 00:24:40.378 --> 00:24:43.780 You can use the same reasoning for choosing K, right? 00:24:43.780 --> 00:24:47.668 Because after E, we could have chosen A 00:24:47.668 --> 00:24:51.080 and said that 9 only a little different-- 00:24:51.080 --> 00:24:52.770 PROFESSOR: But it's strictly worse. 00:24:52.770 --> 00:24:59.348 AUDIENCE: Sorry, sorry, I meant [INAUDIBLE] E to 9 and 10, 00:24:59.348 --> 00:25:02.327 and then we could have chosen 3, 9, and 10, right? 00:25:02.327 --> 00:25:02.827 Because-- 00:25:02.827 --> 00:25:07.051 PROFESSOR: But we chose to use E. Because getting only 3, 00:25:07.051 --> 00:25:08.800 10 wrong is better than getting 3, 9, 00:25:08.800 --> 00:25:10.782 and 10 wrong in any universe. 00:25:10.782 --> 00:25:12.310 You pick. 00:25:12.310 --> 00:25:13.760 You see what I mean? 00:25:13.760 --> 00:25:15.940 It might not be that much worse. 00:25:15.940 --> 00:25:17.980 It might be only a little bit worse to choose K, 00:25:17.980 --> 00:25:20.170 but it's always worse. 00:25:20.170 --> 00:25:22.118 So it-- question? 00:25:22.118 --> 00:25:25.106 AUDIENCE: [INAUDIBLE] the right one by reasoning [INAUDIBLE] 00:25:25.106 --> 00:25:28.094 we need one that doesn't have 3 in it. 00:25:28.094 --> 00:25:32.576 Because right now, [INAUDIBLE] get 3 wrong. 00:25:32.576 --> 00:25:34.700 PROFESSOR: OK, that's a pretty good insight. 00:25:34.700 --> 00:25:36.349 What are you thinking about? 00:25:36.349 --> 00:25:38.217 AUDIENCE: Well, I'm trying to justify D. 00:25:38.217 --> 00:25:39.800 PROFESSOR: You're trying to justify D. 00:25:39.800 --> 00:25:41.070 AUDIENCE: [INAUDIBLE]. 00:25:41.070 --> 00:25:42.199 PROFESSOR: D is huge. 00:25:42.199 --> 00:25:43.740 It gets more than half of them wrong. 00:25:43.740 --> 00:25:44.670 But you know what? 00:25:44.670 --> 00:25:45.600 It gets 3 right. 00:25:45.600 --> 00:25:46.530 You know what? 00:25:46.530 --> 00:25:47.930 It gets 10 right. 00:25:47.930 --> 00:25:52.570 And unlike our things that get 3 and 10 right, which is B, 00:25:52.570 --> 00:25:55.660 it also gets 9 right. 00:25:55.660 --> 00:25:57.450 D is the last classifier. 00:25:57.450 --> 00:25:58.530 You got it. 00:25:58.530 --> 00:26:02.250 It's hard to choose one that has seven of them wrong, 00:26:02.250 --> 00:26:04.620 but D is the last one you might pick. 00:26:04.620 --> 00:26:06.695 It turns out there's nothing better than this 00:26:06.695 --> 00:26:12.120 for classifying correctly those annoying data points of Edward 00:26:12.120 --> 00:26:16.730 Cullen and Squall, and also Circe, who's not that annoying, 00:26:16.730 --> 00:26:20.210 but she tends to be a problem when romance is concerned. 00:26:20.210 --> 00:26:24.240 So these are the six that we might use. 00:26:24.240 --> 00:26:25.720 We can now ignore the rest of them 00:26:25.720 --> 00:26:29.452 forever-- or at least until someone reuses this problem 00:26:29.452 --> 00:26:30.410 or something like that. 00:26:30.410 --> 00:26:32.762 But we can ignore everything except A, B, C, D, E, 00:26:32.762 --> 00:26:34.470 G. In fact, why did I even bring that up? 00:26:34.470 --> 00:26:36.300 All the ones we want are on the front. 00:26:36.300 --> 00:26:38.020 I'll bring it back down. 00:26:38.020 --> 00:26:42.270 Then I'll cross this off with reckless abandon. 00:26:42.270 --> 00:26:43.979 That even broke off a piece of my chalk. 00:26:43.979 --> 00:26:45.770 Now, these are the ones that we're actually 00:26:45.770 --> 00:26:48.110 thinking about using. 00:26:48.110 --> 00:26:50.810 There is a chart over here already 00:26:50.810 --> 00:26:55.590 prepared to do some boosting with these six classifiers, 00:26:55.590 --> 00:26:57.020 all right? 00:26:57.020 --> 00:26:59.710 So let's give it a try. 00:26:59.710 --> 00:27:01.270 Now, remember, in boosting, we always 00:27:01.270 --> 00:27:02.790 try to choose whichever classifier has the least 00:27:02.790 --> 00:27:03.150 errors. 00:27:03.150 --> 00:27:03.720 Is there a question? 00:27:03.720 --> 00:27:04.636 AUDIENCE: Sorry, yeah. 00:27:04.636 --> 00:27:07.450 Before we move on, can you say again a little 00:27:07.450 --> 00:27:09.610 more slowly what exactly we were looking 00:27:09.610 --> 00:27:12.225 for when we were choosing our classifiers, 00:27:12.225 --> 00:27:14.310 like something about the subset? 00:27:17.590 --> 00:27:20.320 PROFESSOR: You generally want to take classifiers. 00:27:20.320 --> 00:27:23.220 So I'll tell you what let's you cross off a classifier. 00:27:23.220 --> 00:27:25.210 That may be a better way to do it. 00:27:25.210 --> 00:27:31.400 You can cross off a classifier as useless if-- and by the way, 00:27:31.400 --> 00:27:34.100 this is only useful if you can do it faster 00:27:34.100 --> 00:27:36.880 than just wasting your time looking at all of them. 00:27:36.880 --> 00:27:39.492 Because if you can't cross off some of them as useless-- 00:27:39.492 --> 00:27:41.200 usually on the test, they won't make you. 00:27:41.200 --> 00:27:42.790 You can just waste your time and have 00:27:42.790 --> 00:27:46.380 14 instead of six possibilities every step of the boosting. 00:27:46.380 --> 00:27:51.530 But take a look at this-- 1, 2, 3, 4, 5, 6, 7. 00:27:51.530 --> 00:27:52.260 Then see. 00:27:52.260 --> 00:27:54.660 Do you have anything here that has 00:27:54.660 --> 00:27:57.610 a strict subset of these wrong? 00:27:57.610 --> 00:27:58.110 Oh, look. 00:27:58.110 --> 00:28:00.220 2, 3, 4, 5 is a strict subset. 00:28:00.220 --> 00:28:01.720 This can be crossed off. 00:28:01.720 --> 00:28:06.170 1, 2, 5, 6, 7, 8, 9-- anything that's a strict subset? 00:28:06.170 --> 00:28:08.419 Yes, 1, 6, 7, 9. 00:28:08.419 --> 00:28:09.460 So it can be crossed off. 00:28:09.460 --> 00:28:12.640 1, 2, 4, 5, 6, 7, 8, 9. 00:28:12.640 --> 00:28:13.660 Let's see. 00:28:13.660 --> 00:28:16.040 1, 6, 7, 9 is a strict subset. 00:28:16.040 --> 00:28:16.820 3, 9, 10. 00:28:16.820 --> 00:28:18.970 3, 10 is a strict subset. 00:28:18.970 --> 00:28:21.310 1, 6, 7, 9-- a strict subset. 00:28:21.310 --> 00:28:25.756 3, 4, 5, 8 is a strict subset, as is 2, 3, 4, 5. 00:28:25.756 --> 00:28:28.000 1, 6, 7, 9 is a strict subset. 00:28:28.000 --> 00:28:31.170 And up here, 3, 4, 10-- 3, 10 is a strict subset. 00:28:31.170 --> 00:28:34.185 But the others don't have one, even 1, 2, 4, 5, 6, 7, 8. 00:28:37.620 --> 00:28:39.440 In general, you want to keep them. 00:28:39.440 --> 00:28:41.930 You want to keep every classifier you might use. 00:28:41.930 --> 00:28:43.400 The only ones you'll never use are 00:28:43.400 --> 00:28:45.816 ones that there's something else that's just better always 00:28:45.816 --> 00:28:50.370 by having a strict subset of them wrong. 00:28:50.370 --> 00:28:52.810 Hopefully, that was more clear. 00:28:52.810 --> 00:28:55.430 It's tricky. 00:28:55.430 --> 00:28:58.140 Very few people realize-- we're brave enough 00:28:58.140 --> 00:29:01.020 to take sparkly even when it got seven things wrong. 00:29:01.020 --> 00:29:03.670 So let's start out some boosting. 00:29:03.670 --> 00:29:04.870 This wasn't boosting. 00:29:04.870 --> 00:29:06.970 This was setting yourself up. 00:29:06.970 --> 00:29:09.040 But it was setting yourself up with the knowledge 00:29:09.040 --> 00:29:10.260 of how boosting works. 00:29:10.260 --> 00:29:11.820 Less knowledge, less search. 00:29:11.820 --> 00:29:13.700 Now we only have to search six things. 00:29:13.700 --> 00:29:16.158 Ah, I mean more knowledge, less search, not less knowledge, 00:29:16.158 --> 00:29:16.980 less search. 00:29:16.980 --> 00:29:23.180 So we start off with all weights being equal. 00:29:23.180 --> 00:29:25.456 And since there's ten data points, 00:29:25.456 --> 00:29:27.330 all ten of the data points are weighted 1/10. 00:29:30.710 --> 00:29:33.495 OK, we're now weighting all of them equally. 00:29:33.495 --> 00:29:34.870 Since we're weighting all of them 00:29:34.870 --> 00:29:36.911 equally, when we want to find the classifier that 00:29:36.911 --> 00:29:38.670 gets the least error, we just want 00:29:38.670 --> 00:29:41.360 to find the one that gets the fewest points wrong. 00:29:41.360 --> 00:29:42.920 Which one is that? 00:29:42.920 --> 00:29:45.020 That's our friend E, the first one that people 00:29:45.020 --> 00:29:46.650 realized was a good one. 00:29:46.650 --> 00:29:49.900 So we're going to choose classifier E. 00:29:49.900 --> 00:29:51.470 What's our error? 00:29:51.470 --> 00:29:53.940 It's just the sum of the ones we get wrong. 00:29:53.940 --> 00:29:56.660 So what's our error this time? 00:29:56.660 --> 00:29:58.050 It's 1/5. 00:29:58.050 --> 00:30:00.057 We got points 3 and 10 wrong. 00:30:00.057 --> 00:30:01.390 They both have a weight of 1/10. 00:30:01.390 --> 00:30:04.430 1/10 plus 1/10 is 1/5. 00:30:04.430 --> 00:30:08.070 So I'll put 1/5, and alpha. 00:30:08.070 --> 00:30:11.760 Alpha is sort of a vote that will be used at the very end 00:30:11.760 --> 00:30:14.480 to aggregate our classifier. 00:30:14.480 --> 00:30:19.090 Alpha is 1/2 natural log of 1 minus the error over the error. 00:30:19.090 --> 00:30:21.580 However, I have a little trick for you. 00:30:21.580 --> 00:30:24.650 It's not that impressive of a trick, but it's a little fun. 00:30:24.650 --> 00:30:30.100 So since error is 1/2-- sorry, not error-- alpha. 00:30:30.100 --> 00:30:33.020 Alpha is 1/2 natural log of 1 minus error over error. 00:30:37.260 --> 00:30:45.403 If error is 1/x, then alpha is 1/2 natural log of x minus 1. 00:30:48.061 --> 00:30:49.980 That just follows from the math. 00:30:49.980 --> 00:30:52.150 It's a little shortcut. 00:30:52.150 --> 00:30:55.490 If error is in the form of 1/x, then it's 00:30:55.490 --> 00:30:57.270 just 1/2 natural log of x minus 1, 00:30:57.270 --> 00:31:00.290 which means since this is in the form of 1/5, everyone, 00:31:00.290 --> 00:31:03.260 alpha is? 00:31:03.260 --> 00:31:04.550 1/2 ln 4. 00:31:12.570 --> 00:31:14.340 OK, 1/2 ln 4. 00:31:14.340 --> 00:31:17.820 So now we come to the part in boosting 00:31:17.820 --> 00:31:19.730 that many people consider the hardest part, 00:31:19.730 --> 00:31:22.090 and I'm going to show you how to do it more easily. 00:31:22.090 --> 00:31:24.060 This is that part where we try to make 00:31:24.060 --> 00:31:27.316 all the ones we got right, we change their weights to be 1/2. 00:31:27.316 --> 00:31:28.690 And all of the ones we got wrong, 00:31:28.690 --> 00:31:30.250 we change their weights to be 1/2. 00:31:30.250 --> 00:31:31.960 Here is my automated process. 00:31:31.960 --> 00:31:34.287 It's called the numerator stays the same method. 00:31:34.287 --> 00:31:35.120 Here's how it works. 00:31:42.460 --> 00:31:43.900 Here's our ten data points. 00:31:43.900 --> 00:31:46.940 Their current weight is 1/10, all of them. 00:31:55.410 --> 00:31:58.900 We're about to re-weight them for the next step. 00:31:58.900 --> 00:32:01.950 So you agree they're all 1/10? 00:32:01.950 --> 00:32:03.600 They're equal, to start off. 00:32:03.600 --> 00:32:08.585 So step one-- erase the denominators. 00:32:11.830 --> 00:32:12.620 Screw fractions. 00:32:12.620 --> 00:32:14.150 I don't like them. 00:32:14.150 --> 00:32:16.940 There's division, multiplication. 00:32:16.940 --> 00:32:17.530 It's a pain. 00:32:17.530 --> 00:32:19.910 I just want to add whole numbers. 00:32:19.910 --> 00:32:21.240 That's what we're going to do. 00:32:21.240 --> 00:32:24.150 So which ones do we get wrong? 00:32:24.150 --> 00:32:25.100 3 and 10. 00:32:25.100 --> 00:32:26.010 Circle those. 00:32:30.940 --> 00:32:33.240 All right. 00:32:33.240 --> 00:32:37.600 Add the numbers in the circles and multiply by 2. 00:32:37.600 --> 00:32:39.800 What does that give you? 00:32:39.800 --> 00:32:40.456 4. 00:32:40.456 --> 00:32:41.580 That's the new denominator. 00:32:44.210 --> 00:32:46.210 AUDIENCE: Do you always multiply by 2, or just-- 00:32:46.210 --> 00:32:48.900 PROFESSOR: You always multiply by 2. 00:32:48.900 --> 00:32:50.760 Add the numbers not in the circles. 00:32:50.760 --> 00:32:51.571 Multiply by 2. 00:32:51.571 --> 00:32:52.570 What does that give you? 00:32:52.570 --> 00:32:53.210 AUDIENCE: 16. 00:32:53.210 --> 00:32:54.270 PROFESSOR: 16. 00:32:54.270 --> 00:32:56.270 That's the new denominator. 00:32:56.270 --> 00:33:00.750 The final, crucial step so that we can do this again next round 00:33:00.750 --> 00:33:02.476 is by far the most mathematically 00:33:02.476 --> 00:33:03.850 complicated thing here because we 00:33:03.850 --> 00:33:06.190 have to actually do something with fractions, 00:33:06.190 --> 00:33:09.450 but it's not too bad-- is we then change everything 00:33:09.450 --> 00:33:11.460 to be with the same denominator. 00:33:11.460 --> 00:33:13.975 So the 1/4's become? 00:33:13.975 --> 00:33:14.600 AUDIENCE: 4/16. 00:33:14.600 --> 00:33:15.266 PROFESSOR: 4/16. 00:33:21.850 --> 00:33:22.770 All right. 00:33:22.770 --> 00:33:26.618 I can also uncircle these for next-- ah. 00:33:26.618 --> 00:33:30.610 I hit the that button. 00:33:30.610 --> 00:33:31.110 All right. 00:33:36.290 --> 00:33:49.072 New weights-- 1/16, 1/16, 4/16, 1/16, 1/16, 1/16, 1/16, 1/16, 00:33:49.072 --> 00:33:50.760 4/16. 00:33:50.760 --> 00:33:52.619 Note, the weights add up to 1. 00:33:52.619 --> 00:33:54.160 The ones you got wrong add up to 1/2. 00:33:54.160 --> 00:33:56.040 The ones you got right add up to 1/2. 00:33:56.040 --> 00:33:56.910 You're happy. 00:33:56.910 --> 00:34:01.700 So now that you get 4/16 of an error for getting 3 wrong, 00:34:01.700 --> 00:34:04.300 4/16 of an error for getting 10 wrong, 00:34:04.300 --> 00:34:05.536 take a look at these six. 00:34:05.536 --> 00:34:06.910 I'm not going to call on someone, 00:34:06.910 --> 00:34:09.690 just whoever's good at math and can 00:34:09.690 --> 00:34:13.400 add these up more quickly-- just 3 and 10 count as 4. 00:34:13.400 --> 00:34:14.530 All the others count as 1. 00:34:14.530 --> 00:34:15.554 Add them up. 00:34:15.554 --> 00:34:16.929 Tell me which one's the lightest. 00:34:16.929 --> 00:34:17.637 What did you say? 00:34:17.637 --> 00:34:18.650 AUDIENCE: I'd go with B. 00:34:18.650 --> 00:34:20.774 PROFESSOR: You'd go with B. It doesn't get 3 wrong. 00:34:20.774 --> 00:34:23.330 That sounds pretty good to me. 00:34:23.330 --> 00:34:25.889 Does everyone else like B as well? 00:34:25.889 --> 00:34:27.159 I like it. 00:34:27.159 --> 00:34:31.989 I mean, all of our ones that don't get 3 wrong or 10 wrong, 00:34:31.989 --> 00:34:36.909 we're only looking at B and D. And D has seven. 00:34:36.909 --> 00:34:38.750 B has four. 00:34:38.750 --> 00:34:41.360 So B is the best. 00:34:41.360 --> 00:34:42.800 B gets 4/16 wrong. 00:34:42.800 --> 00:34:44.540 Does everyone see that? 00:34:44.540 --> 00:34:47.920 Because even getting one of 3 or 10 wrong 00:34:47.920 --> 00:34:50.409 is as bad as all the ones that B gets wrong because 00:34:50.409 --> 00:34:52.540 of the new weights. 00:34:52.540 --> 00:34:53.270 So cool. 00:34:53.270 --> 00:34:55.540 Let's choose B. That's right. 00:34:55.540 --> 00:34:58.820 And I sort of gave it away. 00:34:58.820 --> 00:35:00.740 What's the error that B has? 00:35:00.740 --> 00:35:04.050 It has four wrong, each of which are worth 1/16. 00:35:04.050 --> 00:35:06.300 The error is? 00:35:06.300 --> 00:35:06.900 What? 00:35:06.900 --> 00:35:12.380 4/16, or 1/4, whichever is your favorite, which 00:35:12.380 --> 00:35:14.300 means that the alpha is? 00:35:14.300 --> 00:35:15.530 AUDIENCE: 1/2 ln 3. 00:35:15.530 --> 00:35:16.550 PROFESSOR: 1/2 ln 3. 00:35:21.023 --> 00:35:22.450 Bingo. 00:35:22.450 --> 00:35:24.190 Final round. 00:35:24.190 --> 00:35:26.010 OK, what did we get wrong? 00:35:26.010 --> 00:35:30.090 We got 1, 6, 7, and 9 wrong. 00:35:30.090 --> 00:35:35.405 Oh yeah, we can erase the denominators. 00:35:40.640 --> 00:35:41.970 All right. 00:35:41.970 --> 00:35:45.050 What are the numbers in the circles, summed up, 00:35:45.050 --> 00:35:46.511 multiplied by 2? 00:35:46.511 --> 00:35:47.010 AUDIENCE: 8. 00:35:47.010 --> 00:35:48.093 PROFESSOR: That's 8-- 1/8. 00:35:51.570 --> 00:35:54.800 And what about the numbers not in the circle, summed up, 00:35:54.800 --> 00:35:56.340 multiplied by 2? 00:35:56.340 --> 00:35:57.750 AUDIENCE: 24. 00:35:57.750 --> 00:35:59.660 PROFESSOR: That's right-- 24, which 00:35:59.660 --> 00:36:01.160 means I'm going to have to change 00:36:01.160 --> 00:36:06.960 all the numbers in the circle to 3/24, except I guess I don't 00:36:06.960 --> 00:36:08.754 because this is the last round. 00:36:08.754 --> 00:36:10.670 But if I was going to do another round-- let's 00:36:10.670 --> 00:36:13.510 prepare in case we where, change all of these to 3/24. 00:36:23.100 --> 00:36:25.127 Besides, it makes it easier to calculate 00:36:25.127 --> 00:36:27.710 which one is the best possible classifier because you can just 00:36:27.710 --> 00:36:30.820 use the numerators and sort of add them up. 00:36:30.820 --> 00:36:33.210 So while I'm writing that up, you guys 00:36:33.210 --> 00:36:36.370 figure out which one you like for classifier 00:36:36.370 --> 00:36:39.590 and call it out to me when I'm done. 00:36:39.590 --> 00:36:40.360 3/24. 00:36:40.360 --> 00:36:41.160 1/24. 00:36:41.160 --> 00:36:43.340 4/24. 00:36:43.340 --> 00:36:45.000 1/24. 00:36:45.000 --> 00:36:47.320 1/24. 00:36:47.320 --> 00:36:49.176 3/24. 00:36:49.176 --> 00:36:49.675 3/24. 00:36:52.350 --> 00:36:54.540 1/24. 00:36:54.540 --> 00:36:56.610 3/24-- wait. 00:36:56.610 --> 00:36:58.530 I'm off by one here. 00:36:58.530 --> 00:37:00.940 3, 1, 4-- 00:37:00.940 --> 00:37:03.730 AUDIENCE: It's because w1 is not assigned to anything. 00:37:03.730 --> 00:37:05.600 So w2 is really w1. 00:37:05.600 --> 00:37:06.710 PROFESSOR: Aha. 00:37:06.710 --> 00:37:08.730 You're right. 00:37:08.730 --> 00:37:19.176 w1 is not assigned to anything, so w2 is really w1 Yeah? 00:37:19.176 --> 00:37:22.902 AUDIENCE: There's an extra 1/16 between w1, w2. 00:37:22.902 --> 00:37:25.760 There's an extra 1/16. 00:37:25.760 --> 00:37:28.390 PROFESSOR: Yes, that's true. 00:37:28.390 --> 00:37:29.057 OK, well-- 00:37:29.057 --> 00:37:29.890 AUDIENCE: We get it. 00:37:29.890 --> 00:37:32.160 PROFESSOR: You get it. 00:37:32.160 --> 00:37:34.950 H, so what is the best H? 00:37:34.950 --> 00:37:38.550 You get it because it's right here. 00:37:38.550 --> 00:37:39.420 See? 00:37:39.420 --> 00:37:41.580 The process is so foolproof, even a fool 00:37:41.580 --> 00:37:45.060 like me can get it right while they have the chart wrong. 00:37:45.060 --> 00:37:47.220 All right, so what's the best classifier? 00:37:47.220 --> 00:37:48.300 AUDIENCE: C. 00:37:48.300 --> 00:37:52.382 PROFESSOR: You say C. I say that seems pretty reasonable. 00:37:52.382 --> 00:37:54.770 It only gets 3, 4, 5, and 8 wrong. 00:37:54.770 --> 00:37:56.460 Does anyone else get a different answer? 00:37:56.460 --> 00:37:57.510 AUDIENCE: A. 00:37:57.510 --> 00:38:01.950 PROFESSOR: Someone else gets A. I like A. Who said A? 00:38:01.950 --> 00:38:03.690 A lot of people said A. 00:38:03.690 --> 00:38:05.750 Well, let's figure it out. 00:38:05.750 --> 00:38:12.000 So A gets 1, 5, 6, 7. 00:38:12.000 --> 00:38:14.740 C gets 4, 5, 6, 7. 00:38:14.740 --> 00:38:16.900 They're, in fact, equal. 00:38:16.900 --> 00:38:19.500 Tie-break goes to the lower letter 00:38:19.500 --> 00:38:22.224 because that's what we said. 00:38:22.224 --> 00:38:24.390 In fact, I didn't tell you, but that's what we said. 00:38:24.390 --> 00:38:24.965 Question? 00:38:24.965 --> 00:38:27.090 AUDIENCE: So when we were deciding which classifier 00:38:27.090 --> 00:38:29.540 to use, can we only look at the weights, 00:38:29.540 --> 00:38:32.970 or do we also have to look at the [INAUDIBLE]-- 00:38:32.970 --> 00:38:35.160 PROFESSOR: Ignore all previous rounds. 00:38:35.160 --> 00:38:37.700 The question is, do you only look at the current weights 00:38:37.700 --> 00:38:39.460 when determining a classifier? 00:38:39.460 --> 00:38:42.530 Or do you look at the previous rounds as well? 00:38:42.530 --> 00:38:44.261 You have to ignore the previous rounds. 00:38:44.261 --> 00:38:44.760 Trust me. 00:38:44.760 --> 00:38:47.450 They will be used later in the vote. 00:38:47.450 --> 00:38:50.940 But it's sort of like tainting the jury a little bit 00:38:50.940 --> 00:38:54.250 to use the previous rounds when you're doing the current round. 00:38:54.250 --> 00:38:56.910 Because you want to start fresh with these new weights, 00:38:56.910 --> 00:38:57.930 get a new classifier. 00:38:57.930 --> 00:39:00.460 And then later, everyone will get to make their vote. 00:39:00.460 --> 00:39:02.750 So you only do it based on the current weights. 00:39:02.750 --> 00:39:04.458 AUDIENCE: We don't take any consideration 00:39:04.458 --> 00:39:06.309 if the last round's 6 was wrong or anything. 00:39:06.309 --> 00:39:08.850 PROFESSOR: Nope, although the weights take into consideration 00:39:08.850 --> 00:39:11.471 is when it's wrong, it's going to increase. 00:39:11.471 --> 00:39:12.425 AUDIENCE: OK. 00:39:12.425 --> 00:39:13.379 PROFESSOR: Question? 00:39:13.379 --> 00:39:14.754 AUDIENCE: Could you theoretically 00:39:14.754 --> 00:39:15.764 reuse a classifier? 00:39:15.764 --> 00:39:18.139 PROFESSOR: The question is, could you theoretically reuse 00:39:18.139 --> 00:39:18.895 a classifier? 00:39:18.895 --> 00:39:21.220 Answer-- you absolutely can. 00:39:21.220 --> 00:39:23.457 When that happens, it essentially gets extra weight 00:39:23.457 --> 00:39:24.540 because you used it again. 00:39:24.540 --> 00:39:29.680 But you can never, ever use it twice in a row. 00:39:29.680 --> 00:39:30.460 Here's why. 00:39:30.460 --> 00:39:33.150 Let's say that we want to use-- which was the one we used last 00:39:33.150 --> 00:39:33.650 over there? 00:39:33.650 --> 00:39:34.170 B? 00:39:34.170 --> 00:39:35.628 Let's say we wanted to use B again. 00:39:35.628 --> 00:39:37.780 What does it give us? 00:39:37.780 --> 00:39:39.890 50-50. 00:39:39.890 --> 00:39:43.750 If we wanted to use B and then B-- 3, 6, 9, 12 wrong. 00:39:43.750 --> 00:39:45.820 Always guaranteed to give you 50-50, 00:39:45.820 --> 00:39:48.420 which is the only way that you can 00:39:48.420 --> 00:39:50.030 be sure you'll never use it. 00:39:50.030 --> 00:39:51.880 In fact, that's by design. 00:39:51.880 --> 00:39:54.420 You could reuse it, but not twice in a row. 00:39:54.420 --> 00:39:57.100 It could be used later on down the stream. 00:39:57.100 --> 00:39:58.360 And it will be used. 00:39:58.360 --> 00:40:03.160 Because if you do seven rounds, one of them has to be reused. 00:40:03.160 --> 00:40:05.980 It just gives more weight to whichever one is reused. 00:40:05.980 --> 00:40:09.640 But yes, A wins against C. C was a perfectly good answer 00:40:09.640 --> 00:40:10.140 as well. 00:40:10.140 --> 00:40:10.640 Question? 00:40:10.640 --> 00:40:12.674 AUDIENCE: Wait, can you reuse [INAUDIBLE]? 00:40:12.674 --> 00:40:13.340 PROFESSOR: What? 00:40:13.340 --> 00:40:14.628 AUDIENCE: Instead of A or C. 00:40:14.628 --> 00:40:15.211 PROFESSOR: OK. 00:40:15.211 --> 00:40:18.520 If you could reuse, why doesn't he pick E? 00:40:18.520 --> 00:40:20.380 E gets eight out of 24 wrong. 00:40:20.380 --> 00:40:25.620 It's one worse than A and C. That's the only reason. 00:40:25.620 --> 00:40:27.760 Next step will probably use A-- or sorry. 00:40:27.760 --> 00:40:29.660 Well, next step, we'll probably use E, 00:40:29.660 --> 00:40:35.190 frankly-- although maybe not, because we got 3 wrong on A. 00:40:35.190 --> 00:40:37.060 But pretty soon, we would use E again 00:40:37.060 --> 00:40:38.670 because E's pretty awesome. 00:40:38.670 --> 00:40:43.310 But anyway, here we used A. And we said we got 7/24 wrong. 00:40:43.310 --> 00:40:45.220 Oh, man, we can't use my little shortcut. 00:40:45.220 --> 00:40:49.450 So the answer, it has to be 17/7-- or 1/2 natural log 00:40:49.450 --> 00:40:50.060 of 17/7. 00:40:55.060 --> 00:40:58.260 So there we go. 00:40:58.260 --> 00:41:02.870 Now, we have to ask, what is the final classifier that we 00:41:02.870 --> 00:41:04.330 created from all these things? 00:41:04.330 --> 00:41:09.680 All we do is we sum up all the classifiers we chose. 00:41:09.680 --> 00:41:12.100 And we multiply them times their weight, alpha. 00:41:12.100 --> 00:41:19.020 So 1/2 ln 4 times E, whether or not 00:41:19.020 --> 00:41:28.780 E returns true, plus 1/2 ln 3 times B 00:41:28.780 --> 00:41:36.370 plus 1/2 ln 17/7 times A, is our final classifier, where 00:41:36.370 --> 00:41:41.330 E returns a plus 1 if E thinks it's a vampire, and a minus 1 00:41:41.330 --> 00:41:42.920 if E think it's not. 00:41:42.920 --> 00:41:47.020 Same for B and A, all right? 00:41:47.020 --> 00:41:49.180 And then we take the sign of this. 00:41:49.180 --> 00:41:50.830 And I don't mean sine and cosine. 00:41:50.830 --> 00:41:54.640 I mean just, just is it positive or negative? 00:41:54.640 --> 00:41:56.190 OK? 00:41:56.190 --> 00:42:01.190 So the question now on the exam is, how many of the ten data 00:42:01.190 --> 00:42:04.720 points do you get right if we used this? 00:42:04.720 --> 00:42:05.906 Let's give it a look see. 00:42:05.906 --> 00:42:09.830 E is-- so we have romantic interest greater than 2. 00:42:09.830 --> 00:42:11.610 We have emo yes. 00:42:11.610 --> 00:42:13.140 And we have evil yes. 00:42:13.140 --> 00:42:18.760 So oh my gosh, logarithms, they're sometimes annoying. 00:42:18.760 --> 00:42:20.285 Do we have to actually add them up? 00:42:20.285 --> 00:42:22.290 I claim we don't. 00:42:22.290 --> 00:42:24.010 Here's a nice special case of having 00:42:24.010 --> 00:42:26.420 three logarithms on the board. 00:42:26.420 --> 00:42:28.040 One of two things is true. 00:42:28.040 --> 00:42:30.090 Either one of those three logarithms 00:42:30.090 --> 00:42:33.900 is so large that it's bigger than the other two 00:42:33.900 --> 00:42:38.930 combined, in which case, if that one returns a positive or a 00:42:38.930 --> 00:42:42.830 negative, it's just positive or negative because that one's 00:42:42.830 --> 00:42:44.340 big. 00:42:44.340 --> 00:42:48.370 Or one is not that large, and in which 00:42:48.370 --> 00:42:51.180 case, any two can dominate the other one, 00:42:51.180 --> 00:42:53.710 and so is just equivalent to a majority vote. 00:42:53.710 --> 00:42:57.070 So I claim we never have to add them when there's only three. 00:42:57.070 --> 00:42:58.122 You guys see what I mean? 00:42:58.122 --> 00:43:00.330 Like, let's say one of them was 1/2 log of a billion, 00:43:00.330 --> 00:43:03.320 and the others were 1/2 log of 3 and 1/2 log of 4. 00:43:03.320 --> 00:43:05.500 Obviously, whatever the 1/2 log of a billion 00:43:05.500 --> 00:43:08.570 says, which is multiplied by 1/2 log of a billion, 00:43:08.570 --> 00:43:11.834 is it's just going to be that, and the others will be ignored. 00:43:11.834 --> 00:43:13.750 However, if it's not the case that one of them 00:43:13.750 --> 00:43:15.630 is larger than the other two combined, 00:43:15.630 --> 00:43:17.690 then it's a simple vote between the three, 00:43:17.690 --> 00:43:20.320 because any two can out-vote the other one 00:43:20.320 --> 00:43:21.820 if they work together. 00:43:21.820 --> 00:43:28.120 And in this case, let's see, 17/7 is not quite 3. 00:43:28.120 --> 00:43:32.160 However, log of 4 is certainly not 00:43:32.160 --> 00:43:34.900 better than log of 3 plus log of 17/7. 00:43:34.900 --> 00:43:36.820 It's not even-- log of 4 is equal to log 00:43:36.820 --> 00:43:38.450 of 2 plus log of 2. 00:43:38.450 --> 00:43:40.175 And these are both bigger than log of 2. 00:43:43.270 --> 00:43:46.930 That's rules of logs-- log of 4 equals log of 2 squared, 00:43:46.930 --> 00:43:48.260 and you can take the 2 out. 00:43:48.260 --> 00:43:51.357 So these are not big enough that one of them 00:43:51.357 --> 00:43:52.940 is bigger than the other two combined. 00:43:52.940 --> 00:43:54.630 So it's just going to be a simple vote. 00:43:54.630 --> 00:43:55.800 So let's go through. 00:43:55.800 --> 00:44:00.750 Dracula-- OK, he's got tons of his little vampyrettes. 00:44:00.750 --> 00:44:04.670 He's not emo, so E gets it right. 00:44:04.670 --> 00:44:05.770 He's not emo. 00:44:05.770 --> 00:44:06.950 So that gets it wrong. 00:44:06.950 --> 00:44:08.190 But he is evil. 00:44:08.190 --> 00:44:09.070 That gets it right. 00:44:09.070 --> 00:44:12.780 Two out of three vote that he's a vampire-- correct. 00:44:12.780 --> 00:44:13.660 Next. 00:44:13.660 --> 00:44:17.640 Angel-- OK, well, he was in a long running series. 00:44:17.640 --> 00:44:19.560 He's got plenty of romantic interests. 00:44:19.560 --> 00:44:22.140 So that gets it right. 00:44:22.140 --> 00:44:23.230 He's certainly emo. 00:44:23.230 --> 00:44:24.072 That gets it right. 00:44:24.072 --> 00:44:26.030 And even though he's not evil, two out of three 00:44:26.030 --> 00:44:28.850 says he's a vampire, so correct. 00:44:28.850 --> 00:44:34.010 Next, Edward Cullen-- well, Twilight, here we come. 00:44:34.010 --> 00:44:34.584 Let's see. 00:44:34.584 --> 00:44:36.000 He only has one romantic interest, 00:44:36.000 --> 00:44:37.610 so that gets it wrong. 00:44:37.610 --> 00:44:38.610 OK. 00:44:38.610 --> 00:44:40.410 He's emo, so that gets it right. 00:44:40.410 --> 00:44:42.150 But he's not evil, so two wrong. 00:44:42.150 --> 00:44:44.210 So Edward's not a vampire according 00:44:44.210 --> 00:44:45.820 to our final classifier. 00:44:45.820 --> 00:44:46.880 But he is. 00:44:46.880 --> 00:44:49.620 So we got one of the data points wrong. 00:44:49.620 --> 00:44:50.380 You guys see that? 00:44:50.380 --> 00:44:52.780 Because two out of three of our classifiers 00:44:52.780 --> 00:44:56.405 here said that he was not a vampire. 00:44:56.405 --> 00:44:57.280 All right, let's see. 00:44:57.280 --> 00:45:02.410 Saya-- well, she has more than two romantic interests. 00:45:02.410 --> 00:45:03.610 And she's emo. 00:45:03.610 --> 00:45:06.600 So even though she's not evil, we get it right. 00:45:06.600 --> 00:45:07.290 OK? 00:45:07.290 --> 00:45:08.260 Let's see. 00:45:08.260 --> 00:45:14.950 Lestat-- he also has may love interests, 00:45:14.950 --> 00:45:16.580 is emo, and is not evil. 00:45:16.580 --> 00:45:18.640 So you get it right. 00:45:18.640 --> 00:45:23.140 OK, Bianca is evil with many love interests. 00:45:23.140 --> 00:45:26.620 Even though she's not emo, two out of three, you get it right. 00:45:26.620 --> 00:45:29.670 All right, Carmilla-- I'm going to call her Karnstein-- 00:45:29.670 --> 00:45:31.944 is basically exactly the same as Bianca 00:45:31.944 --> 00:45:34.360 with the number of romantic interests fixed the way it is. 00:45:34.360 --> 00:45:37.460 So she will always do the same thing that Bianca does. 00:45:37.460 --> 00:45:39.790 It's why 6 and 7 always travel together. 00:45:39.790 --> 00:45:41.150 So we get it right. 00:45:41.150 --> 00:45:44.710 Sailor Moon is supposed to be not a vampire. 00:45:44.710 --> 00:45:47.320 So her number of love interests say that she's not a vampire 00:45:47.320 --> 00:45:49.040 because she only has one. 00:45:49.040 --> 00:45:51.150 The fact that she's not evil and not emo 00:45:51.150 --> 00:45:53.550 says that actually, she's perfectly not a vampire. 00:45:53.550 --> 00:45:54.810 They all agree. 00:45:54.810 --> 00:45:56.400 And that's correct. 00:45:56.400 --> 00:45:59.560 Squall has only one love interest, Rinoa. 00:45:59.560 --> 00:46:03.020 And he's not evil, both of which of say he's not a vampire. 00:46:03.020 --> 00:46:03.710 But he is emo. 00:46:03.710 --> 00:46:05.585 But two out of three says he's not a vampire. 00:46:05.585 --> 00:46:06.610 We get it correct. 00:46:06.610 --> 00:46:09.190 And Circe, despite her many romantic interests 00:46:09.190 --> 00:46:10.750 which says she might be a vampire, 00:46:10.750 --> 00:46:13.870 is neither evil nor emo, and is not a vampire. 00:46:13.870 --> 00:46:16.085 So we got everything right except Edward Cullen, 00:46:16.085 --> 00:46:19.190 which perhaps says more about Stephenie Meyers writing 00:46:19.190 --> 00:46:21.770 than about our boosting. 00:46:21.770 --> 00:46:25.130 All right, final question-- Wesley Wyndham, 00:46:25.130 --> 00:46:28.200 a fellow consultant, has noticed a few correlations between some 00:46:28.200 --> 00:46:29.780 of the classifiers you used. 00:46:29.780 --> 00:46:33.270 He suggests using a new set of weak classifiers consisting 00:46:33.270 --> 00:46:40.280 of a pair of your classifiers that are logically 00:46:40.280 --> 00:46:41.744 anded and ored together. 00:46:41.744 --> 00:46:43.410 For instance, two of the new classifiers 00:46:43.410 --> 00:46:48.960 would be emo equals yes or evil equals yes, or sparkly equals 00:46:48.960 --> 00:46:51.470 no and transforms equals yes. 00:46:51.470 --> 00:46:55.230 So that would cut out Sailor Moon from the transforms cloud. 00:46:55.230 --> 00:46:58.000 He believes that you'll be able to classify large vampire data 00:46:58.000 --> 00:46:59.980 sets-- larger than this one, anyway-- 00:46:59.980 --> 00:47:03.160 more quickly with fewer rounds of boosting using his system. 00:47:03.160 --> 00:47:04.710 Do you agree or disagree with Wesley? 00:47:04.710 --> 00:47:06.560 Explain your arguments. 00:47:06.560 --> 00:47:09.890 So this was, you know, the tough concept question. 00:47:09.890 --> 00:47:12.037 Does anyone have just an instinctual thing other 00:47:12.037 --> 00:47:13.370 than like, oh, man, it's Wesley. 00:47:13.370 --> 00:47:14.440 He must be wrong. 00:47:18.851 --> 00:47:20.601 AUDIENCE: You'll probably use fewer rounds 00:47:20.601 --> 00:47:23.487 of boosting because you have more classifiers. 00:47:23.487 --> 00:47:26.800 But you'll have to search through more classifiers. 00:47:26.800 --> 00:47:30.360 PROFESSOR: Aha, that is the rare full-point answer. 00:47:30.360 --> 00:47:33.420 Very few people realize that Wesley was partially right. 00:47:33.420 --> 00:47:35.920 They either said something about him being completely wrong, 00:47:35.920 --> 00:47:38.220 which was wrong, or said that he was completely right. 00:47:38.220 --> 00:47:41.740 Yes, it will use fewer rounds of boosting because of the fact 00:47:41.740 --> 00:47:45.970 that you can-- essentially, one of these boosting already 00:47:45.970 --> 00:47:49.410 does is sort of gets things to vote together 00:47:49.410 --> 00:47:50.860 in an [INAUDIBLE] fashion. 00:47:50.860 --> 00:47:52.662 So it will use approximately half 00:47:52.662 --> 00:47:54.370 the number of rounds of boosting by being 00:47:54.370 --> 00:47:55.770 able to combine into two. 00:47:55.770 --> 00:47:58.300 But there's a lot of ands and ors. 00:47:58.300 --> 00:47:59.830 There's in fact n choose 2, where 00:47:59.830 --> 00:48:01.620 n is the number of vampires. 00:48:01.620 --> 00:48:04.340 And since using half the number of rounds but taking n 00:48:04.340 --> 00:48:08.040 choose 2 time for each round is not less time. 00:48:08.040 --> 00:48:10.200 So that's exactly correct. 00:48:10.200 --> 00:48:13.520 Not that many people got full credit on that one 00:48:13.520 --> 00:48:16.201 because sometimes, they were seduced by Wesley's idea. 00:48:16.201 --> 00:48:17.700 Or they were just like, it's Wesley. 00:48:17.700 --> 00:48:23.080 He's wrong-- or just some other funny answer. 00:48:23.080 --> 00:48:25.080 Any questions about boosting? 00:48:25.080 --> 00:48:25.580 Question? 00:48:25.580 --> 00:48:27.848 AUDIENCE: How do you know how many rounds of boosting 00:48:27.848 --> 00:48:28.677 to take? 00:48:28.677 --> 00:48:30.260 PROFESSOR: The question is, how do you 00:48:30.260 --> 00:48:32.690 know how many rounds of boosting to do? 00:48:32.690 --> 00:48:36.140 The answer is-- so on the quiz, it 00:48:36.140 --> 00:48:38.070 tells you that you have three. 00:48:38.070 --> 00:48:42.830 In real life, you might want to just kind of keep 00:48:42.830 --> 00:48:46.100 it running until it converges. 00:48:46.100 --> 00:48:48.330 That's one possibility-- keep it running 00:48:48.330 --> 00:48:51.204 until it converges to an answer, and it 00:48:51.204 --> 00:48:52.370 doesn't do anything anymore. 00:48:52.370 --> 00:48:56.592 Patrick has a little widget on the 6.034 website, I think, 00:48:56.592 --> 00:48:58.300 that lets you plunk down some data points 00:48:58.300 --> 00:48:59.710 and run boosting on them. 00:48:59.710 --> 00:49:03.640 And you can see that eventually, it converges. 00:49:03.640 --> 00:49:06.932 The boosting converges to an answer, and it doesn't change. 00:49:06.932 --> 00:49:09.524 AUDIENCE: What converges? 00:49:09.524 --> 00:49:12.390 PROFESSOR: Basically, the-- not the classifiers 00:49:12.390 --> 00:49:14.050 you picked, of course, or the weights. 00:49:14.050 --> 00:49:17.100 But what converges is which ones of your data set 00:49:17.100 --> 00:49:18.470 you get correct. 00:49:18.470 --> 00:49:21.720 Because he does his in two-dimensional space 00:49:21.720 --> 00:49:23.114 rather than like this. 00:49:23.114 --> 00:49:24.780 And he shows you the lines that boosting 00:49:24.780 --> 00:49:26.196 is drawing between classification, 00:49:26.196 --> 00:49:29.320 and colors things in green and red or something like that. 00:49:29.320 --> 00:49:31.604 And eventually, it converges where the lines are 00:49:31.604 --> 00:49:33.020 and which ones it's getting right. 00:49:33.020 --> 00:49:36.470 It generally converges to getting everything correct. 00:49:36.470 --> 00:49:40.030 And when that happens, then you can stop. 00:49:40.030 --> 00:49:41.240 But that's a good question. 00:49:41.240 --> 00:49:43.830 And it's not always that easy in the real world. 00:49:43.830 --> 00:49:47.330 You have to sometimes just say, this is enough for me. 00:49:47.330 --> 00:49:50.230 I've given it n number of rounds. 00:49:50.230 --> 00:49:52.320 And that's much more the number of classifiers, 00:49:52.320 --> 00:49:55.300 so maybe it won't get anything better.