WEBVTT 00:00:00.040 --> 00:00:02.460 The following content is provided under a Creative 00:00:02.460 --> 00:00:03.970 Commons license. 00:00:03.970 --> 00:00:06.910 Your support will help MIT OpenCourseWare continue to 00:00:06.910 --> 00:00:08.700 offer high quality, educational 00:00:08.700 --> 00:00:10.660 resources for free. 00:00:10.660 --> 00:00:13.460 To make a donation or view additional materials from 00:00:13.460 --> 00:00:17.390 hundreds of MIT courses, visit MIT OpenCourseWare at 00:00:17.390 --> 00:00:18.640 ocw.mit.edu. 00:00:25.660 --> 00:00:30.320 PROFESSOR: So today, we're showing just a very short 00:00:30.320 --> 00:00:32.760 video I just wanted you to hear. 00:00:32.760 --> 00:00:35.350 And we're going to be talking about Pratham today. 00:00:35.350 --> 00:00:40.280 We've had a very long relationship with Pratham over 00:00:40.280 --> 00:00:42.320 the years, we being J-PAL. 00:00:42.320 --> 00:00:45.140 That is me and Professor Banerjee and 00:00:45.140 --> 00:00:47.090 several graduate students. 00:00:47.090 --> 00:00:51.800 And one of the leaders of is this person, Rukmini Banerji 00:00:51.800 --> 00:00:54.440 And it's sort of useful to hear this. 00:00:54.440 --> 00:00:56.920 It's a very short segment where she explains a little 00:00:56.920 --> 00:01:02.690 bit what she sees the education problem being about. 00:01:02.690 --> 00:01:05.000 And then we'll take it from there. 00:01:05.000 --> 00:01:12.856 [VIDEO WITHHELD] 00:01:12.856 --> 00:01:18.620 PROFESSOR: So I kind of wanted to take one lecture to go in 00:01:18.620 --> 00:01:22.560 detail to all of our work with Pratham, not only because I 00:01:22.560 --> 00:01:27.890 think Pratham is an amazing organization, but also because 00:01:27.890 --> 00:01:32.690 it will expose you to how we learn about the problem, how 00:01:32.690 --> 00:01:35.800 we build from one evolution to the next, one project to the 00:01:35.800 --> 00:01:39.740 next, how one project is bringing some more questions 00:01:39.740 --> 00:01:42.430 then we started with, and how the next project tries to get 00:01:42.430 --> 00:01:46.250 at these questions, also how the partnership with the local 00:01:46.250 --> 00:01:49.370 partners develops and works and all that. 00:01:49.370 --> 00:01:52.080 And to this, we are going to be talking about this quality 00:01:52.080 --> 00:01:55.800 problem, where we are trying to address the basic point 00:01:55.800 --> 00:01:57.330 that she makes. 00:01:57.330 --> 00:01:58.580 Kids are in school. 00:02:01.210 --> 00:02:03.780 At the moment, she talked at the Clinton Global initiative, 00:02:03.780 --> 00:02:05.000 that was 90%. 00:02:05.000 --> 00:02:06.720 But the enrollment rates are even higher 00:02:06.720 --> 00:02:09.669 now, close to 100%. 00:02:09.669 --> 00:02:14.110 In some districts, it's more than 100%, because some of the 00:02:14.110 --> 00:02:16.370 five-year-olds also go to school, because that's kind of 00:02:16.370 --> 00:02:21.090 a convenient way to get babysitting. 00:02:21.090 --> 00:02:24.520 But despite that, the achievement is low. 00:02:24.520 --> 00:02:28.530 She did point out the results from the ASER survey. 00:02:28.530 --> 00:02:33.720 So every year, since 2005, Pratham puts together teams of 00:02:33.720 --> 00:02:36.750 people, local people, who go from local 00:02:36.750 --> 00:02:38.730 universities, et cetera. 00:02:38.730 --> 00:02:42.950 And they go in every single district in India. 00:02:42.950 --> 00:02:46.690 How many districts did she say there was in India? 00:02:46.690 --> 00:02:48.470 She mentioned it at some point. 00:02:48.470 --> 00:02:50.040 You see that everything [INAUDIBLE] 00:02:50.040 --> 00:02:52.520 if you are listening to the video. 00:02:52.520 --> 00:02:56.450 She said it at some point, how many district they work in and 00:02:56.450 --> 00:02:57.700 how many districts they have. 00:03:04.300 --> 00:03:07.170 If I'm not mistaken, she said something like 600. 00:03:07.170 --> 00:03:10.430 So they are sending people to every district in India, teams 00:03:10.430 --> 00:03:12.800 of college students, local volunteers, et cetera. 00:03:12.800 --> 00:03:13.520 They have a sampling. 00:03:13.520 --> 00:03:15.350 Of course they don't go to every village. 00:03:15.350 --> 00:03:17.740 But they do a sample in each district. 00:03:17.740 --> 00:03:21.910 And they administer a very simple test of reading skills 00:03:21.910 --> 00:03:23.520 and math skills. 00:03:23.520 --> 00:03:27.950 So the reading skill test looks like a page. 00:03:27.950 --> 00:03:33.030 On the left side, you have a very simple story. 00:03:33.030 --> 00:03:34.380 Like she said, I go to school. 00:03:34.380 --> 00:03:35.450 My brother goes to school. 00:03:35.450 --> 00:03:37.540 We like going to school. 00:03:37.540 --> 00:03:41.250 On the right side, you will have single words and then 00:03:41.250 --> 00:03:43.300 single letters. 00:03:43.300 --> 00:03:46.890 First, you start the child with a very simple story. 00:03:46.890 --> 00:03:50.350 If they can do that, you bring them to a slightly longer 00:03:50.350 --> 00:03:52.590 story and try to measure their comprehension. 00:03:52.590 --> 00:03:55.160 If they come to the very simple story, you ask them to 00:03:55.160 --> 00:03:56.160 read a word. 00:03:56.160 --> 00:03:59.220 If they can't read a word, you ask them to read a letter. 00:03:59.220 --> 00:04:03.710 So you, in this way, classify your child as a nothing 00:04:03.710 --> 00:04:08.310 reader, a letter reader, a word reader, a small paragraph 00:04:08.310 --> 00:04:11.300 reader, or a story reader. 00:04:11.300 --> 00:04:14.080 And the simple, small paragraph reader is what you 00:04:14.080 --> 00:04:19.470 should be able to do at the end of the first grade. 00:04:19.470 --> 00:04:24.020 And what they find is, on average, in 2005, that about 00:04:24.020 --> 00:04:30.240 35,000 children, aged 7 to 14, most of them are in school and 00:04:30.240 --> 00:04:32.990 could not read a grade 1 paragraph and 60% could not 00:04:32.990 --> 00:04:35.690 read a grade 2 story. 00:04:35.690 --> 00:04:39.330 So these are numbers on one of the ASER survey. 00:04:39.330 --> 00:04:43.520 So blue is better in the maps I'm going to show you. 00:04:43.520 --> 00:04:47.160 And this is the fraction of kids who are out of school. 00:04:47.160 --> 00:04:51.020 And so that's quite a blue map, because there are not 00:04:51.020 --> 00:04:54.000 very many kids who are out of school. 00:04:54.000 --> 00:04:56.210 And the colors here are the same. 00:04:56.210 --> 00:04:58.590 This is now the reading ability. 00:04:58.590 --> 00:05:01.350 This is the fraction of kids who can read a 00:05:01.350 --> 00:05:06.050 standard one text. 00:05:06.050 --> 00:05:08.460 So it is the fraction of kids who can read or understand one 00:05:08.460 --> 00:05:10.880 text among the kids who are enrolled in 00:05:10.880 --> 00:05:13.110 standard three to five. 00:05:13.110 --> 00:05:15.500 So among children who are enrolled in standard three to 00:05:15.500 --> 00:05:19.100 five, how many of them can read this grade one paragraph? 00:05:19.100 --> 00:05:20.720 And you can see that now we have a lot 00:05:20.720 --> 00:05:24.150 of red in this card. 00:05:24.150 --> 00:05:26.770 The red is a bit distributed in different places, but 00:05:26.770 --> 00:05:30.790 mostly it's quite red. 00:05:30.790 --> 00:05:35.430 If you know India, does this map look like what you would 00:05:35.430 --> 00:05:38.995 expect for any social indicator within India? 00:05:50.040 --> 00:05:52.900 For anyone who knew India or about India, because this is a 00:05:52.900 --> 00:05:54.600 question that you might not know the answer to. 00:05:57.260 --> 00:06:02.005 So it doesn't really, because the folklore in India-- and if 00:06:02.005 --> 00:06:05.360 you look at immunization, that's really what you get-- 00:06:05.360 --> 00:06:12.370 is that sort of the bad states are these ones, Uttar Pradesh, 00:06:12.370 --> 00:06:16.900 Bihar, Rajasthan, Jharkhand. 00:06:16.900 --> 00:06:20.180 These are what they call the BIMARU state, Bihar, 00:06:20.180 --> 00:06:21.970 Rajasthan, Uttar Pradesh. 00:06:21.970 --> 00:06:28.420 This is like where you have a lot of boys and not very many 00:06:28.420 --> 00:06:32.610 girls, suggesting infanticide or selective abortion. 00:06:32.610 --> 00:06:36.870 This is where you have very low immunization rate, bad 00:06:36.870 --> 00:06:38.860 outcomes, of course, as I mentioned. 00:06:38.860 --> 00:06:41.670 And these are the good states, Andhra Pradesh, Karnataka, 00:06:41.670 --> 00:06:44.020 Tamil Nadu, Kerala. 00:06:44.020 --> 00:06:47.920 Kerala is, in particular, one of Amartya Sen and John Rawls' 00:06:47.920 --> 00:06:51.640 favorite states for its successful education. 00:06:51.640 --> 00:06:54.870 Kerala does well here, almost everyone can read. 00:06:54.870 --> 00:06:56.920 But what is surprising is you have Tamil Nadu. 00:06:56.920 --> 00:06:59.780 They're very proud of themselves. 00:06:59.780 --> 00:07:01.740 A lot of the kids are in school, et cetera. 00:07:01.740 --> 00:07:02.970 It's very red. 00:07:02.970 --> 00:07:05.363 Less than half the kids enrolled in [INAUDIBLE] three 00:07:05.363 --> 00:07:07.630 to five cannot read a sentence. 00:07:07.630 --> 00:07:10.950 So that upset the government of Tamil Nadu a great deal. 00:07:10.950 --> 00:07:13.910 So when the first ASER report came out, they didn't believe 00:07:13.910 --> 00:07:14.290 the result. 00:07:14.290 --> 00:07:15.350 That is not possible. 00:07:15.350 --> 00:07:18.000 They went ahead and did their own test, but they found the 00:07:18.000 --> 00:07:19.430 same thing again. 00:07:19.430 --> 00:07:21.640 So this suggests that there is something different. 00:07:21.640 --> 00:07:25.230 It's not only like disorganized places that are 00:07:25.230 --> 00:07:28.350 unable to deliver to their kids. 00:07:28.350 --> 00:07:30.900 Their is something maybe deeper than that. 00:07:30.900 --> 00:07:34.220 And Bihar did surprisingly well in not being at the 00:07:34.220 --> 00:07:36.760 absolute bottom of the barrel in this graph. 00:07:36.760 --> 00:07:38.010 So Bihar was happy. 00:07:40.760 --> 00:07:44.030 Another thing that's is maybe even more troubling is this 00:07:44.030 --> 00:07:48.225 survey has been done from 2005 to 2010, every year. 00:07:48.225 --> 00:07:52.680 It's been released in January every year, on the Republic 00:07:52.680 --> 00:07:54.310 Day in India. 00:07:54.310 --> 00:07:56.660 And there is no progress. 00:07:56.660 --> 00:08:00.180 So India is progressing across all these dimensions, becoming 00:08:00.180 --> 00:08:01.860 richer, there are fewer poor people. 00:08:01.860 --> 00:08:05.430 We're seeing they're eating less and less, which is not 00:08:05.430 --> 00:08:06.430 progress, per se. 00:08:06.430 --> 00:08:10.110 But education, nothing, no progress. 00:08:10.110 --> 00:08:14.920 Also troubling is it's not an Indian exception. 00:08:14.920 --> 00:08:18.250 Surveys very similar to ASER have been done in Kenya and 00:08:18.250 --> 00:08:21.170 Uganda and Tanzania and in Pakistan. 00:08:21.170 --> 00:08:24.750 And they all find similar kind of things. 00:08:24.750 --> 00:08:27.015 A lot of children are in school, most of them can't 00:08:27.015 --> 00:08:27.856 read anything. 00:08:27.856 --> 00:08:28.332 Yep? 00:08:28.332 --> 00:08:31.664 AUDIENCE: Is there still progress across the board on 00:08:31.664 --> 00:08:33.568 average, like in the nation? 00:08:33.568 --> 00:08:37.082 Or the states are static as well? 00:08:37.082 --> 00:08:38.010 PROFESSOR: So you're exactly right. 00:08:38.010 --> 00:08:39.200 That's a very good question. 00:08:39.200 --> 00:08:41.130 It's no progress across the board in the nation, but the 00:08:41.130 --> 00:08:43.510 states are not static. 00:08:43.510 --> 00:08:47.670 Some states start doing well for a while, and means some 00:08:47.670 --> 00:08:50.210 states are moving down to compensate. 00:08:50.210 --> 00:08:54.150 So, for example, in the last report, 2010, 00:08:54.150 --> 00:08:58.490 Punjab had a big progress. 00:08:58.490 --> 00:09:00.650 So that's one of the things they are trying to look at is 00:09:00.650 --> 00:09:02.330 why is Punjab suddenly progressing? 00:09:02.330 --> 00:09:04.350 What are they doing that is different? 00:09:04.350 --> 00:09:07.190 And I'm going to be able to talk a bit about that. 00:09:07.190 --> 00:09:10.740 And in fact, in the Punjab example, what they seem to be 00:09:10.740 --> 00:09:13.680 doing that is different is really to try to insist on 00:09:13.680 --> 00:09:17.520 these core competencies in the schools. 00:09:17.520 --> 00:09:21.830 They have a period of time, every school day, two hours a 00:09:21.830 --> 00:09:26.290 day, which are devoted to teaching the kids at whatever 00:09:26.290 --> 00:09:29.190 level they are forgetting the curriculum. 00:09:29.190 --> 00:09:32.990 And that is what they think is the reason why Punjab is 00:09:32.990 --> 00:09:33.830 progressing. 00:09:33.830 --> 00:09:36.760 So what we are going to do today is-- 00:09:36.760 --> 00:09:39.840 do we have evidence to suggest that, in fact, they're right? 00:09:39.840 --> 00:09:43.400 And that is the key problem and, therefore, the key of the 00:09:43.400 --> 00:09:44.840 success of Punjab. 00:09:44.840 --> 00:09:46.910 Meanwhile, some other state, like Uttar Pradesh, would be 00:09:46.910 --> 00:09:48.940 going down. 00:09:48.940 --> 00:09:49.245 Ben? 00:09:49.245 --> 00:09:51.220 AUDIENCE: I have a question. 00:09:51.220 --> 00:09:53.754 Is the student to teacher ratio more or less? 00:09:53.754 --> 00:09:55.158 I know these states are huge. 00:09:55.158 --> 00:09:57.030 But are they more or less the same across 00:09:57.030 --> 00:09:57.970 the different states? 00:09:57.970 --> 00:09:59.590 PROFESSOR: So it's a very good question. 00:09:59.590 --> 00:10:02.080 I think, they are, probably, generally 00:10:02.080 --> 00:10:03.160 more or less the same. 00:10:03.160 --> 00:10:04.480 And they are not horrible. 00:10:04.480 --> 00:10:07.540 Because, actually, being a teacher is not a bad job. 00:10:07.540 --> 00:10:10.140 So there are a number of teachers who are available. 00:10:10.140 --> 00:10:13.080 So you don't have the very, very, large class size that 00:10:13.080 --> 00:10:15.920 you have, for example, in a country like Kenya, where, 00:10:15.920 --> 00:10:19.180 after free primary education, the class size in grade one 00:10:19.180 --> 00:10:24.190 was easily 70 to 80 children per classroom. 00:10:24.190 --> 00:10:25.938 You have less of that in India. 00:10:25.938 --> 00:10:26.416 Yeah? 00:10:26.416 --> 00:10:29.523 AUDIENCE: So it seems like the enrollment is high, but do 00:10:29.523 --> 00:10:32.485 they also track that the attendance is high? 00:10:32.485 --> 00:10:33.675 PROFESSOR: So that's a very good question. 00:10:33.675 --> 00:10:36.300 And the answer is that, yes, they check attendance. 00:10:36.300 --> 00:10:38.340 And attendance is low. 00:10:38.340 --> 00:10:43.980 So attendance is low, both for teachers and for children. 00:10:43.980 --> 00:10:47.140 So in Bihar, for example, we have a survey of children's 00:10:47.140 --> 00:10:48.390 attendance. 00:10:49.950 --> 00:10:55.510 The rate of absence is between 30 to 50% for kids. 00:10:55.510 --> 00:10:57.150 And we've done the same thing in several 00:10:57.150 --> 00:10:58.050 countries of the world. 00:10:58.050 --> 00:11:01.260 Generally, children's absence is very, very high. 00:11:01.260 --> 00:11:04.050 So kids are enrolled, but they don't show up. 00:11:04.050 --> 00:11:07.240 And it's not that they are ghost enrollees. 00:11:07.240 --> 00:11:11.500 This 30% attendance is not concentrated across 30% of 00:11:11.500 --> 00:11:13.570 kids who never come. 00:11:13.570 --> 00:11:17.490 It's every child who is absent one day, coming one day, 00:11:17.490 --> 00:11:19.500 absent on day, coming one day. 00:11:19.500 --> 00:11:21.930 And that is one problem. 00:11:21.930 --> 00:11:24.230 So we have a lot of absence of children. 00:11:24.230 --> 00:11:28.160 And we have a lot of absence of teachers as well. 00:11:28.160 --> 00:11:32.680 So a survey was done in four to five developing countries, 00:11:32.680 --> 00:11:34.450 by the World Bank, including India. 00:11:34.450 --> 00:11:38.610 They found a very large teacher absence rate. 00:11:38.610 --> 00:11:42.110 So in India, it was about 25% absence rate. 00:11:42.110 --> 00:11:45.670 And that's about what ASER found as well, of teachers. 00:11:45.670 --> 00:11:48.980 So you show up to a school during school hours, and you 00:11:48.980 --> 00:11:50.160 have three of four teachers who are 00:11:50.160 --> 00:11:52.090 actually in the school. 00:11:52.090 --> 00:11:54.680 The second problem is that once they are in the school, 00:11:54.680 --> 00:11:57.450 it is not a given that they are actually teaching. 00:11:57.450 --> 00:12:01.310 So another one out of these four teachers is actually in 00:12:01.310 --> 00:12:03.110 school but not teaching. 00:12:03.110 --> 00:12:06.870 There was a funny report, a few years ago, describing what 00:12:06.870 --> 00:12:09.890 they were doing, ranging from drinking tea to drawing 00:12:09.890 --> 00:12:12.040 political posters. 00:12:12.040 --> 00:12:14.890 The reason being that a lot of these teaching jobs are 00:12:14.890 --> 00:12:17.310 patronage jobs, which are filled by the political 00:12:17.310 --> 00:12:20.580 parties, so they need to draw their posters sometimes. 00:12:20.580 --> 00:12:23.830 So that means that the teachers are present teaching 00:12:23.830 --> 00:12:25.030 half the time. 00:12:25.030 --> 00:12:27.850 The students are, themselves, present about half the time. 00:12:27.850 --> 00:12:31.250 That means the students get about a quarter of the time 00:12:31.250 --> 00:12:33.080 that they should get in front of a teacher. 00:12:33.080 --> 00:12:34.940 This is a where, [? Hookman's ?] point, that 00:12:34.940 --> 00:12:37.530 you need a bit of time of student-teacher interaction to 00:12:37.530 --> 00:12:38.005 get teaching. 00:12:38.005 --> 00:12:40.120 It actually becomes quite pertinent. 00:12:40.120 --> 00:12:40.840 Yep, sorry? 00:12:40.840 --> 00:12:43.290 AUDIENCE: I have a question about the standard curriculum. 00:12:43.290 --> 00:12:46.230 What kind of things does it teach? 00:12:46.230 --> 00:12:48.680 Is there any [INAUDIBLE] benefits from it? 00:12:48.680 --> 00:12:51.620 Are there any kids who are up to the curriculum? 00:12:51.620 --> 00:12:52.900 PROFESSOR: So that's a very good question. 00:12:52.900 --> 00:12:54.244 What is the curriculum? 00:12:54.244 --> 00:12:56.020 So the curriculum in India-- 00:12:56.020 --> 00:13:00.350 and that actually is true in the countries I know-- 00:13:00.350 --> 00:13:02.370 tends to be very ambitious. 00:13:02.370 --> 00:13:05.080 And there are various reasons why those 00:13:05.080 --> 00:13:06.330 curriculum are very ambitious. 00:13:06.330 --> 00:13:09.010 They are probably more ambitious than what is being 00:13:09.010 --> 00:13:10.430 taught in the US. 00:13:10.430 --> 00:13:14.630 Actually, significantly more ambitious than what is being 00:13:14.630 --> 00:13:17.310 taught in US primary schools. 00:13:17.310 --> 00:13:23.830 We once showed up to a school. 00:13:23.830 --> 00:13:25.455 We were with the Pratham team. 00:13:25.455 --> 00:13:26.810 The visit was announced. 00:13:26.810 --> 00:13:28.570 The teacher wanted to look good. 00:13:28.570 --> 00:13:31.760 And his idea of looking good was to draw things like that 00:13:31.760 --> 00:13:36.530 on the board, in this grade three class, so all the kids 00:13:36.530 --> 00:13:38.410 were like looking around. 00:13:38.410 --> 00:13:41.340 And it doesn't mean that this is what he actually does on a 00:13:41.340 --> 00:13:43.405 day in day out basis, but it means it is what 00:13:43.405 --> 00:13:45.140 they suggest to do. 00:13:45.140 --> 00:13:47.320 So it's not necessarily a bad curriculum. 00:13:47.320 --> 00:13:51.340 In fact, India has a lot of good, very good, very serious 00:13:51.340 --> 00:13:54.700 education experts who are thinking about educating the 00:13:54.700 --> 00:13:58.510 child to a real, true understanding of science. 00:13:58.510 --> 00:14:00.750 But for most of the children, it's completely above their 00:14:00.750 --> 00:14:02.090 head, very quickly. 00:14:02.090 --> 00:14:05.310 And then some of the children benefit from it. 00:14:05.310 --> 00:14:08.130 We can see an example of that very quickly. 00:14:08.130 --> 00:14:10.070 We'll see an example of that very quickly. 00:14:10.070 --> 00:14:10.420 Yes, [INAUDIBLE]? 00:14:10.420 --> 00:14:13.400 AUDIENCE: Is their proficiency in math as bad as in 00:14:13.400 --> 00:14:13.750 [INAUDIBLE]? 00:14:13.750 --> 00:14:16.076 PROFESSOR: It tends to be worse. 00:14:16.076 --> 00:14:19.440 So math is a bit worse than reading. 00:14:19.440 --> 00:14:21.640 For example, if you look at division, which is the 00:14:21.640 --> 00:14:24.820 equivalent of your standard two kid, you have like 20% of 00:14:24.820 --> 00:14:26.070 kids can do division. 00:14:30.570 --> 00:14:32.460 So that's where we are. 00:14:32.460 --> 00:14:36.630 So this quality of education, I was saying the other day, 00:14:36.630 --> 00:14:40.810 has not concerned the policy establishment, very much, for 00:14:40.810 --> 00:14:41.420 many years. 00:14:41.420 --> 00:14:44.410 The policy establishment has been more concerned about 00:14:44.410 --> 00:14:48.165 getting as many children as possible in schools, enrolled 00:14:48.165 --> 00:14:49.130 in schools. 00:14:49.130 --> 00:14:52.250 With the understanding that, oh, if we can manage to get 00:14:52.250 --> 00:14:54.630 them in schools, somehow learning will happen. 00:14:54.630 --> 00:14:58.070 And the Turkey experiment is, in a sense, an archetypal 00:14:58.070 --> 00:15:00.840 example of that, which is let's make education 00:15:00.840 --> 00:15:02.880 compulsory, let's have boarding schools, let's have 00:15:02.880 --> 00:15:04.570 buses that are going to bring the kids to 00:15:04.570 --> 00:15:05.460 the boarding school. 00:15:05.460 --> 00:15:08.410 And then black box, learning happens. 00:15:08.410 --> 00:15:10.570 You have an educated child. 00:15:10.570 --> 00:15:15.030 But it's been about 10 years since academics have been a 00:15:15.030 --> 00:15:16.210 bit puzzled at that. 00:15:16.210 --> 00:15:19.280 And looking at the quality of education saying, how do we 00:15:19.280 --> 00:15:20.500 make education actually work? 00:15:20.500 --> 00:15:21.720 What's important? 00:15:21.720 --> 00:15:24.070 And the first thing that jumps at you, when you go to a 00:15:24.070 --> 00:15:28.350 school in India, in Kenya, even in rural Morocco, which 00:15:28.350 --> 00:15:30.410 is a richer country, is that there is 00:15:30.410 --> 00:15:31.950 nothing in the school. 00:15:31.950 --> 00:15:34.010 The school is usually very bare. 00:15:34.010 --> 00:15:36.530 You have fewer desks than students. 00:15:36.530 --> 00:15:39.450 People are bunching like three to a desk. 00:15:39.450 --> 00:15:42.900 They will usually have a blackboard, very few 00:15:42.900 --> 00:15:45.480 textbooks, and, of course, nothing like computers and 00:15:45.480 --> 00:15:46.910 things like that. 00:15:46.910 --> 00:15:51.300 So the first generation of studies was about, well, they 00:15:51.300 --> 00:15:52.160 have none of that. 00:15:52.160 --> 00:15:53.040 How can they learn? 00:15:53.040 --> 00:15:55.260 Maybe what they need is more material. 00:15:55.260 --> 00:16:00.580 And in fact, when the idea of randomized evaluation started 00:16:00.580 --> 00:16:04.470 to be applied to development, Michael Kremer, who is a 00:16:04.470 --> 00:16:09.330 professor at Harvard, wanted to do a demonstration project, 00:16:09.330 --> 00:16:12.410 wanted to show that it is possible to do a randomized 00:16:12.410 --> 00:16:13.010 evaluation. 00:16:13.010 --> 00:16:14.330 And it's interesting. 00:16:14.330 --> 00:16:17.460 And he wanted, for demonstration purposes, an 00:16:17.460 --> 00:16:20.790 example where it would clear that it has an effect. 00:16:20.790 --> 00:16:23.030 And so he asked around, et cetera. 00:16:23.030 --> 00:16:26.210 And he found out that what would be obvious is textbooks. 00:16:26.210 --> 00:16:28.100 In Kenya, no one has textbooks. 00:16:28.100 --> 00:16:30.400 If only we give textbooks to the kids, obviously, they are 00:16:30.400 --> 00:16:31.670 going to learn better. 00:16:31.670 --> 00:16:33.880 So that was his first example that he chose, which was 00:16:33.880 --> 00:16:38.680 chosen specifically to demonstrate success. 00:16:38.680 --> 00:16:43.600 And running his experiment, he was learning how to do an 00:16:43.600 --> 00:16:44.570 experiment at the same time. 00:16:44.570 --> 00:16:48.080 So his first experiment was very small, 14 schools, 7 00:16:48.080 --> 00:16:51.340 treatment, 7 control, and found no effect. 00:16:53.880 --> 00:16:56.370 He thought about it, and he said, oh well, of course, it's 00:16:56.370 --> 00:16:57.840 because I have so few schools. 00:16:57.840 --> 00:17:00.360 And you remember, when you have schools, you need to take 00:17:00.360 --> 00:17:02.760 into account, when you calculate your standard error, 00:17:02.760 --> 00:17:05.579 that all these children are similar. 00:17:05.579 --> 00:17:08.390 So he said, oh well, my experiment is not big enough. 00:17:08.390 --> 00:17:11.030 So let's take a bigger sample of schools. 00:17:11.030 --> 00:17:11.849 He took 100 schools. 00:17:11.849 --> 00:17:16.180 He took half of them and again found no effect. 00:17:16.180 --> 00:17:20.099 So there he thought, well, maybe the test is too hard. 00:17:20.099 --> 00:17:21.910 I'm using the test, the normal test. 00:17:21.910 --> 00:17:22.730 Maybe that is too hard. 00:17:22.730 --> 00:17:25.980 Let me use a different test that is going to be able to 00:17:25.980 --> 00:17:29.060 discriminate progress even at a lower level. 00:17:29.060 --> 00:17:30.330 And he got his experiment started. 00:17:30.330 --> 00:17:32.880 And again, he found no effect. 00:17:32.880 --> 00:17:37.060 So there he started saying maybe there is no effect, even 00:17:37.060 --> 00:17:40.360 in a larger sample, even with a better test. 00:17:40.360 --> 00:17:42.380 So maybe there is no effect. 00:17:42.380 --> 00:17:45.910 So eventually, what he found is that, if you focus on the 00:17:45.910 --> 00:17:50.070 children, who were already doing well at the baseline, 00:17:50.070 --> 00:17:51.690 then they benefit from the textbooks. 00:17:51.690 --> 00:17:54.050 So that's a very long winded answer to your question, which 00:17:54.050 --> 00:17:57.260 is there are children who benefit from the curriculum 00:17:57.260 --> 00:17:58.910 and, therefore, benefit from the textbooks. 00:17:58.910 --> 00:18:01.290 But it's a small minority. 00:18:01.290 --> 00:18:05.510 And the explanation he gave is the textbooks are in English, 00:18:05.510 --> 00:18:07.470 which makes sense, because the curriculum in Kenyan schools 00:18:07.470 --> 00:18:08.770 is in English. 00:18:08.770 --> 00:18:11.530 But a lot of children don't even speak English, because 00:18:11.530 --> 00:18:13.600 English is their third language. 00:18:13.600 --> 00:18:16.500 They first speak their mother tongue, the local language. 00:18:16.500 --> 00:18:18.940 Then they learn Swahili in the first few grades. 00:18:18.940 --> 00:18:22.840 And then English is introduced in grade one as a language, 00:18:22.840 --> 00:18:25.450 and then as a language of instruction from grade three 00:18:25.450 --> 00:18:26.650 or grade four. 00:18:26.650 --> 00:18:30.100 But the problem is that because the kids aren't 00:18:30.100 --> 00:18:33.180 actually learning effectively, by the time they reach grade 00:18:33.180 --> 00:18:35.410 five or six, they actually don't know English, so the 00:18:35.410 --> 00:18:36.900 textbooks are of no use. 00:18:36.900 --> 00:18:38.850 So exposed, it could be understood what 00:18:38.850 --> 00:18:40.440 was going on here. 00:18:40.440 --> 00:18:42.230 And then they tried a bunch of stuff. 00:18:42.230 --> 00:18:45.330 Flip charts also have no effect. 00:18:45.330 --> 00:18:47.940 Cutting the class size in two, if you make no other 00:18:47.940 --> 00:18:50.430 differences, also has no effect. 00:18:50.430 --> 00:18:53.560 And a little bit worrisome, things like the deworming 00:18:53.560 --> 00:18:57.280 program, which increases attendance, but it didn't 00:18:57.280 --> 00:18:58.530 increase test scores. 00:18:58.530 --> 00:19:00.230 So it seemed that these extra school 00:19:00.230 --> 00:19:01.750 days also had no effect. 00:19:01.750 --> 00:19:04.810 So changing inputs just didn't work. 00:19:04.810 --> 00:19:08.490 So maybe it's a problem that is just incredibly difficult. 00:19:08.490 --> 00:19:12.440 So we were there a few years ago. 00:19:12.440 --> 00:19:16.270 So what this common with all this intervention is that they 00:19:16.270 --> 00:19:20.100 are just changing the inputs. 00:19:20.100 --> 00:19:23.500 They are doing more of the same, adding more textbooks, 00:19:23.500 --> 00:19:25.880 adding more teachers, adding more resources. 00:19:25.880 --> 00:19:29.270 But they are no change to the pedagogy and no change to the 00:19:29.270 --> 00:19:30.490 incentives. 00:19:30.490 --> 00:19:32.510 So no one is given incentives. 00:19:32.510 --> 00:19:33.940 The teachers are not given incentives. 00:19:33.940 --> 00:19:35.340 The students are not given incentives. 00:19:35.340 --> 00:19:37.060 The parents are not given incentives. 00:19:37.060 --> 00:19:40.650 And the pedagogy and the curriculum stays the same. 00:19:40.650 --> 00:19:43.580 So that's kind of where we were about five, six years 00:19:43.580 --> 00:19:46.150 ago, a little bit depressed. 00:19:46.150 --> 00:19:48.830 Picture us, this was our first set of randomized experiments 00:19:48.830 --> 00:19:50.640 and nothing works. 00:19:50.640 --> 00:19:52.990 So we were thinking, maybe, this is going to be the end of 00:19:52.990 --> 00:19:54.650 randomized experiments, because you can't be a 00:19:54.650 --> 00:19:57.850 doomsayer forever, otherwise people really hate you. 00:19:57.850 --> 00:19:59.750 So that's why we were a little bit sad. 00:20:02.880 --> 00:20:08.670 Rukmini came to visit us, shortly after that happened, a 00:20:08.670 --> 00:20:12.140 few years after the establishment of Pratham, came 00:20:12.140 --> 00:20:13.360 here to MIT. 00:20:13.360 --> 00:20:17.280 Because a former MIT student had gone to work 00:20:17.280 --> 00:20:21.010 for them as an intern. 00:20:21.010 --> 00:20:23.660 A former undergraduate student of ours had a 00:20:23.660 --> 00:20:25.530 relationship with them. 00:20:25.530 --> 00:20:27.780 So Pratham started in 1994. 00:20:27.780 --> 00:20:30.130 At that time, it was established by UNICEF to help 00:20:30.130 --> 00:20:31.690 some kids in Bombay. 00:20:31.690 --> 00:20:35.130 It would be your typical, small NGO doing some work in 00:20:35.130 --> 00:20:37.940 Bombay, particularly what they call bridge classes. 00:20:37.940 --> 00:20:45.370 Which is take a kid who is out of school, give them a course 00:20:45.370 --> 00:20:47.670 for a few months so that they can try and go back to their 00:20:47.670 --> 00:20:48.250 regular school. 00:20:48.250 --> 00:20:50.410 That's what they were doing. 00:20:50.410 --> 00:20:52.070 But they were ambitious. 00:20:52.070 --> 00:20:55.100 So there was the Rukmini that you heard and Madhav Chavan 00:20:55.100 --> 00:20:57.270 wanted to make a real difference. 00:20:57.270 --> 00:21:00.120 So since then, they're grown substantially. 00:21:00.120 --> 00:21:02.160 They've reached millions and millions of children. 00:21:02.160 --> 00:21:05.300 I think there are about 38 million children, who are 00:21:05.300 --> 00:21:07.590 reached by the Pratham program, one way or the other, 00:21:07.590 --> 00:21:10.600 so about half the population of France, overall population 00:21:10.600 --> 00:21:14.100 of France, just in terms of scale. 00:21:14.100 --> 00:21:16.900 And so that's the largest non-governmental organization 00:21:16.900 --> 00:21:19.890 to do education in India, probably in the world. 00:21:19.890 --> 00:21:23.150 And their motto is "every child in school"-- that half 00:21:23.150 --> 00:21:24.810 of the world would agree with them-- 00:21:24.810 --> 00:21:26.910 "and learning well"-- 00:21:26.910 --> 00:21:29.750 is where, maybe, they have a little difference. 00:21:29.750 --> 00:21:31.350 So they came to us towards the beginning. 00:21:34.560 --> 00:21:36.240 And that's the reading that read for today. 00:21:36.240 --> 00:21:40.540 They wanted to evaluate the Balsakhi program, which was 00:21:40.540 --> 00:21:43.450 the flagship program at the time. 00:21:43.450 --> 00:21:46.680 Balsakhi means the friend of the child. 00:21:46.680 --> 00:21:48.540 So the Balsakhi is a young woman, from the 00:21:48.540 --> 00:21:52.220 community, so an adult. 00:21:52.220 --> 00:21:54.190 Like she said, you need some adult time. 00:21:54.190 --> 00:22:00.850 Barely an adult, some 18-year-old, usually having 00:22:00.850 --> 00:22:04.440 only high school education, so grade 10 to 12. 00:22:04.440 --> 00:22:08.480 And Pratham would give them a very short training and then 00:22:08.480 --> 00:22:11.910 dump them into the school, with which they had an 00:22:11.910 --> 00:22:16.590 agreement with the teachers that the school would let them 00:22:16.590 --> 00:22:20.785 pull out the kids who were lagging behind in grade three 00:22:20.785 --> 00:22:22.800 of in grade four. 00:22:22.800 --> 00:22:28.340 So why is this a good demographic, the 18-year-old, 00:22:28.340 --> 00:22:33.620 grade 10 educated woman? 00:22:33.620 --> 00:22:35.330 Why is that a good goal to work with? 00:22:41.635 --> 00:22:42.120 Yep? 00:22:42.120 --> 00:22:44.787 AUDIENCE: She's more familiar with the community, so she can 00:22:44.787 --> 00:22:46.980 probably relate to [INAUDIBLE]. 00:22:46.980 --> 00:22:49.360 PROFESSOR: So that is one reason is that she is more 00:22:49.360 --> 00:22:51.450 known to the community, because she's local. 00:22:51.450 --> 00:22:52.850 She's not much older. 00:22:52.850 --> 00:22:55.550 And she's not intimidating or scary for the parents, who, 00:22:55.550 --> 00:22:57.420 maybe, can work better with the kids. 00:22:57.420 --> 00:22:58.140 That's a very good reason. 00:22:58.140 --> 00:22:58.750 What's another reason? 00:22:58.750 --> 00:23:00.214 AUDIENCE: I would say for the teacher's as well. 00:23:00.214 --> 00:23:02.654 Maybe the teachers are not as intimidated that the balsakhi 00:23:02.654 --> 00:23:05.094 will go and teach these students some really 00:23:05.094 --> 00:23:07.550 innovative thing that would make [INAUDIBLE]. 00:23:07.550 --> 00:23:08.000 PROFESSOR: Yes. 00:23:08.000 --> 00:23:09.920 So for the teachers, she might be not very intimidating, 00:23:09.920 --> 00:23:11.400 which will cut both ways. 00:23:11.400 --> 00:23:12.810 On the one hand, it's more easily 00:23:12.810 --> 00:23:14.050 acceptable for the teachers. 00:23:14.050 --> 00:23:17.080 On the other hand, there is a tendency that teacher would 00:23:17.080 --> 00:23:19.670 use the balsakhi to make the tea or other 00:23:19.670 --> 00:23:21.260 activities like that. 00:23:21.260 --> 00:23:23.520 And we are going to see some of that happening, not 00:23:23.520 --> 00:23:24.360 actually in Baroda. 00:23:24.360 --> 00:23:24.818 Yep? 00:23:24.818 --> 00:23:29.148 AUDIENCE: So she's locally trained, which makes sense in 00:23:29.148 --> 00:23:30.940 kind of [INAUDIBLE] perspective. 00:23:30.940 --> 00:23:32.420 PROFESSOR: Yes. 00:23:32.420 --> 00:23:35.090 She is locally trained, only for two weeks. 00:23:35.090 --> 00:23:36.560 This is very cheap. 00:23:36.560 --> 00:23:38.140 And there is another aspect of the cost. 00:23:38.140 --> 00:23:39.136 Yeah? 00:23:39.136 --> 00:23:41.294 AUDIENCE: Not about the cost, but she might be more 00:23:41.294 --> 00:23:44.614 motivated to help the [INAUDIBLE] children, because 00:23:44.614 --> 00:23:49.096 it's not like this is a second job she can perform. 00:23:49.096 --> 00:23:50.590 It's not a fallback option. 00:23:50.590 --> 00:23:53.744 It's something that she wants to do that is a 00:23:53.744 --> 00:23:54.574 period in her life. 00:23:54.574 --> 00:23:56.070 It seems a bit time sensitive. 00:23:56.070 --> 00:23:56.590 PROFESSOR: Yes. 00:23:56.590 --> 00:23:59.270 So she might be more motivated for this reason. 00:23:59.270 --> 00:24:01.680 Also for what you said earlier, that she's close to 00:24:01.680 --> 00:24:02.440 them, et cetera. 00:24:02.440 --> 00:24:05.220 But also this is something that she effectively chose to 00:24:05.220 --> 00:24:08.440 do at this point and not something that is just putting 00:24:08.440 --> 00:24:10.876 money in a bank account. 00:24:10.876 --> 00:24:13.830 AUDIENCE: She's not highly qualified, so the [INAUDIBLE]. 00:24:13.830 --> 00:24:15.680 PROFESSOR: Right. 00:24:15.680 --> 00:24:18.050 She's not highly qualified, so she's cheap. 00:24:18.050 --> 00:24:21.120 What is another reason why she is cheap? 00:24:21.120 --> 00:24:25.410 Why don't these people go and do something else? 00:24:25.410 --> 00:24:29.330 AUDIENCE: Also there's a high turnover in these [INAUDIBLE] 00:24:29.330 --> 00:24:32.025 make a certain income to teach, so you're not really 00:24:32.025 --> 00:24:34.720 relying on this thing. 00:24:34.720 --> 00:24:36.140 It's [INAUDIBLE] 00:24:36.140 --> 00:24:37.043 quality of the teaching. 00:24:37.043 --> 00:24:39.458 It doesn't have to be like a small, really enthusiastic 00:24:39.458 --> 00:24:40.708 [INAUDIBLE]. 00:24:43.322 --> 00:24:46.220 So it's easy to get [INAUDIBLE]. 00:24:46.220 --> 00:24:48.980 PROFESSOR: Yeah. 00:24:48.980 --> 00:24:51.280 They have high turnover, which could be a plus or a minus. 00:24:51.280 --> 00:24:54.140 On the minus side, of course, you don't get the experience. 00:24:54.140 --> 00:24:58.460 On the plus side, you don't get tired. 00:24:58.460 --> 00:25:01.320 And you don't rely on people who have a huge vocation. 00:25:01.320 --> 00:25:04.260 You rely on people getting enthusiastic for one year. 00:25:04.260 --> 00:25:07.730 So very much along the model of Teach For America, where 00:25:07.730 --> 00:25:15.010 you get MIT, graduating from school, energetic about 00:25:15.010 --> 00:25:15.890 wanting to do this. 00:25:15.890 --> 00:25:18.960 A bit of what you're saying, which is it's what I want to 00:25:18.960 --> 00:25:20.680 do at this point in my life. 00:25:20.680 --> 00:25:22.160 And then by the time you don't want to do it 00:25:22.160 --> 00:25:23.630 any more, you stop. 00:25:23.630 --> 00:25:26.000 And there is one more thing about them, specifically, 00:25:26.000 --> 00:25:29.066 which makes them cheap. 00:25:29.066 --> 00:25:30.518 AUDIENCE: Everything about them. 00:25:30.518 --> 00:25:34.027 I mean I think there's like a good chance that they have 00:25:34.027 --> 00:25:35.842 families or that they're part of a family, and maybe that's 00:25:35.842 --> 00:25:37.294 why they're stuck in the area. 00:25:37.294 --> 00:25:38.988 And they don't necessarily have the ability to go 00:25:38.988 --> 00:25:40.700 somewhere else. 00:25:40.700 --> 00:25:41.560 PROFESSOR: Exactly. 00:25:41.560 --> 00:25:43.410 That's a very important point. 00:25:43.410 --> 00:25:45.280 Most of these women are unmarried. 00:25:45.280 --> 00:25:46.650 They are living at home. 00:25:46.650 --> 00:25:48.420 They are sort of waiting to be married. 00:25:48.420 --> 00:25:50.990 And there is actually not much they can do. 00:25:50.990 --> 00:25:53.290 Because the parents won't let them leave. 00:25:53.290 --> 00:25:56.070 And mostly, their parents won't let them work, because 00:25:56.070 --> 00:26:00.030 working is not becoming for a certain, sort of lower, middle 00:26:00.030 --> 00:26:03.290 class category woman, waiting to be married. 00:26:03.290 --> 00:26:05.350 You shouldn't be like working. 00:26:05.350 --> 00:26:07.230 You should be able to be-- 00:26:07.230 --> 00:26:09.970 it's not a good signal for the family that they're not able 00:26:09.970 --> 00:26:12.290 to provide for you and all that. 00:26:12.290 --> 00:26:14.560 But this is hardly seen as working. 00:26:14.560 --> 00:26:20.210 This is more like volunteering and helping your community and 00:26:20.210 --> 00:26:21.210 things like that. 00:26:21.210 --> 00:26:22.705 So they can do that. 00:26:22.705 --> 00:26:24.210 So they are sort of available. 00:26:24.210 --> 00:26:28.150 So one part of the genius of Pratham is that they identify 00:26:28.150 --> 00:26:30.390 a group of people who are actually sort of 00:26:30.390 --> 00:26:31.890 available for free. 00:26:31.890 --> 00:26:34.880 Here, they pay them, but in future programs, that I'm 00:26:34.880 --> 00:26:37.820 going to talk to you a minute after, they're not even paid. 00:26:37.820 --> 00:26:39.002 They are just unpaid. 00:26:39.002 --> 00:26:40.350 They do that for a while. 00:26:40.350 --> 00:26:46.790 And that get some, presumably, other rewards of doing that, 00:26:46.790 --> 00:26:50.230 which are more in the form of intrinsic reward of seeing the 00:26:50.230 --> 00:26:53.570 kid progress, and, therefore, you also select people who are 00:26:53.570 --> 00:26:57.260 motivated in that dimension as one of you pointed out. 00:26:57.260 --> 00:26:58.610 So that's all of the pluses. 00:26:58.610 --> 00:27:00.440 The minuses you already mentioned. 00:27:00.440 --> 00:27:02.060 The turnover is one of them. 00:27:02.060 --> 00:27:05.490 You can't get them to get experienced. 00:27:05.490 --> 00:27:06.660 The second minus is that they're, of 00:27:06.660 --> 00:27:08.140 course, much less educated. 00:27:08.140 --> 00:27:13.520 Maybe there is some good reasons why a teacher needs a 00:27:13.520 --> 00:27:14.750 college degree. 00:27:14.750 --> 00:27:16.890 Maybe it's actually terrible to take the kid out of the 00:27:16.890 --> 00:27:20.050 classroom to put them in front of someone who knows nothing. 00:27:20.050 --> 00:27:23.870 So that's kind of the risk that would 00:27:23.870 --> 00:27:25.890 potentially be there. 00:27:25.890 --> 00:27:28.930 So that's the program. 00:27:28.930 --> 00:27:36.970 And when they came to see us, we were wondering how to 00:27:36.970 --> 00:27:41.730 evaluate this program, with them, in an experiment, in a 00:27:41.730 --> 00:27:47.800 way that is not going to cause a problem for them in the 00:27:47.800 --> 00:27:51.820 city, that's not going to hinder their work too much. 00:27:51.820 --> 00:27:55.660 And that's experimental design that we decided to 00:27:55.660 --> 00:27:57.860 adopt in a new place. 00:27:57.860 --> 00:28:00.740 So Vadodara and Gujarat, those were places where they had not 00:28:00.740 --> 00:28:02.460 worked at all. 00:28:02.460 --> 00:28:04.040 Had about 100 schools. 00:28:04.040 --> 00:28:08.220 A little more than 100 schools divided in two groups, 00:28:08.220 --> 00:28:15.120 randomly, with the computer, group A 00:28:15.120 --> 00:28:16.820 schools and group B schools. 00:28:16.820 --> 00:28:19.020 The group A schools got the balsakhi 00:28:19.020 --> 00:28:21.310 for grade three children. 00:28:21.310 --> 00:28:23.180 The group B schools got the balsakhi 00:28:23.180 --> 00:28:25.070 for grade four children. 00:28:25.070 --> 00:28:29.170 So what's an advantage of this design from the point of view 00:28:29.170 --> 00:28:31.180 of political acceptability? 00:28:35.640 --> 00:28:38.160 Why did we go like this, instead of doing the more 00:28:38.160 --> 00:28:42.010 standard thing, which would be to say, well, in group A 00:28:42.010 --> 00:28:46.220 treatment, and grade three and grade four get the balsakhi 00:28:46.220 --> 00:28:49.380 and group B, no one gets a balsakhi? 00:28:49.380 --> 00:28:52.500 AUDIENCE: Well, in a way, if you don't give one group any 00:28:52.500 --> 00:28:55.620 treatment at all, you're almost sabotaging your 00:28:55.620 --> 00:28:58.510 education [INAUDIBLE]. 00:28:58.510 --> 00:29:01.360 PROFESSOR: Sabotaging is a strong word. 00:29:01.360 --> 00:29:05.880 But exactly the idea is that this allows you to be present 00:29:05.880 --> 00:29:07.960 in every school. 00:29:07.960 --> 00:29:11.380 So Pratham is not infinitely rich. 00:29:11.380 --> 00:29:14.990 They can easily argue-- and I think that was very true at 00:29:14.990 --> 00:29:16.820 that time-- that look, we can give you one 00:29:16.820 --> 00:29:17.620 balsakhi per school. 00:29:17.620 --> 00:29:19.460 We can't afford more. 00:29:19.460 --> 00:29:21.920 So you either put it in three or you put it in four. 00:29:21.920 --> 00:29:24.520 We can't give you both. 00:29:24.520 --> 00:29:29.670 But at least every school is being engaged with. 00:29:29.670 --> 00:29:32.880 So they are engaged with the system as a whole. 00:29:32.880 --> 00:29:34.780 And no school is disadvantaged. 00:29:34.780 --> 00:29:37.040 That makes it easier for the school system to say, yes, 00:29:37.040 --> 00:29:37.870 they are just working with us. 00:29:37.870 --> 00:29:39.510 They are part of us. 00:29:39.510 --> 00:29:42.480 It also makes data collection much easier, because you're 00:29:42.480 --> 00:29:45.000 involved with everyone. 00:29:45.000 --> 00:29:50.790 What is a potential danger of this design? 00:29:50.790 --> 00:29:52.070 The plus is what I just said. 00:29:52.070 --> 00:29:53.800 What's the potential minus? 00:29:58.080 --> 00:30:00.180 Since you have a balsakhi in every school, what's the 00:30:00.180 --> 00:30:02.592 potential minus? 00:30:02.592 --> 00:30:05.227 AUDIENCE: They might not just be teaching grade three. 00:30:05.227 --> 00:30:08.150 The might be overlapping grade three and four or something. 00:30:08.150 --> 00:30:08.573 PROFESSOR: Exactly. 00:30:08.573 --> 00:30:13.150 The balsakhi might take both groups, so that your control 00:30:13.150 --> 00:30:16.605 group becomes partly treated. 00:30:16.605 --> 00:30:18.710 Or even if it's not the balsakhi, the head teacher 00:30:18.710 --> 00:30:21.920 might say, oh great, we have a balsakhi for grade three. 00:30:21.920 --> 00:30:27.210 So we're going to take the teacher out of grade three and 00:30:27.210 --> 00:30:29.550 use it to divide grade four in two. 00:30:29.550 --> 00:30:32.790 And so grade four would be associated. 00:30:32.790 --> 00:30:35.960 Or they could say, well, since the grade three got the 00:30:35.960 --> 00:30:38.660 balsakhi, when some other organization comes with 00:30:38.660 --> 00:30:41.170 computers, well, we'll put them in grade four. 00:30:41.170 --> 00:30:43.970 So there is a danger of resources being reallocated 00:30:43.970 --> 00:30:47.950 across grades in a way that contaminates your treatment. 00:30:47.950 --> 00:30:52.230 So fortunately, that was not really an issue there, because 00:30:52.230 --> 00:30:55.670 in Vadodara, the way the school system is organized, a 00:30:55.670 --> 00:31:00.460 school is made of one teacher per grade, regardless. 00:31:00.460 --> 00:31:02.460 Sometimes in the same school building, you have more than 00:31:02.460 --> 00:31:03.970 one school. 00:31:03.970 --> 00:31:05.680 And that happens sometimes in the US as well. 00:31:05.680 --> 00:31:08.510 You have this charter school and the regular school sharing 00:31:08.510 --> 00:31:10.720 the same school building. 00:31:10.720 --> 00:31:14.380 In Vadodara what happened is, if it's a big school, it's 00:31:14.380 --> 00:31:16.030 actually a big school building, but under totally 00:31:16.030 --> 00:31:18.090 separate administration. 00:31:18.090 --> 00:31:20.790 So a school is made of five teachers and a head teacher. 00:31:20.790 --> 00:31:21.700 That's it. 00:31:21.700 --> 00:31:24.530 So there is less scope for reallocating, because you have 00:31:24.530 --> 00:31:26.470 your one teacher anyway. 00:31:26.470 --> 00:31:29.370 And they're also not very imaginative in terms of using 00:31:29.370 --> 00:31:32.000 the resources in the most optimal way. 00:31:32.000 --> 00:31:34.800 So that turned out not to be a problem. 00:31:34.800 --> 00:31:37.770 So the way we evaluated the program is then by comparing 00:31:37.770 --> 00:31:39.910 grade three, group A's treatment 00:31:39.910 --> 00:31:41.080 compared with control. 00:31:41.080 --> 00:31:43.850 And in grade four, that's the opposite. 00:31:43.850 --> 00:31:46.800 We also worked in Bombay. 00:31:46.800 --> 00:31:49.220 So that's the second year we have asked a group. 00:31:49.220 --> 00:31:52.930 So if you were a kid who was in grade three, in group A, 00:31:52.930 --> 00:31:57.200 you are treated in grade three and then, again, you are 00:31:57.200 --> 00:32:02.000 treated in grade four, because you would 00:32:02.000 --> 00:32:03.610 have moved one grade. 00:32:03.610 --> 00:32:06.530 If you were a kid who entered in grade four, you would never 00:32:06.530 --> 00:32:07.790 be treated. 00:32:07.790 --> 00:32:10.270 Or if you were a kid who entered grade three in group 00:32:10.270 --> 00:32:11.940 B, you also would never be treated. 00:32:11.940 --> 00:32:14.840 So we have kids treated one year, kids treated two years, 00:32:14.840 --> 00:32:17.830 and kids treated one year. 00:32:17.830 --> 00:32:24.410 And then in Bombay, we did something similar. 00:32:24.410 --> 00:32:27.550 This is Shobhini She used to work at Pratham. 00:32:27.550 --> 00:32:29.880 And now she's one of the executive director of J-PAL 00:32:29.880 --> 00:32:31.310 South Asia. 00:32:31.310 --> 00:32:33.280 You can say, hi, to her. 00:32:33.280 --> 00:32:35.330 And in Bombay, we did something similar, except we 00:32:35.330 --> 00:32:38.160 started in grade two, three, and then moved 00:32:38.160 --> 00:32:40.180 it to three, four. 00:32:40.180 --> 00:32:44.950 The reason why we switched is that they realized that they 00:32:44.950 --> 00:32:48.030 didn't have right pedagogy for engaging with the grade two 00:32:48.030 --> 00:32:51.110 children, that the grade two children were too small, and 00:32:51.110 --> 00:32:53.890 they didn't know how to deal with them. 00:32:53.890 --> 00:32:57.510 The initial plan was to do two, three and then reverse 00:32:57.510 --> 00:32:58.650 again, two, three. 00:32:58.650 --> 00:33:00.500 And then they turned out that they weren't happy with 00:33:00.500 --> 00:33:04.560 they're grade two, so they moved three, four. 00:33:04.560 --> 00:33:05.930 So that's kind of the idea. 00:33:05.930 --> 00:33:07.150 So this is the design. 00:33:07.150 --> 00:33:10.620 So in principle, now, we could say, well, they're good. 00:33:10.620 --> 00:33:12.230 We have the designs. 00:33:12.230 --> 00:33:15.430 Now, all we need to do is to compare treatment and control 00:33:15.430 --> 00:33:16.980 after one year and after two years, and 00:33:16.980 --> 00:33:18.440 we're going to be done. 00:33:18.440 --> 00:33:23.090 But unfortunately, when you run real experiments, you get 00:33:23.090 --> 00:33:24.395 real problems. 00:33:24.395 --> 00:33:27.190 I've yet to meet an experiment where there was no problems. 00:33:27.190 --> 00:33:30.080 And this was, at least, my first big one. 00:33:30.080 --> 00:33:34.020 So we had a lot of problems that I want to talk about. 00:33:34.020 --> 00:33:39.110 The first one is the way to evaluate the program was to 00:33:39.110 --> 00:33:41.720 administer a test in the school. 00:33:41.720 --> 00:33:44.610 And we had some issue with the test. 00:33:44.610 --> 00:33:46.610 The first one is that, as I was saying, a 00:33:46.610 --> 00:33:49.360 lot of kids are absent. 00:33:49.360 --> 00:33:55.670 And if we only tested the kids who were present in school, we 00:33:55.670 --> 00:33:57.920 have some attrition. 00:33:57.920 --> 00:34:01.620 Maybe in urban India, absenteeism is less bad, but 00:34:01.620 --> 00:34:04.780 maybe 20% of kids are absent on a given day, so they don't 00:34:04.780 --> 00:34:06.400 take the test. 00:34:06.400 --> 00:34:12.719 So why is that potentially a problem to have missing kids 00:34:12.719 --> 00:34:13.460 in the test? 00:34:13.460 --> 00:34:14.448 Yep? 00:34:14.448 --> 00:34:16.671 AUDIENCE: Maybe the kids that are absent are those that are 00:34:16.671 --> 00:34:20.376 more likely to not [INAUDIBLE] class, and so they either 00:34:20.376 --> 00:34:22.846 chose or their parents chose that, maybe, they could better 00:34:22.846 --> 00:34:24.822 use that time working from home. 00:34:24.822 --> 00:34:27.786 And so all the results are biased. 00:34:27.786 --> 00:34:29.762 Then you're only really testing the kids that 00:34:29.762 --> 00:34:32.232 understand more and you're ignoring the kids that 00:34:32.232 --> 00:34:33.239 understood the least. 00:34:33.239 --> 00:34:33.389 PROFESSOR: Exactly. 00:34:33.389 --> 00:34:36.300 So that could be the first problem, which is those kids, 00:34:36.300 --> 00:34:38.570 whom we are testing, they are not representative, because 00:34:38.570 --> 00:34:41.290 maybe they're the one who understand the most. 00:34:41.290 --> 00:34:46.520 And that could lead to an even bigger problem, because 00:34:46.520 --> 00:34:50.010 suppose the intervention is effective and most kids now 00:34:50.010 --> 00:34:51.500 understand better. 00:34:51.500 --> 00:34:54.409 Then what could happen with the attrition in the treatment 00:34:54.409 --> 00:34:55.780 group versus the control group? 00:35:01.710 --> 00:35:04.240 Now you all know very well, it's the kids who don't 00:35:04.240 --> 00:35:06.460 understand who don't come. 00:35:06.460 --> 00:35:08.880 And now the kids who understand better are in the 00:35:08.880 --> 00:35:10.120 treatment group. 00:35:10.120 --> 00:35:11.850 What might happen to attrition? 00:35:15.650 --> 00:35:18.025 AUDIENCE: The attrition would be less for the students who 00:35:18.025 --> 00:35:19.460 are in the treatment group. 00:35:19.460 --> 00:35:21.610 PROFESSOR: The attrition would be less for students who are 00:35:21.610 --> 00:35:22.900 in the treatment group. 00:35:22.900 --> 00:35:26.620 And the kids who are coming tends to be the weaker kids. 00:35:26.620 --> 00:35:29.830 So now we are comparing the strong kids in the control 00:35:29.830 --> 00:35:33.530 group to the strong kids plus some of the weak kids in the 00:35:33.530 --> 00:35:34.930 treatment group. 00:35:34.930 --> 00:35:39.120 So that will tend to make our estimate look smaller relative 00:35:39.120 --> 00:35:40.290 to what it should be. 00:35:40.290 --> 00:35:42.720 Because now we have a different population of 00:35:42.720 --> 00:35:44.470 students in the treatment and control groups. 00:35:44.470 --> 00:35:46.150 So we have a bias here. 00:35:46.150 --> 00:35:48.750 It goes in the direction of not finding an effect. 00:35:48.750 --> 00:35:50.560 So if we still find an effect, we're happy. 00:35:50.560 --> 00:35:53.560 But it's not the right estimate. 00:35:53.560 --> 00:35:56.390 So to solve this problem, what did we do? 00:36:00.770 --> 00:36:07.160 Did we rely, uniquely, on the test in the school? 00:36:07.160 --> 00:36:07.640 Yeah? 00:36:07.640 --> 00:36:09.871 AUDIENCE: You found the kids, even if they happened to be at 00:36:09.871 --> 00:36:11.870 home or elsewhere and administrated the test 00:36:11.870 --> 00:36:12.340 [INAUDIBLE]. 00:36:12.340 --> 00:36:15.290 PROFESSOR: We went and looked for them, wherever they were, 00:36:15.290 --> 00:36:18.290 except if they'd left for the village. 00:36:18.290 --> 00:36:22.000 So the attrition ended up being low, less than 10%, and 00:36:22.000 --> 00:36:24.470 similar, now, in treatment and control school. 00:36:24.470 --> 00:36:26.590 Because it's not because people don't understand that 00:36:26.590 --> 00:36:27.520 they leave for the village. 00:36:27.520 --> 00:36:30.800 It's because they have something to do there. 00:36:30.800 --> 00:36:34.350 So that was the first thing. 00:36:34.350 --> 00:36:38.670 The second problem we had is that the testing instrument-- 00:36:38.670 --> 00:36:41.190 actually, not in year 1 but in year 0-- 00:36:41.190 --> 00:36:44.070 we designed a test which was at grade level. 00:36:44.070 --> 00:36:46.390 We designed a test to test the curriculum. 00:36:46.390 --> 00:36:49.490 So it's like, if this is what should be tested, what's on 00:36:49.490 --> 00:36:52.570 the board, here, we tested at that level. 00:36:52.570 --> 00:36:55.640 So if they were supposed to learn Euclidean geometry, 00:36:55.640 --> 00:36:57.230 that's what was tested. 00:36:57.230 --> 00:37:01.290 And we had a big problem, which is that the teachers 00:37:01.290 --> 00:37:03.640 cheated like crazy. 00:37:03.640 --> 00:37:07.220 So the reason we saw that is that all of the 00:37:07.220 --> 00:37:08.950 tests were the same. 00:37:08.950 --> 00:37:11.980 So clearly, the teacher had written the answer on the 00:37:11.980 --> 00:37:14.120 board, and all the students were copying down. 00:37:14.120 --> 00:37:17.170 In one class, all the students had the same name. 00:37:17.170 --> 00:37:20.690 So the teacher clearly wrote the name, like a sample name, 00:37:20.690 --> 00:37:21.240 on the board. 00:37:21.240 --> 00:37:23.970 And all the students dutifully wrote down the name that they 00:37:23.970 --> 00:37:25.670 saw on the broad. 00:37:25.670 --> 00:37:28.050 So we said that's not working. 00:37:28.050 --> 00:37:32.360 There is a paper by Steve Levitt, who shows very subtle 00:37:32.360 --> 00:37:34.180 things, how you can detect cheating. 00:37:34.180 --> 00:37:35.110 That was easy. 00:37:35.110 --> 00:37:37.265 It didn't require a lot of imagination. 00:37:37.265 --> 00:37:41.860 It was pretty obvious that cheated like crazy to make 00:37:41.860 --> 00:37:44.230 their kids look better. 00:37:44.230 --> 00:37:46.730 So we thought about how we can solve this problem. 00:37:46.730 --> 00:37:50.760 We are going to administer a new test without the teacher, 00:37:50.760 --> 00:37:53.450 that some special Pratham staff is going to come to 00:37:53.450 --> 00:37:54.450 administer the test. 00:37:54.450 --> 00:37:56.110 And we did that. 00:37:56.110 --> 00:37:58.780 And then we realized that most of the students now 00:37:58.780 --> 00:38:01.700 had 0 on the test. 00:38:01.700 --> 00:38:05.560 So this test was way too hard, which is why the teacher had 00:38:05.560 --> 00:38:08.110 been cheating, because they were a bit embarrassed. 00:38:08.110 --> 00:38:12.150 So the solution here was to develop a much easier test 00:38:12.150 --> 00:38:15.620 that covered the competencies starting from, can you write 00:38:15.620 --> 00:38:17.740 your own name, to-- 00:38:17.740 --> 00:38:18.110 I don't know-- 00:38:18.110 --> 00:38:25.020 2 plus 2, finishing with a few questions such as as, Nancy 00:38:25.020 --> 00:38:25.800 goes to the market. 00:38:25.800 --> 00:38:30.620 And she buys three onions at 15 rupees. 00:38:30.620 --> 00:38:35.290 And how much change does she have, that kind of thing. 00:38:35.290 --> 00:38:38.000 So that's the problems that we had. 00:38:38.000 --> 00:38:41.430 Further problems we had, we had a problem in Bombay. 00:38:41.430 --> 00:38:44.010 Because the second year of the evaluation turned out to be 00:38:44.010 --> 00:38:46.600 2002 to 2003. 00:38:46.600 --> 00:38:51.790 Before that, September 2001 happened. 00:38:51.790 --> 00:38:55.340 We were walking in a quite Muslim neighborhood of Bombay. 00:38:55.340 --> 00:38:59.100 A lot of kids were named like Osama and things like that. 00:38:59.100 --> 00:39:04.850 People were pretty upset about the US at the time. 00:39:04.850 --> 00:39:10.240 And so Pratham, maybe because of us, maybe just it would 00:39:10.240 --> 00:39:13.100 have been anywhere, was somehow seen, maybe, as an 00:39:13.100 --> 00:39:14.960 American presence. 00:39:14.960 --> 00:39:17.660 So some schools said that they don't want to 00:39:17.660 --> 00:39:18.550 work with the program. 00:39:18.550 --> 00:39:21.920 So about 30% of schools said that they did not want to work 00:39:21.920 --> 00:39:22.830 with the program. 00:39:22.830 --> 00:39:27.490 Or, in another set of schools, someone was identified, but 00:39:27.490 --> 00:39:29.680 they couldn't read, themselves, so they couldn't 00:39:29.680 --> 00:39:32.310 really be entrusted to teach reading to the kids. 00:39:32.310 --> 00:39:35.410 So they also didn't do it. 00:39:35.410 --> 00:39:40.080 So can we just drop all the schools that refuse to 00:39:40.080 --> 00:39:41.410 participate from the program? 00:39:44.340 --> 00:39:46.718 Can we just drop them from the analysis? 00:39:46.718 --> 00:39:48.960 AUDIENCE: Well, that's a selective bias. 00:39:48.960 --> 00:39:51.702 There is a some reasons for which they dropped, which make 00:39:51.702 --> 00:39:54.190 them inherently different from the other schools. 00:39:54.190 --> 00:39:54.455 PROFESSOR: Exactly. 00:39:54.455 --> 00:39:59.490 So we can't do that, because that's a selection bias. 00:39:59.490 --> 00:40:02.060 Maybe these are the weaker schools who are 00:40:02.060 --> 00:40:03.030 refusing the program. 00:40:03.030 --> 00:40:05.510 Maybe they are the stronger schools, because they are 00:40:05.510 --> 00:40:06.540 independently minded. 00:40:06.540 --> 00:40:07.290 Who knows? 00:40:07.290 --> 00:40:09.870 But what is clear is that's it's certainly not random. 00:40:09.870 --> 00:40:14.390 So we lose random control comparison. 00:40:14.390 --> 00:40:16.786 So what we do? 00:40:16.786 --> 00:40:20.340 Well, the first thing you can do is to say, well, I'm going 00:40:20.340 --> 00:40:24.450 to not measure the effect of the balsakhi program. 00:40:24.450 --> 00:40:28.540 I'm going to measure the effect of my intention to 00:40:28.540 --> 00:40:30.430 treat the schools with the balsakhi program. 00:40:30.430 --> 00:40:32.830 So we called that intention to treat. 00:40:32.830 --> 00:40:36.290 What an intention to treat is a Pratham official coming to 00:40:36.290 --> 00:40:38.170 the headmaster and saying, hey, we have a 00:40:38.170 --> 00:40:40.210 balsakhi for you. 00:40:40.210 --> 00:40:44.810 So this, we can do in an unbiased way, right? 00:40:44.810 --> 00:40:49.660 Because if we compare all the test scores in the school 00:40:49.660 --> 00:40:53.300 where someone has come and offered the program to the 00:40:53.300 --> 00:40:55.940 test scores in the school where no one has offered the 00:40:55.940 --> 00:40:58.090 program, there is no selection here. 00:40:58.090 --> 00:41:00.470 But it's not the effect of the program. 00:41:00.470 --> 00:41:04.540 It's the effect of my attempt to provide the program. 00:41:04.540 --> 00:41:07.720 So that's why we call it an intention to treat. 00:41:07.720 --> 00:41:09.890 It's not the treatment effect. 00:41:09.890 --> 00:41:10.880 It's the wishful thinking. 00:41:10.880 --> 00:41:15.620 It's like my attempt to treat you. 00:41:15.620 --> 00:41:20.430 Now, I wrote, here, on the board, if we wanted to know 00:41:20.430 --> 00:41:25.410 the effect of the program itself not the intention to 00:41:25.410 --> 00:41:31.470 treat, we can divide the intention to treat effect by 00:41:31.470 --> 00:41:35.440 the fraction of schools who accept the program. 00:41:35.440 --> 00:41:41.340 So, for example, if 70% of schools accept the program, 00:41:41.340 --> 00:41:50.050 and we find an effect of 10 on the test scores, a gain of 10 00:41:50.050 --> 00:41:54.060 points in the intention to treat, the intention to treat 00:41:54.060 --> 00:41:55.110 is 10 points. 00:41:55.110 --> 00:41:58.450 That's due to only 3/4 of the schools. 00:41:58.450 --> 00:42:02.470 So we can blowup the estimate by dividing by 3/4, which is 00:42:02.470 --> 00:42:05.550 about multiplying by 1.25. 00:42:05.550 --> 00:42:09.530 So instead of 10, the effect is 12.5. 00:42:09.530 --> 00:42:13.070 So this is what's called a Wald estimate. 00:42:13.070 --> 00:42:17.850 When you divide your intention to treat by the fraction of 00:42:17.850 --> 00:42:21.980 the take-up of the program, which, here, we call that the 00:42:21.980 --> 00:42:23.180 first stage. 00:42:23.180 --> 00:42:26.050 So in the first stage, you offer a program, and then some 00:42:26.050 --> 00:42:27.760 fraction of people take it. 00:42:27.760 --> 00:42:30.120 In the second stage, that has an effect. 00:42:30.120 --> 00:42:33.270 And the combination of the first stage and the second 00:42:33.270 --> 00:42:36.050 stage produces the intention to treat. 00:42:36.050 --> 00:42:38.460 So now to go from the intention to treat the effect 00:42:38.460 --> 00:42:40.430 of the program, you do a scale-up by 00:42:40.430 --> 00:42:42.930 dividing by the take-up. 00:42:42.930 --> 00:42:47.050 So this is, in five sentences, you're instrumental viable 00:42:47.050 --> 00:42:52.490 estimate, 00:42:52.490 --> 00:42:56.740 We could spend about a whole semester on it. 00:42:56.740 --> 00:42:58.405 There are conditions under which this 00:42:58.405 --> 00:42:59.310 is valid to do this. 00:42:59.310 --> 00:43:01.080 There are conditions under which this is not valid. 00:43:01.080 --> 00:43:05.020 And this has interpretations which vary. 00:43:05.020 --> 00:43:09.230 So Josh Angrist, in the econ department, is the one who did 00:43:09.230 --> 00:43:11.130 most of the work on this. 00:43:11.130 --> 00:43:14.765 So the point it that-- but think of it in this simple 00:43:14.765 --> 00:43:18.000 way, for this kind of noncompliance problem-- 00:43:18.000 --> 00:43:21.790 is let's just normalize the estimate by 00:43:21.790 --> 00:43:23.380 dividing by the take-up. 00:43:23.380 --> 00:43:26.850 It would also work if you have some school in the control 00:43:26.850 --> 00:43:29.780 group that managed to get the balsakhi anyway. 00:43:29.780 --> 00:43:33.710 Then you could divide by the difference between the take-up 00:43:33.710 --> 00:43:36.780 in the intention to treat group and the intention not to 00:43:36.780 --> 00:43:38.050 treat group. 00:43:38.050 --> 00:43:41.080 So if 10% of schools manage to get the program in the control 00:43:41.080 --> 00:43:45.510 group, you would divide your intention to treat by 27% 00:43:45.510 --> 00:43:47.530 minus 10%, that's 65%. 00:43:47.530 --> 00:43:49.430 And you would blow it up that way. 00:43:49.430 --> 00:43:58.450 What it amounts to doing is to say, this is my estimate of 00:43:58.450 --> 00:44:06.080 the effect of the program, assuming that the only reason 00:44:06.080 --> 00:44:09.680 why they're in the intention to treat effect is because 00:44:09.680 --> 00:44:14.550 those who might try to treat are more likely to be treated. 00:44:14.550 --> 00:44:17.110 So here are the results. 00:44:17.110 --> 00:44:20.080 The results are expressed in standard deviation of the 00:44:20.080 --> 00:44:21.090 unique test scores. 00:44:21.090 --> 00:44:25.330 What this means is that we take the mean of the control 00:44:25.330 --> 00:44:28.700 group and divide by the standard deviation. 00:44:28.700 --> 00:44:31.010 So the advantage of doing that-- maybe you can see that 00:44:31.010 --> 00:44:32.910 more in recitation-- 00:44:32.910 --> 00:44:35.460 is that every education program does that. 00:44:35.460 --> 00:44:39.870 So we can compare our results to what other people did. 00:44:39.870 --> 00:44:43.270 For year one, this is the standard deviation. 00:44:43.270 --> 00:44:46.680 This is the test score in treatment and the test score 00:44:46.680 --> 00:44:47.000 in control. 00:44:47.000 --> 00:44:51.450 That's the difference, 0.17. 00:44:51.450 --> 00:44:55.520 That's the standard error of this difference below. 00:44:55.520 --> 00:44:56.930 So you divide one by the other. 00:44:56.930 --> 00:44:59.720 You get a t-stat of about 1.7. 00:44:59.720 --> 00:45:02.520 So that's significant but at a 10% level. 00:45:02.520 --> 00:45:06.680 In year two, kids progressed. 00:45:06.680 --> 00:45:09.710 And the difference is much larger. 00:45:09.710 --> 00:45:16.080 So you get an effect of 0.4 standard deviation in math and 00:45:16.080 --> 00:45:22.370 a 0.25 standard deviation in language and something similar 00:45:22.370 --> 00:45:24.550 for the standard four classes. 00:45:24.550 --> 00:45:25.860 So those are a large effect. 00:45:29.280 --> 00:45:32.520 To scale that up, one of the most famous education 00:45:32.520 --> 00:45:35.980 experiments is the Tennessee STAR class-size reduction 00:45:35.980 --> 00:45:38.380 experiment, where that reduced class-size in 00:45:38.380 --> 00:45:40.740 Tennessee from 20 to 8. 00:45:40.740 --> 00:45:44.050 It was very, very expensive program that had an effect of 00:45:44.050 --> 00:45:46.280 0.2 standard deviation. 00:45:46.280 --> 00:45:50.300 So 0.2 is considered to be a reasonably large effect. 00:45:50.300 --> 00:45:52.670 Here, in the second year of the program, we are way above 00:45:52.670 --> 00:45:54.110 that for math. 00:45:54.110 --> 00:45:54.603 Yep? 00:45:54.603 --> 00:45:57.561 AUDIENCE: How big is the standard deviation relative to 00:45:57.561 --> 00:45:59.540 the average score? 00:45:59.540 --> 00:46:01.862 PROFESSOR: So this is all normalized. 00:46:01.862 --> 00:46:06.720 So this is, in a sense, already standardized. 00:46:06.720 --> 00:46:10.310 This is already standardized to the average and to the 00:46:10.310 --> 00:46:11.047 standard deviation. 00:46:11.047 --> 00:46:12.270 AUDIENCE: But there is like a wide 00:46:12.270 --> 00:46:14.880 distribution in the classroom? 00:46:14.880 --> 00:46:16.330 PROFESSOR: You mean, is it wide? 00:46:16.330 --> 00:46:22.950 Yeah, so you can see, it's not very wide, in this case. 00:46:22.950 --> 00:46:27.860 Because the students are so weak that, I think, the 00:46:27.860 --> 00:46:30.860 distribution of test scores, in terms of points, is not 00:46:30.860 --> 00:46:35.090 that large compared to what you could find elsewhere. 00:46:35.090 --> 00:46:38.760 It's kind of a relatively tight distribution. 00:46:38.760 --> 00:46:43.660 So in terms of points, it's useful. 00:46:43.660 --> 00:46:45.080 There are other ways to look at this 00:46:45.080 --> 00:46:45.820 than standard deviation. 00:46:45.820 --> 00:46:49.110 For example, the fraction of kids who can do these kinds of 00:46:49.110 --> 00:46:50.680 things, so in percentage points. 00:46:50.680 --> 00:46:52.210 How many more kids can do division? 00:46:52.210 --> 00:46:54.810 How many more kids can do addition? 00:46:54.810 --> 00:46:56.380 Which are also useful to look at, which I 00:46:56.380 --> 00:46:57.180 don't have with me. 00:46:57.180 --> 00:46:58.430 That's a good point. 00:47:03.960 --> 00:47:07.530 So these estimates are large, but they're not very precise, 00:47:07.530 --> 00:47:09.960 because there's a lot of differences between kids. 00:47:09.960 --> 00:47:13.530 And one thing that really helps to control this noise is 00:47:13.530 --> 00:47:17.880 to control for how good where they before. 00:47:17.880 --> 00:47:19.300 Because test scores of children are 00:47:19.300 --> 00:47:21.360 extremely stable over time. 00:47:21.360 --> 00:47:25.150 The biggest predictor, sadly, of how a child does is how 00:47:25.150 --> 00:47:27.020 they did last year. 00:47:27.020 --> 00:47:30.370 And so, when we control for that, we get much more precise 00:47:30.370 --> 00:47:32.460 results, which are here. 00:47:32.460 --> 00:47:35.290 So these are a bunch of results. 00:47:35.290 --> 00:47:39.960 So this is the improvement in year 1, for both cities 00:47:39.960 --> 00:47:43.460 together, where we get an improvement of 0.2 in math, 00:47:43.460 --> 00:47:45.896 0.3 in language-- 00:47:45.896 --> 00:47:51.190 sorry, opposite, 0.2 in math in year 1, 0.3 in math in year 00:47:51.190 --> 00:47:59.350 2, for verbal, 0.7, not significant in math, for 00:47:59.350 --> 00:48:02.990 verbal, 0.7 for year 1, 0.15 for year 2. 00:48:02.990 --> 00:48:06.720 And then we can look at various aspects of it, where 00:48:06.720 --> 00:48:11.690 we get a slightly bigger effect for Vadodara and Bombay 00:48:11.690 --> 00:48:15.350 but, generally, quite similar results for math. 00:48:15.350 --> 00:48:19.600 And language has bigger effect in Vadodara than in Bombay. 00:48:19.600 --> 00:48:21.610 Bombay starts from a higher level. 00:48:21.610 --> 00:48:28.210 But across the broad, except with Bombay year two, you find 00:48:28.210 --> 00:48:33.080 pretty large effects, quite large effects. 00:48:33.080 --> 00:48:35.060 So this is very different from what we had 00:48:35.060 --> 00:48:36.530 with the test scores. 00:48:36.530 --> 00:48:39.490 Now, you might still wonder that these are for all the 00:48:39.490 --> 00:48:42.090 kids involved in the program. 00:48:42.090 --> 00:48:45.440 And you might still wonder whether the effect comes from 00:48:45.440 --> 00:48:48.220 helping the lower achieving kids, which is what you're 00:48:48.220 --> 00:48:51.290 trying to do, or it's coming from helping the higher 00:48:51.290 --> 00:48:54.330 achieving kids by removing these disturbing, lower 00:48:54.330 --> 00:48:58.910 achieving kids from their classroom for two hours a day. 00:48:58.910 --> 00:49:00.180 That would be fine. 00:49:00.180 --> 00:49:04.280 But the distributional effect would be very different. 00:49:04.280 --> 00:49:09.590 If the idea is that you just send the kids to recess, and 00:49:09.590 --> 00:49:12.060 they kind of hang-out, and learn strictly nothing, 00:49:12.060 --> 00:49:15.470 meanwhile, your 20, high achieving kids progress, that 00:49:15.470 --> 00:49:16.980 could give us those mean effects. 00:49:16.980 --> 00:49:19.290 But that is not necessarily something that we would be 00:49:19.290 --> 00:49:20.870 very excited about. 00:49:20.870 --> 00:49:24.000 So for that, what is important is then to look at what is the 00:49:24.000 --> 00:49:26.800 effect by sub-group? 00:49:26.800 --> 00:49:28.520 And what was the finding when we looked at 00:49:28.520 --> 00:49:31.401 sub-groups, who benefited? 00:49:31.401 --> 00:49:34.167 AUDIENCE: The lowest one third benefited the most. 00:49:34.167 --> 00:49:35.950 PROFESSOR: The lowest one benefited the most. 00:49:35.950 --> 00:49:39.730 And who was more likely to go to the balsakhi? 00:49:39.730 --> 00:49:42.180 Also the lowest one, fortunately. 00:49:42.180 --> 00:49:48.090 So when you look at the lowest one, you start finding a 00:49:48.090 --> 00:49:53.090 pretty large effect for the bottom third and a much weaker 00:49:53.090 --> 00:49:54.160 for the top third. 00:49:54.160 --> 00:49:58.270 For example, if you look at year 2, the effect for the 00:49:58.270 --> 00:50:00.740 bottom 1/3 is 0.5 standard deviation. 00:50:00.740 --> 00:50:02.430 That's starting to be a huge effect, 0.5 00:50:02.430 --> 00:50:04.390 standard deviation progress. 00:50:04.390 --> 00:50:07.310 0.3 for the middle one, and 0.03 for 00:50:07.310 --> 00:50:09.170 the top one, no effect. 00:50:09.170 --> 00:50:11.306 And this is your chance to go to the balsakhi. 00:50:11.306 --> 00:50:13.850 So your chance to go to the balsakhi 00:50:13.850 --> 00:50:15.240 declines with the group. 00:50:18.810 --> 00:50:22.340 So the point estimate of the effect declines at about the 00:50:22.340 --> 00:50:25.770 same rate as the point estimate for the balsakhi. 00:50:25.770 --> 00:50:28.880 So the next thing we did is a little bit the same exercise 00:50:28.880 --> 00:50:34.460 as adjusting the Bombay estimate, to go from this 00:50:34.460 --> 00:50:36.910 estimate of the effect of putting a balsakhi in the 00:50:36.910 --> 00:50:40.470 classroom, to are you going to the balsakhi? 00:50:40.470 --> 00:50:45.350 So basically, the effect, this 0.5 standard deviation, is due 00:50:45.350 --> 00:50:47.610 to about 20% of kids who are actually 00:50:47.610 --> 00:50:49.210 going to the balsakhi. 00:50:49.210 --> 00:50:51.790 Because there doesn't seem to be an effect for the other 00:50:51.790 --> 00:50:55.340 children, which we see from looking at the pattern 00:50:55.340 --> 00:50:58.870 declining as the pattern of take-up declines. 00:50:58.870 --> 00:51:02.460 So now you can divide this 0.5 by 0.2. 00:51:02.460 --> 00:51:04.760 And you get the effect for people who go to the balsakhi 00:51:04.760 --> 00:51:08.970 of one standard deviation, which are very, very, very 00:51:08.970 --> 00:51:14.530 large effects, in the education literature. 00:51:14.530 --> 00:51:17.350 And if you divide 0.32 by 0.16, you'll 00:51:17.350 --> 00:51:19.470 also find about 1. 00:51:19.470 --> 00:51:21.070 And this is, of course, unsignificant. 00:51:21.070 --> 00:51:24.520 But if you divided 0.4 by 0.06, taking the estimate 00:51:24.520 --> 00:51:29.830 seriously, again, you would find the same type of effect. 00:51:29.830 --> 00:51:34.570 So this is a program that was highly effective, for kids who 00:51:34.570 --> 00:51:39.580 were sent to the program, but had no effect on the kids who, 00:51:39.580 --> 00:51:42.155 in principle, should have also benefited from the reduction 00:51:42.155 --> 00:51:44.240 in class size. 00:51:44.240 --> 00:51:45.490 So what did we learn? 00:51:48.220 --> 00:51:53.630 We learned that it is possible to make a lot of progress, 00:51:53.630 --> 00:51:58.490 with a grade 10 educated woman trained for two weeks. 00:51:58.490 --> 00:52:01.340 And we also learned that teachers are not very good at 00:52:01.340 --> 00:52:03.710 exploiting freed-up resources. 00:52:03.710 --> 00:52:04.698 Yep? 00:52:04.698 --> 00:52:08.156 AUDIENCE: How much variability was there between the 00:52:08.156 --> 00:52:09.406 different thirds? 00:52:13.096 --> 00:52:16.260 If the top third isn't that much better off than then the 00:52:16.260 --> 00:52:19.185 bottom third, then that would say something different than, 00:52:19.185 --> 00:52:21.390 if the top third is very well educated whereas the bottom 00:52:21.390 --> 00:52:23.850 third is really not. 00:52:23.850 --> 00:52:25.430 PROFESSOR: So the bottom third, basically, knows 00:52:25.430 --> 00:52:30.190 nothing at all, for starters. 00:52:30.190 --> 00:52:31.900 Like they can't do anything at the beginning. 00:52:31.900 --> 00:52:34.350 They can't recognize letters, can't recognize numbers. 00:52:34.350 --> 00:52:37.555 Whereas the top third is not at grade level but can at 00:52:37.555 --> 00:52:39.000 least do something. 00:52:39.000 --> 00:52:42.250 So what I'm saying is not that we shouldn't care 00:52:42.250 --> 00:52:43.050 about the top third. 00:52:43.050 --> 00:52:44.620 We should care about them. 00:52:44.620 --> 00:52:46.990 But that program was really targeted toward the bottom. 00:52:46.990 --> 00:52:49.380 And so the big difference, in the context of this program, 00:52:49.380 --> 00:52:51.890 is that the top third didn't get the benefit from the 00:52:51.890 --> 00:52:55.690 program, because they weren't sent to the program. 00:52:55.690 --> 00:52:58.930 Except that there could have been indirect benefits from 00:52:58.930 --> 00:53:00.690 the fact that the teacher's now have a much 00:53:00.690 --> 00:53:01.960 smaller class size. 00:53:01.960 --> 00:53:06.140 Instead of 40 kids of very heterogeneous levels, they now 00:53:06.140 --> 00:53:09.100 have 20 of better level. 00:53:09.100 --> 00:53:11.360 So they could have adjusted their teaching to do better by 00:53:11.360 --> 00:53:12.140 these kids. 00:53:12.140 --> 00:53:14.060 And it seems that they didn't. 00:53:14.060 --> 00:53:16.750 So in terms of the whether this is an optimistic or 00:53:16.750 --> 00:53:20.640 pessimistic conclusion, it's kind of the glass is half-full 00:53:20.640 --> 00:53:21.500 or half-empty. 00:53:21.500 --> 00:53:24.120 The half-full part is you can do this. 00:53:24.120 --> 00:53:27.410 It's reasonably easy to make a lot of progress. 00:53:27.410 --> 00:53:29.860 Perhaps because kids start from such a low level, to go 00:53:29.860 --> 00:53:32.820 back to your question, initially. 00:53:32.820 --> 00:53:35.965 If the standard deviation is low, because everyone is at 0, 00:53:35.965 --> 00:53:37.950 then it's easy to make some progress. 00:53:37.950 --> 00:53:40.990 And you're going to see it very quickly. 00:53:40.990 --> 00:53:44.310 On the other hand, the teachers seem to not really be 00:53:44.310 --> 00:53:47.230 using the resources in a way that allows them to take 00:53:47.230 --> 00:53:49.010 advantage of this. 00:53:49.010 --> 00:53:51.280 So that's where we were at the end of this. 00:53:51.280 --> 00:53:53.570 And we had a bunch more questions that 00:53:53.570 --> 00:53:55.520 we wanted to ask. 00:53:55.520 --> 00:53:57.450 What do we still need to know? 00:53:57.450 --> 00:54:00.230 So suppose we would want to go from this program to, say, 00:54:00.230 --> 00:54:03.490 well, let's do a policy for all of India. 00:54:03.490 --> 00:54:05.530 What else do we need to know before we move further? 00:54:05.530 --> 00:54:06.446 Yeah? 00:54:06.446 --> 00:54:10.692 AUDIENCE: The thing, for me, is to understand what is the 00:54:10.692 --> 00:54:15.400 relation between attendance and how much those kids, those 00:54:15.400 --> 00:54:17.280 programs are actually effective? 00:54:17.280 --> 00:54:20.800 Because, I mean, you showed that enrollment, as a whole, 00:54:20.800 --> 00:54:22.380 doesn't have too much of an influence. 00:54:22.380 --> 00:54:24.140 But maybe attendance would help. 00:54:24.140 --> 00:54:24.510 PROFESSOR: Right. 00:54:24.510 --> 00:54:26.510 So we could say, well, maybe there are other things to look 00:54:26.510 --> 00:54:28.270 at, children's attendance, teacher's attendance. 00:54:28.270 --> 00:54:29.995 Maybe the big difference with the balsakhi is 00:54:29.995 --> 00:54:31.710 that they were there. 00:54:31.710 --> 00:54:34.040 Maybe that's just it. 00:54:34.040 --> 00:54:37.450 Can we change their learning just with incentives? 00:54:37.450 --> 00:54:39.900 AUDIENCE: Also, maybe look at the curriculum. 00:54:39.900 --> 00:54:41.696 Maybe the reason why the children are learning is 00:54:41.696 --> 00:54:44.310 because the material is more basic, whereas if you just 00:54:44.310 --> 00:54:46.760 stay in the classroom, the material is already too hard, 00:54:46.760 --> 00:54:47.760 and you're not able to do it. 00:54:47.760 --> 00:54:50.340 PROFESSOR: That's the other big contender. 00:54:50.340 --> 00:54:52.390 So you have both contenders. 00:54:52.390 --> 00:54:54.940 One is, OK, the balsakhi are actually there. 00:54:54.940 --> 00:54:56.100 The kids are going. 00:54:56.100 --> 00:54:57.200 That's why they are learning. 00:54:57.200 --> 00:55:00.245 The second is that's not-- the incentive to this is really 00:55:00.245 --> 00:55:03.730 the pedagogy, which is about the learning material. 00:55:03.730 --> 00:55:06.690 And if we could train the teachers to train at that 00:55:06.690 --> 00:55:08.320 level, we would have the same effects. 00:55:08.320 --> 00:55:11.266 AUDIENCE: The cost effectiveness of the program, 00:55:11.266 --> 00:55:13.230 [INAUDIBLE]. 00:55:13.230 --> 00:55:15.280 PROFESSOR: Right, so we could say, what's the cost 00:55:15.280 --> 00:55:17.050 effectiveness in the city? 00:55:17.050 --> 00:55:21.445 We could look at the cost effectiveness elsewhere. 00:55:21.445 --> 00:55:24.180 AUDIENCE: Also, it could be interesting to see the effects 00:55:24.180 --> 00:55:30.510 of having more strict, I guess, passing policy. 00:55:30.510 --> 00:55:33.840 Because a lot of kids are advancing to the next grade 00:55:33.840 --> 00:55:35.300 without actually being prepared. 00:55:35.300 --> 00:55:38.030 And maybe making that a little bit more strict, so that they 00:55:38.030 --> 00:55:41.380 don't go to next level until they're actually prepared. 00:55:41.380 --> 00:55:42.750 That could improve. 00:55:42.750 --> 00:55:42.870 PROFESSOR: Yeah. 00:55:42.870 --> 00:55:44.270 It's a very good point. 00:55:44.270 --> 00:55:47.260 In fact, the policy is not strict at all at the moment. 00:55:47.260 --> 00:55:50.900 Rukmini says it the short video, everybody passes, 00:55:50.900 --> 00:55:52.280 regardless. 00:55:52.280 --> 00:55:55.390 So maybe having something where, actually, repeating is 00:55:55.390 --> 00:55:57.740 allowed or remedial education of some kind is 00:55:57.740 --> 00:56:01.831 provided would help. 00:56:01.831 --> 00:56:05.200 The other question one might ask is, is it only an urban 00:56:05.200 --> 00:56:09.780 phenomenon or would it also work in rural schools? 00:56:09.780 --> 00:56:12.190 Do you need to pay the people or could you have volunteers? 00:56:12.190 --> 00:56:15.330 Would it be sufficient to distribute materials, if it's 00:56:15.330 --> 00:56:16.450 a material question? 00:56:16.450 --> 00:56:20.180 Would you be able to motivate the teachers to do it, to 00:56:20.180 --> 00:56:21.905 focus on the students? 00:56:26.730 --> 00:56:30.630 Can you concentrate even more on the basics and make even 00:56:30.630 --> 00:56:33.370 more progress on the basics, by not trying to do remedial 00:56:33.370 --> 00:56:36.250 of everything, but just focusing on learning? 00:56:36.250 --> 00:56:38.160 So those were the questions that Pratham had. 00:56:38.160 --> 00:56:39.430 Those were the questions we had. 00:56:42.560 --> 00:56:45.510 Another question you could ask is, on the basics, you could 00:56:45.510 --> 00:56:48.740 ask the opposite question. 00:56:48.740 --> 00:56:50.960 You were saying that the students at the top, it's not 00:56:50.960 --> 00:56:55.470 that they are exactly ready to come to MIT. 00:56:55.470 --> 00:56:57.480 They are still way below grade level. 00:56:57.480 --> 00:56:59.770 Is there something that can be done for them? 00:56:59.770 --> 00:57:03.190 Or does this very simple pedagogy that Pratham has work 00:57:03.190 --> 00:57:06.020 only for to very low achieving kids? 00:57:06.020 --> 00:57:07.630 So we had all these questions. 00:57:07.630 --> 00:57:08.590 But it was many years ago. 00:57:08.590 --> 00:57:09.800 We had a lot of time. 00:57:09.800 --> 00:57:13.990 So we started looking at them. 00:57:13.990 --> 00:57:20.240 So the first thing we did was a new evaluation in Jaunpur, 00:57:20.240 --> 00:57:21.490 in Uttar Pradesh. 00:57:24.480 --> 00:57:26.450 In the first program, Pratham was running this program. 00:57:26.450 --> 00:57:28.150 And they called us to evaluate it. 00:57:28.150 --> 00:57:30.970 And we went and evaluated what it is they were doing. 00:57:30.970 --> 00:57:34.010 But with the Jaunpur program, we were working together. 00:57:34.010 --> 00:57:38.320 So this study is, actually, authored by Rukmini Banerji, 00:57:38.320 --> 00:57:39.480 along with the rest of us. 00:57:39.480 --> 00:57:42.780 We were now all working together to try to figure out, 00:57:42.780 --> 00:57:44.250 can we learn more what's going on? 00:57:48.290 --> 00:57:51.090 So Pratham renamed the balsakhi program as the Read 00:57:51.090 --> 00:57:52.440 India Program. 00:57:52.440 --> 00:57:56.410 And as they renamed it, they shifted to focus it on 00:57:56.410 --> 00:58:00.850 something even more basic and simple, on reading, and tried 00:58:00.850 --> 00:58:05.330 to spread out much more everywhere, not focusing on 00:58:05.330 --> 00:58:07.990 cities but also working in rural areas and working on a 00:58:07.990 --> 00:58:09.400 much larger scale. 00:58:09.400 --> 00:58:13.370 So we worked in rural Uttar Pradesh. 00:58:13.370 --> 00:58:16.470 We had three groups of villages, here, in Jaunpur. 00:58:16.470 --> 00:58:20.040 One group, where we just went to see all the parents, had 00:58:20.040 --> 00:58:21.890 village meetings, the women and Pratham. 00:58:21.890 --> 00:58:24.570 We went to see the parents and say, you know what? 00:58:24.570 --> 00:58:26.030 There are things you can do. 00:58:26.030 --> 00:58:26.940 You can advocate. 00:58:26.940 --> 00:58:30.520 We can lobby for more resources for your schools. 00:58:30.520 --> 00:58:32.940 You can get an extra teacher that the government will have 00:58:32.940 --> 00:58:34.060 to pay for. 00:58:34.060 --> 00:58:36.850 You can get scholarships, et cetera. 00:58:36.850 --> 00:58:41.360 That's the first treatment, to see whether parents would be 00:58:41.360 --> 00:58:43.770 able to engage with the system. 00:58:43.770 --> 00:58:47.460 The second thing we had is we realized that parents were 00:58:47.460 --> 00:58:51.380 overestimating how much their kids knew. 00:58:51.380 --> 00:58:54.730 So a first step, that Pratham has discovered, to get people 00:58:54.730 --> 00:58:59.180 excited about reading, is to train parents to administer 00:58:59.180 --> 00:59:01.210 the small Pratham test. 00:59:01.210 --> 00:59:03.820 So you go and say, Ben, read this. 00:59:03.820 --> 00:59:05.820 And Ben is like staring blankly. 00:59:05.820 --> 00:59:08.750 And then they realize, oh, my god, the kids can't read. 00:59:08.750 --> 00:59:11.780 And they have been giving their kids to the schools, 00:59:11.780 --> 00:59:16.300 faithfully, for years, and assuming that something would 00:59:16.300 --> 00:59:16.930 come out of it. 00:59:16.930 --> 00:59:18.930 And then they realize in this exercise that 00:59:18.930 --> 00:59:19.990 actually, not really. 00:59:19.990 --> 00:59:22.290 So the parents prepare a report card for the village. 00:59:22.290 --> 00:59:24.550 And then there was a lot of discussion. 00:59:24.550 --> 00:59:26.500 And the third was the Read India volunteer. 00:59:26.500 --> 00:59:30.730 At the end of this process, Pratham asked anybody, are 00:59:30.730 --> 00:59:32.570 their volunteers to learn Read India? 00:59:32.570 --> 00:59:34.590 In this case, they were not paid. 00:59:34.590 --> 00:59:37.790 People just came up, boys and girls, usually 00:59:37.790 --> 00:59:39.430 youngish boys and girls. 00:59:39.430 --> 00:59:42.410 They got trained by Pratham and stared running these camps 00:59:42.410 --> 00:59:44.432 for the students, Read India camps. 00:59:44.432 --> 00:59:46.740 And what did we find? 00:59:46.740 --> 00:59:52.510 That's kind of the result in one graph, kids who, at 00:59:52.510 --> 00:59:55.020 baseline, could not read anything. 00:59:55.020 --> 00:59:57.540 And this their result at end-line. 00:59:57.540 --> 00:59:59.755 These three lines look the same, control, information 00:59:59.755 --> 01:00:04.060 only, information plus test provided no significant 01:00:04.060 --> 01:00:04.970 difference. 01:00:04.970 --> 01:00:07.430 But there's a somewhat bigger jump for 01:00:07.430 --> 01:00:09.170 the Read India program. 01:00:09.170 --> 01:00:11.330 That jump, you might think, is really not that big. 01:00:11.330 --> 01:00:12.970 So first, I have to tell you, it's statistically 01:00:12.970 --> 01:00:14.140 significant. 01:00:14.140 --> 01:00:15.980 This is different than that. 01:00:15.980 --> 01:00:20.890 But second is, this jump is only due to 13% of kids who 01:00:20.890 --> 01:00:22.040 actually showed up. 01:00:22.040 --> 01:00:24.840 So only 13% of kids who couldn't read went to the 01:00:24.840 --> 01:00:26.230 reading camp. 01:00:26.230 --> 01:00:28.870 So we can do the same exercise, the same Josh 01:00:28.870 --> 01:00:31.660 Angrist exercise that we did for Bombay or we did for 01:00:31.660 --> 01:00:34.350 looking at the effect of the balsakhi, to look at the 01:00:34.350 --> 01:00:36.060 effect of Read India. 01:00:36.060 --> 01:00:39.120 So we divide this little bar by 13%. 01:00:39.120 --> 01:00:42.570 What is it going to do to my little bar. 01:00:42.570 --> 01:00:44.580 It's going to make it look much bigger. 01:00:44.580 --> 01:00:46.236 So this is what we have here. 01:00:48.770 --> 01:00:50.760 And that's what we find. 01:00:50.760 --> 01:00:54.030 So these are kids who couldn't read letters at baseline. 01:00:54.030 --> 01:00:59.630 And now I'm adding to the control group the little bar 01:00:59.630 --> 01:01:01.220 that have now become bigger. 01:01:01.220 --> 01:01:04.040 And we get to exactly 100%. 01:01:04.040 --> 01:01:08.240 So this suggests that 100% of the kids who actually attended 01:01:08.240 --> 01:01:12.060 the camp are able to read letters at end-line. 01:01:12.060 --> 01:01:14.340 So this is a program which is trying to 01:01:14.340 --> 01:01:15.860 get the kids to read. 01:01:15.860 --> 01:01:18.505 And it doesn't get them to read full paragraphs, but it 01:01:18.505 --> 01:01:20.700 put them one level up. 01:01:20.700 --> 01:01:22.320 And you can do the same exercise. 01:01:22.320 --> 01:01:26.650 Kids who were at letter levels are able to read paragraphs 01:01:26.650 --> 01:01:28.660 and kids who were at paragraph level are 01:01:28.660 --> 01:01:30.430 able to read stories. 01:01:30.430 --> 01:01:32.850 So this is a program that, as a program, is 01:01:32.850 --> 01:01:34.760 tremendously effective. 01:01:34.760 --> 01:01:41.650 But there were still some issues, which is that very few 01:01:41.650 --> 01:01:42.880 kids attended the camp. 01:01:42.880 --> 01:01:46.490 Why did only 13% of kids attend the camp? 01:01:46.490 --> 01:01:50.590 And that is something that was a puzzle for us. 01:01:50.590 --> 01:01:52.850 Next time, we'll spend much more time on why did only 13% 01:01:52.850 --> 01:01:55.850 of kids come to the camp. 01:01:55.850 --> 01:01:57.470 Therefore, the overall effect was low. 01:01:57.470 --> 01:01:59.810 Next time, we'll try to understand that. 01:01:59.810 --> 01:02:03.300 But today, I'll tell you what happened when we saw this 01:02:03.300 --> 01:02:05.970 result, which is we saw an effective pedagogy, where, if 01:02:05.970 --> 01:02:10.280 I managed to grab Ben by the collar and try to inculcate 01:02:10.280 --> 01:02:13.560 him how to read with Pratham technique, I can do it. 01:02:13.560 --> 01:02:14.810 He can learn letters. 01:02:18.460 --> 01:02:21.160 But on the other hand, if it's left to volunteers and left to 01:02:21.160 --> 01:02:24.240 the effort of the parents, we only managed to 01:02:24.240 --> 01:02:26.200 get 13% of the kids. 01:02:26.200 --> 01:02:29.040 So the next step was is it possible to integrate this 01:02:29.040 --> 01:02:30.850 within the school system? 01:02:30.850 --> 01:02:34.250 Because enrollment is already high. 01:02:34.250 --> 01:02:36.700 Students are already a captive audience in the school. 01:02:36.700 --> 01:02:39.550 If we could use the teachers to do this, then that would 01:02:39.550 --> 01:02:41.500 work better. 01:02:41.500 --> 01:02:43.280 So off we went. 01:02:43.280 --> 01:02:47.130 Pratham got a bunch of money, by the Gates and Hewlett 01:02:47.130 --> 01:02:49.490 Foundations, to expand the program 01:02:49.490 --> 01:02:52.340 in about 100 districts. 01:02:52.340 --> 01:02:55.490 That's how they reach 38 million kids today. 01:02:55.490 --> 01:02:57.120 But they felt that they should try and 01:02:57.120 --> 01:02:59.290 work through the states. 01:02:59.290 --> 01:03:02.570 And so here the trade-off was, well, if it does work, if we 01:03:02.570 --> 01:03:05.590 can make it work with regular teachers, then we will have 01:03:05.590 --> 01:03:07.590 many more kids reached. 01:03:07.590 --> 01:03:08.590 So that's the advantage. 01:03:08.590 --> 01:03:11.390 The 13% will become 80%. 01:03:11.390 --> 01:03:14.760 On the other hand, the treatment effect might go from 01:03:14.760 --> 01:03:18.020 this very large treatment effect we have to a much lower 01:03:18.020 --> 01:03:22.000 treatment effect if the teachers are not willing or 01:03:22.000 --> 01:03:23.660 able to carry out the program. 01:03:23.660 --> 01:03:25.710 So that's the question. 01:03:25.710 --> 01:03:28.900 So the next experiment, we worked in two states, Bihar 01:03:28.900 --> 01:03:31.080 and Uttarakhand. 01:03:31.080 --> 01:03:33.700 In the meantime, UP got cut in two. 01:03:33.700 --> 01:03:39.170 And Uttarakhand is one of the big, mountainous states, a 01:03:39.170 --> 01:03:42.310 former part of UP, very beautiful place with 01:03:42.310 --> 01:03:45.240 mountains and stuff. 01:03:45.240 --> 01:03:48.170 So in Bihar, we contrasted four models in different 01:03:48.170 --> 01:03:52.620 state, a summer camp taught by government teachers, trained 01:03:52.620 --> 01:03:55.530 teachers to implement Pratham as part of their regular 01:03:55.530 --> 01:03:58.970 teaching, trained volunteers roughly like the Jaunpur 01:03:58.970 --> 01:04:01.880 model, and distribute only the material. 01:04:01.880 --> 01:04:05.200 Can you just distribute the material, the textbooks, and 01:04:05.200 --> 01:04:06.200 get an effect from that? 01:04:06.200 --> 01:04:08.400 That would be the best, because that's very cheap. 01:04:08.400 --> 01:04:11.360 In Uttarakhand, we tried this same teacher training as in 01:04:11.360 --> 01:04:13.530 Bihar and the volunteers. 01:04:13.530 --> 01:04:17.160 But the volunteers were put in the school, with the idea 01:04:17.160 --> 01:04:20.600 that, maybe, in this way, the kids would come. 01:04:20.600 --> 01:04:23.620 And what did we find? 01:04:23.620 --> 01:04:27.310 What we found is that, again, when you have the volunteer 01:04:27.310 --> 01:04:30.840 intervention, that again works extremely well, with effects 01:04:30.840 --> 01:04:35.180 similar to what we had in Jaunpur. 01:04:35.180 --> 01:04:37.500 So what we found in Jaunpur, we find the 01:04:37.500 --> 01:04:38.610 same thing in Bihar. 01:04:38.610 --> 01:04:41.900 It worked in the same way, with not very many kids going, 01:04:41.900 --> 01:04:45.620 but the kids who went really benefited. 01:04:45.620 --> 01:04:49.150 And what is interesting here is that it's not only basic 01:04:49.150 --> 01:04:50.550 reading level any more. 01:04:50.550 --> 01:04:52.950 It also covers more advanced skills. 01:04:52.950 --> 01:04:56.210 And the gains were felt at all levels. 01:04:56.210 --> 01:05:00.160 So the same approach of trying to teach what the kids don't 01:05:00.160 --> 01:05:02.250 know, it works at reading, but it also works at 01:05:02.250 --> 01:05:03.600 more advanced levels. 01:05:03.600 --> 01:05:04.100 So great. 01:05:04.100 --> 01:05:05.520 We had a big effect on learning. 01:05:05.520 --> 01:05:08.160 We were happy. 01:05:08.160 --> 01:05:14.290 However, the teacher intervention had very, very 01:05:14.290 --> 01:05:15.520 little effect. 01:05:15.520 --> 01:05:18.690 So training the teachers had very, very little effect. 01:05:18.690 --> 01:05:21.620 There is one effect on Hindi written tests. 01:05:21.620 --> 01:05:24.200 So if you squint, you find a little bit of an effect of the 01:05:24.200 --> 01:05:26.880 teachers but, basically, not much. 01:05:26.880 --> 01:05:30.690 And moreover, when you put the volunteers in the schools, the 01:05:30.690 --> 01:05:32.300 volunteers had no effect. 01:05:32.300 --> 01:05:34.855 In fact, what we found is when you put the volunteers in 01:05:34.855 --> 01:05:36.930 schools, the teacher is absent more. 01:05:36.930 --> 01:05:39.180 So they just stay home more. 01:05:39.180 --> 01:05:41.150 And the volunteers start teaching the kids. 01:05:41.150 --> 01:05:45.270 And then it has no impact. 01:05:45.270 --> 01:05:48.300 So what do we conclude? 01:05:48.300 --> 01:05:50.980 Is it that the teachers are just horrible, and they can 01:05:50.980 --> 01:05:53.950 absolutely, never teach anything to the kids? 01:05:53.950 --> 01:05:56.040 Well, we don't think that's the case, because of the 01:05:56.040 --> 01:05:58.995 summer camp. 01:05:58.995 --> 01:06:01.340 In the summer camp program, it's the teachers who were 01:06:01.340 --> 01:06:03.360 trained to teach. 01:06:03.360 --> 01:06:06.380 A teacher had to volunteer, and they were paid extra to 01:06:06.380 --> 01:06:07.490 teach summer camps. 01:06:07.490 --> 01:06:09.010 But they were still teachers. 01:06:09.010 --> 01:06:09.710 And they were trained. 01:06:09.710 --> 01:06:10.750 And they did it. 01:06:10.750 --> 01:06:13.270 And the summer camp effect was as large as 01:06:13.270 --> 01:06:14.720 the volunteer treatment. 01:06:14.720 --> 01:06:17.510 It means that teachers are able to teach kids to read if 01:06:17.510 --> 01:06:19.110 they want to. 01:06:19.110 --> 01:06:22.250 So the problem is that it's not that they can't. 01:06:22.250 --> 01:06:26.330 It's that usually they choose not to. 01:06:26.330 --> 01:06:28.710 And why do they choose not to? 01:06:28.710 --> 01:06:31.010 That's what we are going to see next time. 01:06:31.010 --> 01:06:33.060 Where are we? 01:06:33.060 --> 01:06:36.600 We know that we can improve the quality of education. 01:06:36.600 --> 01:06:39.620 Because you can take a high school graduate, train them 01:06:39.620 --> 01:06:42.850 for two weeks, and they can do it. 01:06:42.850 --> 01:06:46.470 And the puzzle is that, why is it not taken up more? 01:06:46.470 --> 01:06:48.670 Why is it not taken up more by the school system? 01:06:48.670 --> 01:06:51.140 When the teacher's are trained, it doesn't work. 01:06:51.140 --> 01:06:53.990 Why is not taken more by teachers themselves? 01:06:53.990 --> 01:06:59.160 Why is it not taken more by parents, who are not sending 01:06:59.160 --> 01:07:01.060 their kids to the reading camp when they have a chance? 01:07:01.060 --> 01:07:03.800 I mean some do but not that many. 01:07:03.800 --> 01:07:05.790 And finally, they are private schools. 01:07:05.790 --> 01:07:07.650 A lot of these kids are going to private schools. 01:07:07.650 --> 01:07:09.940 The private schools are somewhat more effective than 01:07:09.940 --> 01:07:12.760 the public schools but not tremendously more. 01:07:12.760 --> 01:07:14.760 So why aren't the private schools using those 01:07:14.760 --> 01:07:15.120 techniques? 01:07:15.120 --> 01:07:16.960 After all, nothing stops them. 01:07:16.960 --> 01:07:19.810 They can fire the teacher if they don't do what they do. 01:07:19.810 --> 01:07:21.600 They are more flexible. 01:07:21.600 --> 01:07:24.960 So why aren't the private school adopting that? 01:07:24.960 --> 01:07:26.960 So that's kind of where we are, which is there is an 01:07:26.960 --> 01:07:29.580 effective methodology, of very effective, you know. 01:07:29.580 --> 01:07:33.090 You can get everybody to read at least one more level, from 01:07:33.090 --> 01:07:35.190 where they were, in three months of a volunteer. 01:07:35.190 --> 01:07:38.580 We can increase their skills by one standard deviation. 01:07:38.580 --> 01:07:40.340 And yet this is not taken up. 01:07:40.340 --> 01:07:42.530 So this is not the technology. 01:07:42.530 --> 01:07:44.660 The technology exists. 01:07:44.660 --> 01:07:48.515 But it's not adopted, neither at the individual level nor at 01:07:48.515 --> 01:07:49.560 the system level. 01:07:49.560 --> 01:07:51.550 So we'll see where we are next time. 01:07:51.550 --> 01:07:52.740 And we'll conclude with you. 01:07:52.740 --> 01:07:53.990 AUDIENCE: How would you [INAUDIBLE]? 01:07:57.436 --> 01:07:59.570 PROFESSOR: Teachers are very well educated. 01:07:59.570 --> 01:08:05.320 Teachers have a BA in teaching plus a discipline. 01:08:05.320 --> 01:08:07.830 And it's a pretty competitive job to get. 01:08:07.830 --> 01:08:09.560 And it's very well paid. 01:08:09.560 --> 01:08:11.860 But it's very well paid at entry, and then you don't 01:08:11.860 --> 01:08:12.700 increase much. 01:08:12.700 --> 01:08:15.630 And you can never, ever be fired. 01:08:15.630 --> 01:08:16.880 So it's a very desirable job.