WEBVTT 00:00:00.000 --> 00:00:01.984 align:middle line:90% [SQUEAKING] 00:00:01.984 --> 00:00:03.968 align:middle line:90% [RUSTLING] 00:00:03.968 --> 00:00:06.944 align:middle line:90% [CLICKING] 00:00:06.944 --> 00:00:18.858 align:middle line:90% 00:00:18.858 --> 00:00:20.650 align:middle line:84% ESTHER DUFLO: So where we left it last time 00:00:20.650 --> 00:00:24.340 align:middle line:84% is that we found some theoretical justification 00:00:24.340 --> 00:00:27.670 align:middle line:84% for why people might be able to tell you 00:00:27.670 --> 00:00:31.330 align:middle line:84% who the influential people in a network are, 00:00:31.330 --> 00:00:34.810 align:middle line:84% even if they have no idea really of the shape of the network. 00:00:34.810 --> 00:00:37.990 align:middle line:84% Simply because it happens to be that there 00:00:37.990 --> 00:00:41.830 align:middle line:84% is a natural link between how many times we hear 00:00:41.830 --> 00:00:46.450 align:middle line:84% about someone and the ability from any message originating 00:00:46.450 --> 00:00:49.258 align:middle line:84% from that person to reach a lot of people. 00:00:49.258 --> 00:00:50.800 align:middle line:84% So although it's not the same object, 00:00:50.800 --> 00:00:54.580 align:middle line:84% they are related and potentially quite closely related 00:00:54.580 --> 00:00:57.370 align:middle line:84% if they are-- once we are starting to look at processes 00:00:57.370 --> 00:01:00.560 align:middle line:90% that continue for a long time. 00:01:00.560 --> 00:01:05.470 align:middle line:84% So the faster, now, let's put that in a context 00:01:05.470 --> 00:01:08.300 align:middle line:90% of thinking about immunization. 00:01:08.300 --> 00:01:12.630 align:middle line:84% So that was in fact only just the second time we tested it. 00:01:12.630 --> 00:01:14.990 align:middle line:84% We tested this idea for immunization 00:01:14.990 --> 00:01:18.950 align:middle line:84% was so in a large scale project undertaken in collaboration 00:01:18.950 --> 00:01:21.680 align:middle line:84% with the government of Haryana in India. 00:01:21.680 --> 00:01:24.350 align:middle line:84% And the objective was to increase for immunization 00:01:24.350 --> 00:01:26.750 align:middle line:84% in a context where immunization rate was low 00:01:26.750 --> 00:01:28.400 align:middle line:90% and the demand was quite good. 00:01:28.400 --> 00:01:29.660 align:middle line:90% The supply was quite good. 00:01:29.660 --> 00:01:34.580 align:middle line:84% They invested in a lot of infrastructure to make it easy. 00:01:34.580 --> 00:01:38.420 align:middle line:84% Going back to the point that you made about Chile, to make 00:01:38.420 --> 00:01:40.190 align:middle line:90% it easy for people to go. 00:01:40.190 --> 00:01:43.160 align:middle line:84% And yet, it got stuck at full immunization rate 00:01:43.160 --> 00:01:48.020 align:middle line:84% in the high 40s according to the report 00:01:48.020 --> 00:01:51.620 align:middle line:84% by the parents, which are probably overestimated anyway. 00:01:51.620 --> 00:02:00.800 align:middle line:84% So what we had in our hands was this thing 00:02:00.800 --> 00:02:03.897 align:middle line:84% that maybe gossips would work, as well as 00:02:03.897 --> 00:02:05.480 align:middle line:84% the previous experiment that I already 00:02:05.480 --> 00:02:08.930 align:middle line:84% showed you on the impact of small incentives 00:02:08.930 --> 00:02:11.410 align:middle line:90% on immunization. 00:02:11.410 --> 00:02:16.360 align:middle line:84% And so we decided that we could put the two things together, as 00:02:16.360 --> 00:02:21.330 align:middle line:84% well as a third intervention, which is very frequent. 00:02:21.330 --> 00:02:24.370 align:middle line:84% And probably even for your own health, 00:02:24.370 --> 00:02:26.620 align:middle line:84% you get reminders of going to the doctor. 00:02:26.620 --> 00:02:28.930 align:middle line:84% If you've gone once to the eye doctor, 00:02:28.930 --> 00:02:30.460 align:middle line:84% you will get a reminder once a year 00:02:30.460 --> 00:02:32.860 align:middle line:84% that it is time to go to the eye doctor. 00:02:32.860 --> 00:02:36.110 align:middle line:84% So SMS reminder have also become quite popular. 00:02:36.110 --> 00:02:37.705 align:middle line:84% So we wanted to try the three things. 00:02:37.705 --> 00:02:40.480 align:middle line:90% 00:02:40.480 --> 00:02:44.370 align:middle line:84% And to do that, we first had to set up an MIS. 00:02:44.370 --> 00:02:46.920 align:middle line:84% They didn't have a way of collecting immunization 00:02:46.920 --> 00:02:47.940 align:middle line:90% in a reliable way. 00:02:47.940 --> 00:02:52.020 align:middle line:84% It was done in registries, paper registries that 00:02:52.020 --> 00:02:52.948 align:middle line:90% were not collected. 00:02:52.948 --> 00:02:54.990 align:middle line:84% So the first thing we did is to work with a group 00:02:54.990 --> 00:02:59.490 align:middle line:84% here at MIT to set up a small mHealth platform. 00:02:59.490 --> 00:03:02.370 align:middle line:84% I'm saying small in that it was very barebones. 00:03:02.370 --> 00:03:05.270 align:middle line:90% It was just for immunization. 00:03:05.270 --> 00:03:07.430 align:middle line:84% Incidentally, I thought this would be really easy. 00:03:07.430 --> 00:03:09.230 align:middle line:84% And for whatever reason, it's not. 00:03:09.230 --> 00:03:10.860 align:middle line:90% It took them a long time. 00:03:10.860 --> 00:03:12.210 align:middle line:90% And it wasn't working fast. 00:03:12.210 --> 00:03:14.660 align:middle line:84% And so it took months to develop that thing, which 00:03:14.660 --> 00:03:16.190 align:middle line:90% is really basic. 00:03:16.190 --> 00:03:22.610 align:middle line:90% It's like an Android form. 00:03:22.610 --> 00:03:24.230 align:middle line:90% There is a registration aspect. 00:03:24.230 --> 00:03:26.210 align:middle line:90% And then-- for a new kid. 00:03:26.210 --> 00:03:29.070 align:middle line:84% And then, the follow up as kids come up. 00:03:29.070 --> 00:03:32.820 align:middle line:84% For whatever reason, it's harder than you'd think. 00:03:32.820 --> 00:03:34.470 align:middle line:90% I still don't understand why. 00:03:34.470 --> 00:03:37.320 align:middle line:84% But it's worth keeping that in mind. 00:03:37.320 --> 00:03:40.170 align:middle line:84% If you want to get into any things 00:03:40.170 --> 00:03:41.692 align:middle line:84% like that involving software, it's 00:03:41.692 --> 00:03:42.900 align:middle line:90% always harder than you think. 00:03:42.900 --> 00:03:45.650 align:middle line:90% 00:03:45.650 --> 00:03:49.100 align:middle line:84% Maybe, also we wanted to work at scale. 00:03:49.100 --> 00:03:55.280 align:middle line:84% So we worked in seven districts, a little over 2,000 villages. 00:03:55.280 --> 00:03:58.160 align:middle line:84% In total, we had 300,000 unique children 00:03:58.160 --> 00:04:04.520 align:middle line:84% in our system and almost 500,000 vaccines. 00:04:04.520 --> 00:04:06.080 align:middle line:84% What I'm going to present today, it's 00:04:06.080 --> 00:04:09.290 align:middle line:84% all based on the administrative data from that system 00:04:09.290 --> 00:04:10.910 align:middle line:90% that we owned. 00:04:10.910 --> 00:04:14.870 align:middle line:84% So one issue is whether that system, that data is reliable, 00:04:14.870 --> 00:04:18.660 align:middle line:84% or whether people lie about for example, 00:04:18.660 --> 00:04:21.140 align:middle line:84% whether kids that have never been immunized that 00:04:21.140 --> 00:04:22.580 align:middle line:84% are registered as being immunized. 00:04:22.580 --> 00:04:25.190 align:middle line:84% That's particularly relevant for the incentive. 00:04:25.190 --> 00:04:28.700 align:middle line:84% But because you could imagine a nurse 00:04:28.700 --> 00:04:32.630 align:middle line:84% deciding that they are going to immunize a lot of children, 00:04:32.630 --> 00:04:34.820 align:middle line:84% giving their phone number, or their cousin's phone 00:04:34.820 --> 00:04:37.640 align:middle line:84% number, or everyone, their friend's phone number, 00:04:37.640 --> 00:04:40.460 align:middle line:84% and in this way collecting the small incentives, 00:04:40.460 --> 00:04:42.920 align:middle line:84% which were air time on the phone. 00:04:42.920 --> 00:04:52.840 align:middle line:84% So what we did there is we sent people to the field 00:04:52.840 --> 00:04:55.150 align:middle line:90% to collect-- 00:04:55.150 --> 00:05:00.790 align:middle line:84% to contact all of the families that had-- 00:05:00.790 --> 00:05:01.420 align:middle line:90% not all. 00:05:01.420 --> 00:05:03.220 align:middle line:84% Some random sample of the families 00:05:03.220 --> 00:05:06.040 align:middle line:84% that had been registered in the system. 00:05:06.040 --> 00:05:09.070 align:middle line:84% Find them, checked they existed, and then if they existed, 00:05:09.070 --> 00:05:11.410 align:middle line:84% whether they remembered going to the camp, and the kind 00:05:11.410 --> 00:05:13.530 align:middle line:90% of vaccinations they got. 00:05:13.530 --> 00:05:16.600 align:middle line:84% And the good news is that we could find everyone. 00:05:16.600 --> 00:05:19.020 align:middle line:84% Unlike in similar exercises that have 00:05:19.020 --> 00:05:22.770 align:middle line:84% been performed from the MNREGA program or welfare 00:05:22.770 --> 00:05:24.582 align:middle line:84% program where you go to the village 00:05:24.582 --> 00:05:26.790 align:middle line:84% to look for the beneficiaries of the welfare program, 00:05:26.790 --> 00:05:28.510 align:middle line:90% and you find half of them. 00:05:28.510 --> 00:05:30.090 align:middle line:90% So we find all of them. 00:05:30.090 --> 00:05:33.090 align:middle line:84% And then, when we find them, they at least 00:05:33.090 --> 00:05:34.560 align:middle line:90% remember going to the camp. 00:05:34.560 --> 00:05:37.440 align:middle line:84% Although sometimes, they don't remember which shot they got. 00:05:37.440 --> 00:05:40.090 align:middle line:84% But then, it's not clear who is wrong at this point. 00:05:40.090 --> 00:05:45.240 align:middle line:84% So our take away from that is by and large, the data is-- 00:05:45.240 --> 00:05:47.280 align:middle line:84% the administrative data is reliable enough. 00:05:47.280 --> 00:05:49.740 align:middle line:84% And that's what we are going to use as our measure. 00:05:49.740 --> 00:05:50.320 align:middle line:90% Yeah? 00:05:50.320 --> 00:05:51.150 align:middle line:90% AUDIENCE: Question. 00:05:51.150 --> 00:05:52.760 align:middle line:84% When you're running an experiment 00:05:52.760 --> 00:05:57.350 align:middle line:84% with multiple treatment ops, how do you think about 00:05:57.350 --> 00:05:59.600 align:middle line:90% how many ops to have and-- 00:05:59.600 --> 00:06:01.820 align:middle line:84% ESTHER DUFLO: It's an excellent question. 00:06:01.820 --> 00:06:03.902 align:middle line:84% And I thank you very much for asking it to me. 00:06:03.902 --> 00:06:05.360 align:middle line:84% And that's basically what I'm going 00:06:05.360 --> 00:06:07.850 align:middle line:90% to talk about in five minutes. 00:06:07.850 --> 00:06:15.720 align:middle line:84% In fact, I hope you're going to read that paper one day. 00:06:15.720 --> 00:06:16.810 align:middle line:90% This is the paper. 00:06:16.810 --> 00:06:20.490 align:middle line:84% I should have put on the syllabus for you to read now. 00:06:20.490 --> 00:06:22.242 align:middle line:84% It's asking exactly that question. 00:06:22.242 --> 00:06:24.450 align:middle line:84% Instead I put the "Gossip" paper, which is fine, too. 00:06:24.450 --> 00:06:28.530 align:middle line:84% And then, it's-- but which highlights just one part 00:06:28.530 --> 00:06:29.030 align:middle line:90% of that. 00:06:29.030 --> 00:06:32.240 align:middle line:90% And not this question about-- 00:06:32.240 --> 00:06:33.673 align:middle line:90% there are many treatments there. 00:06:33.673 --> 00:06:35.840 align:middle line:84% But let me come back to that in a minute, just close 00:06:35.840 --> 00:06:38.730 align:middle line:84% the "Gossip" part that we are done with that. 00:06:38.730 --> 00:06:42.400 align:middle line:90% So in the "Gossip" part-- 00:06:42.400 --> 00:06:45.280 align:middle line:84% so in our first gossip experiment, which in the paper 00:06:45.280 --> 00:06:46.400 align:middle line:90% you read comes second. 00:06:46.400 --> 00:06:50.470 align:middle line:84% But chronologically it was first. 00:06:50.470 --> 00:06:54.700 align:middle line:84% We just had this question of if I want to see people 00:06:54.700 --> 00:06:58.360 align:middle line:90% for fair, who should I go see? 00:06:58.360 --> 00:07:00.400 align:middle line:84% So the idea was to capture something, 00:07:00.400 --> 00:07:06.460 align:middle line:84% which is not loaded and is all about diffusion of information. 00:07:06.460 --> 00:07:09.710 align:middle line:84% And then, in that first experiment, 00:07:09.710 --> 00:07:11.540 align:middle line:84% the experiment was a bit artificial. 00:07:11.540 --> 00:07:13.240 align:middle line:84% We were trying to transfer information 00:07:13.240 --> 00:07:17.532 align:middle line:90% about a cell phone raffle. 00:07:17.532 --> 00:07:20.430 align:middle line:90% 00:07:20.430 --> 00:07:23.880 align:middle line:84% But here, when we are now in a real world setting 00:07:23.880 --> 00:07:26.710 align:middle line:84% and in particular a health setting, there might be-- 00:07:26.710 --> 00:07:29.600 align:middle line:84% there is this trust issue, which is the people whom 00:07:29.600 --> 00:07:34.040 align:middle line:84% you might trust to tell you when the next affair is 00:07:34.040 --> 00:07:37.460 align:middle line:84% or when the next cell phone raffle is might not 00:07:37.460 --> 00:07:41.580 align:middle line:84% be the one you're going to talk to about health. 00:07:41.580 --> 00:07:44.160 align:middle line:84% And although we have a theory for saying 00:07:44.160 --> 00:07:47.340 align:middle line:84% people might be good at knowing who is good at spreading 00:07:47.340 --> 00:07:50.190 align:middle line:84% information, we don't have a similar theory 00:07:50.190 --> 00:07:54.300 align:middle line:84% to tell us whether people are good at spreading information 00:07:54.300 --> 00:07:57.960 align:middle line:84% that people will believe, that people trust. 00:07:57.960 --> 00:08:02.860 align:middle line:84% So what we decided to do is to just test it. 00:08:02.860 --> 00:08:04.620 align:middle line:90% So we asked. 00:08:04.620 --> 00:08:08.640 align:middle line:84% People either were randomly at random seeds. 00:08:08.640 --> 00:08:10.590 align:middle line:84% Someone asked, by the way, in the paper 00:08:10.590 --> 00:08:12.140 align:middle line:84% should you have had random seeds? 00:08:12.140 --> 00:08:13.980 align:middle line:90% So we had random seeds. 00:08:13.980 --> 00:08:19.200 align:middle line:84% Or seeds that were where we elicited 00:08:19.200 --> 00:08:23.160 align:middle line:84% gossips, where we listed the trusted people or trusted 00:08:23.160 --> 00:08:24.510 align:middle line:90% gossips. 00:08:24.510 --> 00:08:29.970 align:middle line:84% So how do we elicit any of these people? 00:08:29.970 --> 00:08:31.140 align:middle line:90% We go and find out. 00:08:31.140 --> 00:08:33.809 align:middle line:84% Talk to 15 people, and we ask them. 00:08:33.809 --> 00:08:37.500 align:middle line:84% So for gossip, that's the question. 00:08:37.500 --> 00:08:40.230 align:middle line:84% If they share information about the music festival, street 00:08:40.230 --> 00:08:44.640 align:middle line:84% play, fair, many people would learn about it. 00:08:44.640 --> 00:08:49.140 align:middle line:84% That's because-- and then we have the theory for how, why. 00:08:49.140 --> 00:08:52.450 align:middle line:90% Could you name these people? 00:08:52.450 --> 00:08:55.510 align:middle line:84% Then for trusted people, it's who are the people 00:08:55.510 --> 00:08:58.810 align:middle line:84% in this village that you and many villagers 00:08:58.810 --> 00:09:02.080 align:middle line:84% trust within and outside this neighborhood? 00:09:02.080 --> 00:09:03.970 align:middle line:84% That means when I give advice on something, 00:09:03.970 --> 00:09:05.650 align:middle line:84% many people believe that it's correct. 00:09:05.650 --> 00:09:08.720 align:middle line:90% And they tend to follow it. 00:09:08.720 --> 00:09:11.410 align:middle line:84% So, of course, choosing the right fertilizer for the crop, 00:09:11.410 --> 00:09:13.100 align:middle line:90% and keeping your child healthy. 00:09:13.100 --> 00:09:15.160 align:middle line:84% So we specifically added the health example 00:09:15.160 --> 00:09:17.530 align:middle line:84% because we were going to transmit information 00:09:17.530 --> 00:09:20.610 align:middle line:90% about health. 00:09:20.610 --> 00:09:22.350 align:middle line:84% And then finally, trusted gossip. 00:09:22.350 --> 00:09:24.210 align:middle line:90% It's a mouthful. 00:09:24.210 --> 00:09:28.860 align:middle line:84% But many people get to know about a piece of information. 00:09:28.860 --> 00:09:32.740 align:middle line:84% And among those people, who are trusted? 00:09:32.740 --> 00:09:34.290 align:middle line:84% So you need to first be a gossip, 00:09:34.290 --> 00:09:35.535 align:middle line:90% and then also be trusted. 00:09:35.535 --> 00:09:39.970 align:middle line:90% 00:09:39.970 --> 00:09:43.840 align:middle line:84% And then, the intervention was once these people were named, 00:09:43.840 --> 00:09:48.010 align:middle line:84% so we found this result that very few people are named. 00:09:48.010 --> 00:09:51.100 align:middle line:84% And there is a lot of consensus of who they might be. 00:09:51.100 --> 00:09:52.990 align:middle line:90% We pick the five-- 00:09:52.990 --> 00:09:58.930 align:middle line:84% the common top of the 15 out of all the domination from the 15 00:09:58.930 --> 00:10:00.460 align:middle line:90% random people we interviewed. 00:10:00.460 --> 00:10:03.700 align:middle line:90% 00:10:03.700 --> 00:10:04.850 align:middle line:90% And we go for that. 00:10:04.850 --> 00:10:08.380 align:middle line:84% So someone was asking also whether it's pretty expensive 00:10:08.380 --> 00:10:10.510 align:middle line:90% to go visit 15 people. 00:10:10.510 --> 00:10:12.800 align:middle line:84% But it's not really expensive to go visit 15 people, 00:10:12.800 --> 00:10:16.030 align:middle line:84% especially if we don't really care who they are. 00:10:16.030 --> 00:10:17.980 align:middle line:90% Because it's not that we are-- 00:10:17.980 --> 00:10:21.268 align:middle line:84% under this theory, it's anybody would be fine. 00:10:21.268 --> 00:10:22.810 align:middle line:84% It doesn't need to be random, random. 00:10:22.810 --> 00:10:26.860 align:middle line:90% It could be just anyone. 00:10:26.860 --> 00:10:29.350 align:middle line:84% So it goes quite quickly to go visit 15 people. 00:10:29.350 --> 00:10:31.780 align:middle line:84% That said, we are now interested in the scale 00:10:31.780 --> 00:10:37.220 align:middle line:84% up of this program, which I'm short cutting, 00:10:37.220 --> 00:10:40.700 align:middle line:84% but which turned out to be effective and cost-effective. 00:10:40.700 --> 00:10:43.070 align:middle line:84% And now, we actually exactly stumbled 00:10:43.070 --> 00:10:44.810 align:middle line:84% upon this issue, which is for us, 00:10:44.810 --> 00:10:48.590 align:middle line:84% it's pretty easy to go see 15 people, and record the answer, 00:10:48.590 --> 00:10:50.660 align:middle line:90% and be done in our gossips. 00:10:50.660 --> 00:10:52.760 align:middle line:84% But for the government if you want 00:10:52.760 --> 00:10:56.030 align:middle line:84% to do that for every single village, it becomes a thing. 00:10:56.030 --> 00:10:57.427 align:middle line:90% So who is going to do it? 00:10:57.427 --> 00:10:58.760 align:middle line:90% And how they are going to do it? 00:10:58.760 --> 00:11:00.980 align:middle line:90% And are they going-- 00:11:00.980 --> 00:11:03.860 align:middle line:84% if we ask the frontline health worker to do it the ashes, 00:11:03.860 --> 00:11:06.320 align:middle line:84% are they just going to make up the names? 00:11:06.320 --> 00:11:09.030 align:middle line:84% And so basically, it's a huge stumbling block. 00:11:09.030 --> 00:11:13.280 align:middle line:84% So in fact, when I read your question I was like, well, 00:11:13.280 --> 00:11:14.820 align:middle line:90% it's not that expensive. 00:11:14.820 --> 00:11:20.100 align:middle line:84% But then I say, well, actually, it is logistically a nightmare. 00:11:20.100 --> 00:11:26.240 align:middle line:84% And so what we are trying to do in the scale up of that is we 00:11:26.240 --> 00:11:28.590 align:middle line:90% are not doing that anymore. 00:11:28.590 --> 00:11:33.910 align:middle line:84% We are asking the parents when they-- since now they 00:11:33.910 --> 00:11:37.210 align:middle line:84% have a little better mHealth intervention, 00:11:37.210 --> 00:11:40.900 align:middle line:84% we are asking the parents to tell us. 00:11:40.900 --> 00:11:44.320 align:middle line:84% So that's what we were hoping to do is to using-- 00:11:44.320 --> 00:11:48.700 align:middle line:84% to add a module to their registration camp 00:11:48.700 --> 00:11:51.040 align:middle line:84% when people come to say, hey, by the way, 00:11:51.040 --> 00:11:55.510 align:middle line:84% do you know someone who is the gossip? 00:11:55.510 --> 00:11:57.900 align:middle line:84% And so we thought that would be very smart. 00:11:57.900 --> 00:12:01.020 align:middle line:84% But that didn't-- so that was the idea. 00:12:01.020 --> 00:12:02.670 align:middle line:84% That it's going to be very cheap. 00:12:02.670 --> 00:12:04.830 align:middle line:84% It would in fact be very cheap, but it 00:12:04.830 --> 00:12:11.300 align:middle line:84% involves modifying the software, which turns out 00:12:11.300 --> 00:12:13.850 align:middle line:90% to be a huge nightmare. 00:12:13.850 --> 00:12:17.510 align:middle line:84% And getting the nurse to use the software 00:12:17.510 --> 00:12:20.600 align:middle line:84% and to use that modified version. 00:12:20.600 --> 00:12:27.650 align:middle line:84% So we've also been stuck on that for some month. 00:12:27.650 --> 00:12:30.120 align:middle line:90% So the current version-- 00:12:30.120 --> 00:12:32.000 align:middle line:90% so this was version 2. 00:12:32.000 --> 00:12:37.280 align:middle line:84% So the current version is that we 00:12:37.280 --> 00:12:41.930 align:middle line:84% have a large database of phone numbers 00:12:41.930 --> 00:12:44.930 align:middle line:84% because there is a huge health insurance 00:12:44.930 --> 00:12:47.780 align:middle line:84% programs where many people are in need, 00:12:47.780 --> 00:12:50.150 align:middle line:90% and they have phone numbers. 00:12:50.150 --> 00:12:54.830 align:middle line:84% And these health insurance people have a call center. 00:12:54.830 --> 00:12:55.990 align:middle line:90% They call them for bias. 00:12:55.990 --> 00:12:58.340 align:middle line:84% People call them to ask for insurance stuff, 00:12:58.340 --> 00:13:00.080 align:middle line:90% and they call people in turn. 00:13:00.080 --> 00:13:03.770 align:middle line:84% And these people in the call center can do a survey, 00:13:03.770 --> 00:13:05.960 align:middle line:84% can do a short survey to a few people 00:13:05.960 --> 00:13:08.990 align:middle line:90% in each village on the phone. 00:13:08.990 --> 00:13:12.420 align:middle line:90% So that's where we are now. 00:13:12.420 --> 00:13:15.260 align:middle line:90% That's what seems feasible. 00:13:15.260 --> 00:13:17.420 align:middle line:84% Stay tuned, I tell you because we haven't yet 00:13:17.420 --> 00:13:18.830 align:middle line:90% tried and therefore, failed. 00:13:18.830 --> 00:13:21.350 align:middle line:90% But that might be an issue. 00:13:21.350 --> 00:13:22.860 align:middle line:90% Anyway so that seems simple. 00:13:22.860 --> 00:13:24.140 align:middle line:90% But it's not that simple. 00:13:24.140 --> 00:13:29.780 align:middle line:84% But once we find them, then we go in and visit them, 00:13:29.780 --> 00:13:31.760 align:middle line:84% try to enroll them in the program, 00:13:31.760 --> 00:13:36.150 align:middle line:84% and we tell them that we will send them these messages. 00:13:36.150 --> 00:13:38.750 align:middle line:84% So we have incentive and non-incentive villages. 00:13:38.750 --> 00:13:40.370 align:middle line:84% We saw that the incentive villages, 00:13:40.370 --> 00:13:42.650 align:middle line:84% they were well-- they would be well-placed to mention 00:13:42.650 --> 00:13:44.990 align:middle line:84% that there is an incentive program. 00:13:44.990 --> 00:13:47.240 align:middle line:84% That's a concrete piece of information people probably 00:13:47.240 --> 00:13:50.510 align:middle line:84% don't know because the advertising of the incentive 00:13:50.510 --> 00:13:52.560 align:middle line:90% program is not huge. 00:13:52.560 --> 00:13:54.440 align:middle line:84% And then, in no incentive villages 00:13:54.440 --> 00:13:57.300 align:middle line:84% it's just a general immunization is good for you, 00:13:57.300 --> 00:14:01.230 align:middle line:90% go ahead and do it. 00:14:01.230 --> 00:14:04.070 align:middle line:90% So here is the basic results. 00:14:04.070 --> 00:14:08.240 align:middle line:84% I'll tell you about more results when I get to Kayla's questions 00:14:08.240 --> 00:14:10.220 align:middle line:90% in more details. 00:14:10.220 --> 00:14:12.530 align:middle line:84% But the basic result is that if we 00:14:12.530 --> 00:14:13.970 align:middle line:84% look at measles that's going to be 00:14:13.970 --> 00:14:17.750 align:middle line:84% my main outcomes because that's the last vaccination 00:14:17.750 --> 00:14:20.030 align:middle line:90% in the sequence. 00:14:20.030 --> 00:14:21.830 align:middle line:84% That's the log number of people who 00:14:21.830 --> 00:14:29.150 align:middle line:84% receive the measles vaccine in any monthly camp in a village. 00:14:29.150 --> 00:14:30.552 align:middle line:90% So here, it's in logs. 00:14:30.552 --> 00:14:32.510 align:middle line:84% Later, I'm going to show you results in levels. 00:14:32.510 --> 00:14:34.927 align:middle line:84% But for some reason in these people, we show them in logs. 00:14:34.927 --> 00:14:40.580 align:middle line:84% And it's 18% higher in places in a village that have a gossip. 00:14:40.580 --> 00:14:44.973 align:middle line:84% These two, the trusted and trusted gossip are not-- 00:14:44.973 --> 00:14:46.640 align:middle line:84% neither of them is actually significant. 00:14:46.640 --> 00:14:49.410 align:middle line:84% They're also not significantly different. 00:14:49.410 --> 00:14:51.590 align:middle line:90% So I can't say that it's worth. 00:14:51.590 --> 00:14:55.295 align:middle line:84% But what is relevant is that it's no better. 00:14:55.295 --> 00:14:57.420 align:middle line:84% And in particular, you might think that the trusted 00:14:57.420 --> 00:15:00.370 align:middle line:84% gossip would be obviously better because it's both a gossip 00:15:00.370 --> 00:15:01.920 align:middle line:84% and it's trusted, but of course, when 00:15:01.920 --> 00:15:03.600 align:middle line:84% you're trying to find a trusted gossip, 00:15:03.600 --> 00:15:06.850 align:middle line:84% you lose some on the gossip dimension. 00:15:06.850 --> 00:15:09.090 align:middle line:84% So if in fact, it turns out that you 00:15:09.090 --> 00:15:11.580 align:middle line:84% don't get the most gossipy gossip potentially 00:15:11.580 --> 00:15:13.830 align:middle line:84% because they are not necessarily the most trusted. 00:15:13.830 --> 00:15:16.620 align:middle line:84% So if in fact, the trust dimension is not that important 00:15:16.620 --> 00:15:20.280 align:middle line:84% compared to repeating the thing, then you might lose something. 00:15:20.280 --> 00:15:25.410 align:middle line:84% So we can say that they look-- they all look fairly similar. 00:15:25.410 --> 00:15:27.810 align:middle line:84% And then, these are the incentive results. 00:15:27.810 --> 00:15:31.910 align:middle line:84% But I'll come to that in a second. 00:15:31.910 --> 00:15:34.850 align:middle line:90% So putting it all together. 00:15:34.850 --> 00:15:39.940 align:middle line:84% And so that's the other paper written on the same experiment. 00:15:39.940 --> 00:15:44.050 align:middle line:84% So the first one really just was more for use this experiment 00:15:44.050 --> 00:15:46.220 align:middle line:90% just for the gossip aspect. 00:15:46.220 --> 00:15:48.250 align:middle line:90% But then, we exactly have-- 00:15:48.250 --> 00:15:50.900 align:middle line:84% Kyla asked this question-- is there are a lot of treatment 00:15:50.900 --> 00:15:51.400 align:middle line:90% here. 00:15:51.400 --> 00:15:54.700 align:middle line:90% 00:15:54.700 --> 00:15:57.810 align:middle line:84% And why are there a lot of treatment? 00:15:57.810 --> 00:15:59.810 align:middle line:84% There are a lot of treatment because we are not, 00:15:59.810 --> 00:16:02.240 align:middle line:84% in this case, we didn't go to this project 00:16:02.240 --> 00:16:04.310 align:middle line:90% for testing a particular theory. 00:16:04.310 --> 00:16:06.920 align:middle line:84% Sometimes, you go on and you have something very simple 00:16:06.920 --> 00:16:08.240 align:middle line:90% in mind, very clear in mind. 00:16:08.240 --> 00:16:10.070 align:middle line:84% You want to test a particular theory, 00:16:10.070 --> 00:16:15.490 align:middle line:84% or you want to test whether intervention A works. 00:16:15.490 --> 00:16:17.710 align:middle line:84% But our mandate here with the gun 00:16:17.710 --> 00:16:19.930 align:middle line:84% was a bit different, which is that can you 00:16:19.930 --> 00:16:23.890 align:middle line:84% come up with the best possible intervention for us 00:16:23.890 --> 00:16:28.180 align:middle line:84% in terms of increasing immunization rates based only 00:16:28.180 --> 00:16:29.270 align:middle line:90% on the men intervention? 00:16:29.270 --> 00:16:31.250 align:middle line:84% At least they are taking care of the supply. 00:16:31.250 --> 00:16:33.040 align:middle line:90% So we had the supply sorted out. 00:16:33.040 --> 00:16:36.640 align:middle line:84% But based on what you know on the literature, what's the best 00:16:36.640 --> 00:16:40.270 align:middle line:84% we can do both for increasing immunization overall, and also 00:16:40.270 --> 00:16:44.100 align:middle line:84% in terms of cost effectiveness and cost per dollar? 00:16:44.100 --> 00:16:49.740 align:middle line:84% And when we started brainstorming with them, 00:16:49.740 --> 00:16:52.710 align:middle line:90% the uncaring thing on social-- 00:16:52.710 --> 00:16:54.340 align:middle line:90% the bracelet was not there yet. 00:16:54.340 --> 00:16:56.400 align:middle line:84% So we didn't have that part of it. 00:16:56.400 --> 00:17:00.690 align:middle line:84% But what we had was the idea of cash incentive, which 00:17:00.690 --> 00:17:03.640 align:middle line:84% we had found to be effective in [INAUDIBLE] The reminders, 00:17:03.640 --> 00:17:06.480 align:middle line:84% which everybody is using, which has been endorsed 00:17:06.480 --> 00:17:09.900 align:middle line:84% by the Indian Academy of Pediatrics, 00:17:09.900 --> 00:17:13.750 align:middle line:90% and this new idea of gossip. 00:17:13.750 --> 00:17:19.599 align:middle line:84% And then, so there is evidence from the rest of the literature 00:17:19.599 --> 00:17:22.839 align:middle line:84% that each of these strategy may improve take up. 00:17:22.839 --> 00:17:26.480 align:middle line:84% But we don't know which one is the most effective, 00:17:26.480 --> 00:17:31.310 align:middle line:84% which one is the most cost effective, how 00:17:31.310 --> 00:17:32.720 align:middle line:90% they should be used. 00:17:32.720 --> 00:17:34.220 align:middle line:84% For example, in [INAUDIBLE] we tried 00:17:34.220 --> 00:17:38.540 align:middle line:84% one type of incentive, lentils, and then an increase, 00:17:38.540 --> 00:17:45.820 align:middle line:84% which was slopy and the equivalent of maybe about $1 00:17:45.820 --> 00:17:49.720 align:middle line:84% a shot into this term, in PPP term. 00:17:49.720 --> 00:17:52.230 align:middle line:90% 00:17:52.230 --> 00:17:55.770 align:middle line:90% Maybe we could get less. 00:17:55.770 --> 00:17:58.170 align:middle line:84% Maybe you could just use less money. 00:17:58.170 --> 00:18:01.380 align:middle line:84% Maybe it doesn't need to be slopy. 00:18:01.380 --> 00:18:03.930 align:middle line:84% With SMS, do you need to SMS everyone? 00:18:03.930 --> 00:18:07.830 align:middle line:84% Or you SMS some people with the gossip. 00:18:07.830 --> 00:18:09.810 align:middle line:84% As you know, we didn't know if we 00:18:09.810 --> 00:18:13.320 align:middle line:84% should use gossips, or trusted people, or trusted gossips. 00:18:13.320 --> 00:18:17.140 align:middle line:84% So they are different types of variants. 00:18:17.140 --> 00:18:19.000 align:middle line:84% And then you start asking yourself 00:18:19.000 --> 00:18:22.550 align:middle line:84% whether the combination of these variants would work well. 00:18:22.550 --> 00:18:25.570 align:middle line:84% So for example, the gossip might work 00:18:25.570 --> 00:18:27.760 align:middle line:84% better when there is incentive as well because there 00:18:27.760 --> 00:18:32.140 align:middle line:84% is a clear message to convey than not. 00:18:32.140 --> 00:18:36.280 align:middle line:84% So at least this one was potentially a clear hypothesis. 00:18:36.280 --> 00:18:38.380 align:middle line:90% But it's a mess. 00:18:38.380 --> 00:18:40.520 align:middle line:90% There is a jumble of things. 00:18:40.520 --> 00:18:42.790 align:middle line:84% And so here, it comes to Kyla's question 00:18:42.790 --> 00:18:45.590 align:middle line:90% is, how do you proceed? 00:18:45.590 --> 00:18:50.560 align:middle line:84% So in the literature, the way you typically get to that, 00:18:50.560 --> 00:18:55.660 align:middle line:84% and that's the best way we have most of the time, 00:18:55.660 --> 00:18:59.200 align:middle line:84% is you are collecting estimates from a lot of different papers 00:18:59.200 --> 00:19:01.390 align:middle line:84% and putting them on a common scale. 00:19:01.390 --> 00:19:03.850 align:middle line:84% So that's really the Campbell Collaborative 00:19:03.850 --> 00:19:06.010 align:middle line:84% and the Cochrane Review is what they 00:19:06.010 --> 00:19:08.050 align:middle line:84% do for health intervention, and now 00:19:08.050 --> 00:19:10.150 align:middle line:84% also for some social science intervention. 00:19:10.150 --> 00:19:13.900 align:middle line:84% They're basically saying, on vitamin A, we have 20 studies. 00:19:13.900 --> 00:19:15.940 align:middle line:90% And we can put them on a scale. 00:19:15.940 --> 00:19:17.440 align:middle line:84% In fact, once you have that, you can 00:19:17.440 --> 00:19:19.480 align:middle line:84% try to do meta analysis like the ones 00:19:19.480 --> 00:19:24.640 align:middle line:84% that we found for the-- we discussed for the [INAUDIBLE] 00:19:24.640 --> 00:19:26.880 align:middle line:90% paper. 00:19:26.880 --> 00:19:32.830 align:middle line:84% J-PAL is doing that, trying to accumulate 00:19:32.830 --> 00:19:36.040 align:middle line:84% the impact of different intervention on something, 00:19:36.040 --> 00:19:37.720 align:middle line:84% and then put them on a common scale 00:19:37.720 --> 00:19:41.710 align:middle line:84% to find out what seems to be most cost effective to increase 00:19:41.710 --> 00:19:45.220 align:middle line:84% test scores, or attendance, or anything you're interested in. 00:19:45.220 --> 00:19:47.830 align:middle line:84% The problem, of course, is the population intervention 00:19:47.830 --> 00:19:49.760 align:middle line:84% vary considerably across studies. 00:19:49.760 --> 00:19:52.900 align:middle line:90% So it's comparable or not. 00:19:52.900 --> 00:19:54.580 align:middle line:84% When you test different intervention 00:19:54.580 --> 00:19:58.090 align:middle line:84% in different contexts, you can't get to the interaction, 00:19:58.090 --> 00:19:59.980 align:middle line:90% which makes learning-- 00:19:59.980 --> 00:20:02.482 align:middle line:84% if you want to answer a question in particular, 00:20:02.482 --> 00:20:03.940 align:middle line:84% in this case with high error, which 00:20:03.940 --> 00:20:05.917 align:middle line:84% is the view was that whatever you tell us, 00:20:05.917 --> 00:20:07.375 align:middle line:84% we're going to try and scale it up. 00:20:07.375 --> 00:20:11.020 align:middle line:90% 00:20:11.020 --> 00:20:14.440 align:middle line:84% The idea in experiment there is not one experiment 00:20:14.440 --> 00:20:15.700 align:middle line:90% with one treatment. 00:20:15.700 --> 00:20:20.050 align:middle line:84% It's to run a single, large RCT, which has 00:20:20.050 --> 00:20:21.550 align:middle line:90% everything you can think of. 00:20:21.550 --> 00:20:25.900 align:middle line:90% 00:20:25.900 --> 00:20:30.220 align:middle line:84% The problem is that even though the experiment, 00:20:30.220 --> 00:20:33.470 align:middle line:84% this experiment was pretty large-- 00:20:33.470 --> 00:20:34.810 align:middle line:90% so it's 2,000 villages. 00:20:34.810 --> 00:20:40.065 align:middle line:84% But in fact, 900 villages where we did the census, where we 00:20:40.065 --> 00:20:40.940 align:middle line:90% could do the gossips. 00:20:40.940 --> 00:20:43.630 align:middle line:90% 00:20:43.630 --> 00:20:45.730 align:middle line:90% But it's not infinite. 00:20:45.730 --> 00:20:49.850 align:middle line:84% And the number of possible combinations is large. 00:20:49.850 --> 00:20:50.840 align:middle line:90% So what do you do? 00:20:50.840 --> 00:20:53.770 align:middle line:84% So the first thing you could do is 00:20:53.770 --> 00:20:56.863 align:middle line:90% to say, well, it's too large. 00:20:56.863 --> 00:20:58.780 align:middle line:84% That's going to be too many things to look at. 00:20:58.780 --> 00:21:01.450 align:middle line:84% You're never going to be power to look at every single cell 00:21:01.450 --> 00:21:02.770 align:middle line:90% of interaction. 00:21:02.770 --> 00:21:04.480 align:middle line:90% So run very few treatments. 00:21:04.480 --> 00:21:07.890 align:middle line:90% 00:21:07.890 --> 00:21:10.740 align:middle line:84% There is a paper by Karthik Muralidharan and others 00:21:10.740 --> 00:21:15.050 align:middle line:84% that point out that once you have interaction, if you have, 00:21:15.050 --> 00:21:16.800 align:middle line:84% say, two treatments and their interaction, 00:21:16.800 --> 00:21:19.080 align:middle line:84% it gets messy to ignore the interaction, which 00:21:19.080 --> 00:21:21.000 align:middle line:90% is what a lot of people do. 00:21:21.000 --> 00:21:23.770 align:middle line:84% And then, therefore, it's confusing. 00:21:23.770 --> 00:21:27.240 align:middle line:84% So the McKenzie solution is just don't do it. 00:21:27.240 --> 00:21:29.520 align:middle line:90% Run one treatment. 00:21:29.520 --> 00:21:31.290 align:middle line:90% That's nice. 00:21:31.290 --> 00:21:36.930 align:middle line:90% But that means, which one? 00:21:36.930 --> 00:21:39.150 align:middle line:90% How will you pick which one? 00:21:39.150 --> 00:21:43.380 align:middle line:84% Or you could include all combos, but potentially 00:21:43.380 --> 00:21:47.640 align:middle line:84% lose power because you have a lot of different cells. 00:21:47.640 --> 00:21:49.590 align:middle line:84% And the cells can become quite small. 00:21:49.590 --> 00:21:52.890 align:middle line:90% 00:21:52.890 --> 00:21:55.408 align:middle line:84% I'll tell you exactly how small in our case, for example, 00:21:55.408 --> 00:21:56.200 align:middle line:90% when I get to that. 00:21:56.200 --> 00:21:58.710 align:middle line:84% So in practice, what people do is sometimes 00:21:58.710 --> 00:22:00.660 align:middle line:84% run many experiments, and then pull 00:22:00.660 --> 00:22:03.270 align:middle line:84% exposed if it looks like the interactions are not 00:22:03.270 --> 00:22:04.820 align:middle line:90% significant. 00:22:04.820 --> 00:22:07.410 align:middle line:84% But then that poses a problem of statistics, 00:22:07.410 --> 00:22:11.070 align:middle line:84% which is if you first run all interactions then remove them 00:22:11.070 --> 00:22:13.710 align:middle line:84% when they are not significant, then 00:22:13.710 --> 00:22:17.820 align:middle line:84% it's not clear how to read the test statistics because it 00:22:17.820 --> 00:22:19.740 align:middle line:84% might just have been a fluke that it 00:22:19.740 --> 00:22:21.760 align:middle line:90% was or wasn't significant. 00:22:21.760 --> 00:22:23.550 align:middle line:84% So we can't interpret these statistics 00:22:23.550 --> 00:22:28.720 align:middle line:84% unless we go with a clear frame in mind. 00:22:28.720 --> 00:22:30.930 align:middle line:84% So in the literature, there is a lot 00:22:30.930 --> 00:22:35.580 align:middle line:84% of push towards going simple and pre-analysis plan. 00:22:35.580 --> 00:22:39.360 align:middle line:84% And therefore, going towards one or saying in 00:22:39.360 --> 00:22:41.580 align:middle line:84% advance exactly what you're going to run. 00:22:41.580 --> 00:22:44.600 align:middle line:90% 00:22:44.600 --> 00:22:50.040 align:middle line:84% The issue is that it's assuming the conclusion. 00:22:50.040 --> 00:22:52.790 align:middle line:84% So this is an image from The Matrix. 00:22:52.790 --> 00:22:56.510 align:middle line:84% Well, The Oracle, so this is The Oracle problem, 00:22:56.510 --> 00:22:57.530 align:middle line:90% The Oracle point. 00:22:57.530 --> 00:23:01.320 align:middle line:84% It's that if you know what it is that you want to test, then 00:23:01.320 --> 00:23:03.320 align:middle line:84% first of all, there is a question of whether you 00:23:03.320 --> 00:23:05.210 align:middle line:90% should test it anyway. 00:23:05.210 --> 00:23:11.900 align:middle line:84% But you run the experiment to justify your prior. 00:23:11.900 --> 00:23:13.760 align:middle line:84% So you didn't come here to make the choice. 00:23:13.760 --> 00:23:15.020 align:middle line:90% You've already made it. 00:23:15.020 --> 00:23:18.440 align:middle line:84% You have to understand why you made it. 00:23:18.440 --> 00:23:19.880 align:middle line:90% So that's the issue with Oracle. 00:23:19.880 --> 00:23:23.390 align:middle line:84% And in a lot of cases, for example, in this case, 00:23:23.390 --> 00:23:25.040 align:middle line:90% we didn't have an Oracle. 00:23:25.040 --> 00:23:26.940 align:middle line:90% So that's the problem. 00:23:26.940 --> 00:23:31.730 align:middle line:84% So that's the tension that Kyla's question was asking. 00:23:31.730 --> 00:23:33.300 align:middle line:84% And then there is another problem, 00:23:33.300 --> 00:23:34.850 align:middle line:84% which is when you-- even if you had 00:23:34.850 --> 00:23:40.250 align:middle line:84% a large, large, large, large, large, large set of sample 00:23:40.250 --> 00:23:43.340 align:middle line:84% size, so you could run many, many things individually, 00:23:43.340 --> 00:23:46.640 align:middle line:84% you can think of them as all of the combos, and variants, 00:23:46.640 --> 00:23:52.550 align:middle line:84% and variant combos, or variants as individual treatment. 00:23:52.550 --> 00:23:54.050 align:middle line:84% And then, you tell the government 00:23:54.050 --> 00:23:55.350 align:middle line:90% this is the best policy. 00:23:55.350 --> 00:23:56.040 align:middle line:90% You go for it. 00:23:56.040 --> 00:23:58.760 align:middle line:84% This is the effect you're going to get for it. 00:23:58.760 --> 00:23:59.480 align:middle line:90% OK. 00:23:59.480 --> 00:24:02.600 align:middle line:84% Now, that's nice, except that it will 00:24:02.600 --> 00:24:12.540 align:middle line:84% make the policy is chosen to be the best in that sample. 00:24:12.540 --> 00:24:15.920 align:middle line:84% So by definition, it performed pretty well in that sample. 00:24:15.920 --> 00:24:18.530 align:middle line:90% And it could be a fluke. 00:24:18.530 --> 00:24:22.520 align:middle line:84% It's like the best basketball player. 00:24:22.520 --> 00:24:24.440 align:middle line:84% If you rank people by basketball player, 00:24:24.440 --> 00:24:26.180 align:middle line:84% the best basketball player is someone 00:24:26.180 --> 00:24:29.090 align:middle line:84% who's on the day you were ranking has 00:24:29.090 --> 00:24:30.800 align:middle line:90% scored many, many, many goals. 00:24:30.800 --> 00:24:35.750 align:middle line:84% But it might not be that much best overall. 00:24:35.750 --> 00:24:38.510 align:middle line:84% It's just that day he performed so well that it 00:24:38.510 --> 00:24:40.010 align:middle line:90% looks absolutely fabulous. 00:24:40.010 --> 00:24:43.880 align:middle line:84% That's not his-- you might be very disappointed 00:24:43.880 --> 00:24:46.450 align:middle line:84% once you get him to your team that on most days 00:24:46.450 --> 00:24:49.450 align:middle line:90% is not as fabulous. 00:24:49.450 --> 00:24:52.210 align:middle line:84% Similarly, for our policy, if you picked it 00:24:52.210 --> 00:24:55.120 align:middle line:84% among a bigger range of policy to be the best, 00:24:55.120 --> 00:24:58.000 align:middle line:90% it will look really great. 00:24:58.000 --> 00:25:00.220 align:middle line:84% By definition, it performs really well 00:25:00.220 --> 00:25:02.680 align:middle line:90% in your small sample. 00:25:02.680 --> 00:25:08.800 align:middle line:84% The question is, you're giving it credit for the epsilon. 00:25:08.800 --> 00:25:12.800 align:middle line:84% And so that's not a conceptually difficult problem. 00:25:12.800 --> 00:25:15.040 align:middle line:84% There's a very nice paper by Isaiah 00:25:15.040 --> 00:25:17.170 align:middle line:84% and co-authors that looks at that. 00:25:17.170 --> 00:25:20.750 align:middle line:84% And basically, this just can be corrected. 00:25:20.750 --> 00:25:21.820 align:middle line:90% And so this is-- 00:25:21.820 --> 00:25:24.370 align:middle line:84% they have a paper called "Inference on Winners." 00:25:24.370 --> 00:25:26.530 align:middle line:84% Now, the problem is the more policies 00:25:26.530 --> 00:25:32.950 align:middle line:84% you are trying to test against, and the more similar 00:25:32.950 --> 00:25:36.470 align:middle line:84% they are to each other, the more you're 00:25:36.470 --> 00:25:39.780 align:middle line:84% going to be penalized by the inference on winners. 00:25:39.780 --> 00:25:44.580 align:middle line:84% So naturally, because to be the best 00:25:44.580 --> 00:25:48.480 align:middle line:84% you'd have to be particularly lucky. 00:25:48.480 --> 00:25:51.540 align:middle line:84% So that means that that's one more reason to not 00:25:51.540 --> 00:25:54.960 align:middle line:84% have a lot and lot and lot of treatments in principle, 00:25:54.960 --> 00:25:56.535 align:middle line:84% but then, we still would sometimes 00:25:56.535 --> 00:25:58.410 align:middle line:84% would like to have a lot, a lot of treatments 00:25:58.410 --> 00:25:59.730 align:middle line:90% because we really don't know. 00:25:59.730 --> 00:26:01.530 align:middle line:84% And we would like to run one experiment 00:26:01.530 --> 00:26:04.080 align:middle line:84% and have a coherent answer that doesn't 00:26:04.080 --> 00:26:06.082 align:middle line:90% assume the solution in advance. 00:26:06.082 --> 00:26:06.790 align:middle line:90% AUDIENCE: Esther? 00:26:06.790 --> 00:26:08.028 align:middle line:90% ESTHER DUFLO: Yeah? 00:26:08.028 --> 00:26:09.070 align:middle line:90% AUDIENCE: So I'm curious. 00:26:09.070 --> 00:26:10.380 align:middle line:90% This is random, right? 00:26:10.380 --> 00:26:12.078 align:middle line:90% It's if I happen to run? 00:26:12.078 --> 00:26:12.870 align:middle line:90% ESTHER DUFLO: Yeah. 00:26:12.870 --> 00:26:16.240 align:middle line:84% So it's just you happen to-- in that sample, 00:26:16.240 --> 00:26:18.970 align:middle line:90% you happen to do quite well. 00:26:18.970 --> 00:26:21.180 align:middle line:84% So just imagine, for example, a setting 00:26:21.180 --> 00:26:25.330 align:middle line:84% where in fact, all of the policies have the same effect. 00:26:25.330 --> 00:26:30.490 align:middle line:84% You're still going to rank them according to the epsilon. 00:26:30.490 --> 00:26:32.230 align:middle line:84% So that's what we have to get rid of. 00:26:32.230 --> 00:26:34.673 align:middle line:84% AUDIENCE: So this is not like the scale up problem? 00:26:34.673 --> 00:26:36.590 align:middle line:84% ESTHER DUFLO: That's not the scale up problem. 00:26:36.590 --> 00:26:38.720 align:middle line:84% It's just I have tested a lot of policy. 00:26:38.720 --> 00:26:40.660 align:middle line:90% I'm picking the best. 00:26:40.660 --> 00:26:43.728 align:middle line:84% And therefore, and it is a standard thing. 00:26:43.728 --> 00:26:45.520 align:middle line:84% That's exactly what we're going to do here. 00:26:45.520 --> 00:26:47.145 align:middle line:84% That's exactly what we want to do here. 00:26:47.145 --> 00:26:50.230 align:middle line:84% And that's a common way to think about policy problems. 00:26:50.230 --> 00:26:51.040 align:middle line:90% What's the best? 00:26:51.040 --> 00:26:52.690 align:middle line:84% And then, the best is also the one 00:26:52.690 --> 00:26:55.120 align:middle line:84% who performed the best in that sample. 00:26:55.120 --> 00:26:58.570 align:middle line:84% And hence, that has benefited from the most fluke 00:26:58.570 --> 00:26:59.810 align:middle line:90% by construction. 00:26:59.810 --> 00:27:01.990 align:middle line:90% So this is correctable. 00:27:01.990 --> 00:27:03.940 align:middle line:84% This is pretty easy to correct, actually. 00:27:03.940 --> 00:27:05.890 align:middle line:84% There is nothing-- it's just very 00:27:05.890 --> 00:27:07.850 align:middle line:90% classical statistics in there. 00:27:07.850 --> 00:27:10.143 align:middle line:84% The problem is the correction is going to penalize you 00:27:10.143 --> 00:27:11.560 align:middle line:84% for having a lot of options you're 00:27:11.560 --> 00:27:14.590 align:middle line:84% trying against each other, as well as for having options 00:27:14.590 --> 00:27:16.240 align:middle line:90% that are in fact, similar. 00:27:16.240 --> 00:27:16.910 align:middle line:90% Yep? 00:27:16.910 --> 00:27:18.910 align:middle line:84% AUDIENCE: Yeah, I was just thinking that-- well, 00:27:18.910 --> 00:27:21.670 align:middle line:84% it wouldn't be the same because according to the performance 00:27:21.670 --> 00:27:25.390 align:middle line:84% against people control, that between each other because I 00:27:25.390 --> 00:27:29.710 align:middle line:84% don't know if I can see that two interventions are one is-- 00:27:29.710 --> 00:27:32.260 align:middle line:84% ESTHER DUFLO: Yeah, so the control here doesn't-- 00:27:32.260 --> 00:27:34.900 align:middle line:84% the control here is the same for everyone. 00:27:34.900 --> 00:27:37.705 align:middle line:90% So when you're going against-- 00:27:37.705 --> 00:27:40.000 align:middle line:84% if you do the treatment effect, you're 00:27:40.000 --> 00:27:42.940 align:middle line:84% going to do the treatment effect against the control. 00:27:42.940 --> 00:27:44.770 align:middle line:84% And you're saying intervention A are 00:27:44.770 --> 00:27:47.710 align:middle line:84% the biggest treatment effect, B as the second biggest treatment 00:27:47.710 --> 00:27:48.460 align:middle line:90% effect, et cetera. 00:27:48.460 --> 00:27:50.830 align:middle line:84% They're all ranked against the same control. 00:27:50.830 --> 00:27:53.530 align:middle line:84% AUDIENCE: Yeah, but then, for example, the first, 00:27:53.530 --> 00:27:58.090 align:middle line:84% the best run of the second run between each other, not be-- 00:27:58.090 --> 00:27:59.440 align:middle line:90% not statistically. 00:27:59.440 --> 00:28:01.420 align:middle line:84% If I'm comparing those two, they could be-- 00:28:01.420 --> 00:28:04.660 align:middle line:84% I couldn't rule out that one is better than the other. 00:28:04.660 --> 00:28:06.310 align:middle line:84% ESTHER DUFLO: So that's possible. 00:28:06.310 --> 00:28:09.370 align:middle line:84% But it's in that case, you don't know which is the best. 00:28:09.370 --> 00:28:12.250 align:middle line:84% But even if they are, it might be 00:28:12.250 --> 00:28:15.670 align:middle line:84% because it is in part because your first one happened 00:28:15.670 --> 00:28:18.950 align:middle line:84% to do well in the sample compared to the second one. 00:28:18.950 --> 00:28:20.298 align:middle line:90% AUDIENCE: If they are? 00:28:20.298 --> 00:28:22.590 align:middle line:84% ESTHER DUFLO: Suppose they are statistically different. 00:28:22.590 --> 00:28:25.490 align:middle line:84% It could still be in part because the first one happened 00:28:25.490 --> 00:28:29.960 align:middle line:84% to have a very good draw of the epsilons in the treatment, 00:28:29.960 --> 00:28:32.170 align:middle line:90% and the two didn't. 00:28:32.170 --> 00:28:33.828 align:middle line:90% So that's the problem. 00:28:33.828 --> 00:28:36.370 align:middle line:84% It's not the problem that you might think they are different, 00:28:36.370 --> 00:28:38.412 align:middle line:84% but in fact they are not statistically different, 00:28:38.412 --> 00:28:39.610 align:middle line:90% which is a good point. 00:28:39.610 --> 00:28:41.800 align:middle line:84% But even if they are, it could be just 00:28:41.800 --> 00:28:44.210 align:middle line:84% for a fluke performance in that sample. 00:28:44.210 --> 00:28:49.390 align:middle line:84% So it's not conceptually difficult. 00:28:49.390 --> 00:28:55.690 align:middle line:84% But it's not difficult to deal with at all, even in its data. 00:28:55.690 --> 00:28:59.890 align:middle line:84% But it's just that how much you penalize yourself. 00:28:59.890 --> 00:29:01.690 align:middle line:84% It depends on how many policies you 00:29:01.690 --> 00:29:04.330 align:middle line:84% run against each other, which makes sense because you 00:29:04.330 --> 00:29:05.860 align:middle line:90% have more ignorance. 00:29:05.860 --> 00:29:08.630 align:middle line:90% So that's the tension. 00:29:08.630 --> 00:29:11.500 align:middle line:84% That's why your question needed more than a 1 minute answer. 00:29:11.500 --> 00:29:14.480 align:middle line:90% It's what do you do? 00:29:14.480 --> 00:29:16.810 align:middle line:90% So what we did is that we tried. 00:29:16.810 --> 00:29:20.890 align:middle line:84% Basically, we tried to navigate getting the best of both worlds 00:29:20.890 --> 00:29:25.060 align:middle line:84% by saying, look, most policies don't work anyways. 00:29:25.060 --> 00:29:29.970 align:middle line:84% And most policy variation might not matter. 00:29:29.970 --> 00:29:33.010 align:middle line:90% So for example, dosage. 00:29:33.010 --> 00:29:36.550 align:middle line:84% Dosage might not be that relevant in the gossip world. 00:29:36.550 --> 00:29:39.430 align:middle line:84% The fact that the gossip is trusted, or trusted gossip 00:29:39.430 --> 00:29:43.002 align:middle line:90% might be similar. 00:29:43.002 --> 00:29:44.460 align:middle line:84% We don't know whether it's similar. 00:29:44.460 --> 00:29:45.610 align:middle line:90% You say maybe it is. 00:29:45.610 --> 00:29:49.330 align:middle line:84% So maybe a lot of my possible treatments 00:29:49.330 --> 00:29:55.330 align:middle line:84% can be eliminated from my consideration set. 00:29:55.330 --> 00:29:58.230 align:middle line:90% 00:29:58.230 --> 00:30:02.240 align:middle line:84% And therefore, I can run many, many treatments 00:30:02.240 --> 00:30:06.680 align:middle line:84% and get rid of anything that doesn't work at all. 00:30:06.680 --> 00:30:14.330 align:middle line:84% And then, pull together what is similar. 00:30:14.330 --> 00:30:21.560 align:middle line:84% And then, be left only with intervention 00:30:21.560 --> 00:30:23.900 align:middle line:84% that seems promising at the end of the day 00:30:23.900 --> 00:30:29.670 align:middle line:84% and estimate my regression in that set of promising policies. 00:30:29.670 --> 00:30:33.190 align:middle line:90% So what do I mean by pooling? 00:30:33.190 --> 00:30:36.320 align:middle line:90% We're only going to pool-- 00:30:36.320 --> 00:30:42.860 align:middle line:84% and that's why cannot pool in some setting and not get 00:30:42.860 --> 00:30:48.520 align:middle line:84% the full play of the problem that Karthik Muralidharan 00:30:48.520 --> 00:30:54.310 align:middle line:84% identified, which is if I pool ex-post when the interaction is 00:30:54.310 --> 00:30:55.750 align:middle line:90% negative, that's a little-- 00:30:55.750 --> 00:30:59.050 align:middle line:90% that's fishy. 00:30:59.050 --> 00:31:00.880 align:middle line:90% We are only going to pool-- 00:31:00.880 --> 00:31:03.010 align:middle line:84% or we are only going to ask the data whether they 00:31:03.010 --> 00:31:08.740 align:middle line:84% want to pool things that are dosages of the same or variance 00:31:08.740 --> 00:31:10.490 align:middle line:90% of the same treatment. 00:31:10.490 --> 00:31:13.450 align:middle line:90% So I'm putting in some theory. 00:31:13.450 --> 00:31:16.360 align:middle line:84% It's not really theory, some amount of structure 00:31:16.360 --> 00:31:20.920 align:middle line:84% there to say, well, any incentives-- 00:31:20.920 --> 00:31:24.280 align:middle line:84% I want to try a different incentive because who? 00:31:24.280 --> 00:31:26.300 align:middle line:90% Knows they might differ. 00:31:26.300 --> 00:31:29.560 align:middle line:84% But I'm going to allow the data to pool them 00:31:29.560 --> 00:31:32.580 align:middle line:84% if it turns out that all the effects are the same. 00:31:32.580 --> 00:31:34.210 align:middle line:84% So example, high and low incentive 00:31:34.210 --> 00:31:35.560 align:middle line:90% will be allowed to pool. 00:31:35.560 --> 00:31:39.550 align:middle line:84% Slope and flat, and incentive will be allowed to pool. 00:31:39.550 --> 00:31:43.330 align:middle line:84% Slope and flat, and high and low will be allowed to pool. 00:31:43.330 --> 00:31:49.000 align:middle line:84% And so that creates a treatment profile of any incentives. 00:31:49.000 --> 00:31:52.600 align:middle line:90% And similarly, any gossips. 00:31:52.600 --> 00:31:57.490 align:middle line:90% And then, any reminder. 00:31:57.490 --> 00:32:00.820 align:middle line:84% A lot of reminders versus fewer reminders. 00:32:00.820 --> 00:32:03.190 align:middle line:84% So I'm telling the data that they can pull. 00:32:03.190 --> 00:32:07.100 align:middle line:84% I'm going to permit the data they can pool that way. 00:32:07.100 --> 00:32:12.330 align:middle line:84% And then, you run a LASSO on that. 00:32:12.330 --> 00:32:14.790 align:middle line:84% You run into a small problem, which 00:32:14.790 --> 00:32:17.980 align:middle line:84% is that when you run a LASSO on that, by definition, 00:32:17.980 --> 00:32:19.890 align:middle line:84% you have a lot of correlation between. 00:32:19.890 --> 00:32:21.040 align:middle line:90% You're going to structure. 00:32:21.040 --> 00:32:25.620 align:middle line:84% So think of writing the regression in order to-- 00:32:25.620 --> 00:32:28.830 align:middle line:84% allowing pooling instead of have now 00:32:28.830 --> 00:32:31.710 align:middle line:84% run all of your different possible treatments 00:32:31.710 --> 00:32:33.660 align:middle line:90% and interactions. 00:32:33.660 --> 00:32:35.580 align:middle line:84% Instead of running one regression 00:32:35.580 --> 00:32:38.100 align:middle line:84% for each possible treatment, you run a regression 00:32:38.100 --> 00:32:41.580 align:middle line:84% of the form, any incentive, and then 00:32:41.580 --> 00:32:43.530 align:middle line:84% marginal effect of high incentive. 00:32:43.530 --> 00:32:45.690 align:middle line:90% So then that might be-- 00:32:45.690 --> 00:32:48.270 align:middle line:84% that might become zero, as opposed 00:32:48.270 --> 00:32:49.830 align:middle line:84% to high incentive, low incentive, 00:32:49.830 --> 00:32:51.300 align:middle line:84% and then you test the difference. 00:32:51.300 --> 00:32:53.040 align:middle line:84% Why it's relevant to write it this way? 00:32:53.040 --> 00:32:56.160 align:middle line:84% It's because you are going to run a LASSO on this regression. 00:32:56.160 --> 00:33:01.050 align:middle line:84% And LASSO is going to penalize you for having a lot of terms. 00:33:01.050 --> 00:33:04.147 align:middle line:84% And it's going to just ask you to cull a bunch of terms 00:33:04.147 --> 00:33:05.730 align:middle line:84% saying they are completely irrelevant. 00:33:05.730 --> 00:33:06.938 align:middle line:90% They shouldn't even be there. 00:33:06.938 --> 00:33:09.330 align:middle line:90% 00:33:09.330 --> 00:33:12.300 align:middle line:84% So if you had two policies, high and low incentive, that 00:33:12.300 --> 00:33:14.190 align:middle line:84% in fact had the same effect, LASSO 00:33:14.190 --> 00:33:17.640 align:middle line:84% would want them both if they were both effective. 00:33:17.640 --> 00:33:22.920 align:middle line:84% If you write it as any incentive and an incentive-- 00:33:22.920 --> 00:33:27.120 align:middle line:84% the extra effect of high versus low, then LASSO 00:33:27.120 --> 00:33:29.620 align:middle line:90% is going to get rid of that. 00:33:29.620 --> 00:33:31.440 align:middle line:90% So it will allow get rid of it. 00:33:31.440 --> 00:33:32.880 align:middle line:84% But there is a problem with doing 00:33:32.880 --> 00:33:36.000 align:middle line:84% that, which is as soon as you do that, you 00:33:36.000 --> 00:33:41.160 align:middle line:84% create a lot of correlation between the variable because-- 00:33:41.160 --> 00:33:45.240 align:middle line:84% and by definition, any incentive and incentive high 00:33:45.240 --> 00:33:49.320 align:middle line:84% are very correlated because incentive high 00:33:49.320 --> 00:33:51.930 align:middle line:90% will have a one every time. 00:33:51.930 --> 00:33:54.390 align:middle line:84% Incentive will have a one every time. 00:33:54.390 --> 00:33:57.450 align:middle line:84% Incentive one has high as the one. 00:33:57.450 --> 00:33:59.280 align:middle line:84% And in addition, it will have some zeros 00:33:59.280 --> 00:34:01.110 align:middle line:84% when the incentive is, in fact low. 00:34:01.110 --> 00:34:03.900 align:middle line:84% So by construction, you're going to have a lot of collision, 00:34:03.900 --> 00:34:06.510 align:middle line:84% which is precisely when you cannot run LASSO. 00:34:06.510 --> 00:34:08.969 align:middle line:84% Intuitively, it's that the intuition 00:34:08.969 --> 00:34:12.330 align:middle line:84% for it being that LASSO is not going 00:34:12.330 --> 00:34:16.830 align:middle line:84% to know what to pick if the two things are correlated. 00:34:16.830 --> 00:34:19.770 align:middle line:90% Statistically, it fails. 00:34:19.770 --> 00:34:23.730 align:middle line:84% Irrepresentability from TVA, which is saying you cannot-- 00:34:23.730 --> 00:34:27.540 align:middle line:84% the irrepresentability is that you can only run Lasso 00:34:27.540 --> 00:34:31.889 align:middle line:84% if no variable can be approximated very well 00:34:31.889 --> 00:34:34.940 align:middle line:90% by a combination of other ones. 00:34:34.940 --> 00:34:38.199 align:middle line:84% So but there is a solution for this problem, which 00:34:38.199 --> 00:34:41.739 align:middle line:84% is a preconditioning of the variable that basically 00:34:41.739 --> 00:34:43.270 align:middle line:90% organizes them. 00:34:43.270 --> 00:34:47.080 align:middle line:84% It penalizes you for doing that, but it works. 00:34:47.080 --> 00:34:49.929 align:middle line:84% And then, once you've done that, you run your LASSO. 00:34:49.929 --> 00:34:54.010 align:middle line:84% The LASSO is going to select some variable. 00:34:54.010 --> 00:34:56.770 align:middle line:84% It's going to kick out everything that is ineffective. 00:34:56.770 --> 00:35:00.220 align:middle line:84% And it's going to also kick out things 00:35:00.220 --> 00:35:04.970 align:middle line:84% that are marginal difference that are not pertinent. 00:35:04.970 --> 00:35:08.300 align:middle line:84% And once you have that, you have very few variables left. 00:35:08.300 --> 00:35:11.090 align:middle line:84% And you can do your post-processing. 00:35:11.090 --> 00:35:12.740 align:middle line:84% That is running, which is basically 00:35:12.740 --> 00:35:14.630 align:middle line:84% running your less regression happily 00:35:14.630 --> 00:35:17.340 align:middle line:84% on a very small set of variables. 00:35:17.340 --> 00:35:22.320 align:middle line:84% And now, you can run your Andrews correction 00:35:22.320 --> 00:35:23.940 align:middle line:84% without being penalized by the fact 00:35:23.940 --> 00:35:27.450 align:middle line:84% that there was a lot of variables in the first place. 00:35:27.450 --> 00:35:29.950 align:middle line:90% And you go ahead. 00:35:29.950 --> 00:35:32.305 align:middle line:90% So that strategy works. 00:35:32.305 --> 00:35:35.050 align:middle line:90% 00:35:35.050 --> 00:35:39.190 align:middle line:90% And it's nicely compatible. 00:35:39.190 --> 00:35:43.360 align:middle line:84% You adjust-- you don't have a flu case that's too high 00:35:43.360 --> 00:35:46.300 align:middle line:90% because you've-- 00:35:46.300 --> 00:35:50.560 align:middle line:84% with Andrews, it's if you wanted to really say, 00:35:50.560 --> 00:35:52.962 align:middle line:84% this is going back to Salu's point, 00:35:52.962 --> 00:35:55.420 align:middle line:84% which is suppose you have high incentive and low incentive. 00:35:55.420 --> 00:35:57.800 align:middle line:84% And in fact, they have the same effect. 00:35:57.800 --> 00:35:59.510 align:middle line:84% They are not statistically different. 00:35:59.510 --> 00:36:01.180 align:middle line:84% So one of them will come out ahead 00:36:01.180 --> 00:36:04.030 align:middle line:84% and will be penalized a lot by the fact 00:36:04.030 --> 00:36:06.280 align:middle line:84% that it's very close to the other one. 00:36:06.280 --> 00:36:08.680 align:middle line:84% But in fact, if you know that they should be pooled, 00:36:08.680 --> 00:36:10.450 align:middle line:84% then you won't have that problem. 00:36:10.450 --> 00:36:14.020 align:middle line:90% You'll just have incentives. 00:36:14.020 --> 00:36:17.270 align:middle line:84% So that's why we run it in these steps. 00:36:17.270 --> 00:36:21.520 align:middle line:84% So if we go back with how many treatments we have, 00:36:21.520 --> 00:36:24.880 align:middle line:84% so incentives are crossed two by two, by high and low, 00:36:24.880 --> 00:36:26.950 align:middle line:90% and flat and slope. 00:36:26.950 --> 00:36:28.705 align:middle line:90% Reminders have to. 00:36:28.705 --> 00:36:32.340 align:middle line:90% 00:36:32.340 --> 00:36:36.180 align:middle line:90% Smaller, a few versus a lot. 00:36:36.180 --> 00:36:44.040 align:middle line:84% The ambassadors of random, plus information seed, 00:36:44.040 --> 00:36:47.940 align:middle line:84% plus trusted seed, plus I think there's one from the board 00:36:47.940 --> 00:36:49.290 align:middle line:90% information times trust. 00:36:49.290 --> 00:36:51.000 align:middle line:90% That makes four. 00:36:51.000 --> 00:36:55.232 align:middle line:84% So in total, that's 75 unique policy combinations. 00:36:55.232 --> 00:36:56.940 align:middle line:84% Everything is interacted with everything. 00:36:56.940 --> 00:37:00.450 align:middle line:84% So it's 75 unique policy combinations. 00:37:00.450 --> 00:37:04.870 align:middle line:84% So with 914, which are highly used, this is useful. 00:37:04.870 --> 00:37:07.920 align:middle line:84% This is a sample that where we do a minor in incentive, 00:37:07.920 --> 00:37:10.670 align:middle line:84% but the gossip is only done in 900. 00:37:10.670 --> 00:37:14.430 align:middle line:84% The resulting sample size would actually be quite low. 00:37:14.430 --> 00:37:19.530 align:middle line:90% 00:37:19.530 --> 00:37:25.060 align:middle line:84% That's how everything is crossed with everything. 00:37:25.060 --> 00:37:28.390 align:middle line:90% It doesn't really matter. 00:37:28.390 --> 00:37:29.470 align:middle line:90% So we could start. 00:37:29.470 --> 00:37:32.790 align:middle line:84% We start by doing-- if we start by doing the naive 00:37:32.790 --> 00:37:36.830 align:middle line:84% thing of not interacting with everything, 00:37:36.830 --> 00:37:39.810 align:middle line:90% you run this regression. 00:37:39.810 --> 00:37:43.040 align:middle line:84% So this is a dummy for each thing. 00:37:43.040 --> 00:37:46.130 align:middle line:84% It's already giving you some interesting results. 00:37:46.130 --> 00:37:49.370 align:middle line:84% We already saw the gossip results. 00:37:49.370 --> 00:37:52.970 align:middle line:84% And then, in terms of incentives it 00:37:52.970 --> 00:37:55.340 align:middle line:84% looks like the high slope incentive works. 00:37:55.340 --> 00:37:59.150 align:middle line:84% Although, maybe, you can say that it's so statistically 00:37:59.150 --> 00:38:01.310 align:middle line:90% different from low slope. 00:38:01.310 --> 00:38:05.520 align:middle line:84% High slope giving flat incentive doesn't seem to work. 00:38:05.520 --> 00:38:11.520 align:middle line:84% SMS, there's not significant impact. 00:38:11.520 --> 00:38:13.280 align:middle line:90% This isn't in the entire sample. 00:38:13.280 --> 00:38:17.750 align:middle line:84% And that's in the smaller sample. 00:38:17.750 --> 00:38:20.660 align:middle line:84% So if you just looked at that you would say, 00:38:20.660 --> 00:38:24.160 align:middle line:84% well, gossip looked effective, perhaps as effective 00:38:24.160 --> 00:38:26.333 align:middle line:90% as giving high slope incentive. 00:38:26.333 --> 00:38:28.000 align:middle line:84% And otherwise, we are not very sure what 00:38:28.000 --> 00:38:29.170 align:middle line:90% will pool, what will not pool. 00:38:29.170 --> 00:38:31.628 align:middle line:84% And of course, we have nothing to say about the interaction 00:38:31.628 --> 00:38:33.530 align:middle line:90% from looking in that. 00:38:33.530 --> 00:38:34.734 align:middle line:90% So that's the procedure. 00:38:34.734 --> 00:38:37.840 align:middle line:84% You define what's allowed to pool potentially. 00:38:37.840 --> 00:38:39.340 align:middle line:84% So in our case, it's pretty simple 00:38:39.340 --> 00:38:42.610 align:middle line:84% because it's by category of intervention. 00:38:42.610 --> 00:38:45.850 align:middle line:84% Once you do that, you define-- you specify the regression 00:38:45.850 --> 00:38:47.840 align:middle line:90% in this marginal way. 00:38:47.840 --> 00:38:52.330 align:middle line:84% So to me, that looks very intuitive. 00:38:52.330 --> 00:38:55.150 align:middle line:84% But in fact, it's a bit tricky when you 00:38:55.150 --> 00:38:56.680 align:middle line:90% have a lot of possibilities. 00:38:56.680 --> 00:38:59.590 align:middle line:84% If two by two, it's pretty clear. 00:38:59.590 --> 00:39:02.620 align:middle line:90% There's only one way to do it. 00:39:02.620 --> 00:39:06.010 align:middle line:84% If you start having many, potentially a large number 00:39:06.010 --> 00:39:10.090 align:middle line:84% of different treatments and different treatment profiles, 00:39:10.090 --> 00:39:12.580 align:middle line:84% it takes a bit of combinatorics to say 00:39:12.580 --> 00:39:14.380 align:middle line:84% how to make sure not to forget some, 00:39:14.380 --> 00:39:16.870 align:middle line:84% and how they should be pooled, and why it still works. 00:39:16.870 --> 00:39:20.000 align:middle line:84% And that's described in the paper. 00:39:20.000 --> 00:39:21.790 align:middle line:84% Then, you use the Puffer transformation 00:39:21.790 --> 00:39:24.520 align:middle line:90% to make it LASSO compatible. 00:39:24.520 --> 00:39:26.320 align:middle line:90% So that's just a way of de-- 00:39:26.320 --> 00:39:28.180 align:middle line:84% basically changing the expression 00:39:28.180 --> 00:39:32.458 align:middle line:84% of the variable to make them orthogonal to each other. 00:39:32.458 --> 00:39:34.000 align:middle line:84% It doesn't come for free, by the way. 00:39:34.000 --> 00:39:36.760 align:middle line:84% It adds noise to the data, as it should. 00:39:36.760 --> 00:39:39.040 align:middle line:90% Then you run your-- 00:39:39.040 --> 00:39:41.710 align:middle line:84% so you make your LASSO, then your post LASSO, and finally 00:39:41.710 --> 00:39:45.810 align:middle line:90% the winner curse correction. 00:39:45.810 --> 00:39:49.410 align:middle line:84% So that's the pooling specification, looks something 00:39:49.410 --> 00:39:54.120 align:middle line:84% like that where you have, for example, SMS, any SMS, and then 00:39:54.120 --> 00:39:58.710 align:middle line:84% the marginal effect of SMS, any slope, 00:39:58.710 --> 00:40:01.470 align:middle line:84% and then whether it's high, any flat incentive, 00:40:01.470 --> 00:40:04.900 align:middle line:84% and whether it's a flat incentive. 00:40:04.900 --> 00:40:07.500 align:middle line:90% Random is separate. 00:40:07.500 --> 00:40:08.730 align:middle line:90% Info Hub is separate. 00:40:08.730 --> 00:40:09.660 align:middle line:90% Trust is separate. 00:40:09.660 --> 00:40:15.880 align:middle line:84% And then Trusted Info is on top of trust. 00:40:15.880 --> 00:40:17.520 align:middle line:84% And then, all of the interactions that 00:40:17.520 --> 00:40:20.060 align:middle line:84% are in the axes of all of these things. 00:40:20.060 --> 00:40:25.440 align:middle line:84% So that's the main effect, as in the main treatment profile. 00:40:25.440 --> 00:40:30.150 align:middle line:84% So you see that we already took some decisions in some sense 00:40:30.150 --> 00:40:32.340 align:middle line:84% because for example, we don't allow-- 00:40:32.340 --> 00:40:35.100 align:middle line:90% we don't do an any-- 00:40:35.100 --> 00:40:36.480 align:middle line:90% Random is its own. 00:40:36.480 --> 00:40:39.130 align:middle line:84% We don't allow random to pool with the rest 00:40:39.130 --> 00:40:42.660 align:middle line:84% because we think that seed is different than-- 00:40:42.660 --> 00:40:45.210 align:middle line:84% a chosen seed is different than a random seed. 00:40:45.210 --> 00:40:48.120 align:middle line:84% So there is already a little bit of theory in there. 00:40:48.120 --> 00:40:55.480 align:middle line:84% Theory in there, structured in there, even in doing just that. 00:40:55.480 --> 00:40:58.080 align:middle line:84% So what do we find when we do all this? 00:40:58.080 --> 00:41:01.830 align:middle line:84% So very few things survive LASSO. 00:41:01.830 --> 00:41:06.660 align:middle line:84% Only four combinations survive Lasso, 00:41:06.660 --> 00:41:10.720 align:middle line:84% in terms of increase of immunization. 00:41:10.720 --> 00:41:12.520 align:middle line:84% So this, we just do it very simple. 00:41:12.520 --> 00:41:13.560 align:middle line:90% It's a number. 00:41:13.560 --> 00:41:15.450 align:middle line:90% It's number of shots given-- 00:41:15.450 --> 00:41:19.210 align:middle line:84% number of measles shots given in the village every month. 00:41:19.210 --> 00:41:23.980 align:middle line:84% So the average, it's 7.32 in the control group. 00:41:23.980 --> 00:41:30.490 align:middle line:84% What appears to be the most effective is to do everything. 00:41:30.490 --> 00:41:31.980 align:middle line:84% So that's even after all this work, 00:41:31.980 --> 00:41:33.230 align:middle line:90% maybe you could have known it. 00:41:33.230 --> 00:41:35.370 align:middle line:90% But that was not obvious. 00:41:35.370 --> 00:41:40.800 align:middle line:90% And all of the incentives pool. 00:41:40.800 --> 00:41:43.260 align:middle line:84% Now, all of the slopy incentives pool. 00:41:43.260 --> 00:41:46.560 align:middle line:90% So high and low incentive pull. 00:41:46.560 --> 00:41:48.090 align:middle line:90% The Info Hub pool. 00:41:48.090 --> 00:41:52.890 align:middle line:84% So the trusted gossip and trusted gossip pull together. 00:41:52.890 --> 00:41:57.160 align:middle line:90% And the SMS pool. 00:41:57.160 --> 00:41:58.800 align:middle line:84% So it's pretty nice that the variance 00:41:58.800 --> 00:42:01.300 align:middle line:90% seems to make no difference. 00:42:01.300 --> 00:42:03.395 align:middle line:90% So you could go with-- 00:42:03.395 --> 00:42:04.770 align:middle line:84% if you want to pick the cheapest, 00:42:04.770 --> 00:42:09.960 align:middle line:84% you could pick low, slopy incentive SMS, 00:42:09.960 --> 00:42:17.810 align:middle line:84% low level of SMS, and gossips because that pools 00:42:17.810 --> 00:42:20.550 align:middle line:84% with the more ambitious version of it. 00:42:20.550 --> 00:42:24.780 align:middle line:84% And that increases immunization level by four, 00:42:24.780 --> 00:42:28.710 align:middle line:84% relative to control, which was a 7.32. 00:42:28.710 --> 00:42:33.510 align:middle line:84% So it's a pretty large proportional increase. 00:42:33.510 --> 00:42:37.740 align:middle line:84% The other things-- so the other thing that also works 00:42:37.740 --> 00:42:41.790 align:middle line:84% is something very similar, but using the trusted seed 00:42:41.790 --> 00:42:45.510 align:middle line:90% instead of the info hubs. 00:42:45.510 --> 00:42:47.160 align:middle line:90% That's also significant. 00:42:47.160 --> 00:42:49.830 align:middle line:84% These other two, although they were selected by LASSO 00:42:49.830 --> 00:42:52.800 align:middle line:84% and the post Lasso appears to be insignificant on their own, 00:42:52.800 --> 00:42:54.540 align:middle line:84% although you can make them-- you can 00:42:54.540 --> 00:42:57.870 align:middle line:84% set a difference, which is no seed at all, 00:42:57.870 --> 00:42:59.910 align:middle line:90% and high slope, and SMS. 00:42:59.910 --> 00:43:02.442 align:middle line:90% And then, this one is-- 00:43:02.442 --> 00:43:03.900 align:middle line:84% actually, it has a negative impact. 00:43:03.900 --> 00:43:06.670 align:middle line:84% It's selected by LASSO, but it does have a negative impact 00:43:06.670 --> 00:43:07.170 align:middle line:90% here. 00:43:07.170 --> 00:43:10.090 align:middle line:90% 00:43:10.090 --> 00:43:11.766 align:middle line:90% Once you look at-- yeah? 00:43:11.766 --> 00:43:14.910 align:middle line:84% AUDIENCE: [INAUDIBLE] I guess two questions. 00:43:14.910 --> 00:43:17.760 align:middle line:84% That line, does that mean that a whole lot of treatments 00:43:17.760 --> 00:43:19.613 align:middle line:90% are pooled in with control? 00:43:19.613 --> 00:43:21.280 align:middle line:84% ESTHER DUFLO: Yes, all of the treatments 00:43:21.280 --> 00:43:23.860 align:middle line:90% are pooled with the control. 00:43:23.860 --> 00:43:25.940 align:middle line:90% I mean, LASSO doesn't want them. 00:43:25.940 --> 00:43:27.400 align:middle line:90% So basically, they are. 00:43:27.400 --> 00:43:29.274 align:middle line:90% They become-- 00:43:29.274 --> 00:43:30.756 align:middle line:90% AUDIENCE: OK. 00:43:30.756 --> 00:43:36.360 align:middle line:84% And then on that last [INAUDIBLE] 00:43:36.360 --> 00:43:39.480 align:middle line:84% ESTHER DUFLO: So LASSO selects some variable, 00:43:39.480 --> 00:43:42.930 align:middle line:84% and then you post-- you run your OLS regression. 00:43:42.930 --> 00:43:48.570 align:middle line:84% And then, it doesn't turn out to be significant. 00:43:48.570 --> 00:43:50.100 align:middle line:90% LASSO penalizes you, basically. 00:43:50.100 --> 00:43:52.680 align:middle line:84% What LASSO does is that it penalizes you 00:43:52.680 --> 00:43:56.310 align:middle line:84% for every variable you introduce. 00:43:56.310 --> 00:43:59.130 align:middle line:90% And then, it lets you keep-- 00:43:59.130 --> 00:44:02.250 align:middle line:84% so at some point, depending on how-- 00:44:02.250 --> 00:44:05.790 align:middle line:84% depending on the penalty level you choose, 00:44:05.790 --> 00:44:08.880 align:middle line:84% it says how many variables you should introduce. 00:44:08.880 --> 00:44:11.700 align:middle line:84% And then, you introduce them by the fact 00:44:11.700 --> 00:44:13.860 align:middle line:84% that they contribute more to the variance. 00:44:13.860 --> 00:44:16.440 align:middle line:84% To this case, it's after that day, 00:44:16.440 --> 00:44:19.260 align:middle line:84% which doesn't mean they are that-- it 00:44:19.260 --> 00:44:22.320 align:middle line:84% means these other things that have even smaller coefficients. 00:44:22.320 --> 00:44:28.150 align:middle line:90% 00:44:28.150 --> 00:44:32.770 align:middle line:84% Also, once you rerun the regression with only the four, 00:44:32.770 --> 00:44:34.893 align:middle line:90% you get different coefficient. 00:44:34.893 --> 00:44:36.310 align:middle line:84% You don't get the same coefficient 00:44:36.310 --> 00:44:38.380 align:middle line:84% as when you run if you had attempted 00:44:38.380 --> 00:44:42.610 align:middle line:84% to run the full regression with OLS because all of the rest 00:44:42.610 --> 00:44:44.424 align:middle line:90% is sent to control. 00:44:44.424 --> 00:44:53.090 align:middle line:84% AUDIENCE: [INAUDIBLE] even then just 00:44:53.090 --> 00:44:56.220 align:middle line:84% changing your [INAUDIBLE] was trusted [INAUDIBLE] 00:44:56.220 --> 00:45:02.130 align:middle line:90% and that means that [INAUDIBLE] 00:45:02.130 --> 00:45:03.660 align:middle line:84% ESTHER DUFLO: This one to this one. 00:45:03.660 --> 00:45:04.860 align:middle line:90% Yeah. 00:45:04.860 --> 00:45:07.290 align:middle line:90% I don't think the trusted-- 00:45:07.290 --> 00:45:10.020 align:middle line:90% yeah, I don't know. 00:45:10.020 --> 00:45:11.670 align:middle line:90% I don't think that the trust-- 00:45:11.670 --> 00:45:18.310 align:middle line:90% 00:45:18.310 --> 00:45:22.130 align:middle line:84% it's the trusted one or not trusted. 00:45:22.130 --> 00:45:22.810 align:middle line:90% So let me think. 00:45:22.810 --> 00:45:28.060 align:middle line:90% 00:45:28.060 --> 00:45:31.165 align:middle line:84% Yeah, it's just trusted by the-- trusted info on their own. 00:45:31.165 --> 00:45:35.980 align:middle line:90% 00:45:35.980 --> 00:45:38.440 align:middle line:84% Yeah, I don't have a great interpretation 00:45:38.440 --> 00:45:41.440 align:middle line:84% for why they seem to pull with the other one. 00:45:41.440 --> 00:45:44.950 align:middle line:84% But they produce no effect on their own. 00:45:44.950 --> 00:45:47.200 align:middle line:84% This is, by the way, that's the result we had earlier. 00:45:47.200 --> 00:45:50.470 align:middle line:84% That the trusted gossip on their own seems to have no effect. 00:45:50.470 --> 00:45:53.080 align:middle line:84% But they have no effect in a way that it's not 00:45:53.080 --> 00:45:56.350 align:middle line:84% significantly different from putting them 00:45:56.350 --> 00:45:59.020 align:middle line:90% with the other info hubs. 00:45:59.020 --> 00:46:01.792 align:middle line:90% 00:46:01.792 --> 00:46:05.718 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:46:05.718 --> 00:46:06.885 align:middle line:90% ESTHER DUFLO: Yeah, exactly. 00:46:06.885 --> 00:46:13.850 align:middle line:90% 00:46:13.850 --> 00:46:16.100 align:middle line:84% Oh, incidentally, when you run this regression, 00:46:16.100 --> 00:46:19.440 align:middle line:84% you rerun it now as in each of them is an individual package. 00:46:19.440 --> 00:46:21.110 align:middle line:84% So it's the effect of-- the combo 00:46:21.110 --> 00:46:24.222 align:middle line:84% of each of these is the effect of the combo policy, not 00:46:24.222 --> 00:46:25.055 align:middle line:90% the marginal effect. 00:46:25.055 --> 00:46:31.030 align:middle line:90% 00:46:31.030 --> 00:46:33.890 align:middle line:84% When you look at immunizations per dollars, 00:46:33.890 --> 00:46:35.440 align:middle line:90% LASSO selects a few more things. 00:46:35.440 --> 00:46:37.870 align:middle line:90% But most of them are negative. 00:46:37.870 --> 00:46:38.740 align:middle line:90% Why? 00:46:38.740 --> 00:46:41.980 align:middle line:84% Because you are spending money on these programs. 00:46:41.980 --> 00:46:44.870 align:middle line:84% And we are now comparing to the control group. 00:46:44.870 --> 00:46:48.460 align:middle line:84% So you could well say that you are perfectly willing to spend 00:46:48.460 --> 00:46:50.230 align:middle line:84% money on getting more kids immunized 00:46:50.230 --> 00:46:53.960 align:middle line:84% even if the cost per immunization increases. 00:46:53.960 --> 00:46:56.830 align:middle line:84% But it could also be, like we had in the pool setting, 00:46:56.830 --> 00:46:59.470 align:middle line:84% that there is some treatment where the cost per immunization 00:46:59.470 --> 00:47:01.300 align:middle line:84% decreases because there are fixed 00:47:01.300 --> 00:47:03.170 align:middle line:90% cost of running your program. 00:47:03.170 --> 00:47:05.680 align:middle line:84% So in the cost per immunization, we include both fixed cost 00:47:05.680 --> 00:47:06.820 align:middle line:90% and variable cost. 00:47:06.820 --> 00:47:08.680 align:middle line:84% Any intervention like incentives, 00:47:08.680 --> 00:47:11.200 align:middle line:84% et cetera, increases the variable cost. 00:47:11.200 --> 00:47:13.390 align:middle line:84% But means that this fixed cost is spread 00:47:13.390 --> 00:47:15.980 align:middle line:90% over a larger group of people. 00:47:15.980 --> 00:47:18.280 align:middle line:84% So in the-- that's why in the [INAUDIBLE] pool results 00:47:18.280 --> 00:47:21.100 align:middle line:84% we had that seemingly paradoxical result. That 00:47:21.100 --> 00:47:23.780 align:middle line:84% it's cheaper to give incentive, not to give them. 00:47:23.780 --> 00:47:25.490 align:middle line:84% But it didn't have to be the case. 00:47:25.490 --> 00:47:27.790 align:middle line:84% And in fact, in the entire sample, 00:47:27.790 --> 00:47:34.092 align:middle line:84% all of the incentive results, all of the incentive effects 00:47:34.092 --> 00:47:36.710 align:middle line:84% are selected because they have this very negative treatment 00:47:36.710 --> 00:47:37.210 align:middle line:90% effect. 00:47:37.210 --> 00:47:39.610 align:middle line:84% They are expensive, the cost per dollar. 00:47:39.610 --> 00:47:41.740 align:middle line:90% But what emerges? 00:47:41.740 --> 00:47:44.260 align:middle line:84% And in fact, in this case is the only policy 00:47:44.260 --> 00:47:48.730 align:middle line:84% that is cost-effective compared to doing nothing, 00:47:48.730 --> 00:47:51.130 align:middle line:84% in the sense that it increased the cost on immunization 00:47:51.130 --> 00:47:57.820 align:middle line:84% per dollar is the doing SMS combined within info hubs. 00:47:57.820 --> 00:48:00.720 align:middle line:90% 00:48:00.720 --> 00:48:05.870 align:middle line:90% So it's gossip plus SMS. 00:48:05.870 --> 00:48:07.400 align:middle line:90% It makes sense. 00:48:07.400 --> 00:48:10.070 align:middle line:90% It's cheap to do. 00:48:10.070 --> 00:48:15.500 align:middle line:84% And therefore, the marginal cost is not 00:48:15.500 --> 00:48:20.600 align:middle line:84% very large to the extent they are impacts. 00:48:20.600 --> 00:48:23.390 align:middle line:84% The impacts means that all your fixed infrastructure 00:48:23.390 --> 00:48:25.095 align:middle line:90% is used more effectively. 00:48:25.095 --> 00:48:28.900 align:middle line:90% 00:48:28.900 --> 00:48:31.890 align:middle line:84% So now, we can do the best policy correction. 00:48:31.890 --> 00:48:36.450 align:middle line:84% So if you remember, we had an initial effect of about four 00:48:36.450 --> 00:48:39.210 align:middle line:84% for the combination for the best policy, 00:48:39.210 --> 00:48:40.680 align:middle line:84% in case of number of shots, which 00:48:40.680 --> 00:48:42.910 align:middle line:90% was the combo of everything. 00:48:42.910 --> 00:48:46.650 align:middle line:84% So we penalize it for being close to the next one. 00:48:46.650 --> 00:48:49.135 align:middle line:84% And we arrived, therefore, at the effect 00:48:49.135 --> 00:48:50.760 align:middle line:84% of an increase in immunization compared 00:48:50.760 --> 00:48:53.320 align:middle line:90% to control of about three. 00:48:53.320 --> 00:48:54.570 align:middle line:90% Four million shots per dollar. 00:48:54.570 --> 00:48:57.810 align:middle line:84% Since there is only one policy that has a positive impact, 00:48:57.810 --> 00:48:59.610 align:middle line:90% it doesn't get penalized. 00:48:59.610 --> 00:49:01.200 align:middle line:90% It's on its own. 00:49:01.200 --> 00:49:04.305 align:middle line:84% So the penalty impact is the impact. 00:49:04.305 --> 00:49:09.550 align:middle line:90% 00:49:09.550 --> 00:49:11.590 align:middle line:84% So that's a summary of the results. 00:49:11.590 --> 00:49:15.320 align:middle line:84% But I think we went through this already. 00:49:15.320 --> 00:49:16.983 align:middle line:84% So what's the policy prescription? 00:49:16.983 --> 00:49:19.400 align:middle line:84% So at some point, all these people from high end actually, 00:49:19.400 --> 00:49:20.060 align:middle line:90% they came here. 00:49:20.060 --> 00:49:22.470 align:middle line:84% And the delegation will present them the results. 00:49:22.470 --> 00:49:27.200 align:middle line:84% And so obviously, it would be-- they 00:49:27.200 --> 00:49:29.420 align:middle line:90% could do gossip SMS everywhere. 00:49:29.420 --> 00:49:32.130 align:middle line:84% It's cheap, and it's effective, and it's cost effective. 00:49:32.130 --> 00:49:34.970 align:middle line:90% So fine. 00:49:34.970 --> 00:49:37.310 align:middle line:84% But if they're also interested in increasing-- 00:49:37.310 --> 00:49:40.650 align:middle line:90% in some places doing-- 00:49:40.650 --> 00:49:42.440 align:middle line:84% really having a big push on immunization. 00:49:42.440 --> 00:49:44.000 align:middle line:90% They are willing to pay for it. 00:49:44.000 --> 00:49:49.130 align:middle line:84% The most effective policy is the combination. 00:49:49.130 --> 00:49:54.230 align:middle line:84% So then, are there places they can identify where 00:49:54.230 --> 00:49:57.360 align:middle line:90% this policy would be effective? 00:49:57.360 --> 00:49:59.950 align:middle line:84% And that's the second paper that you read, 00:49:59.950 --> 00:50:04.230 align:middle line:84% which is trying to do this predictive medicine, 00:50:04.230 --> 00:50:07.710 align:middle line:84% which is can I figure out a group of people for whom 00:50:07.710 --> 00:50:10.380 align:middle line:90% this intervention will work? 00:50:10.380 --> 00:50:12.210 align:middle line:84% And the issue is predictive medicine 00:50:12.210 --> 00:50:14.880 align:middle line:84% is even more than with multiple treatment, 00:50:14.880 --> 00:50:19.740 align:middle line:84% is this possibility of specification searching, which 00:50:19.740 --> 00:50:20.970 align:middle line:90% is exposed. 00:50:20.970 --> 00:50:23.820 align:middle line:84% You will always find a group for which it looks 00:50:23.820 --> 00:50:27.160 align:middle line:90% like the treatment was working. 00:50:27.160 --> 00:50:29.370 align:middle line:84% So that's why the FDA does not allow 00:50:29.370 --> 00:50:35.300 align:middle line:84% you to report any subgroup that were not pre-specified. 00:50:35.300 --> 00:50:36.290 align:middle line:90% You just cannot. 00:50:36.290 --> 00:50:40.190 align:middle line:84% Otherwise, you say, well, my drug doesn't work on average. 00:50:40.190 --> 00:50:43.970 align:middle line:84% But it worked on people which have read charts 00:50:43.970 --> 00:50:46.310 align:middle line:84% on the day of the trial, and then you can always 00:50:46.310 --> 00:50:48.000 align:middle line:90% come up with a reason. 00:50:48.000 --> 00:50:52.070 align:middle line:84% So in economics, we are a little less stressed about that 00:50:52.070 --> 00:50:56.660 align:middle line:84% probably because there are less financial incentive reasons 00:50:56.660 --> 00:50:58.460 align:middle line:90% to push a particular results. 00:50:58.460 --> 00:51:01.550 align:middle line:90% And there is always-- 00:51:01.550 --> 00:51:04.670 align:middle line:84% we have more tolerance, I think, for theorizing ex-post 00:51:04.670 --> 00:51:07.400 align:middle line:84% about results, partly because we like theory, and we should. 00:51:07.400 --> 00:51:09.710 align:middle line:84% And we've learned something about the world, 00:51:09.710 --> 00:51:11.960 align:middle line:84% and even if the subgroups are experts, 00:51:11.960 --> 00:51:14.760 align:middle line:84% you may have learned something about them. 00:51:14.760 --> 00:51:17.220 align:middle line:84% So it would be a bit sad not to use them. 00:51:17.220 --> 00:51:19.880 align:middle line:84% So that's the exact same-- it's not exactly 00:51:19.880 --> 00:51:21.620 align:middle line:84% the same trade off as before, but there 00:51:21.620 --> 00:51:23.700 align:middle line:90% is a trade off of that nature. 00:51:23.700 --> 00:51:26.840 align:middle line:84% So going all the way to the medicine standard and say, 00:51:26.840 --> 00:51:31.670 align:middle line:84% we are just going to allow a very few pre-analysis plan 00:51:31.670 --> 00:51:41.120 align:middle line:84% and penalize multiple comparison with q-values or other forms 00:51:41.120 --> 00:51:43.700 align:middle line:84% of adjusting for multiple comparisons. 00:51:43.700 --> 00:51:47.630 align:middle line:84% Then, you would get a bit boring. 00:51:47.630 --> 00:51:49.940 align:middle line:84% You wouldn't be able to get as much information. 00:51:49.940 --> 00:51:52.860 align:middle line:90% 00:51:52.860 --> 00:51:55.230 align:middle line:84% And often, we have a lot of data. 00:51:55.230 --> 00:51:57.630 align:middle line:84% And we would like to use it, but we would like 00:51:57.630 --> 00:51:59.850 align:middle line:90% to use it in a disciplined way. 00:51:59.850 --> 00:52:03.360 align:middle line:84% Hence, machine learning has the advantage 00:52:03.360 --> 00:52:07.620 align:middle line:84% of potentially giving you some rules, some algorithm that you 00:52:07.620 --> 00:52:09.480 align:middle line:84% can specify in advance, that you could 00:52:09.480 --> 00:52:11.250 align:middle line:90% put in your pre-analysis plan. 00:52:11.250 --> 00:52:14.550 align:middle line:84% I'm going to use so-and-so's machine learning technique 00:52:14.550 --> 00:52:17.250 align:middle line:84% to look for possible-- for whether they 00:52:17.250 --> 00:52:20.590 align:middle line:84% are heterogeneous effect and who seemed to be the most affected. 00:52:20.590 --> 00:52:24.300 align:middle line:84% So in principle, the hope is you get the best of both worlds 00:52:24.300 --> 00:52:27.150 align:middle line:84% because you have the ability to look 00:52:27.150 --> 00:52:30.330 align:middle line:84% for differences in a statistically disciplined way. 00:52:30.330 --> 00:52:33.720 align:middle line:84% And you won't do too much data mining. 00:52:33.720 --> 00:52:38.100 align:middle line:84% The problem is minor problem, machine learning 00:52:38.100 --> 00:52:42.960 align:middle line:90% is really not a tool for saying. 00:52:42.960 --> 00:52:45.060 align:middle line:90% It's a tool for prediction. 00:52:45.060 --> 00:52:47.190 align:middle line:84% It's a tool for predicting what's going to happen, 00:52:47.190 --> 00:52:50.400 align:middle line:84% but it's not a tool to interpret any differences 00:52:50.400 --> 00:52:52.050 align:middle line:90% in treatment effect we find. 00:52:52.050 --> 00:52:54.720 align:middle line:84% And in particular, in the absence of very 00:52:54.720 --> 00:52:58.360 align:middle line:90% strict conditions to be met. 00:52:58.360 --> 00:53:01.140 align:middle line:84% We don't know how to estimate the causal average treatment 00:53:01.140 --> 00:53:04.020 align:middle line:84% effect in a way that is interpretable. 00:53:04.020 --> 00:53:06.780 align:middle line:84% So the idea that, OK, I'm going to-- 00:53:06.780 --> 00:53:11.040 align:middle line:84% the full predictive medicine ideal for anybody 00:53:11.040 --> 00:53:13.500 align:middle line:84% with any set of x characteristics, 00:53:13.500 --> 00:53:15.900 align:middle line:84% and I can predict their treatment effect. 00:53:15.900 --> 00:53:17.490 align:middle line:84% We just don't know how to do that, 00:53:17.490 --> 00:53:20.200 align:middle line:84% unless we are willing to be super restrictive. 00:53:20.200 --> 00:53:23.970 align:middle line:84% And then, Susan [INAUDIBLE] has a paper 00:53:23.970 --> 00:53:29.010 align:middle line:84% with Steven Wagner with enough under sufficiently harsh 00:53:29.010 --> 00:53:33.420 align:middle line:84% conditions, you can do things like trees 00:53:33.420 --> 00:53:37.250 align:middle line:84% to give an answer to this question. 00:53:37.250 --> 00:53:39.490 align:middle line:84% But these conditions are pretty hard, 00:53:39.490 --> 00:53:41.860 align:middle line:84% and they are not really testable. 00:53:41.860 --> 00:53:44.540 align:middle line:90% So what can you do? 00:53:44.540 --> 00:53:50.260 align:middle line:84% Well, you can say, let's give up on estimate the whole function. 00:53:50.260 --> 00:53:52.490 align:middle line:84% Can we say at least a few things? 00:53:52.490 --> 00:53:57.070 align:middle line:84% So we give up on estimating the full condition of a treatment 00:53:57.070 --> 00:54:01.000 align:middle line:84% effect as opposed unrestricted as a function of all those x's 00:54:01.000 --> 00:54:03.490 align:middle line:84% that people or villages might have. 00:54:03.490 --> 00:54:06.110 align:middle line:84% But I want to answer a few simple questions. 00:54:06.110 --> 00:54:09.250 align:middle line:84% Number one, is there any heterogeneity? 00:54:09.250 --> 00:54:11.050 align:middle line:90% So that's one parameter. 00:54:11.050 --> 00:54:15.280 align:middle line:84% Number two, if there is any-- or is it just any heterogeneity 00:54:15.280 --> 00:54:17.320 align:middle line:84% that I think is there is in fact, not there, 00:54:17.320 --> 00:54:21.310 align:middle line:84% number two if there is any heterogeneity, can I put-- 00:54:21.310 --> 00:54:26.890 align:middle line:84% can I make say, five groups where I estimate the treatment 00:54:26.890 --> 00:54:28.760 align:middle line:90% effect for these five groups. 00:54:28.760 --> 00:54:30.850 align:middle line:84% So it's now five parameters, still 00:54:30.850 --> 00:54:33.130 align:middle line:90% a small number of parameters. 00:54:33.130 --> 00:54:35.710 align:middle line:84% And then, maybe one more step is saying, well, 00:54:35.710 --> 00:54:37.750 align:middle line:84% now that I have these five groups, 00:54:37.750 --> 00:54:40.660 align:middle line:84% maybe I can describe the characteristics of these five 00:54:40.660 --> 00:54:41.440 align:middle line:90% groups. 00:54:41.440 --> 00:54:43.033 align:middle line:90% It's not going to be causal. 00:54:43.033 --> 00:54:44.200 align:middle line:90% It's going to be predictive. 00:54:44.200 --> 00:54:46.990 align:middle line:84% But maybe it's interesting, anyway. 00:54:46.990 --> 00:54:50.680 align:middle line:84% So that's what was in the paper was Victor. 00:54:50.680 --> 00:54:53.920 align:middle line:84% We use the machinery of the Bell machine 00:54:53.920 --> 00:54:58.210 align:middle line:84% learning that Victor Whitney had co-developed that we already 00:54:58.210 --> 00:55:01.570 align:middle line:84% discussed in the [? Gana ?] paper to apply to this problem. 00:55:01.570 --> 00:55:04.090 align:middle line:90% 00:55:04.090 --> 00:55:07.540 align:middle line:84% Not important to answer to understand 00:55:07.540 --> 00:55:09.270 align:middle line:90% the detail of how it works. 00:55:09.270 --> 00:55:11.020 align:middle line:84% The most important thing is what I already 00:55:11.020 --> 00:55:12.270 align:middle line:90% told you, which is very clear. 00:55:12.270 --> 00:55:14.430 align:middle line:90% Reduce your ambitions. 00:55:14.430 --> 00:55:17.850 align:middle line:84% The second most important thing is as in the first thing, 00:55:17.850 --> 00:55:23.460 align:middle line:84% as in that first paper, use sample splitting. 00:55:23.460 --> 00:55:25.980 align:middle line:84% And basically, think, for example, 00:55:25.980 --> 00:55:27.380 align:middle line:90% is there any heterogeneity? 00:55:27.380 --> 00:55:30.110 align:middle line:90% How is it going to work? 00:55:30.110 --> 00:55:32.330 align:middle line:90% Well, I split my sample in two. 00:55:32.330 --> 00:55:35.930 align:middle line:84% In half of the sample, I use any machine 00:55:35.930 --> 00:55:40.040 align:middle line:84% learning tool I like to come up with my best 00:55:40.040 --> 00:55:43.190 align:middle line:84% guess for the causal average treatment effect. 00:55:43.190 --> 00:55:47.270 align:middle line:84% That is, think of it if you think of it in some version 00:55:47.270 --> 00:55:48.450 align:middle line:90% of LASSO, for example. 00:55:48.450 --> 00:55:51.980 align:middle line:84% It's interacting all my x's and all of the interaction 00:55:51.980 --> 00:55:57.910 align:middle line:90% possible with the treatment. 00:55:57.910 --> 00:55:59.470 align:middle line:84% And then, that gives in principle 00:55:59.470 --> 00:56:01.240 align:middle line:84% a prediction for each person of what 00:56:01.240 --> 00:56:03.270 align:middle line:90% the treatment effect should be. 00:56:03.270 --> 00:56:07.740 align:middle line:84% I know that this is unlikely to be great because it's full 00:56:07.740 --> 00:56:12.990 align:middle line:84% of flux, and noise, and stuff like that that are going to be 00:56:12.990 --> 00:56:17.190 align:middle line:84% wrongly attributed to the effect of the x's. 00:56:17.190 --> 00:56:19.155 align:middle line:90% But I still do it. 00:56:19.155 --> 00:56:22.110 align:middle line:84% You could use LASSO, random forest, neural network, 00:56:22.110 --> 00:56:26.850 align:middle line:84% anything you like to make that prediction for each combination 00:56:26.850 --> 00:56:29.740 align:middle line:90% of covariates. 00:56:29.740 --> 00:56:33.300 align:middle line:84% And then, in the sample that you left behind, 00:56:33.300 --> 00:56:35.580 align:middle line:84% so note that it's different from sample 00:56:35.580 --> 00:56:39.540 align:middle line:84% splitting as it's used in ML typically because for example, 00:56:39.540 --> 00:56:41.880 align:middle line:84% in order to do even the exercise that I'm doing, 00:56:41.880 --> 00:56:43.920 align:middle line:84% usually, you will need sample splitting to tune 00:56:43.920 --> 00:56:45.160 align:middle line:90% the parameter and all that. 00:56:45.160 --> 00:56:47.160 align:middle line:84% So that's not-- that's sample splitting. 00:56:47.160 --> 00:56:50.830 align:middle line:84% My sample splitting is that in my half of the sample, 00:56:50.830 --> 00:56:55.920 align:middle line:84% I put my best foot forward to estimate the thing. 00:56:55.920 --> 00:57:00.720 align:middle line:84% And then, in the half of the sample that I kept, 00:57:00.720 --> 00:57:02.760 align:middle line:84% I'm going to regress the treatment 00:57:02.760 --> 00:57:06.180 align:middle line:84% effect on this characteristic and see 00:57:06.180 --> 00:57:09.510 align:middle line:90% whether I get just linearly. 00:57:09.510 --> 00:57:12.630 align:middle line:84% And say, if there is any effect, I should find one. 00:57:12.630 --> 00:57:17.445 align:middle line:84% If the two effect was as estimated in-- 00:57:17.445 --> 00:57:21.570 align:middle line:84% if the two heterogeneity in effect 00:57:21.570 --> 00:57:25.260 align:middle line:84% was as estimated in my machine learning sample, 00:57:25.260 --> 00:57:28.110 align:middle line:84% in the test sample I should find a coefficient of one 00:57:28.110 --> 00:57:31.980 align:middle line:84% when I try to predict it for each person. 00:57:31.980 --> 00:57:35.140 align:middle line:84% It should line up in the same way. 00:57:35.140 --> 00:57:38.010 align:middle line:84% And if in fact, it was all noise, 00:57:38.010 --> 00:57:39.970 align:middle line:84% it's not going to line up in the same way. 00:57:39.970 --> 00:57:43.380 align:middle line:84% So I'm going to find a big fat zero. 00:57:43.380 --> 00:57:46.830 align:middle line:84% Empirically, in my experience of trying this, most of the time 00:57:46.830 --> 00:57:49.140 align:middle line:84% you stop there because you find zero. 00:57:49.140 --> 00:57:51.630 align:middle line:84% That where you think that you're heterogeneity 00:57:51.630 --> 00:57:55.590 align:middle line:84% is maybe you even told yourself some nice stories about why 00:57:55.590 --> 00:57:57.060 align:middle line:90% it would be heterogenous. 00:57:57.060 --> 00:58:00.120 align:middle line:84% And in fact, once you do it in the other sample, 00:58:00.120 --> 00:58:02.760 align:middle line:84% and you regress the predicted treatment effect 00:58:02.760 --> 00:58:08.970 align:middle line:84% on the actual treatment effect for the person 00:58:08.970 --> 00:58:13.690 align:middle line:84% with those characteristics, you don't find any difference. 00:58:13.690 --> 00:58:16.860 align:middle line:84% If you do, then you can say, oh, great. 00:58:16.860 --> 00:58:22.470 align:middle line:84% Then now, I can separate my groups 00:58:22.470 --> 00:58:26.460 align:middle line:84% into who is the most likely to be affected by the treatment, 00:58:26.460 --> 00:58:32.730 align:middle line:84% or by quintile, and run and calculate the treatment 00:58:32.730 --> 00:58:35.400 align:middle line:84% effect for the people who are predicted 00:58:35.400 --> 00:58:40.560 align:middle line:84% to be the least affected, and the second least affected, 00:58:40.560 --> 00:58:41.970 align:middle line:90% and the third least affected. 00:58:41.970 --> 00:58:44.730 align:middle line:84% And so I make the separation based 00:58:44.730 --> 00:58:46.110 align:middle line:90% on my machine learning sample. 00:58:46.110 --> 00:58:51.570 align:middle line:84% And then, I estimate the treatment effect in that group. 00:58:51.570 --> 00:58:54.470 align:middle line:84% So this is a treatment effect in that group in the leftover 00:58:54.470 --> 00:58:57.000 align:middle line:90% sample. 00:58:57.000 --> 00:58:59.850 align:middle line:84% And then once you've done that once, you can do it many, 00:58:59.850 --> 00:59:02.540 align:middle line:84% many, many times, many of the splits many times. 00:59:02.540 --> 00:59:05.560 align:middle line:84% And then, you even get a standard, therefore, 00:59:05.560 --> 00:59:07.460 align:middle line:90% for that effect. 00:59:07.460 --> 00:59:11.120 align:middle line:84% That reflects both the actual noise you have in your data set 00:59:11.120 --> 00:59:14.420 align:middle line:84% and the fact that your sample split. 00:59:14.420 --> 00:59:16.788 align:middle line:84% So each sample still introduce its own noise that 00:59:16.788 --> 00:59:18.080 align:middle line:90% needs to be taken into account. 00:59:18.080 --> 00:59:18.830 align:middle line:90% Yep? 00:59:18.830 --> 00:59:20.210 align:middle line:84% AUDIENCE: What exactly is the difference 00:59:20.210 --> 00:59:22.585 align:middle line:84% between what you do in the training set and the test set? 00:59:22.585 --> 00:59:24.418 align:middle line:84% So in the training set, it's the first time. 00:59:24.418 --> 00:59:26.793 align:middle line:84% ESTHER DUFLO: You will stimulate the treatment-- you will 00:59:26.793 --> 00:59:28.210 align:middle line:90% stimulate the treatment effect. 00:59:28.210 --> 00:59:29.350 align:middle line:84% AUDIENCE: Interactive regression. 00:59:29.350 --> 00:59:31.058 align:middle line:84% ESTHER DUFLO: The interactive regression. 00:59:31.058 --> 00:59:33.370 align:middle line:84% For example, think of it this way. 00:59:33.370 --> 00:59:41.420 align:middle line:84% In the second set you predict for each person 00:59:41.420 --> 00:59:44.960 align:middle line:84% what's there as a function of your estimate 00:59:44.960 --> 00:59:49.350 align:middle line:84% in the z, what their treatment effect would have been. 00:59:49.350 --> 00:59:50.390 align:middle line:90% And, then you regress. 00:59:50.390 --> 01:00:01.650 align:middle line:84% For example, you run a linear regression of why your outcome. 01:00:01.650 --> 01:00:04.600 align:middle line:90% 01:00:04.600 --> 01:00:08.370 align:middle line:84% So think of running the following regression. 01:00:08.370 --> 01:00:11.040 align:middle line:84% On your outcome, on treatment, treatment interaction 01:00:11.040 --> 01:00:13.140 align:middle line:84% interacted with this predicted effect 01:00:13.140 --> 01:00:16.260 align:middle line:90% and the predicted effect. 01:00:16.260 --> 01:00:18.080 align:middle line:84% So this is now a linear coefficient. 01:00:18.080 --> 01:00:22.980 align:middle line:84% And just asking, are the people that I 01:00:22.980 --> 01:00:26.640 align:middle line:84% predict to have the largest treatment effect, 01:00:26.640 --> 01:00:29.960 align:middle line:84% do they in fact have a larger treatment effect? 01:00:29.960 --> 01:00:34.490 align:middle line:84% That linear parameter is of no interest in and of itself. 01:00:34.490 --> 01:00:38.270 align:middle line:84% It just say, it should be-- ideally, 01:00:38.270 --> 01:00:41.570 align:middle line:84% if that estimate was very, very precise, 01:00:41.570 --> 01:00:44.480 align:middle line:90% that coefficient would be one. 01:00:44.480 --> 01:00:47.126 align:middle line:84% If that estimate is junk, that estimate would be zero. 01:00:47.126 --> 01:00:50.430 align:middle line:90% 01:00:50.430 --> 01:00:53.190 align:middle line:84% That's the main use of that linear thing. 01:00:53.190 --> 01:00:58.680 align:middle line:84% Then, what you're doing here is similar, but dummified, 01:00:58.680 --> 01:01:00.570 align:middle line:90% dummified version. 01:01:00.570 --> 01:01:07.110 align:middle line:84% You basically on the basis of the estimate in your machine 01:01:07.110 --> 01:01:11.310 align:middle line:84% learning sample, you've made five groups 01:01:11.310 --> 01:01:14.400 align:middle line:84% of the people who are predicted to be the most affected 01:01:14.400 --> 01:01:19.170 align:middle line:84% based on their X characteristics and their baseline 01:01:19.170 --> 01:01:20.990 align:middle line:90% characteristics. 01:01:20.990 --> 01:01:25.160 align:middle line:84% And then, in your test sample, you 01:01:25.160 --> 01:01:27.950 align:middle line:84% are actually estimating the treatment effect 01:01:27.950 --> 01:01:30.110 align:middle line:84% for the people who have these characteristics, 01:01:30.110 --> 01:01:33.850 align:middle line:84% this combination of characteristics. 01:01:33.850 --> 01:01:36.010 align:middle line:84% In that quintile of the people who 01:01:36.010 --> 01:01:38.360 align:middle line:84% are predicted to have the lowest treatment effect. 01:01:38.360 --> 01:01:42.010 align:middle line:84% So again, if it was all junk, it would just 01:01:42.010 --> 01:01:44.980 align:middle line:84% be a line because the people who have the lowest treatment 01:01:44.980 --> 01:01:47.110 align:middle line:84% effect in your machine learning sample, in fact, 01:01:47.110 --> 01:01:48.850 align:middle line:90% don't in real life. 01:01:48.850 --> 01:01:53.540 align:middle line:90% 01:01:53.540 --> 01:01:55.380 align:middle line:90% Makes sense? 01:01:55.380 --> 01:01:58.200 align:middle line:90% And then, so you can do that. 01:01:58.200 --> 01:02:00.905 align:middle line:84% So here, these are the results for the immunization sample. 01:02:00.905 --> 01:02:03.790 align:middle line:90% 01:02:03.790 --> 01:02:07.920 align:middle line:84% So they are exceptionally nice in that they actually-- 01:02:07.920 --> 01:02:10.450 align:middle line:90% something comes out of it. 01:02:10.450 --> 01:02:12.360 align:middle line:84% It's not always the case, as I said. 01:02:12.360 --> 01:02:15.180 align:middle line:84% And what you find is very, very strong heterogeneity 01:02:15.180 --> 01:02:18.540 align:middle line:84% to the point that the places that are most affected 01:02:18.540 --> 01:02:19.620 align:middle line:90% are really affected. 01:02:19.620 --> 01:02:21.930 align:middle line:90% You have huge treatment effect. 01:02:21.930 --> 01:02:25.020 align:middle line:84% And the places which are not, which are least effective, 01:02:25.020 --> 01:02:26.561 align:middle line:90% have negative treatment effect. 01:02:26.561 --> 01:02:30.710 align:middle line:90% 01:02:30.710 --> 01:02:33.170 align:middle line:84% So then, what you can do is two things. 01:02:33.170 --> 01:02:35.470 align:middle line:84% First, I will discuss that for a minute 01:02:35.470 --> 01:02:40.480 align:middle line:84% of why is it even possible to have negative treatment effect. 01:02:40.480 --> 01:02:45.460 align:middle line:84% And secondly, you could say that the last thing that we do 01:02:45.460 --> 01:02:50.030 align:middle line:84% is to say, well, we can do a classification analysis, 01:02:50.030 --> 01:02:52.900 align:middle line:84% which is to say, what are the characters-- this 01:02:52.900 --> 01:02:55.270 align:middle line:84% is like a bunch of these interacted with each other 01:02:55.270 --> 01:02:56.590 align:middle line:90% in a complicated way. 01:02:56.590 --> 01:02:59.590 align:middle line:84% But if I just run a summary statistics, 01:02:59.590 --> 01:03:04.480 align:middle line:84% just a summary statistics table of the sum characteristic 01:03:04.480 --> 01:03:07.690 align:middle line:84% of these different groups, do they look different, 01:03:07.690 --> 01:03:09.112 align:middle line:90% the groups? 01:03:09.112 --> 01:03:10.570 align:middle line:84% So there you have to be careful not 01:03:10.570 --> 01:03:12.610 align:middle line:84% to overinterpret this difference because again, they're 01:03:12.610 --> 01:03:13.150 align:middle line:90% not causal. 01:03:13.150 --> 01:03:15.430 align:middle line:84% But at least it gives you a description. 01:03:15.430 --> 01:03:17.320 align:middle line:84% And what we find that's pretty interesting 01:03:17.320 --> 01:03:19.360 align:middle line:84% is that the places that are the most affected 01:03:19.360 --> 01:03:22.060 align:middle line:84% are the ones where immunization rates were very, very 01:03:22.060 --> 01:03:24.380 align:middle line:90% low to start with. 01:03:24.380 --> 01:03:26.410 align:middle line:84% And the places that were the least affected 01:03:26.410 --> 01:03:28.160 align:middle line:84% are the places that have high immunization 01:03:28.160 --> 01:03:29.600 align:middle line:90% rates to start with. 01:03:29.600 --> 01:03:32.030 align:middle line:84% So I remember I first presented that in the bank, 01:03:32.030 --> 01:03:36.320 align:middle line:84% in the World Bank and my friend [? Jishnu Des, ?] 01:03:36.320 --> 01:03:40.700 align:middle line:84% whose work we are going to talk a lot about in the rest 01:03:40.700 --> 01:03:43.300 align:middle line:90% of today and next week. 01:03:43.300 --> 01:03:44.230 align:middle line:90% I was like, come on. 01:03:44.230 --> 01:03:47.590 align:middle line:84% You cannot write a paper where you do this whole thing and you 01:03:47.590 --> 01:03:50.333 align:middle line:84% just tell me the people who are most affected by-- 01:03:50.333 --> 01:03:52.750 align:middle line:84% and immediately forgot about the people whose immunization 01:03:52.750 --> 01:03:55.390 align:middle line:90% level were low to start with. 01:03:55.390 --> 01:03:57.990 align:middle line:84% And first I was like, yeah, yeah, you're right. 01:03:57.990 --> 01:03:58.990 align:middle line:90% That sounds a bit silly. 01:03:58.990 --> 01:04:02.050 align:middle line:84% But then I thought about it, and it's like, why? 01:04:02.050 --> 01:04:04.132 align:middle line:84% Even the people with high immunization rate, 01:04:04.132 --> 01:04:06.340 align:middle line:84% it's not that they have super-high immunization rate. 01:04:06.340 --> 01:04:09.080 align:middle line:84% They have plenty of scope to improve. 01:04:09.080 --> 01:04:10.900 align:middle line:90% So it's not mechanical. 01:04:10.900 --> 01:04:12.190 align:middle line:90% They are not bounded. 01:04:12.190 --> 01:04:15.350 align:middle line:84% They are not at all in the range where they would be bounded. 01:04:15.350 --> 01:04:18.010 align:middle line:84% So nothing says that the impact of intervention 01:04:18.010 --> 01:04:20.120 align:middle line:90% are concave, in general. 01:04:20.120 --> 01:04:22.498 align:middle line:84% In fact, in many cases, it's for example, 01:04:22.498 --> 01:04:24.040 align:middle line:84% in the ultra poor program, the people 01:04:24.040 --> 01:04:27.220 align:middle line:84% who benefit the most are the richest of the ultra poor. 01:04:27.220 --> 01:04:32.610 align:middle line:84% So I don't think it is mechanical, but maybe it is. 01:04:32.610 --> 01:04:35.130 align:middle line:84% It's true that a little bit like the first paper, 01:04:35.130 --> 01:04:40.110 align:middle line:84% it sounds a bit sad after all this exercise 01:04:40.110 --> 01:04:42.600 align:middle line:84% to come up with something obvious. 01:04:42.600 --> 01:04:45.330 align:middle line:84% And then, the other thing that is clearly not mechanical, 01:04:45.330 --> 01:04:46.770 align:middle line:84% is that you get a negative impact. 01:04:46.770 --> 01:04:51.450 align:middle line:90% 01:04:51.450 --> 01:04:54.075 align:middle line:84% Which might be the most interesting-- 01:04:54.075 --> 01:04:56.580 align:middle line:84% or is probably the most interesting and scary results 01:04:56.580 --> 01:04:57.310 align:middle line:90% of here. 01:04:57.310 --> 01:05:00.810 align:middle line:84% So why do you think the result might be-- 01:05:00.810 --> 01:05:04.750 align:middle line:90% 01:05:04.750 --> 01:05:07.705 align:middle line:84% why do you think some people might be negatively affected? 01:05:07.705 --> 01:05:11.740 align:middle line:90% 01:05:11.740 --> 01:05:14.970 align:middle line:84% Some villages and therefore, some people 01:05:14.970 --> 01:05:19.640 align:middle line:84% might be negatively affected by a program like that. 01:05:19.640 --> 01:05:27.605 align:middle line:90% 01:05:27.605 --> 01:05:33.545 align:middle line:90% AUDIENCE: [INAUDIBLE] 01:05:33.545 --> 01:05:36.160 align:middle line:84% ESTHER DUFLO: Do you mean they were already doing something 01:05:36.160 --> 01:05:36.700 align:middle line:90% and-- 01:05:36.700 --> 01:05:38.470 align:middle line:84% AUDIENCE: If other people are positioning 01:05:38.470 --> 01:05:42.017 align:middle line:90% themselves [INAUDIBLE] 01:05:42.017 --> 01:05:43.600 align:middle line:84% ESTHER DUFLO: It's the number of shots 01:05:43.600 --> 01:05:47.690 align:middle line:84% that are given in the month that's the outcome. 01:05:47.690 --> 01:05:50.030 align:middle line:84% So there might be crowding out of other things. 01:05:50.030 --> 01:05:52.690 align:middle line:84% But here, for example, if people come from other villages, 01:05:52.690 --> 01:05:54.970 align:middle line:90% they would be counted. 01:05:54.970 --> 01:05:58.682 align:middle line:84% This is including any crowding of other people 01:05:58.682 --> 01:05:59.515 align:middle line:90% from other villages. 01:05:59.515 --> 01:06:02.770 align:middle line:90% 01:06:02.770 --> 01:06:03.760 align:middle line:90% Yep. 01:06:03.760 --> 01:06:05.177 align:middle line:84% AUDIENCE: One hypothesis, I guess, 01:06:05.177 --> 01:06:07.052 align:middle line:84% that was mentioned briefly in the paper which 01:06:07.052 --> 01:06:08.890 align:middle line:84% is that some people are very intrinsically 01:06:08.890 --> 01:06:10.170 align:middle line:90% motivated to get vaccinated. 01:06:10.170 --> 01:06:13.930 align:middle line:84% And once you introduce this external encouragement, 01:06:13.930 --> 01:06:17.610 align:middle line:84% then there's maybe some behavioral explanation 01:06:17.610 --> 01:06:19.720 align:middle line:84% of actually I don't want to do it anymore. 01:06:19.720 --> 01:06:21.220 align:middle line:84% ESTHER DUFLO: Yes, maybe this is one 01:06:21.220 --> 01:06:28.950 align:middle line:84% of these example of weaknesses as a series of paper. 01:06:28.950 --> 01:06:31.000 align:middle line:84% One is called "Finding the Price" 01:06:31.000 --> 01:06:38.410 align:middle line:84% that shows that when a school penalizes parents, 01:06:38.410 --> 01:06:41.050 align:middle line:84% gives the fine to people for parents 01:06:41.050 --> 01:06:45.100 align:middle line:84% when they collect their kids late at daycare, 01:06:45.100 --> 01:06:47.860 align:middle line:84% the number of parents who send their kids-- who pick up 01:06:47.860 --> 01:06:51.220 align:middle line:84% their kids late actually increases 01:06:51.220 --> 01:06:55.300 align:middle line:84% because before, people thought, oh, this is the deadline. 01:06:55.300 --> 01:06:56.500 align:middle line:90% I have to pick up my kids. 01:06:56.500 --> 01:07:03.940 align:middle line:84% And now, it's you're actually sending a signal that you 01:07:03.940 --> 01:07:08.170 align:middle line:90% can do it against some price. 01:07:08.170 --> 01:07:11.816 align:middle line:90% Other example-- 01:07:11.816 --> 01:07:15.720 align:middle line:84% AUDIENCE: Another reason why all this stuff happened 01:07:15.720 --> 01:07:18.960 align:middle line:84% was if everyone in the village had already 01:07:18.960 --> 01:07:20.710 align:middle line:84% been treated with the [INAUDIBLE] 01:07:20.710 --> 01:07:23.370 align:middle line:84% people for vaccinations, and everyone else needs lots 01:07:23.370 --> 01:07:25.350 align:middle line:90% of information [INAUDIBLE]. 01:07:25.350 --> 01:07:28.360 align:middle line:84% But with financial incentive, they might wonder like, 01:07:28.360 --> 01:07:32.448 align:middle line:84% why is the government or whoever it is trying to get money 01:07:32.448 --> 01:07:35.343 align:middle line:90% [INAUDIBLE] people-- 01:07:35.343 --> 01:07:36.510 align:middle line:90% ESTHER DUFLO: Yeah, exactly. 01:07:36.510 --> 01:07:40.680 align:middle line:84% So the people-- so some version of it, which behavioral or not 01:07:40.680 --> 01:07:43.200 align:middle line:84% could be that people were already doing it. 01:07:43.200 --> 01:07:46.530 align:middle line:84% And then, suddenly, there is an incentive. 01:07:46.530 --> 01:07:48.060 align:middle line:90% Why do they need me? 01:07:48.060 --> 01:07:49.860 align:middle line:84% Why do they need to pay me to do that? 01:07:49.860 --> 01:07:52.430 align:middle line:90% 01:07:52.430 --> 01:07:55.980 align:middle line:84% It must be really bad if they need to pay me to do that. 01:07:55.980 --> 01:07:58.110 align:middle line:84% I shouldn't want to do it by myself. 01:07:58.110 --> 01:08:00.870 align:middle line:84% In the context of the COVID 19 vaccine, for example, 01:08:00.870 --> 01:08:05.040 align:middle line:84% in discussion in France of making it mandatory 01:08:05.040 --> 01:08:10.297 align:middle line:84% or quasi-mandatory, which is what it is now, 01:08:10.297 --> 01:08:11.130 align:middle line:90% it certainly wasn't. 01:08:11.130 --> 01:08:13.770 align:middle line:84% It was there that if you make something mandatory, then 01:08:13.770 --> 01:08:15.000 align:middle line:90% it says-- 01:08:15.000 --> 01:08:19.069 align:middle line:84% it sends a signal that it's not something you should desire. 01:08:19.069 --> 01:08:22.700 align:middle line:84% And instead, in fact, people were extremely suspicious 01:08:22.700 --> 01:08:23.899 align:middle line:90% of the vaccine in France. 01:08:23.899 --> 01:08:27.020 align:middle line:90% It was one of the highest-- 01:08:27.020 --> 01:08:29.689 align:middle line:84% before the vaccination campaign even started, 01:08:29.689 --> 01:08:35.960 align:middle line:84% it was one of the highest suspicion countries. 01:08:35.960 --> 01:08:39.260 align:middle line:84% And then, the beginning of the campaign was totally butched. 01:08:39.260 --> 01:08:41.180 align:middle line:90% It was really slow. 01:08:41.180 --> 01:08:44.720 align:middle line:84% In the first two weeks, they immunized 12 people. 01:08:44.720 --> 01:08:49.609 align:middle line:84% Since they immunized some woman called Morrisette in an old age 01:08:49.609 --> 01:08:50.720 align:middle line:90% home person. 01:08:50.720 --> 01:08:53.180 align:middle line:84% They immunized 12 people in two weeks. 01:08:53.180 --> 01:08:54.986 align:middle line:90% It was a disaster. 01:08:54.986 --> 01:08:56.569 align:middle line:84% The whole thing was about there is not 01:08:56.569 --> 01:08:58.520 align:middle line:84% going to be enough vaccine, and we are not 01:08:58.520 --> 01:09:00.410 align:middle line:90% going to be able to do it. 01:09:00.410 --> 01:09:06.000 align:middle line:84% And then, the view on whether people wanted to get vaccinated 01:09:06.000 --> 01:09:07.270 align:middle line:90% changed. 01:09:07.270 --> 01:09:09.000 align:middle line:84% And one possible interpretation is 01:09:09.000 --> 01:09:11.430 align:middle line:84% that that became to be seen like something rare. 01:09:11.430 --> 01:09:14.310 align:middle line:84% That you should desire because there is not enough of it. 01:09:14.310 --> 01:09:17.420 align:middle line:84% And there is not-- and these idiots of the government, 01:09:17.420 --> 01:09:20.760 align:middle line:84% they are not able to deliver this good that we really need. 01:09:20.760 --> 01:09:25.830 align:middle line:84% And therefore, your mistrust in the government paradoxically, 01:09:25.830 --> 01:09:28.350 align:middle line:84% I think, played in there in the favor 01:09:28.350 --> 01:09:32.710 align:middle line:84% of generating a consensus more than anything else. 01:09:32.710 --> 01:09:36.630 align:middle line:84% And then, I'm not saying they did that on purpose. 01:09:36.630 --> 01:09:39.210 align:middle line:90% I know they didn't. 01:09:39.210 --> 01:09:41.760 align:middle line:84% And I don't know whether my explanation is correct, 01:09:41.760 --> 01:09:43.800 align:middle line:90% but it could be. 01:09:43.800 --> 01:09:46.290 align:middle line:84% What I know they did do on purpose 01:09:46.290 --> 01:09:48.479 align:middle line:84% is to not have a mandate, or anything looking 01:09:48.479 --> 01:09:53.100 align:middle line:84% like that, or incentives until quite late because they 01:09:53.100 --> 01:09:55.260 align:middle line:84% were worried about that backlash effect. 01:09:55.260 --> 01:09:56.910 align:middle line:84% That if you put an incentive, people 01:09:56.910 --> 01:10:03.613 align:middle line:84% are just going to not do it just to spite the government. 01:10:03.613 --> 01:10:05.280 align:middle line:84% And here, that might be a bit like that. 01:10:05.280 --> 01:10:07.280 align:middle line:84% It's why do they need to pay me to do the thing? 01:10:07.280 --> 01:10:11.160 align:middle line:84% I was going to do it anyway, but it must be good 01:10:11.160 --> 01:10:14.800 align:middle line:90% if they pay me to do it. 01:10:14.800 --> 01:10:16.840 align:middle line:84% And that's consistent with the fact 01:10:16.840 --> 01:10:19.510 align:middle line:84% that the effects are the smallest in places 01:10:19.510 --> 01:10:23.140 align:middle line:84% that have a lot of immunization already because it leaves 01:10:23.140 --> 01:10:27.970 align:middle line:84% a bunch of people who would otherwise have done it 01:10:27.970 --> 01:10:30.280 align:middle line:90% who can be unconvinced. 01:10:30.280 --> 01:10:31.180 align:middle line:90% A lot of people-- 01:10:31.180 --> 01:10:33.160 align:middle line:84% whereas, it was in the effect in the places 01:10:33.160 --> 01:10:35.290 align:middle line:90% where nobody gets vaccinated. 01:10:35.290 --> 01:10:37.030 align:middle line:84% There is nobody who was intrinsically 01:10:37.030 --> 01:10:41.320 align:middle line:84% motivated to start with, so your extrinsic motivation 01:10:41.320 --> 01:10:42.988 align:middle line:90% is helping it. 01:10:42.988 --> 01:10:44.530 align:middle line:84% AUDIENCE: In this particular setting, 01:10:44.530 --> 01:10:50.380 align:middle line:84% do you have any evidence or prior to this intervention when 01:10:50.380 --> 01:10:53.897 align:middle line:84% people were getting immunized, was it previously associated 01:10:53.897 --> 01:10:54.730 align:middle line:90% with the government? 01:10:54.730 --> 01:10:57.100 align:middle line:84% So could it be that now the government wants 01:10:57.100 --> 01:11:00.880 align:middle line:84% to give me the-- previously, I was having to go to my doctor. 01:11:00.880 --> 01:11:03.400 align:middle line:84% There wasn't as much of a link between immunizations 01:11:03.400 --> 01:11:05.220 align:middle line:84% and the government wanting you to do it. 01:11:05.220 --> 01:11:08.460 align:middle line:84% So maybe it's specifically about, 01:11:08.460 --> 01:11:10.067 align:middle line:84% I'm mistrustful of the government. 01:11:10.067 --> 01:11:11.400 align:middle line:90% But not necessarily of medicine. 01:11:11.400 --> 01:11:14.020 align:middle line:84% But now that the government is imposing itself, 01:11:14.020 --> 01:11:15.330 align:middle line:90% I don't know if-- 01:11:15.330 --> 01:11:19.080 align:middle line:84% ESTHER DUFLO: I don't know because the incentive was 01:11:19.080 --> 01:11:23.490 align:middle line:84% done as part of the government structures anyway. 01:11:23.490 --> 01:11:26.430 align:middle line:84% So it could be an instance of keep the government out 01:11:26.430 --> 01:11:29.160 align:middle line:90% of my Medicare. 01:11:29.160 --> 01:11:31.320 align:middle line:84% That is, suddenly people realized 01:11:31.320 --> 01:11:34.530 align:middle line:84% the importance of the government in earning their care, 01:11:34.530 --> 01:11:36.720 align:middle line:90% but I'm not sure. 01:11:36.720 --> 01:11:38.350 align:middle line:90% It could be. 01:11:38.350 --> 01:11:39.850 align:middle line:84% AUDIENCE: You get a result like this 01:11:39.850 --> 01:11:43.390 align:middle line:84% and you're communicating it back to the call center. 01:11:43.390 --> 01:11:49.497 align:middle line:84% Do you translate it back into [INAUDIBLE] 01:11:49.497 --> 01:11:51.830 align:middle line:84% ESTHER DUFLO: Yes, so the way that-- this one translates 01:11:51.830 --> 01:11:53.510 align:middle line:90% nicely. 01:11:53.510 --> 01:11:58.220 align:middle line:84% Basically, you have a table like that, 01:11:58.220 --> 01:12:02.570 align:middle line:84% except you can express it in graphs to say, well, 01:12:02.570 --> 01:12:05.270 align:middle line:84% you get the biggest effect in places where there is 01:12:05.270 --> 01:12:08.480 align:middle line:90% a lot of vaccination already. 01:12:08.480 --> 01:12:11.030 align:middle line:84% These are five dimensions of vaccination, 01:12:11.030 --> 01:12:13.190 align:middle line:90% but you can pick just one. 01:12:13.190 --> 01:12:15.650 align:middle line:90% And none of the other-- 01:12:15.650 --> 01:12:20.120 align:middle line:84% and then, that's the one thing that comes out. 01:12:20.120 --> 01:12:22.340 align:middle line:84% And for example, for the government, very important 01:12:22.340 --> 01:12:25.250 align:middle line:84% to tell them that it's not about religion. 01:12:25.250 --> 01:12:26.220 align:middle line:90% This is Haryana. 01:12:26.220 --> 01:12:27.200 align:middle line:90% That's like Hindu. 01:12:27.200 --> 01:12:29.870 align:middle line:90% 01:12:29.870 --> 01:12:34.700 align:middle line:84% The government is like Hindu dominated. 01:12:34.700 --> 01:12:37.687 align:middle line:84% So they are very prompt to say that this is are Muslims 01:12:37.687 --> 01:12:39.020 align:middle line:90% who don't want to get immunized. 01:12:39.020 --> 01:12:42.245 align:middle line:84% So you can show this effect doesn't vary by Muslim, Hindu. 01:12:42.245 --> 01:12:45.380 align:middle line:90% 01:12:45.380 --> 01:12:48.552 align:middle line:90% That kind of things. 01:12:48.552 --> 01:12:49.260 align:middle line:90% But you're right. 01:12:49.260 --> 01:12:52.250 align:middle line:84% That's why we want to get to the [INAUDIBLE] 01:12:52.250 --> 01:12:55.430 align:middle line:84% to give something that is concrete, actionable, 01:12:55.430 --> 01:12:58.460 align:middle line:90% understandable. 01:12:58.460 --> 01:13:03.080 align:middle line:84% But they are still not doing it, just full one. 01:13:03.080 --> 01:13:04.770 align:middle line:90% It's not there. 01:13:04.770 --> 01:13:08.260 align:middle line:84% We're very keen to scale up the gossip plus SMS. 01:13:08.260 --> 01:13:12.040 align:middle line:84% But this one is more complicated. 01:13:12.040 --> 01:13:14.260 align:middle line:84% The fact that it cost money, so even 01:13:14.260 --> 01:13:17.020 align:middle line:84% though another important message for policy 01:13:17.020 --> 01:13:21.700 align:middle line:84% is that if you focus on the places that are the most-- 01:13:21.700 --> 01:13:24.160 align:middle line:84% that have the largest effect, it becomes 01:13:24.160 --> 01:13:27.430 align:middle line:90% cost-effective in those places. 01:13:27.430 --> 01:13:30.340 align:middle line:90% Not tremendously so. 01:13:30.340 --> 01:13:39.610 align:middle line:84% It's not terribly-- it's not terribly much cheaper 01:13:39.610 --> 01:13:41.130 align:middle line:84% than the control, which is what we 01:13:41.130 --> 01:13:43.320 align:middle line:90% do in which we find in Udaipur. 01:13:43.320 --> 01:13:47.490 align:middle line:84% But in the top groups, you cannot project that you 01:13:47.490 --> 01:13:50.610 align:middle line:84% don't lose any money per shot compared to not doing 01:13:50.610 --> 01:13:52.540 align:middle line:90% the incentives. 01:13:52.540 --> 01:13:58.830 align:middle line:84% So that's also why it's important to have that analysis 01:13:58.830 --> 01:14:00.420 align:middle line:84% because the policy message becomes 01:14:00.420 --> 01:14:02.070 align:middle line:90% a little bit more subtle. 01:14:02.070 --> 01:14:04.980 align:middle line:84% It's A, you don't want to do it where it's going to backfire. 01:14:04.980 --> 01:14:07.350 align:middle line:84% But B, in places where it actually works, 01:14:07.350 --> 01:14:10.110 align:middle line:90% you might as well do it. 01:14:10.110 --> 01:14:12.960 align:middle line:84% It's not even going to cost you more per immunization. 01:14:12.960 --> 01:14:18.210 align:middle line:84% But I totally take the point that. 01:14:18.210 --> 01:14:21.150 align:middle line:84% My empirical observation is that that's 01:14:21.150 --> 01:14:24.150 align:middle line:84% a message that's harder to convey than the simple message. 01:14:24.150 --> 01:14:26.640 align:middle line:84% This is what you can do everywhere. 01:14:26.640 --> 01:14:29.640 align:middle line:84% This is cheap to do, and go for it. 01:14:29.640 --> 01:14:36.080 align:middle line:90% 01:14:36.080 --> 01:14:38.780 align:middle line:84% So all that means that-- so first of all, there 01:14:38.780 --> 01:14:41.660 align:middle line:84% is a lot we don't fully understand for health. 01:14:41.660 --> 01:14:47.130 align:middle line:84% But it's really a great set of topics. 01:14:47.130 --> 01:14:48.740 align:middle line:90% Many of them still open. 01:14:48.740 --> 01:14:52.310 align:middle line:84% And then, it makes for a pretty complicated supply response. 01:14:52.310 --> 01:14:56.510 align:middle line:84% In an environment where the demand is complicated-- 01:14:56.510 --> 01:14:58.110 align:middle line:84% has all of these things going on, 01:14:58.110 --> 01:15:00.620 align:middle line:84% the trust, the hyperbolic discounting, 01:15:00.620 --> 01:15:03.020 align:middle line:90% the misinformation, et cetera. 01:15:03.020 --> 01:15:05.030 align:middle line:84% You might expect that's where the supply is 01:15:05.030 --> 01:15:08.130 align:middle line:90% going to be hard to discipline. 01:15:08.130 --> 01:15:12.800 align:middle line:84% So we don't really have time to cover all of the supply today. 01:15:12.800 --> 01:15:16.050 align:middle line:84% But I can just start a little bit, and then continue. 01:15:16.050 --> 01:15:17.930 align:middle line:84% I do want to teach about health care supply, 01:15:17.930 --> 01:15:22.580 align:middle line:90% so I'm going to continue later. 01:15:22.580 --> 01:15:23.470 align:middle line:90% Where do we start? 01:15:23.470 --> 01:15:26.200 align:middle line:90% 01:15:26.200 --> 01:15:29.530 align:middle line:90% So the motivation is from Arrow. 01:15:29.530 --> 01:15:34.660 align:middle line:84% Health care is a credence good with a lot 01:15:34.660 --> 01:15:36.190 align:middle line:90% of asymmetric information. 01:15:36.190 --> 01:15:38.740 align:middle line:84% The provider knows more than the patient. 01:15:38.740 --> 01:15:41.740 align:middle line:84% For all of the reason we described, that's clearly true. 01:15:41.740 --> 01:15:44.350 align:middle line:84% Learning is very difficult. And there 01:15:44.350 --> 01:15:45.920 align:middle line:90% are a lot of externalities. 01:15:45.920 --> 01:15:48.850 align:middle line:84% So even if you knew everything, you 01:15:48.850 --> 01:15:52.022 align:middle line:84% will have fairly low demand for some goods, like prevention. 01:15:52.022 --> 01:15:53.980 align:middle line:84% And then, demand too high for some other goods, 01:15:53.980 --> 01:15:54.940 align:middle line:90% like antibiotic. 01:15:54.940 --> 01:15:57.430 align:middle line:84% Even if you understand everything, just 01:15:57.430 --> 01:15:59.790 align:middle line:84% because the antibiotic is going to help you now, 01:15:59.790 --> 01:16:04.700 align:middle line:84% whereas the immunization is going to help all other people. 01:16:04.700 --> 01:16:08.920 align:middle line:84% So you cannot rely on the individuals to choose the right 01:16:08.920 --> 01:16:11.220 align:middle line:90% outcome. 01:16:11.220 --> 01:16:14.850 align:middle line:84% For all of these reasons, unregulated private health care 01:16:14.850 --> 01:16:18.390 align:middle line:84% will tend to treat badly, and to over provide some medication 01:16:18.390 --> 01:16:21.510 align:middle line:84% and to under provide some medication. 01:16:21.510 --> 01:16:25.320 align:middle line:84% Hence, our conclusion is not talking about India. 01:16:25.320 --> 01:16:28.050 align:middle line:84% It's talking about the US, but that applies to India. 01:16:28.050 --> 01:16:31.140 align:middle line:84% It is the general social consensus, clearly, 01:16:31.140 --> 01:16:35.250 align:middle line:84% that the laissez-faire solution for medicine is intolerable. 01:16:35.250 --> 01:16:35.775 align:middle line:90% We agree. 01:16:35.775 --> 01:16:39.090 align:middle line:90% 01:16:39.090 --> 01:16:46.245 align:middle line:84% Except that-- I'm going to skip that. 01:16:46.245 --> 01:16:49.790 align:middle line:90% 01:16:49.790 --> 01:16:52.830 align:middle line:90% Well, maybe not. 01:16:52.830 --> 01:16:58.350 align:middle line:84% So indeed, learning about the quality of health 01:16:58.350 --> 01:17:02.100 align:middle line:84% care you receive is very difficult, both because of what 01:17:02.100 --> 01:17:05.440 align:middle line:84% we discussed of self-limiting diseases, 01:17:05.440 --> 01:17:08.420 align:middle line:90% and because of externalities. 01:17:08.420 --> 01:17:12.380 align:middle line:84% As a result, people have a very poor understanding 01:17:12.380 --> 01:17:13.220 align:middle line:90% of their health. 01:17:13.220 --> 01:17:14.690 align:middle line:84% And they have a poor understanding 01:17:14.690 --> 01:17:16.565 align:middle line:84% of whether they are getting good health care. 01:17:16.565 --> 01:17:20.540 align:middle line:84% So we discussed self-limited diseases already. 01:17:20.540 --> 01:17:25.610 align:middle line:84% And that leads potentially to overuse of antibiotic 01:17:25.610 --> 01:17:27.710 align:middle line:84% or [INAUDIBLE] for COVID because you 01:17:27.710 --> 01:17:29.210 align:middle line:84% think that they have been effective, 01:17:29.210 --> 01:17:32.430 align:middle line:84% when in fact, it's just that the disease became later. 01:17:32.430 --> 01:17:35.630 align:middle line:84% The other problem is externalities. 01:17:35.630 --> 01:17:42.470 align:middle line:84% That's when a lot of people get immunized or get dewormed, 01:17:42.470 --> 01:17:43.790 align:middle line:90% nobody gets sick. 01:17:43.790 --> 01:17:45.530 align:middle line:84% And you don't get less sick because you 01:17:45.530 --> 01:17:49.090 align:middle line:84% got the immunization or the deworming. 01:17:49.090 --> 01:17:52.360 align:middle line:84% The result is that there is a lot of-- 01:17:52.360 --> 01:17:58.090 align:middle line:84% it's very easy for a provider to take advantage of the situation 01:17:58.090 --> 01:18:00.100 align:middle line:90% to give very, very bad care. 01:18:00.100 --> 01:18:05.020 align:middle line:84% And in particular, to give very, very, very bad medicines. 01:18:05.020 --> 01:18:08.630 align:middle line:84% In an environment where many people take malaria treatment 01:18:08.630 --> 01:18:10.090 align:middle line:84% even when they don't have malaria, 01:18:10.090 --> 01:18:12.700 align:middle line:84% and people are very naive about the model, 01:18:12.700 --> 01:18:15.490 align:middle line:84% there is very strong incentive to sell fake treatment 01:18:15.490 --> 01:18:19.420 align:middle line:84% because usually, you can get away with it. 01:18:19.420 --> 01:18:23.020 align:middle line:84% So to investigate that, Bjorkman, Svensson, 01:18:23.020 --> 01:18:26.110 align:middle line:84% and Yanakizawa-Drott sent mystery shopper 01:18:26.110 --> 01:18:29.710 align:middle line:84% to buy a anti-malaria drug, the new one, ACT, 01:18:29.710 --> 01:18:33.850 align:middle line:84% artemisinin combination therapy at 99 markets in Uganda, 01:18:33.850 --> 01:18:36.280 align:middle line:90% and then test them in the lab. 01:18:36.280 --> 01:18:39.310 align:middle line:84% And what they find is that 37% of the shop 01:18:39.310 --> 01:18:43.370 align:middle line:84% sells some counterfeits, that the price is 01:18:43.370 --> 01:18:45.290 align:middle line:84% unrelated with whether it's counterfeits 01:18:45.290 --> 01:18:49.850 align:middle line:84% or not, and that the share-- the proportion of counterfeits 01:18:49.850 --> 01:18:54.380 align:middle line:84% is increasing in the share of naive consumers or people who 01:18:54.380 --> 01:18:55.830 align:middle line:90% misunderstand malaria. 01:18:55.830 --> 01:18:59.120 align:middle line:84% So they ask people questions about what you think 01:18:59.120 --> 01:19:00.330 align:middle line:90% causes malaria. 01:19:00.330 --> 01:19:03.540 align:middle line:84% So some people think that rain causes malaria. 01:19:03.540 --> 01:19:06.140 align:middle line:84% And some people know that it's mosquitoes. 01:19:06.140 --> 01:19:08.840 align:middle line:84% And there are mosquitoes when it rains. 01:19:08.840 --> 01:19:12.510 align:middle line:84% And the more of these naive household you have, 01:19:12.510 --> 01:19:15.140 align:middle line:90% the more counterfeits there is. 01:19:15.140 --> 01:19:20.280 align:middle line:90% People are more likely to have-- 01:19:20.280 --> 01:19:20.990 align:middle line:90% sorry. 01:19:20.990 --> 01:19:21.770 align:middle line:90% This is it here. 01:19:21.770 --> 01:19:26.840 align:middle line:84% The more naive people you have, the more likely 01:19:26.840 --> 01:19:30.110 align:middle line:84% it is that a place sell counterfeit malaria. 01:19:30.110 --> 01:19:32.960 align:middle line:84% And that's related to the fact that the more naive people are, 01:19:32.960 --> 01:19:35.360 align:middle line:84% the more people are convinced that the drug they get 01:19:35.360 --> 01:19:38.160 align:middle line:84% is good, perhaps because they tried it, 01:19:38.160 --> 01:19:41.320 align:middle line:84% and it worked as far as they're concerned. 01:19:41.320 --> 01:19:45.910 align:middle line:84% So what they do in this paper is that they ask themselves 01:19:45.910 --> 01:19:49.730 align:middle line:84% whether a good quality can drive bad ones. 01:19:49.730 --> 01:19:53.890 align:middle line:84% So they work with BRAC, the same of ultra poor fame 01:19:53.890 --> 01:19:59.470 align:middle line:84% and otherwise extremely reputable and competent 01:19:59.470 --> 01:20:02.590 align:middle line:84% organization, who now doesn't work 01:20:02.590 --> 01:20:05.290 align:middle line:84% just in Bangladesh, but also in Uganda and other East 01:20:05.290 --> 01:20:06.940 align:middle line:90% African countries. 01:20:06.940 --> 01:20:11.770 align:middle line:84% And BRAC sets up pharmacies, which sells bonded malaria 01:20:11.770 --> 01:20:14.080 align:middle line:90% medicine of good quality. 01:20:14.080 --> 01:20:15.880 align:middle line:84% And their question is, is it going 01:20:15.880 --> 01:20:19.570 align:middle line:84% to lead the other people to disappear? 01:20:19.570 --> 01:20:26.450 align:middle line:84% And the question-- the answer is in two steps. 01:20:26.450 --> 01:20:28.850 align:middle line:84% On average, the quality in the shop 01:20:28.850 --> 01:20:32.820 align:middle line:84% increases, and the price of the malaria medicine decreases. 01:20:32.820 --> 01:20:36.470 align:middle line:84% So on average, the competition effect, in fact, works out. 01:20:36.470 --> 01:20:39.050 align:middle line:84% If you introduce this bonded product, 01:20:39.050 --> 01:20:41.520 align:middle line:90% people will flock to that. 01:20:41.520 --> 01:20:43.680 align:middle line:84% And then, put pressure on the other guys. 01:20:43.680 --> 01:20:46.670 align:middle line:84% And they will tend to either go for good medicine 01:20:46.670 --> 01:20:48.660 align:middle line:90% and reduce the prices. 01:20:48.660 --> 01:20:52.610 align:middle line:84% However, this works less in villages where 01:20:52.610 --> 01:20:54.360 align:middle line:90% they are more naive customers. 01:20:54.360 --> 01:20:56.225 align:middle line:90% So when you interact-- 01:20:56.225 --> 01:21:00.200 align:middle line:84% you use this table, when you interact 01:21:00.200 --> 01:21:03.680 align:middle line:84% the fraction the NGO sells drug on average, 01:21:03.680 --> 01:21:09.500 align:middle line:84% there is less likely to be fake drug in the village. 01:21:09.500 --> 01:21:11.120 align:middle line:84% If there are more native people as we 01:21:11.120 --> 01:21:15.890 align:middle line:84% saw before, that's this one, they are more fake drugs. 01:21:15.890 --> 01:21:19.790 align:middle line:84% And then, if there is more native people in the village, 01:21:19.790 --> 01:21:21.965 align:middle line:84% the treatment effect is less strong. 01:21:21.965 --> 01:21:27.500 align:middle line:90% 01:21:27.500 --> 01:21:29.690 align:middle line:84% So number one, when you have more native people 01:21:29.690 --> 01:21:31.760 align:middle line:84% in the village who don't understand the model, 01:21:31.760 --> 01:21:33.260 align:middle line:90% you have more fake drugs. 01:21:33.260 --> 01:21:35.540 align:middle line:84% Number two, it makes it difficult to-- 01:21:35.540 --> 01:21:38.810 align:middle line:84% more difficult to address it by putting good drugs. 01:21:38.810 --> 01:21:41.440 align:middle line:90% So that's just to say that-- 01:21:41.440 --> 01:21:45.086 align:middle line:90% 01:21:45.086 --> 01:21:48.065 align:middle line:84% to comfort the point that Arrow made. 01:21:48.065 --> 01:21:50.900 align:middle line:90% 01:21:50.900 --> 01:21:55.010 align:middle line:84% It's hard to learn about health and provider take 01:21:55.010 --> 01:21:56.640 align:middle line:90% advantage of it. 01:21:56.640 --> 01:22:01.330 align:middle line:84% And therefore, the private health care sector 01:22:01.330 --> 01:22:04.690 align:middle line:84% is poor countries is really awful. 01:22:04.690 --> 01:22:07.360 align:middle line:84% And I'll tell you a bit more about how awful it 01:22:07.360 --> 01:22:09.647 align:middle line:90% is next time. 01:22:09.647 --> 01:22:12.780 align:middle line:90% 01:22:12.780 --> 01:22:15.010 align:middle line:90% So we'll stop here. 01:22:15.010 --> 01:22:16.620 align:middle line:84% I think we'll have time to finish 01:22:16.620 --> 01:22:19.470 align:middle line:84% that in about a half lecture, which still will give us 01:22:19.470 --> 01:22:22.230 align:middle line:84% a one and a half lecture on gender. 01:22:22.230 --> 01:22:23.560 align:middle line:90% So that will be fine. 01:22:23.560 --> 01:22:26.920 align:middle line:90% But on Monday, Ben is teaching. 01:22:26.920 --> 01:22:30.900 align:middle line:84% So you're going to leave that in the fridge for a minute, 01:22:30.900 --> 01:22:34.100 align:middle line:84% for one lecture, and we'll go back together. 01:22:34.100 --> 01:22:46.000 align:middle line:90%