WEBVTT 00:00:00.000 --> 00:00:01.960 align:middle line:90% [SQUEAKING] 00:00:01.960 --> 00:00:03.430 align:middle line:90% [RUSTLING] 00:00:03.430 --> 00:00:19.120 align:middle line:90% [CLICKING] 00:00:19.120 --> 00:00:21.380 align:middle line:90% PROFESSOR: Welcome, everyone. 00:00:21.380 --> 00:00:26.500 align:middle line:84% So last time, we left it at saying, well, one fast way 00:00:26.500 --> 00:00:28.630 align:middle line:84% in which you could test whether or not 00:00:28.630 --> 00:00:31.600 align:middle line:84% there is a potential for a poverty trap 00:00:31.600 --> 00:00:34.900 align:middle line:84% or for multiple steady state, which is typically interpreted 00:00:34.900 --> 00:00:39.160 align:middle line:84% as poverty trap, is think about the mechanism that would lead 00:00:39.160 --> 00:00:42.520 align:middle line:84% to a mapping between income today and income tomorrow 00:00:42.520 --> 00:00:44.870 align:middle line:84% or between asset today and asset tomorrow. 00:00:44.870 --> 00:00:48.190 align:middle line:84% And then look at whether this mapping is sufficiently strong. 00:00:48.190 --> 00:00:51.250 align:middle line:84% So that's the exercise we did at the end of last time. 00:00:51.250 --> 00:00:56.320 align:middle line:84% When we're looking about the effect of income on nutrition, 00:00:56.320 --> 00:00:57.910 align:middle line:84% we spend most of the time on that. 00:00:57.910 --> 00:01:02.350 align:middle line:84% And I asserted that the effect of nutrition on productivity 00:01:02.350 --> 00:01:05.080 align:middle line:84% and therefore future income is there, but not so 00:01:05.080 --> 00:01:08.380 align:middle line:84% strong that it would outweigh the fact 00:01:08.380 --> 00:01:12.880 align:middle line:84% that the effect of income of-- the elasticity of nutrition 00:01:12.880 --> 00:01:15.160 align:middle line:84% with respect to income is not one. 00:01:15.160 --> 00:01:16.870 align:middle line:84% So we left it as thinking that maybe 00:01:16.870 --> 00:01:20.200 align:middle line:84% the nutrition-based poverty trap, as described 00:01:20.200 --> 00:01:23.290 align:middle line:84% as the fundamental phenomenon of Dasgupta and Ray, is not there. 00:01:23.290 --> 00:01:28.630 align:middle line:84% But nonetheless, my response to I think a question by Recca, 00:01:28.630 --> 00:01:31.270 align:middle line:84% I pointed out that this is a paper that everybody reads. 00:01:31.270 --> 00:01:33.070 align:middle line:90% This is like a modern classic. 00:01:33.070 --> 00:01:34.810 align:middle line:84% And so people read it not so much 00:01:34.810 --> 00:01:38.110 align:middle line:84% because you necessarily believed in that very mechanism, 00:01:38.110 --> 00:01:43.990 align:middle line:84% but because the whole idea of poverty trap based 00:01:43.990 --> 00:01:49.560 align:middle line:84% on this funky mapping between income or asset to the optimal 00:01:49.560 --> 00:01:53.830 align:middle line:84% always is something that we keep finding 00:01:53.830 --> 00:01:56.740 align:middle line:90% in different areas of the field. 00:01:56.740 --> 00:02:00.382 align:middle line:84% Now, so then you could go to another approach, which 00:02:00.382 --> 00:02:01.840 align:middle line:84% is what we're going to do today, is 00:02:01.840 --> 00:02:04.360 align:middle line:84% to say, well, irrespective of the mechanism, 00:02:04.360 --> 00:02:05.890 align:middle line:90% is it just there? 00:02:05.890 --> 00:02:07.810 align:middle line:84% Can we find a setting where we can 00:02:07.810 --> 00:02:14.930 align:middle line:84% find a relationship between today's assets and assets 00:02:14.930 --> 00:02:17.530 align:middle line:84% next year, for example, where there is not 00:02:17.530 --> 00:02:21.400 align:middle line:84% only that nonlinearity, but that crossing of the 45-degree line 00:02:21.400 --> 00:02:23.710 align:middle line:84% from below, which ensure that there 00:02:23.710 --> 00:02:29.950 align:middle line:84% is an unstable, steady state and two steady state, one low, one 00:02:29.950 --> 00:02:32.110 align:middle line:90% high. 00:02:32.110 --> 00:02:36.070 align:middle line:84% Another thing is, of course, no matter what 00:02:36.070 --> 00:02:39.160 align:middle line:84% we think as economists sitting in these rooms, 00:02:39.160 --> 00:02:42.910 align:middle line:84% there is also the policy-making world, the activist world, 00:02:42.910 --> 00:02:43.750 align:middle line:90% the NGO world. 00:02:43.750 --> 00:02:46.900 align:middle line:84% And some aspect some part of this world believes in poverty 00:02:46.900 --> 00:02:50.050 align:middle line:84% trap and has, in fact, created programs 00:02:50.050 --> 00:02:54.490 align:middle line:84% that responds to the idea that there might be a poverty trap. 00:02:54.490 --> 00:02:58.090 align:middle line:84% And the two ideas met together nicely 00:02:58.090 --> 00:03:00.580 align:middle line:84% in the context of the ultra poor programs, 00:03:00.580 --> 00:03:03.880 align:middle line:84% which are very much a program inspired by the idea 00:03:03.880 --> 00:03:08.260 align:middle line:84% that there might be a poverty trap, which were designed 00:03:08.260 --> 00:03:10.180 align:middle line:84% and rolled out in the last few years, 00:03:10.180 --> 00:03:13.150 align:middle line:84% and which have given an opportunity to actually 00:03:13.150 --> 00:03:19.840 align:middle line:84% directly present evidence for against a reduced form view of, 00:03:19.840 --> 00:03:20.860 align:middle line:90% is there a poverty trap? 00:03:20.860 --> 00:03:24.650 align:middle line:84% So that's what we're going to do today. 00:03:24.650 --> 00:03:27.130 align:middle line:84% So the idea that there might be a poverty trap 00:03:27.130 --> 00:03:30.670 align:middle line:84% is at the heart of a relatively recent set 00:03:30.670 --> 00:03:34.570 align:middle line:84% of antipoverty programs, poverty-fighting programs, 00:03:34.570 --> 00:03:39.220 align:middle line:84% promoted very much from the South itself, 00:03:39.220 --> 00:03:42.520 align:middle line:84% called a graduation approach, graduation as 00:03:42.520 --> 00:03:45.120 align:middle line:90% in graduating out of poverty. 00:03:45.120 --> 00:03:47.860 align:middle line:84% And so the idea is that you are undergraduate in poverty, 00:03:47.860 --> 00:03:51.590 align:middle line:84% and then you graduate, and you're out of poverty. 00:03:51.590 --> 00:03:54.160 align:middle line:84% So the graduation approach was designed 00:03:54.160 --> 00:03:57.640 align:middle line:84% by a microfinance organization slash 00:03:57.640 --> 00:04:01.640 align:middle line:90% NGO called BRAC in Bangladesh. 00:04:01.640 --> 00:04:04.720 align:middle line:84% So unless you're very familiar with development economics, 00:04:04.720 --> 00:04:08.890 align:middle line:84% you've surely heard of Grameen and not necessarily of BRAC. 00:04:08.890 --> 00:04:13.120 align:middle line:84% But BRAC is actually the bigger and more 00:04:13.120 --> 00:04:15.490 align:middle line:84% comprehensive organization of the two. 00:04:15.490 --> 00:04:19.014 align:middle line:84% Its founder just passed away a couple of years ago, 00:04:19.014 --> 00:04:22.420 align:middle line:84% and was a wonderful, wonderful individual. 00:04:22.420 --> 00:04:27.370 align:middle line:84% And so they have a microcredit program. 00:04:27.370 --> 00:04:28.690 align:middle line:90% They also run schools. 00:04:28.690 --> 00:04:29.980 align:middle line:90% They run health center. 00:04:29.980 --> 00:04:33.070 align:middle line:84% They run antidiarrheal program, et cetera. 00:04:33.070 --> 00:04:35.020 align:middle line:84% I'm sure they're doing a lot of COVID. 00:04:35.020 --> 00:04:36.910 align:middle line:84% And they've now, in fact, expanded 00:04:36.910 --> 00:04:39.430 align:middle line:84% in a lot of other countries in the global South. 00:04:39.430 --> 00:04:41.510 align:middle line:90% But they started in Bangladesh. 00:04:41.510 --> 00:04:43.540 align:middle line:84% So their idea, what happened with them 00:04:43.540 --> 00:04:46.840 align:middle line:84% is that when they were running the microcredit program, 00:04:46.840 --> 00:04:48.850 align:middle line:84% they realized that there is a set of people 00:04:48.850 --> 00:04:52.390 align:middle line:84% who are not even interested in getting a microcredit loan. 00:04:52.390 --> 00:04:56.600 align:middle line:84% And those people are some of the poorest in the villages. 00:04:56.600 --> 00:04:58.570 align:middle line:84% So they decided that they need to do something 00:04:58.570 --> 00:04:59.980 align:middle line:90% to reach out to these people. 00:04:59.980 --> 00:05:01.990 align:middle line:84% And they designed a program which 00:05:01.990 --> 00:05:07.150 align:middle line:84% they conceived as potentially a first entry 00:05:07.150 --> 00:05:10.570 align:middle line:84% into the world of very, very small business for people, 00:05:10.570 --> 00:05:13.750 align:middle line:84% and potentially a steppingstone into microcredit 00:05:13.750 --> 00:05:15.560 align:middle line:90% or, in general, in the world. 00:05:15.560 --> 00:05:19.670 align:middle line:84% So their idea was that by giving people an infusion of capital 00:05:19.670 --> 00:05:23.480 align:middle line:84% as a gift, not as a loan, and helping them start it, 00:05:23.480 --> 00:05:28.560 align:middle line:84% you can help them escape poverty in a durable way. 00:05:28.560 --> 00:05:29.700 align:middle line:90% So the question for today-- 00:05:29.700 --> 00:05:31.700 align:middle line:84% I'll describe the program in a bit more detail-- 00:05:31.700 --> 00:05:33.950 align:middle line:84% and the question we'll try to answer today 00:05:33.950 --> 00:05:37.670 align:middle line:84% is whether this works, whether there 00:05:37.670 --> 00:05:41.510 align:middle line:84% seem to be prima facie evidence for a poverty trap, 00:05:41.510 --> 00:05:43.920 align:middle line:90% and how we build such evidence. 00:05:43.920 --> 00:05:46.490 align:middle line:84% And then do we know where the poverty 00:05:46.490 --> 00:05:49.290 align:middle line:90% trap might be coming from? 00:05:49.290 --> 00:05:53.430 align:middle line:84% So let me cut short to that in case we don't get at the end. 00:05:53.430 --> 00:05:57.590 align:middle line:84% And the answer is, not completely. 00:05:57.590 --> 00:06:00.650 align:middle line:84% I think we'll have pretty good evidence that it works. 00:06:00.650 --> 00:06:03.260 align:middle line:84% It has worked, both in the short run, 00:06:03.260 --> 00:06:05.360 align:middle line:84% but also in the long run, which is prima facie 00:06:05.360 --> 00:06:08.480 align:middle line:84% evidence of poverty trap, probably 00:06:08.480 --> 00:06:11.270 align:middle line:84% because of a phenomenon of escaping 00:06:11.270 --> 00:06:13.310 align:middle line:90% some kind of too small assets. 00:06:13.310 --> 00:06:15.020 align:middle line:90% But we don't know exactly why. 00:06:15.020 --> 00:06:17.480 align:middle line:84% And there is a lot of recent research 00:06:17.480 --> 00:06:22.290 align:middle line:84% to try to get into that and into various mechanism. 00:06:22.290 --> 00:06:25.940 align:middle line:84% So how does the graduation approach work in a nutshell? 00:06:25.940 --> 00:06:28.700 align:middle line:84% The community is brought together 00:06:28.700 --> 00:06:33.500 align:middle line:84% and identified what they think of as the poorest 00:06:33.500 --> 00:06:35.180 align:middle line:90% of the poor in the village. 00:06:35.180 --> 00:06:39.300 align:middle line:84% So that's what sometimes call as a participatory resource 00:06:39.300 --> 00:06:39.800 align:middle line:90% mapping. 00:06:39.800 --> 00:06:40.970 align:middle line:90% They all go together. 00:06:40.970 --> 00:06:42.800 align:middle line:90% They rank people. 00:06:42.800 --> 00:06:46.220 align:middle line:84% There is a whole exercise with cards involving ranking people 00:06:46.220 --> 00:06:48.460 align:middle line:90% by wealth. 00:06:48.460 --> 00:06:51.020 align:middle line:84% First half of the people are cut in half 00:06:51.020 --> 00:06:52.743 align:middle line:84% by wealth, the richest and the poor. 00:06:52.743 --> 00:06:54.410 align:middle line:84% And then the poor are cut again and then 00:06:54.410 --> 00:06:57.500 align:middle line:84% cut again and then cut again until you arrive 00:06:57.500 --> 00:06:59.807 align:middle line:90% at four or five people, maybe. 00:06:59.807 --> 00:07:00.890 align:middle line:90% It depends on the village. 00:07:00.890 --> 00:07:02.750 align:middle line:84% But on average, it's four or five people 00:07:02.750 --> 00:07:04.160 align:middle line:84% in the village who are considered 00:07:04.160 --> 00:07:07.830 align:middle line:84% by their fellow villagers to be the ultra poor. 00:07:07.830 --> 00:07:10.160 align:middle line:84% And when you look at these families, 00:07:10.160 --> 00:07:15.470 align:middle line:84% there is often something pretty clear in their history. 00:07:15.470 --> 00:07:17.990 align:middle line:84% There are a lot of female-headed households, where 00:07:17.990 --> 00:07:22.640 align:middle line:84% the husband died, or the husband is alive 00:07:22.640 --> 00:07:24.500 align:middle line:90% but incapacitated somehow. 00:07:24.500 --> 00:07:30.530 align:middle line:84% A lot of households like that or other households 00:07:30.530 --> 00:07:31.460 align:middle line:90% where there was-- 00:07:31.460 --> 00:07:35.900 align:middle line:84% you can almost always identify the tragedy that brought them 00:07:35.900 --> 00:07:37.760 align:middle line:90% to an extreme set of poverty. 00:07:37.760 --> 00:07:41.630 align:middle line:84% Typically, a lot of these women-- 00:07:41.630 --> 00:07:43.740 align:middle line:84% these are often women, a lot of these. 00:07:43.740 --> 00:07:46.250 align:middle line:84% Didn't have to be, depending on the problem, that it's 00:07:46.250 --> 00:07:48.060 align:middle line:90% a woman who is targeted. 00:07:48.060 --> 00:07:51.170 align:middle line:84% But it's often happen to be women 00:07:51.170 --> 00:07:57.950 align:middle line:84% who are begging or working as casual laborer, 00:07:57.950 --> 00:08:01.190 align:middle line:84% living maybe on the outskirts of the village, 00:08:01.190 --> 00:08:05.010 align:middle line:84% moderately included in the social fabric. 00:08:05.010 --> 00:08:07.300 align:middle line:84% Once these people are identified, 00:08:07.300 --> 00:08:10.650 align:middle line:84% they are visited by the worker firm BRAC. 00:08:10.650 --> 00:08:14.460 align:middle line:84% Or now it's been diffused in a lot of other countries, 00:08:14.460 --> 00:08:16.830 align:middle line:90% as I'll show you in a moment. 00:08:16.830 --> 00:08:18.600 align:middle line:90% And they receive income. 00:08:18.600 --> 00:08:20.790 align:middle line:90% They receive an asset. 00:08:20.790 --> 00:08:25.320 align:middle line:84% It's a productive asset, which they choose from a menu. 00:08:25.320 --> 00:08:31.560 align:middle line:84% But it often happens to be a pair of cows or a few goats. 00:08:31.560 --> 00:08:33.285 align:middle line:90% In Peru, Guinea pigs. 00:08:33.285 --> 00:08:36.750 align:middle line:90% 00:08:36.750 --> 00:08:41.669 align:middle line:84% Bees in Ethiopia, like honeybees. 00:08:41.669 --> 00:08:44.500 align:middle line:84% It could also be a sewing machine. 00:08:44.500 --> 00:08:47.880 align:middle line:84% It can be a beginning, like some money 00:08:47.880 --> 00:08:51.660 align:middle line:84% to start a small business, to buy the first set of things 00:08:51.660 --> 00:08:52.930 align:middle line:90% to sell, that kind of thing. 00:08:52.930 --> 00:08:55.320 align:middle line:84% So in principle, there is a menu, and people choose. 00:08:55.320 --> 00:09:00.480 align:middle line:84% Although in practice, there is a huge preponderance of cows. 00:09:00.480 --> 00:09:04.647 align:middle line:84% Then they get income support for a few weeks, 00:09:04.647 --> 00:09:06.730 align:middle line:84% depending on the assets, depending on the country, 00:09:06.730 --> 00:09:10.038 align:middle line:84% depending on the program to make sure 00:09:10.038 --> 00:09:12.330 align:middle line:84% that they don't sell-- they don't dispose of the assets 00:09:12.330 --> 00:09:13.800 align:middle line:84% too quickly, I think, is the idea 00:09:13.800 --> 00:09:15.870 align:middle line:84% is that, OK, we just got them in. 00:09:15.870 --> 00:09:18.330 align:middle line:84% So they get a stipend for a few weeks. 00:09:18.330 --> 00:09:21.970 align:middle line:84% They get technical support to take care of their assets. 00:09:21.970 --> 00:09:24.090 align:middle line:84% So if it's a cow, the cow get vaccinated. 00:09:24.090 --> 00:09:27.060 align:middle line:84% If it's a small business, the employee 00:09:27.060 --> 00:09:34.710 align:middle line:84% will go to the market with them to help them buy things, teach 00:09:34.710 --> 00:09:36.990 align:middle line:84% them to take the bus, whatever it takes 00:09:36.990 --> 00:09:39.450 align:middle line:90% to help them with the activity. 00:09:39.450 --> 00:09:42.930 align:middle line:84% Then there are group meetings, where there is literacy, 00:09:42.930 --> 00:09:45.360 align:middle line:90% small health component. 00:09:45.360 --> 00:09:48.960 align:middle line:84% They are also encouraged slash forced to have regular savings. 00:09:48.960 --> 00:09:53.370 align:middle line:84% So it's a very multifaceted intervention. 00:09:53.370 --> 00:09:58.320 align:middle line:84% And the additional components are so intensive in labor 00:09:58.320 --> 00:10:02.520 align:middle line:84% that it roughly doubles the cost of the transfer program. 00:10:02.520 --> 00:10:03.900 align:middle line:90% The asset program is about-- 00:10:03.900 --> 00:10:10.050 align:middle line:84% the cost of the assets turned out to be $1,000, so $250 PP, 00:10:10.050 --> 00:10:12.370 align:middle line:90% depending on the countries. 00:10:12.370 --> 00:10:18.920 align:middle line:84% And then that's about double by the human cost. 00:10:18.920 --> 00:10:20.990 align:middle line:84% AUDIENCE: What's the prediction that you 00:10:20.990 --> 00:10:25.670 align:middle line:84% want to prove the poverty trap model, as opposed to just 00:10:25.670 --> 00:10:30.058 align:middle line:84% giving someone capital increases their income the next period. 00:10:30.058 --> 00:10:33.360 align:middle line:84% Is there some key result that we [INAUDIBLE]?? 00:10:33.360 --> 00:10:34.860 align:middle line:84% PROFESSOR: Yeah, so the question is, 00:10:34.860 --> 00:10:36.990 align:middle line:90% what are the key predictions? 00:10:36.990 --> 00:10:41.490 align:middle line:84% So what form a poverty trap, as opposed to-- 00:10:41.490 --> 00:10:43.020 align:middle line:90% money is good. 00:10:43.020 --> 00:10:45.760 align:middle line:90% Money is better than no money. 00:10:45.760 --> 00:10:49.020 align:middle line:84% So let me turn the question back to you, if you don't mind. 00:10:49.020 --> 00:10:52.350 align:middle line:90% 00:10:52.350 --> 00:10:53.850 align:middle line:84% AUDIENCE: I guess for the last week, 00:10:53.850 --> 00:10:56.150 align:middle line:90% there's super high returns. 00:10:56.150 --> 00:10:58.910 align:middle line:84% Got that elasticity, I guess, out of poverty. 00:10:58.910 --> 00:11:00.260 align:middle line:90% PROFESSOR: Yes, so that's one. 00:11:00.260 --> 00:11:05.310 align:middle line:84% What's another one that's pretty clear? 00:11:05.310 --> 00:11:07.460 align:middle line:84% Because this is a program that lasts-- 00:11:07.460 --> 00:11:09.410 align:middle line:84% the entire thing until graduation 00:11:09.410 --> 00:11:10.970 align:middle line:90% lasts about 18 months. 00:11:10.970 --> 00:11:12.050 align:middle line:90% You get your assets. 00:11:12.050 --> 00:11:15.680 align:middle line:84% And you're supported by this group meeting and all that 00:11:15.680 --> 00:11:16.970 align:middle line:90% for about 18 months. 00:11:16.970 --> 00:11:19.990 align:middle line:90% And then it's shuts down. 00:11:19.990 --> 00:11:21.207 align:middle line:90% AUDIENCE: Poverty trap? 00:11:21.207 --> 00:11:22.290 align:middle line:90% PROFESSOR: Right, exactly. 00:11:22.290 --> 00:11:25.380 align:middle line:84% So the idea is that if it's poverty trap, 00:11:25.380 --> 00:11:28.410 align:middle line:90% the effect should not diminish. 00:11:28.410 --> 00:11:33.300 align:middle line:84% Otherwise, you would see them slowly diminish over the longer 00:11:33.300 --> 00:11:38.040 align:middle line:84% term as shocks happen and other people get positive shocks, 00:11:38.040 --> 00:11:39.480 align:middle line:90% our guys get negative shocks. 00:11:39.480 --> 00:11:43.900 align:middle line:84% We see the effect slowly receding, 00:11:43.900 --> 00:11:46.050 align:middle line:84% whereas if it were a poverty trap, 00:11:46.050 --> 00:11:49.270 align:middle line:84% and they were permanently pushed out, 00:11:49.270 --> 00:11:51.840 align:middle line:84% we would see the effect to persist or even 00:11:51.840 --> 00:11:53.500 align:middle line:90% to increase over time. 00:11:53.500 --> 00:11:57.450 align:middle line:84% I think that would be a first-order reduction. 00:11:57.450 --> 00:12:00.280 align:middle line:84% And then as well is what you're saying, 00:12:00.280 --> 00:12:04.950 align:middle line:84% which is you should see that if it's a poverty trap 00:12:04.950 --> 00:12:08.650 align:middle line:84% effect, since it's the same asset for everyone, 00:12:08.650 --> 00:12:13.360 align:middle line:84% but some people are even poorer than other, even in this set. 00:12:13.360 --> 00:12:21.240 align:middle line:84% So if we have a poverty trap model based on this low return 00:12:21.240 --> 00:12:24.450 align:middle line:84% to capital at very level, followed by a steep increase, 00:12:24.450 --> 00:12:28.080 align:middle line:84% followed by low again, what would we 00:12:28.080 --> 00:12:32.400 align:middle line:84% expect in terms of heterogeneity of asset accumulation post 00:12:32.400 --> 00:12:35.140 align:middle line:84% the program or potentially at the originity of program 00:12:35.140 --> 00:12:35.640 align:middle line:90% impact? 00:12:35.640 --> 00:12:38.710 align:middle line:90% 00:12:38.710 --> 00:12:39.383 align:middle line:90% Yes. 00:12:39.383 --> 00:12:41.050 align:middle line:84% AUDIENCE: As for the poorest individuals 00:12:41.050 --> 00:12:43.810 align:middle line:84% who receive the transfer, would the poor and those 00:12:43.810 --> 00:12:46.810 align:middle line:84% who are closer to what we see as this steeper for 00:12:46.810 --> 00:12:48.388 align:middle line:90% [? transcapitalism? ?] 00:12:48.388 --> 00:12:49.180 align:middle line:90% PROFESSOR: Exactly. 00:12:49.180 --> 00:12:51.580 align:middle line:84% So potentially, you would see a divergence, 00:12:51.580 --> 00:12:56.320 align:middle line:84% with the program effect being much larger for households that 00:12:56.320 --> 00:13:01.630 align:middle line:84% are sufficiently not so poor that the base 00:13:01.630 --> 00:13:06.700 align:middle line:84% level of asset plus the asset bring them above the poverty 00:13:06.700 --> 00:13:07.690 align:middle line:90% trap. 00:13:07.690 --> 00:13:10.510 align:middle line:84% And we could see that by looking at long-term program effect. 00:13:10.510 --> 00:13:12.040 align:middle line:84% We could see that even by looking 00:13:12.040 --> 00:13:15.340 align:middle line:84% at the dynamic of the asset accumulation 00:13:15.340 --> 00:13:20.720 align:middle line:84% for households that end up post transfer to be above or below. 00:13:20.720 --> 00:13:24.948 align:middle line:84% So we have a chance, in a sense, to see that S-curve in action. 00:13:24.948 --> 00:13:26.740 align:middle line:84% AUDIENCE: Rather than just the traditional. 00:13:26.740 --> 00:13:28.360 align:middle line:84% PROFESSOR: Rather than just the traditional, 00:13:28.360 --> 00:13:30.850 align:middle line:84% this is increasing because more assets is better than less 00:13:30.850 --> 00:13:32.560 align:middle line:90% assets, but declining. 00:13:32.560 --> 00:13:38.590 align:middle line:84% So those are the two that we are hunting for today. 00:13:38.590 --> 00:13:40.845 align:middle line:84% Of course, first the short-term effect, 00:13:40.845 --> 00:13:42.970 align:middle line:84% which are not obvious to start with, and then that. 00:13:42.970 --> 00:13:44.560 align:middle line:90% Yes, [INAUDIBLE], and then-- 00:13:44.560 --> 00:13:47.980 align:middle line:84% AUDIENCE: Typically, when this type of program is introduced, 00:13:47.980 --> 00:13:51.310 align:middle line:84% and there's this participatory method-- 00:13:51.310 --> 00:13:54.490 align:middle line:84% process, does that take place with a knowledge 00:13:54.490 --> 00:13:57.180 align:middle line:84% that the results of the process are-- 00:13:57.180 --> 00:13:59.430 align:middle line:84% the people who are identified are 00:13:59.430 --> 00:14:01.770 align:middle line:84% going to receive this very valuable program? 00:14:01.770 --> 00:14:04.312 align:middle line:84% Because we could imagine there could be some kind of capture. 00:14:04.312 --> 00:14:06.980 align:middle line:90% 00:14:06.980 --> 00:14:09.973 align:middle line:84% Is that taken into account when [INAUDIBLE] designed this? 00:14:09.973 --> 00:14:10.640 align:middle line:90% PROFESSOR: Yeah. 00:14:10.640 --> 00:14:11.640 align:middle line:90% That's a great question. 00:14:11.640 --> 00:14:14.670 align:middle line:84% So the answer is yes, everybody knows this is what's going on. 00:14:14.670 --> 00:14:17.580 align:middle line:84% I mean, you could maintain the surprise maybe the first time. 00:14:17.580 --> 00:14:19.910 align:middle line:84% But in any case, that's not sustainable. 00:14:19.910 --> 00:14:23.940 align:middle line:84% Or it's visible when someone gets something like that. 00:14:23.940 --> 00:14:25.530 align:middle line:90% So the news would go in. 00:14:25.530 --> 00:14:27.260 align:middle line:84% So this really is done with the view 00:14:27.260 --> 00:14:33.440 align:middle line:84% that people know when they are doing the exercise that this is 00:14:33.440 --> 00:14:35.060 align:middle line:90% going to be used for resources. 00:14:35.060 --> 00:14:39.740 align:middle line:84% So that's definitely a concern that people have expressed. 00:14:39.740 --> 00:14:43.730 align:middle line:84% And I'm going to ask you to suspend your disbelief 00:14:43.730 --> 00:14:46.910 align:middle line:84% until you get to Ben's part of the section, 00:14:46.910 --> 00:14:49.460 align:middle line:84% where he's going to talk about targeting. 00:14:49.460 --> 00:14:53.060 align:middle line:84% So in this particular instance, what they do as well is 00:14:53.060 --> 00:14:57.230 align:middle line:84% a verification ex-post to check that-- 00:14:57.230 --> 00:14:58.800 align:middle line:90% to actually go to the household. 00:14:58.800 --> 00:15:01.310 align:middle line:84% And then they actually bookend some of the households 00:15:01.310 --> 00:15:04.010 align:middle line:90% that are not poor enough. 00:15:04.010 --> 00:15:06.670 align:middle line:84% Another thing that-- because this was replicated in India. 00:15:06.670 --> 00:15:08.423 align:middle line:84% I was involved in the India study. 00:15:08.423 --> 00:15:10.340 align:middle line:84% That we found in India is that a lot of people 00:15:10.340 --> 00:15:12.920 align:middle line:90% also refused the transfer. 00:15:12.920 --> 00:15:15.680 align:middle line:84% In our setting, about half the people refused to transfer. 00:15:15.680 --> 00:15:17.773 align:middle line:84% But that was a bit of an Indian-specific thing. 00:15:17.773 --> 00:15:19.190 align:middle line:84% So when you visit them, it's like, 00:15:19.190 --> 00:15:20.315 align:middle line:90% what are you talking about? 00:15:20.315 --> 00:15:22.580 align:middle line:84% I don't want this, which is surprising 00:15:22.580 --> 00:15:26.010 align:middle line:90% since it's a lot of money. 00:15:26.010 --> 00:15:28.640 align:middle line:84% So I think the long and short of it, 00:15:28.640 --> 00:15:30.590 align:middle line:84% you're going to, again, talk about targeting 00:15:30.590 --> 00:15:34.430 align:middle line:84% in a lot of detail because it's a key issue, 00:15:34.430 --> 00:15:38.590 align:middle line:84% is that when everything is said and done, 00:15:38.590 --> 00:15:43.270 align:middle line:84% these problems of capture and potential manipulation 00:15:43.270 --> 00:15:44.890 align:middle line:90% are probably not that big. 00:15:44.890 --> 00:15:48.280 align:middle line:84% They are not inexistent, but they are not enormous, 00:15:48.280 --> 00:15:52.750 align:middle line:84% less first order than the economist, a knee-jerk reaction 00:15:52.750 --> 00:15:53.620 align:middle line:90% happens to be. 00:15:53.620 --> 00:15:56.300 align:middle line:90% Yep. 00:15:56.300 --> 00:15:58.200 align:middle line:84% AUDIENCE: I have a twofold question. 00:15:58.200 --> 00:16:00.560 align:middle line:84% The first one is going back to last class. 00:16:00.560 --> 00:16:04.440 align:middle line:84% We can't have poverty traps without financial frictions. 00:16:04.440 --> 00:16:06.780 align:middle line:90% PROFESSOR: Yeah. 00:16:06.780 --> 00:16:08.340 align:middle line:84% AUDIENCE: Conditional on that, is 00:16:08.340 --> 00:16:14.130 align:middle line:84% this a test of financial frictions 00:16:14.130 --> 00:16:18.875 align:middle line:84% and/or financial friction can induce poverty traps to join? 00:16:18.875 --> 00:16:20.250 align:middle line:84% PROFESSOR: No, I don't think it's 00:16:20.250 --> 00:16:22.120 align:middle line:90% a test of financial friction. 00:16:22.120 --> 00:16:24.210 align:middle line:84% And in fact, I think we can almost 00:16:24.210 --> 00:16:27.090 align:middle line:84% rule out that financial friction at the heart of this 00:16:27.090 --> 00:16:29.670 align:middle line:84% in this setting because this program is 00:16:29.670 --> 00:16:33.480 align:middle line:84% run by a microcredit company who is already 00:16:33.480 --> 00:16:36.000 align:middle line:84% present in these villages and would like nothing 00:16:36.000 --> 00:16:38.220 align:middle line:84% more than to give money to these people. 00:16:38.220 --> 00:16:40.980 align:middle line:84% And in fact, the inspiration for this program 00:16:40.980 --> 00:16:44.760 align:middle line:84% was that these guys are not willing to go. 00:16:44.760 --> 00:16:45.930 align:middle line:90% They don't have a project. 00:16:45.930 --> 00:16:47.740 align:middle line:90% They don't want to do it. 00:16:47.740 --> 00:16:53.310 align:middle line:84% So they're not willing or slash able to go. 00:16:53.310 --> 00:16:57.330 align:middle line:84% So I think, in a sense, we can rule out the financial friction 00:16:57.330 --> 00:16:59.940 align:middle line:84% with a certain amount of hand waving. 00:16:59.940 --> 00:17:02.857 align:middle line:84% This is a market where credit would 00:17:02.857 --> 00:17:04.940 align:middle line:84% be available for these people should they want it. 00:17:04.940 --> 00:17:08.480 align:middle line:90% 00:17:08.480 --> 00:17:10.849 align:middle line:84% So in this setting, that's not what we are testing. 00:17:10.849 --> 00:17:16.020 align:middle line:84% And most of today is going to stay terribly useful, 00:17:16.020 --> 00:17:18.925 align:middle line:84% which is I'm going to show you that-- 00:17:18.925 --> 00:17:20.300 align:middle line:84% previewing the result-- I'm going 00:17:20.300 --> 00:17:22.819 align:middle line:84% to show you that these programs are large effect that 00:17:22.819 --> 00:17:26.089 align:middle line:84% are very persistent over time, over 10 years, 00:17:26.089 --> 00:17:30.980 align:middle line:84% suggesting that there might be a poverty trap at play. 00:17:30.980 --> 00:17:32.930 align:middle line:84% Furthermore, we are going to see that there 00:17:32.930 --> 00:17:36.470 align:middle line:84% is, in fact, in this setting, this apparently, 00:17:36.470 --> 00:17:38.720 align:middle line:84% this S-shape relationship between capital today 00:17:38.720 --> 00:17:40.700 align:middle line:84% and capital tomorrow, which could underlying 00:17:40.700 --> 00:17:42.470 align:middle line:90% this persistent effect. 00:17:42.470 --> 00:17:44.630 align:middle line:84% What I'm not going to tell you today 00:17:44.630 --> 00:17:47.730 align:middle line:90% is where it's coming from. 00:17:47.730 --> 00:17:49.740 align:middle line:84% I'm willing to rule out nutrition because 00:17:49.740 --> 00:17:51.840 align:middle line:90% of what we discussed last time. 00:17:51.840 --> 00:17:55.320 align:middle line:84% I'm willing to rule out credit markets because of what I just 00:17:55.320 --> 00:17:55.980 align:middle line:90% told you. 00:17:55.980 --> 00:18:00.150 align:middle line:84% But what it actually is I think is very much open for her 00:18:00.150 --> 00:18:01.350 align:middle line:90% for discussion. 00:18:01.350 --> 00:18:04.630 align:middle line:84% And it's not going to be solved today. 00:18:04.630 --> 00:18:07.328 align:middle line:84% So this is, at this point, just like-- but already 00:18:07.328 --> 00:18:08.120 align:middle line:90% knowing it's there. 00:18:08.120 --> 00:18:11.880 align:middle line:84% And then we can go back to mechanism and say, where is it? 00:18:11.880 --> 00:18:15.630 align:middle line:84% Where is that map incoming that is so strong to come-- 00:18:15.630 --> 00:18:20.600 align:middle line:84% to lead to that, lead to that [INAUDIBLE]?? 00:18:20.600 --> 00:18:23.740 align:middle line:84% So for this intervention is super expensive, 00:18:23.740 --> 00:18:25.580 align:middle line:84% compared to doing almost everything, 00:18:25.580 --> 00:18:27.490 align:middle line:84% including giving just cash to people, which 00:18:27.490 --> 00:18:32.560 align:middle line:84% is another popular alternative, or including doing other stuff 00:18:32.560 --> 00:18:36.760 align:middle line:84% that could directly affect the health or education 00:18:36.760 --> 00:18:37.940 align:middle line:90% of these guys. 00:18:37.940 --> 00:18:40.060 align:middle line:84% So of course, it's an interesting question, 00:18:40.060 --> 00:18:43.360 align:middle line:84% whether or not it is effective, not only in the short term. 00:18:43.360 --> 00:18:49.150 align:middle line:84% It's only valuable if that idea of poverty trap is there, 00:18:49.150 --> 00:18:50.530 align:middle line:90% and it's, in fact, justified. 00:18:50.530 --> 00:18:54.310 align:middle line:84% The theory of change underlying it is that it's one big push. 00:18:54.310 --> 00:18:55.840 align:middle line:90% It's super worth it. 00:18:55.840 --> 00:18:57.975 align:middle line:84% It's super expensive, but it is worth it 00:18:57.975 --> 00:18:59.350 align:middle line:84% because it's going to be paid off 00:18:59.350 --> 00:19:02.440 align:middle line:84% over the lifetime of this person and potentially 00:19:02.440 --> 00:19:04.740 align:middle line:90% their children as well. 00:19:04.740 --> 00:19:07.037 align:middle line:84% So the first question is, does it work? 00:19:07.037 --> 00:19:08.870 align:middle line:84% So here we're going to take you a small step 00:19:08.870 --> 00:19:12.868 align:middle line:84% aside to think about how we even answer this question. 00:19:12.868 --> 00:19:14.660 align:middle line:84% For some of you, it's going to be a review. 00:19:14.660 --> 00:19:18.300 align:middle line:84% For some of you, it's going to be new material. 00:19:18.300 --> 00:19:20.840 align:middle line:84% But even the review, I find that-- 00:19:20.840 --> 00:19:21.890 align:middle line:90% every year, I teach this. 00:19:21.890 --> 00:19:24.380 align:middle line:84% And every year, I learn something 00:19:24.380 --> 00:19:27.140 align:middle line:90% by just staring at this number. 00:19:27.140 --> 00:19:29.390 align:middle line:84% So I hope that even for people for whom it's a review, 00:19:29.390 --> 00:19:32.110 align:middle line:90% you're going to find it useful. 00:19:32.110 --> 00:19:35.010 align:middle line:84% So it's a review of what it even means to have an effect 00:19:35.010 --> 00:19:39.150 align:middle line:90% and what impact means. 00:19:39.150 --> 00:19:41.633 align:middle line:90% So I'll consider our treatment. 00:19:41.633 --> 00:19:43.800 align:middle line:84% We want to know whether our particular treatment has 00:19:43.800 --> 00:19:44.790 align:middle line:90% an impact. 00:19:44.790 --> 00:19:49.750 align:middle line:84% And let's call it W and an outcome Y. 00:19:49.750 --> 00:19:51.930 align:middle line:84% For example, the outcome here is whether you 00:19:51.930 --> 00:19:54.930 align:middle line:84% got the ultra poor program is the treatment, 00:19:54.930 --> 00:19:59.520 align:middle line:84% and then whether or not you are able to feed yourself, 00:19:59.520 --> 00:20:01.230 align:middle line:90% food consumption or anything. 00:20:01.230 --> 00:20:03.300 align:middle line:84% Actually, there could be a range of outcomes. 00:20:03.300 --> 00:20:07.210 align:middle line:84% And we'll go back to this question is the outcome. 00:20:07.210 --> 00:20:11.280 align:middle line:84% So the way that Imbens and Rubin hold 00:20:11.280 --> 00:20:17.580 align:middle line:84% this up is to say, well, the outcome that I'm observing 00:20:17.580 --> 00:20:21.990 align:middle line:84% is something that I'm going to call potential outcome 00:20:21.990 --> 00:20:26.150 align:middle line:84% if treated, which is opt out only 00:20:26.150 --> 00:20:27.620 align:middle line:90% if you happen to be treated. 00:20:27.620 --> 00:20:30.830 align:middle line:84% So it's multiplied by this dummy, which is equal to 1 00:20:30.830 --> 00:20:32.970 align:middle line:90% if you're treated. 00:20:32.970 --> 00:20:37.230 align:middle line:84% And then it's your potential outcome untreated 00:20:37.230 --> 00:20:40.690 align:middle line:84% multiplied by a dummy, which is 1 if you're untreated, 00:20:40.690 --> 00:20:43.710 align:middle line:84% which is 1 minus Wi, which is dummy 00:20:43.710 --> 00:20:45.500 align:middle line:90% indicated that you're treated. 00:20:45.500 --> 00:20:50.300 align:middle line:84% So everybody is only observed in one of the two situation. 00:20:50.300 --> 00:20:54.840 align:middle line:84% But the dummies of treatment shuts things on and off. 00:20:54.840 --> 00:20:57.290 align:middle line:90% And so I'm not-- 00:20:57.290 --> 00:21:00.230 align:middle line:84% at least conceptually, I can always 00:21:00.230 --> 00:21:03.740 align:middle line:84% define the potential outcome for both situation. 00:21:03.740 --> 00:21:05.570 align:middle line:84% Before the automation occurred, you 00:21:05.570 --> 00:21:08.000 align:middle line:84% could have been a treated person or a control person. 00:21:08.000 --> 00:21:10.520 align:middle line:84% You have a potential outcome for these two, 00:21:10.520 --> 00:21:13.530 align:middle line:90% for these two settings. 00:21:13.530 --> 00:21:16.200 align:middle line:84% Before I move further, this might 00:21:16.200 --> 00:21:17.610 align:middle line:90% be familiar to a lot of you. 00:21:17.610 --> 00:21:19.200 align:middle line:84% But there is an assumption that's 00:21:19.200 --> 00:21:25.250 align:middle line:84% buried in there already in the idea 00:21:25.250 --> 00:21:28.490 align:middle line:90% that I can write this this way. 00:21:28.490 --> 00:21:33.600 align:middle line:84% Can someone tell me what the assumption is buried in there? 00:21:33.600 --> 00:21:34.480 align:middle line:90% Yeah, Recca. 00:21:34.480 --> 00:21:35.897 align:middle line:84% AUDIENCE: You assign the treatment 00:21:35.897 --> 00:21:37.764 align:middle line:84% to one group that's not effecting 00:21:37.764 --> 00:21:39.192 align:middle line:84% the experience of the other group 00:21:39.192 --> 00:21:41.328 align:middle line:84% so there's no network effect, for instance. 00:21:41.328 --> 00:21:42.120 align:middle line:90% PROFESSOR: Exactly. 00:21:42.120 --> 00:21:48.090 align:middle line:84% So this is what goes under the jargon of Stable Unit Treatment 00:21:48.090 --> 00:21:52.800 align:middle line:84% Value Assumption, or SUTVA, which basically says, 00:21:52.800 --> 00:21:54.750 align:middle line:84% if the fact that someone else is treated 00:21:54.750 --> 00:21:57.620 align:middle line:90% doesn't affect my treatment. 00:21:57.620 --> 00:21:59.380 align:middle line:84% And this is the only thing that allows 00:21:59.380 --> 00:22:03.370 align:middle line:84% me to write the potential outcome as just my treatment as 00:22:03.370 --> 00:22:06.940 align:middle line:84% opposed to my treatment and that of everyone apart. 00:22:06.940 --> 00:22:10.330 align:middle line:84% Otherwise, suppose you had two people. 00:22:10.330 --> 00:22:12.460 align:middle line:84% And one is treated, one is nontreated. 00:22:12.460 --> 00:22:14.410 align:middle line:84% But the treated person gets upset when 00:22:14.410 --> 00:22:16.400 align:middle line:90% the other person is untreated. 00:22:16.400 --> 00:22:18.670 align:middle line:84% Then the happiness of the treated person 00:22:18.670 --> 00:22:21.280 align:middle line:84% is a function of their untreated status 00:22:21.280 --> 00:22:24.140 align:middle line:84% and the treatment status of the other person. 00:22:24.140 --> 00:22:27.100 align:middle line:84% So just with two, we already have four potential outcomes, 00:22:27.100 --> 00:22:28.840 align:middle line:90% not two. 00:22:28.840 --> 00:22:31.240 align:middle line:84% Again, we are ruling them out here, which, as Recca, 00:22:31.240 --> 00:22:34.070 align:middle line:84% I points out, means we are rooting out network effects, 00:22:34.070 --> 00:22:37.780 align:middle line:84% for example, the possibility that someone would help someone 00:22:37.780 --> 00:22:38.860 align:middle line:90% else. 00:22:38.860 --> 00:22:43.210 align:middle line:84% We are ruling out invidious comparison effect, 00:22:43.210 --> 00:22:43.990 align:middle line:90% like I just did. 00:22:43.990 --> 00:22:46.840 align:middle line:84% It might make me upset that someone else is helped. 00:22:46.840 --> 00:22:48.730 align:middle line:90% We are rolling out learning. 00:22:48.730 --> 00:22:52.210 align:middle line:84% We are rolling out straight, externality, contagion, 00:22:52.210 --> 00:22:54.220 align:middle line:84% this type of things, which become more relevant 00:22:54.220 --> 00:22:58.370 align:middle line:84% for medical treatment and that kind of thing. 00:22:58.370 --> 00:23:01.440 align:middle line:84% So we are not ruling out because we think they never exist. 00:23:01.440 --> 00:23:05.770 align:middle line:84% We're ruling out because we found out we are ruling out. 00:23:05.770 --> 00:23:09.140 align:middle line:84% And then we'll bring them back at some point. 00:23:09.140 --> 00:23:11.890 align:middle line:84% So that's an important thing to say. 00:23:11.890 --> 00:23:15.820 align:middle line:84% It becomes hard to tackle if we don't have to SUTVA. 00:23:15.820 --> 00:23:20.710 align:middle line:84% So when we don't have SUTVA, we try and model in what way 00:23:20.710 --> 00:23:24.070 align:middle line:84% we are affected by the other units. 00:23:24.070 --> 00:23:26.430 align:middle line:84% So if we can write it this way, then there 00:23:26.430 --> 00:23:27.723 align:middle line:90% is a well-defined idea. 00:23:27.723 --> 00:23:28.890 align:middle line:90% What's the treatment effect? 00:23:28.890 --> 00:23:31.020 align:middle line:84% It's my potential outcome if I'm treated 00:23:31.020 --> 00:23:34.850 align:middle line:84% versus my potential outcome if I'm not. 00:23:34.850 --> 00:23:39.800 align:middle line:84% Are we able to estimate that thing for any single person? 00:23:39.800 --> 00:23:43.010 align:middle line:84% No because we are never observing both. 00:23:43.010 --> 00:23:46.400 align:middle line:84% So that's not happening because ex-post, 00:23:46.400 --> 00:23:48.830 align:middle line:84% we only observe one of the two potential outcomes-- 00:23:48.830 --> 00:23:50.600 align:middle line:84% the individual ends up being treated 00:23:50.600 --> 00:23:52.130 align:middle line:90% and ends up not being treated. 00:23:52.130 --> 00:23:56.000 align:middle line:84% So we'll never know the individual treatment effect 00:23:56.000 --> 00:23:57.700 align:middle line:90% for each individual. 00:23:57.700 --> 00:24:01.750 align:middle line:90% We might get close to that. 00:24:01.750 --> 00:24:07.030 align:middle line:84% If we have enough variation in, say, observable characteristics 00:24:07.030 --> 00:24:11.260 align:middle line:84% about an individual, and we have sufficiently much data, 00:24:11.260 --> 00:24:14.380 align:middle line:84% we can try to get to somewhere in saying, well, the treatment 00:24:14.380 --> 00:24:17.140 align:middle line:84% effect for someone who looks like Recca, 00:24:17.140 --> 00:24:20.830 align:middle line:84% in terms of being a young woman who is studying economics, 00:24:20.830 --> 00:24:25.310 align:middle line:84% who grew up in eight countries, that tends to be that. 00:24:25.310 --> 00:24:27.250 align:middle line:84% But we would have to have enough of people 00:24:27.250 --> 00:24:30.610 align:middle line:84% like that to be able to say, this is the treatment 00:24:30.610 --> 00:24:32.120 align:middle line:90% effect for someone like that. 00:24:32.120 --> 00:24:35.420 align:middle line:90% So this, we might be able to do. 00:24:35.420 --> 00:24:37.550 align:middle line:84% So we could be interested in the average treatment 00:24:37.550 --> 00:24:39.380 align:middle line:84% effect for the population, which would 00:24:39.380 --> 00:24:43.220 align:middle line:84% be the expectation across the globe of the difference 00:24:43.220 --> 00:24:45.780 align:middle line:90% of the treatment effects. 00:24:45.780 --> 00:24:47.790 align:middle line:84% We could be interested in the one for whom 00:24:47.790 --> 00:24:48.820 align:middle line:90% received the treatment. 00:24:48.820 --> 00:24:50.550 align:middle line:84% So that's the average treatment effect. 00:24:50.550 --> 00:24:54.080 align:middle line:84% That's the treatment on the treated. 00:24:54.080 --> 00:24:56.450 align:middle line:84% And then as the discussion, I just 00:24:56.450 --> 00:25:02.000 align:middle line:84% had said we could be interested in the impact of the treatment 00:25:02.000 --> 00:25:04.520 align:middle line:84% for those who have some characteristics 00:25:04.520 --> 00:25:06.170 align:middle line:90% observed or unobserved. 00:25:06.170 --> 00:25:09.470 align:middle line:84% So this becomes still an average treatment effect, 00:25:09.470 --> 00:25:13.950 align:middle line:84% but for a person who has some vector 00:25:13.950 --> 00:25:17.130 align:middle line:84% or whose characteristics are some vector x. 00:25:17.130 --> 00:25:19.410 align:middle line:84% So when we talk about personalized medicine, 00:25:19.410 --> 00:25:21.660 align:middle line:84% machine learning, et cetera, that's really what we are 00:25:21.660 --> 00:25:28.330 align:middle line:84% talking about, which is trying to bypass this thing that we 00:25:28.330 --> 00:25:30.730 align:middle line:84% cannot do personalized medicine because we only observe 00:25:30.730 --> 00:25:36.100 align:middle line:84% the same person ever once, by exploiting the old information 00:25:36.100 --> 00:25:38.548 align:middle line:90% we have about them. 00:25:38.548 --> 00:25:40.090 align:middle line:84% So that's things we might want to do. 00:25:40.090 --> 00:25:44.640 align:middle line:84% This is all a flavor of average treatment effect. 00:25:44.640 --> 00:25:47.220 align:middle line:84% We might want to know how the treatment affects 00:25:47.220 --> 00:25:50.520 align:middle line:84% not only the mean across groups, but the distribution 00:25:50.520 --> 00:25:51.940 align:middle line:90% of the treatment effects. 00:25:51.940 --> 00:25:54.510 align:middle line:84% So that's we are going to get from quantile treatment effect 00:25:54.510 --> 00:25:58.220 align:middle line:84% by comparing people, by comparing the quantile. 00:25:58.220 --> 00:25:59.220 align:middle line:90% Could be the median. 00:25:59.220 --> 00:26:02.310 align:middle line:84% Could be the 90th percentile, the 5th percentile. 00:26:02.310 --> 00:26:06.150 align:middle line:84% We might be interested in the quantile of the treatment 00:26:06.150 --> 00:26:13.360 align:middle line:84% effect, which is not the same thing as this 00:26:13.360 --> 00:26:17.920 align:middle line:84% because unlike the expectation, which are nicely walking out 00:26:17.920 --> 00:26:23.310 align:middle line:84% in both sense, the quantile of a difference 00:26:23.310 --> 00:26:25.080 align:middle line:84% is not the difference in the quantile 00:26:25.080 --> 00:26:28.320 align:middle line:84% unless you are willing to make another assumption. 00:26:28.320 --> 00:26:30.255 align:middle line:90% So I'm pointing that out. 00:26:30.255 --> 00:26:31.130 align:middle line:90% You already heard it. 00:26:31.130 --> 00:26:34.840 align:middle line:84% But we forget it all the time because if you 00:26:34.840 --> 00:26:38.800 align:middle line:84% add a convenient assumption that the treatment doesn't 00:26:38.800 --> 00:26:43.130 align:middle line:84% affect people's ranks, then this start 00:26:43.130 --> 00:26:44.430 align:middle line:90% to look very much like that. 00:26:44.430 --> 00:26:48.170 align:middle line:84% So under this assumption, which is very strong, 00:26:48.170 --> 00:26:50.090 align:middle line:84% then you can get at something that 00:26:50.090 --> 00:26:53.220 align:middle line:84% looks more like the quantile of the treatment effect. 00:26:53.220 --> 00:26:57.560 align:middle line:84% But we have to remember that unless you're 00:26:57.560 --> 00:27:00.170 align:middle line:84% willing to make that assumption, that's not necessarily 00:27:00.170 --> 00:27:01.160 align:middle line:90% a very good-- 00:27:01.160 --> 00:27:02.880 align:middle line:90% that doesn't work that way. 00:27:02.880 --> 00:27:07.620 align:middle line:84% For example, if you go back to the ultra poor program, 00:27:07.620 --> 00:27:11.260 align:middle line:84% it could well be that the program not only affects 00:27:11.260 --> 00:27:17.820 align:middle line:84% people's wealth level, consumption, et cetera, 00:27:17.820 --> 00:27:21.690 align:middle line:84% but also their rank in where they stand in the distribution. 00:27:21.690 --> 00:27:25.260 align:middle line:84% There is nothing that assures us that we have this rank 00:27:25.260 --> 00:27:28.330 align:middle line:90% and variance assumption. 00:27:28.330 --> 00:27:29.620 align:middle line:90% So now, suppose we have that. 00:27:29.620 --> 00:27:31.960 align:middle line:84% We go back to the simpler problem 00:27:31.960 --> 00:27:36.740 align:middle line:84% of trying to know what's the average treatment effect. 00:27:36.740 --> 00:27:38.290 align:middle line:84% And we have a group of treated people 00:27:38.290 --> 00:27:39.760 align:middle line:90% and a group of control people. 00:27:39.760 --> 00:27:43.690 align:middle line:84% So, well, the best we can do is to start 00:27:43.690 --> 00:27:48.208 align:middle line:84% by looking at the difference between the mean. 00:27:48.208 --> 00:27:50.500 align:middle line:84% We would like the difference between the mean, which we 00:27:50.500 --> 00:27:52.670 align:middle line:90% replace by the sample average. 00:27:52.670 --> 00:27:57.220 align:middle line:84% So that's the expectation of Yi treated 00:27:57.220 --> 00:28:00.220 align:middle line:84% for the treated minus the expectation of Yi 00:28:00.220 --> 00:28:03.610 align:middle line:90% control for the control. 00:28:03.610 --> 00:28:07.900 align:middle line:84% I'm going to add and subtract something, an object that 00:28:07.900 --> 00:28:11.360 align:middle line:84% is not observed but conceptually exist, 00:28:11.360 --> 00:28:17.920 align:middle line:84% which is the average of the untreated outcome 00:28:17.920 --> 00:28:21.160 align:middle line:90% for the treated population. 00:28:21.160 --> 00:28:24.960 align:middle line:90% So I'm subtracting it here. 00:28:24.960 --> 00:28:30.690 align:middle line:84% And then I'm adding it in the part that is missing. 00:28:30.690 --> 00:28:32.010 align:middle line:90% So what does it give me? 00:28:32.010 --> 00:28:34.050 align:middle line:84% Well, now that I have that, I can just 00:28:34.050 --> 00:28:36.360 align:middle line:90% rearrange a little bit. 00:28:36.360 --> 00:28:42.990 align:middle line:84% And I'm getting the expectation of Yi1 minus Yi0 00:28:42.990 --> 00:28:43.990 align:middle line:90% for the treated. 00:28:43.990 --> 00:28:46.250 align:middle line:90% So what is this? 00:28:46.250 --> 00:28:47.165 align:middle line:90% What's that quantity? 00:28:47.165 --> 00:28:54.390 align:middle line:90% 00:28:54.390 --> 00:28:57.900 align:middle line:84% Is it an object that we are potentially interested? 00:28:57.900 --> 00:28:58.400 align:middle line:90% Yep. 00:28:58.400 --> 00:29:00.192 align:middle line:84% AUDIENCE: It's the average treatment effect 00:29:00.192 --> 00:29:01.470 align:middle line:90% on people who were treated. 00:29:01.470 --> 00:29:03.387 align:middle line:84% PROFESSOR: So treatment effect on the treated. 00:29:03.387 --> 00:29:04.190 align:middle line:90% That's great. 00:29:04.190 --> 00:29:05.990 align:middle line:90% But then we have that. 00:29:05.990 --> 00:29:07.520 align:middle line:90% And what's that difference? 00:29:07.520 --> 00:29:11.256 align:middle line:90% 00:29:11.256 --> 00:29:13.591 align:middle line:84% AUDIENCE: Whatever differences already 00:29:13.591 --> 00:29:16.808 align:middle line:84% existed between the people who were selected for treatment. 00:29:16.808 --> 00:29:17.600 align:middle line:90% PROFESSOR: Exactly. 00:29:17.600 --> 00:29:19.070 align:middle line:90% So it's the potential-- 00:29:19.070 --> 00:29:20.990 align:middle line:84% rather than already exists, that would 00:29:20.990 --> 00:29:22.820 align:middle line:84% have existed anyways or something 00:29:22.820 --> 00:29:26.570 align:middle line:84% in the absence of the treatment because it 00:29:26.570 --> 00:29:29.000 align:middle line:84% could be captured by a baseline, or it couldn't be. 00:29:29.000 --> 00:29:30.590 align:middle line:84% It's not necessarily a temporal thing. 00:29:30.590 --> 00:29:34.910 align:middle line:84% But it's saying that in term of potential outcome, 00:29:34.910 --> 00:29:38.790 align:middle line:84% these two groups are potentially already very different. 00:29:38.790 --> 00:29:42.860 align:middle line:84% So take within the village, if I run the ultra poor program, 00:29:42.860 --> 00:29:47.240 align:middle line:84% and I run my participatory resource exercise. 00:29:47.240 --> 00:29:50.510 align:middle line:90% And they identify 10 people. 00:29:50.510 --> 00:29:54.300 align:middle line:84% And then I go, and I survey these 10 people. 00:29:54.300 --> 00:29:58.340 align:middle line:84% And finally, the survey staff decides 00:29:58.340 --> 00:30:00.590 align:middle line:84% they're only going to take 5 out of the 10 people 00:30:00.590 --> 00:30:03.560 align:middle line:84% because the other one don't quite make the cut. 00:30:03.560 --> 00:30:06.980 align:middle line:84% Someone could say-- and someone has said and done 00:30:06.980 --> 00:30:09.290 align:middle line:84% a first analysis of this program by saying, 00:30:09.290 --> 00:30:11.900 align:middle line:84% let me compare the five people who got in the program 00:30:11.900 --> 00:30:14.365 align:middle line:84% to the five people who didn't get in. 00:30:14.365 --> 00:30:15.740 align:middle line:84% The five people who didn't get in 00:30:15.740 --> 00:30:18.020 align:middle line:84% were presumably quite poor because they were selected 00:30:18.020 --> 00:30:19.550 align:middle line:90% by their fellow villagers. 00:30:19.550 --> 00:30:22.880 align:middle line:90% 00:30:22.880 --> 00:30:25.400 align:middle line:84% So if I still find a difference between the two, 00:30:25.400 --> 00:30:27.890 align:middle line:84% it must be the fault of the poor. 00:30:27.890 --> 00:30:31.610 align:middle line:84% Of course, what we don't know is whether there 00:30:31.610 --> 00:30:33.560 align:middle line:84% is something different about them 00:30:33.560 --> 00:30:41.110 align:middle line:84% that means that the staff of the NGO decided not to take them. 00:30:41.110 --> 00:30:43.690 align:middle line:84% It could be observable, or it could be unobservable. 00:30:43.690 --> 00:30:45.140 align:middle line:90% And that's the problem we have. 00:30:45.140 --> 00:30:46.990 align:middle line:90% So that's the selection problem. 00:30:46.990 --> 00:30:49.960 align:middle line:84% So basically, we have three cases. 00:30:49.960 --> 00:30:52.825 align:middle line:84% Either we are in the best case scenario 00:30:52.825 --> 00:30:54.700 align:middle line:84% from the point of view of this problem, which 00:30:54.700 --> 00:30:57.320 align:middle line:84% is that the program was randomly assigned. 00:30:57.320 --> 00:31:03.010 align:middle line:84% So for example, a frequent thing in the [INAUDIBLE] study, 00:31:03.010 --> 00:31:06.400 align:middle line:84% there are two kind of random assignment. 00:31:06.400 --> 00:31:10.080 align:middle line:84% Either the villages were randomly selected, 00:31:10.080 --> 00:31:13.920 align:middle line:84% or the villages we are not randomly selected. 00:31:13.920 --> 00:31:15.600 align:middle line:90% The identification was done. 00:31:15.600 --> 00:31:18.000 align:middle line:84% And then within the people who were identified, 00:31:18.000 --> 00:31:21.540 align:middle line:84% people were or were not selected randomly 00:31:21.540 --> 00:31:22.920 align:middle line:90% within qualified people. 00:31:22.920 --> 00:31:24.990 align:middle line:84% In that case, there is nothing systematically 00:31:24.990 --> 00:31:30.060 align:middle line:84% different about the people who got selected once by lot, 00:31:30.060 --> 00:31:33.130 align:middle line:90% once I do all of this thing. 00:31:33.130 --> 00:31:35.730 align:middle line:84% So in that case, the selection should be 0. 00:31:35.730 --> 00:31:38.908 align:middle line:84% And I should get an unbiased effect 00:31:38.908 --> 00:31:41.200 align:middle line:84% estimate of the effect of the treatment on the treated. 00:31:41.200 --> 00:31:42.867 align:middle line:84% That's what we are going to study today. 00:31:42.867 --> 00:31:45.660 align:middle line:90% 00:31:45.660 --> 00:31:55.100 align:middle line:84% Another setting is one where it's not 00:31:55.100 --> 00:31:58.670 align:middle line:90% a known function of covariates. 00:31:58.670 --> 00:32:03.200 align:middle line:84% But I'm willing to say that once I know everything about them, 00:32:03.200 --> 00:32:06.712 align:middle line:84% enough things about them, something happened, 00:32:06.712 --> 00:32:08.420 align:middle line:84% which means that some people got treated, 00:32:08.420 --> 00:32:10.100 align:middle line:90% and some people didn't. 00:32:10.100 --> 00:32:13.040 align:middle line:84% And I'm willing to say that conditional on the caveat 00:32:13.040 --> 00:32:16.850 align:middle line:84% that I observe, despite my not knowing 00:32:16.850 --> 00:32:18.950 align:middle line:84% exactly how the sausage was made, 00:32:18.950 --> 00:32:20.840 align:middle line:84% I am willing to make the assumption 00:32:20.840 --> 00:32:25.040 align:middle line:84% that the potential outcomes are independent. 00:32:25.040 --> 00:32:27.740 align:middle line:84% The two potential outcomes conditional on the x 00:32:27.740 --> 00:32:30.990 align:middle line:84% are independent of the treatment status. 00:32:30.990 --> 00:32:34.191 align:middle line:90% And that leads you to-- 00:32:34.191 --> 00:32:38.250 align:middle line:84% that would make you confident to try and control adequately 00:32:38.250 --> 00:32:39.900 align:middle line:90% for these different observables. 00:32:39.900 --> 00:32:43.560 align:middle line:84% As long as you control fully for this set of x, even though you 00:32:43.560 --> 00:32:46.770 align:middle line:84% don't really know why someone with this particular 00:32:46.770 --> 00:32:49.380 align:middle line:84% characteristic was treated, another was not, 00:32:49.380 --> 00:32:54.137 align:middle line:84% you are reasonably, sufficiently satisfied. 00:32:54.137 --> 00:32:56.220 align:middle line:84% So if you do that, for example, that's the example 00:32:56.220 --> 00:32:58.750 align:middle line:84% I gave you with this idea is that conditional had 00:32:58.750 --> 00:33:01.380 align:middle line:90% been selected by the people. 00:33:01.380 --> 00:33:03.090 align:middle line:84% You're willing to say that you don't know 00:33:03.090 --> 00:33:04.650 align:middle line:90% exactly how the staff chose. 00:33:04.650 --> 00:33:08.340 align:middle line:84% But whatever they did, that's as good as on them. 00:33:08.340 --> 00:33:10.773 align:middle line:84% Then if I'm able to control fully in this case, 00:33:10.773 --> 00:33:12.690 align:middle line:84% that would be easy because I have the selected 00:33:12.690 --> 00:33:15.330 align:middle line:84% and the nonselected people, I can control for that status, 00:33:15.330 --> 00:33:19.800 align:middle line:84% and within that, proceed almost as things had been randomly 00:33:19.800 --> 00:33:21.460 align:middle line:84% assigned, even though they weren't. 00:33:21.460 --> 00:33:23.580 align:middle line:90% And I know they weren't. 00:33:23.580 --> 00:33:27.810 align:middle line:84% So that's, of course, the standard justification 00:33:27.810 --> 00:33:30.490 align:middle line:84% for a lot of [INAUDIBLE] regression, 00:33:30.490 --> 00:33:33.030 align:middle line:84% which adds on top of that a bunch of functional 00:33:33.030 --> 00:33:36.420 align:middle line:84% form assumption; matching regression, which 00:33:36.420 --> 00:33:40.410 align:middle line:84% adds less strong functional form assumption; 00:33:40.410 --> 00:33:43.050 align:middle line:84% or sometimes no functional form assumption. 00:33:43.050 --> 00:33:49.140 align:middle line:84% For example, if the number of cells that you have of excess 00:33:49.140 --> 00:33:51.960 align:middle line:84% is not that large, and the number of observations 00:33:51.960 --> 00:33:54.270 align:middle line:84% is very large, then you can calculate 00:33:54.270 --> 00:33:57.520 align:middle line:84% the difference between treatment and control in each cell. 00:33:57.520 --> 00:33:59.607 align:middle line:84% And then take the weighted average of this cell. 00:33:59.607 --> 00:34:01.440 align:middle line:84% And there you go with your treatment effect. 00:34:01.440 --> 00:34:02.920 align:middle line:84% All your assumptions are correct. 00:34:02.920 --> 00:34:06.090 align:middle line:84% So Josh, for example, has a very well-known paper 00:34:06.090 --> 00:34:08.639 align:middle line:84% on the effect of the military service, where he's saying, 00:34:08.639 --> 00:34:11.429 align:middle line:84% I'm controlling for everything that could potentially 00:34:11.429 --> 00:34:14.429 align:middle line:84% enter in the decision in why someone 00:34:14.429 --> 00:34:17.590 align:middle line:84% does the military service within the cells. 00:34:17.590 --> 00:34:19.020 align:middle line:90% I'm looking at each cell. 00:34:19.020 --> 00:34:20.969 align:middle line:84% I'm looking at different between has 00:34:20.969 --> 00:34:22.560 align:middle line:90% done it versus has not done it. 00:34:22.560 --> 00:34:23.790 align:middle line:90% And then I'm reaggregating. 00:34:23.790 --> 00:34:25.318 align:middle line:90% So that's a matching estimate. 00:34:25.318 --> 00:34:27.360 align:middle line:84% And you have different type of matching estimates 00:34:27.360 --> 00:34:31.500 align:middle line:84% that are more or less parametric, and therefore more 00:34:31.500 --> 00:34:32.550 align:middle line:90% or less believable. 00:34:32.550 --> 00:34:37.020 align:middle line:84% But the underlying assumption is always this of unconfoundedness 00:34:37.020 --> 00:34:39.239 align:middle line:90% once you know enough. 00:34:39.239 --> 00:34:42.290 align:middle line:84% And then third method with the same idea 00:34:42.290 --> 00:34:47.060 align:middle line:84% is a double machine learning estimators, double post-lasso, 00:34:47.060 --> 00:34:53.780 align:middle line:84% that Victor Chernozhukov and Belloni developed and has 00:34:53.780 --> 00:34:56.300 align:middle line:84% become quite popular in the applied literature, 00:34:56.300 --> 00:34:59.840 align:middle line:84% and the more generic double machine learning method 00:34:59.840 --> 00:35:03.140 align:middle line:84% that Whitney, Victor, me, and a bunch of other people 00:35:03.140 --> 00:35:04.670 align:middle line:90% worked on. 00:35:04.670 --> 00:35:05.930 align:middle line:90% And that is also quite nice. 00:35:05.930 --> 00:35:11.150 align:middle line:84% But it's conditional on this assumption. 00:35:11.150 --> 00:35:13.850 align:middle line:84% So it adds bells and whistles and sophistication 00:35:13.850 --> 00:35:16.110 align:middle line:84% in terms of the econometrics analysis. 00:35:16.110 --> 00:35:17.510 align:middle line:84% But at the end of the day, that's 00:35:17.510 --> 00:35:19.730 align:middle line:84% always the same thing, which is not really 00:35:19.730 --> 00:35:22.300 align:middle line:84% a testable assumption in most cases. 00:35:22.300 --> 00:35:25.870 align:middle line:84% So this is your observation-- observer-- 00:35:25.870 --> 00:35:30.100 align:middle line:84% I would call it without prejudice no research design 00:35:30.100 --> 00:35:32.335 align:middle line:84% strategy because my strategy is no strategy. 00:35:32.335 --> 00:35:35.780 align:middle line:90% 00:35:35.780 --> 00:35:40.016 align:middle line:84% And then there is a strategy where you're saying, well, 00:35:40.016 --> 00:35:42.740 align:middle line:84% you're worried that the potential outcome depends 00:35:42.740 --> 00:35:44.053 align:middle line:90% on the assignment. 00:35:44.053 --> 00:35:46.220 align:middle line:84% You're not willing to make the assumption that there 00:35:46.220 --> 00:35:48.840 align:middle line:90% is a set of x that solves it. 00:35:48.840 --> 00:35:52.310 align:middle line:84% Then they have to do something to try and find 00:35:52.310 --> 00:35:56.880 align:middle line:84% a corner of the world who are not subject to this. 00:35:56.880 --> 00:36:00.450 align:middle line:84% And that's where you get to the research design strategy. 00:36:00.450 --> 00:36:03.140 align:middle line:84% So we'll spend a lot of the class looking at examples 00:36:03.140 --> 00:36:07.850 align:middle line:84% of that difference-in-differences one, 00:36:07.850 --> 00:36:13.040 align:middle line:84% regression discontinuity design, and instrumental variable. 00:36:13.040 --> 00:36:15.080 align:middle line:84% A special kind of instrumented variable 00:36:15.080 --> 00:36:20.480 align:middle line:84% is one where you have a latent variable that might 00:36:20.480 --> 00:36:24.000 align:middle line:90% be randomly assigned itself. 00:36:24.000 --> 00:36:29.250 align:middle line:84% And you use that to form an instrument for the treatment 00:36:29.250 --> 00:36:31.240 align:middle line:84% that you're actually interested in. 00:36:31.240 --> 00:36:34.830 align:middle line:84% So there is no Wi that is randomly assigned or as good 00:36:34.830 --> 00:36:36.010 align:middle line:90% as randomly assigned. 00:36:36.010 --> 00:36:39.960 align:middle line:84% But there is a Zi that is potentially randomly assigned, 00:36:39.960 --> 00:36:42.510 align:middle line:84% affects Wi, and unless you're willing to make 00:36:42.510 --> 00:36:48.550 align:middle line:84% some additional assumption to recover the effect of the Wi. 00:36:48.550 --> 00:36:50.850 align:middle line:90% So that's what we are going to-- 00:36:50.850 --> 00:36:54.660 align:middle line:84% short preview of how the different strategy 00:36:54.660 --> 00:36:58.060 align:middle line:84% that we are going to see in the course of the semester line up. 00:36:58.060 --> 00:37:00.870 align:middle line:84% So today we are starting with the easiest one, 00:37:00.870 --> 00:37:02.400 align:middle line:84% at least the conceptually easiest 00:37:02.400 --> 00:37:05.790 align:middle line:84% one, which is random assignment, which, 00:37:05.790 --> 00:37:09.750 align:middle line:84% so by definition, random assignments 00:37:09.750 --> 00:37:13.860 align:middle line:84% solve the selection bias in your sample 00:37:13.860 --> 00:37:17.560 align:middle line:84% because it's a known assignment, known assignment rule, 00:37:17.560 --> 00:37:19.650 align:middle line:84% which is either fully randomized or randomized 00:37:19.650 --> 00:37:21.870 align:middle line:84% conditional unbiased, but randomized anyway, 00:37:21.870 --> 00:37:25.350 align:middle line:84% so you can compare things directly without worry 00:37:25.350 --> 00:37:27.420 align:middle line:90% about the selection bias. 00:37:27.420 --> 00:37:32.580 align:middle line:84% That said, that's still not like a panacea. 00:37:32.580 --> 00:37:36.210 align:middle line:84% There are issues with randomized evaluation 00:37:36.210 --> 00:37:37.920 align:middle line:90% that many people have mentioned. 00:37:37.920 --> 00:37:40.860 align:middle line:90% 00:37:40.860 --> 00:37:42.960 align:middle line:84% And we are going to keep seeing those. 00:37:42.960 --> 00:37:45.510 align:middle line:84% And we're going to see some of this today. 00:37:45.510 --> 00:37:47.160 align:middle line:84% What are some of the things that people 00:37:47.160 --> 00:37:49.770 align:middle line:84% worry about in terms of validity of results 00:37:49.770 --> 00:37:54.010 align:middle line:84% in the case of randomized evaluations? 00:37:54.010 --> 00:37:54.510 align:middle line:90% Recca. 00:37:54.510 --> 00:37:57.480 align:middle line:84% AUDIENCE: You can conduct an RCT in a specific setting. 00:37:57.480 --> 00:37:59.482 align:middle line:84% But you might think that if you wanted 00:37:59.482 --> 00:38:01.774 align:middle line:84% to apply that within a different country, for instance, 00:38:01.774 --> 00:38:05.278 align:middle line:84% that so would we have that effect elsewhere? 00:38:05.278 --> 00:38:06.070 align:middle line:90% PROFESSOR: Exactly. 00:38:06.070 --> 00:38:07.450 align:middle line:90% So that's the first issue. 00:38:07.450 --> 00:38:09.430 align:middle line:84% And in fact, one that we're going to talk today 00:38:09.430 --> 00:38:16.150 align:middle line:84% is the extent to which the results are context dependent 00:38:16.150 --> 00:38:19.060 align:middle line:84% and would be heterogeneous from context to context. 00:38:19.060 --> 00:38:22.000 align:middle line:84% And therefore, the result of one is predictive or not 00:38:22.000 --> 00:38:23.820 align:middle line:84% of what would happen in another setting. 00:38:23.820 --> 00:38:26.070 align:middle line:84% So that's the first concern that's definitely present. 00:38:26.070 --> 00:38:27.790 align:middle line:84% What are other concerns people have? 00:38:27.790 --> 00:38:28.600 align:middle line:90% Yeah. 00:38:28.600 --> 00:38:30.760 align:middle line:84% AUDIENCE: When people don't behave following 00:38:30.760 --> 00:38:33.010 align:middle line:84% the assignment that we're giving and treat it 00:38:33.010 --> 00:38:34.480 align:middle line:90% as if they were controls. 00:38:34.480 --> 00:38:36.460 align:middle line:84% And controls act as if they were treated. 00:38:36.460 --> 00:38:40.480 align:middle line:84% For example, if you give them a pill, I can take the pill, 00:38:40.480 --> 00:38:42.142 align:middle line:90% or I can decide not to take it. 00:38:42.142 --> 00:38:42.850 align:middle line:90% PROFESSOR: Right. 00:38:42.850 --> 00:38:45.393 align:middle line:84% So there are various noncompliance issue, 00:38:45.393 --> 00:38:46.810 align:middle line:84% for example, the one I was talking 00:38:46.810 --> 00:38:49.850 align:middle line:84% about in the case of the ultra poor, 00:38:49.850 --> 00:38:51.970 align:middle line:84% which is some people just say no to the program. 00:38:51.970 --> 00:38:53.740 align:middle line:84% Alternatively, some people might manage 00:38:53.740 --> 00:38:56.890 align:middle line:84% to sneak in, which means that your actual treatment is not 00:38:56.890 --> 00:39:00.230 align:middle line:90% randomly assigned anymore. 00:39:00.230 --> 00:39:02.755 align:middle line:84% So then you have to deal with that, 00:39:02.755 --> 00:39:09.910 align:middle line:84% that you lose the world of perfect [INAUDIBLE] Yep. 00:39:09.910 --> 00:39:11.890 align:middle line:84% AUDIENCE: Following up on the first comments, 00:39:11.890 --> 00:39:15.430 align:middle line:84% there's also a more realistic issue here. 00:39:15.430 --> 00:39:18.220 align:middle line:84% In the realm of heterogeneous treatment effects, 00:39:18.220 --> 00:39:21.220 align:middle line:84% the ability to run an RCT may depend 00:39:21.220 --> 00:39:23.380 align:middle line:84% on people knowing that, oh, they are 00:39:23.380 --> 00:39:25.300 align:middle line:84% in the right tail of the treatment effects. 00:39:25.300 --> 00:39:28.570 align:middle line:84% People say, I'm incapable of running an experiment 00:39:28.570 --> 00:39:29.590 align:middle line:90% in a village. 00:39:29.590 --> 00:39:32.200 align:middle line:84% Maybe that's because the governors in that village 00:39:32.200 --> 00:39:35.037 align:middle line:84% are aware of the fact that, oh, we are actually 00:39:35.037 --> 00:39:36.620 align:middle line:84% on the right tail of the distribution. 00:39:36.620 --> 00:39:38.060 align:middle line:90% It would be very good for us. 00:39:38.060 --> 00:39:40.480 align:middle line:84% So it's not only that the treatment 00:39:40.480 --> 00:39:43.090 align:middle line:84% effect in that village is not predictable of others, 00:39:43.090 --> 00:39:45.550 align:middle line:84% it's that the the treatment in the village 00:39:45.550 --> 00:39:47.290 align:middle line:84% is the highest possible treatment 00:39:47.290 --> 00:39:49.270 align:middle line:90% effect compared to the-- 00:39:49.270 --> 00:39:50.440 align:middle line:90% PROFESSOR: Right. 00:39:50.440 --> 00:39:51.010 align:middle line:90% Excellent. 00:39:51.010 --> 00:39:53.440 align:middle line:84% So that's what Heckman called the randomization 00:39:53.440 --> 00:39:57.620 align:middle line:84% bias, which is that not only is it originating in the world. 00:39:57.620 --> 00:40:02.260 align:middle line:84% So the results of one experiment may or may not generalize. 00:40:02.260 --> 00:40:04.810 align:middle line:84% But there is something specific about 00:40:04.810 --> 00:40:08.900 align:middle line:84% randomized controlled trial that may lead you to have, 00:40:08.900 --> 00:40:10.870 align:middle line:84% for example, higher treatment effect. 00:40:10.870 --> 00:40:15.100 align:middle line:84% And one of them is site selection, which is someone, 00:40:15.100 --> 00:40:17.470 align:middle line:84% either the researcher because they are willing to-- they 00:40:17.470 --> 00:40:19.390 align:middle line:90% would like to publish a paper-- 00:40:19.390 --> 00:40:22.540 align:middle line:84% or the partners because they would like to look good, 00:40:22.540 --> 00:40:24.850 align:middle line:84% will pick the sites where the treatment 00:40:24.850 --> 00:40:26.470 align:middle line:90% effect are the highest. 00:40:26.470 --> 00:40:31.580 align:middle line:84% So you will tend to be biased in that direction. 00:40:31.580 --> 00:40:35.650 align:middle line:84% So there is a very nice paper by Hunt Alcott in environment 00:40:35.650 --> 00:40:40.130 align:middle line:84% which shows that the first experiment by the firm 00:40:40.130 --> 00:40:44.200 align:middle line:84% Opower, which is one where you get this little leaflet 00:40:44.200 --> 00:40:47.440 align:middle line:90% about your energy consumption. 00:40:47.440 --> 00:40:51.020 align:middle line:84% And it tells you how you compare with your neighbors. 00:40:51.020 --> 00:40:52.510 align:middle line:90% So you got them? 00:40:52.510 --> 00:40:54.670 align:middle line:90% We always look terrible. 00:40:54.670 --> 00:40:57.790 align:middle line:84% When you guys lived in a house, we were terrible plus, 00:40:57.790 --> 00:41:00.670 align:middle line:90% even worse than usual. 00:41:00.670 --> 00:41:04.700 align:middle line:84% So I wanted to tell them, but there are six of them. 00:41:04.700 --> 00:41:06.550 align:middle line:84% So in any case, you receive these things. 00:41:06.550 --> 00:41:09.610 align:middle line:84% And it tells you, you read, please reduce your consumption. 00:41:09.610 --> 00:41:10.990 align:middle line:90% Or it doesn't say that. 00:41:10.990 --> 00:41:16.580 align:middle line:84% It just say-- so it finds pretty large effect in the first site. 00:41:16.580 --> 00:41:19.790 align:middle line:84% And then the experiments got replicated 00:41:19.790 --> 00:41:21.560 align:middle line:90% across hundreds of sites. 00:41:21.560 --> 00:41:26.840 align:middle line:84% And then what you see is that the results in the next 95 00:41:26.840 --> 00:41:29.180 align:middle line:84% sites are actually much lower than the results 00:41:29.180 --> 00:41:31.410 align:middle line:90% in the first few sites. 00:41:31.410 --> 00:41:33.930 align:middle line:84% And that could be exactly because of that, 00:41:33.930 --> 00:41:35.550 align:middle line:84% because the first sites were selected 00:41:35.550 --> 00:41:39.030 align:middle line:84% to be places where maybe people would be believed-- 00:41:39.030 --> 00:41:43.690 align:middle line:84% where people believe that it would have the biggest impact. 00:41:43.690 --> 00:41:46.030 align:middle line:90% So those are some of the issues. 00:41:46.030 --> 00:41:51.390 align:middle line:84% Another issue is that when you have many, many outcomes, 00:41:51.390 --> 00:41:54.210 align:middle line:84% not everything can be really presented in a people. 00:41:54.210 --> 00:41:55.740 align:middle line:90% People have limited attention. 00:41:55.740 --> 00:41:59.470 align:middle line:84% So there is a risk of cherry picking for very many outcomes. 00:41:59.470 --> 00:42:01.560 align:middle line:84% In the case of a program like the ultra poor, 00:42:01.560 --> 00:42:04.230 align:middle line:84% you have that issue in droves because it's not 00:42:04.230 --> 00:42:06.600 align:middle line:84% that you have one thing that you really want to follow. 00:42:06.600 --> 00:42:09.120 align:middle line:84% You want to see whether it improves their lives, which 00:42:09.120 --> 00:42:11.680 align:middle line:90% is measured in many, many ways. 00:42:11.680 --> 00:42:16.320 align:middle line:84% And so how do you deal with many outcomes? 00:42:16.320 --> 00:42:18.050 align:middle line:84% So I think these are some of the-- 00:42:18.050 --> 00:42:19.520 align:middle line:84% and then, of course, uncertainty. 00:42:19.520 --> 00:42:20.630 align:middle line:90% We mentioned compliance. 00:42:20.630 --> 00:42:24.890 align:middle line:84% But there is also just the sample size is not enormous. 00:42:24.890 --> 00:42:26.720 align:middle line:90% So they are not infinite. 00:42:26.720 --> 00:42:29.420 align:middle line:84% And therefore, you have potentially power issues, 00:42:29.420 --> 00:42:32.880 align:middle line:84% which I think you'll spend some time on that. 00:42:32.880 --> 00:42:34.790 align:middle line:84% And a criticism against experiment 00:42:34.790 --> 00:42:36.590 align:middle line:84% is that they are generally too small. 00:42:36.590 --> 00:42:38.090 align:middle line:90% So their power is too low. 00:42:38.090 --> 00:42:41.900 align:middle line:84% And they can only get effects if the-- 00:42:41.900 --> 00:42:45.030 align:middle line:84% get results that are significant if the effects are very large, 00:42:45.030 --> 00:42:49.570 align:middle line:84% which might lead people to miss some effect. 00:42:49.570 --> 00:42:51.630 align:middle line:84% So we discussed that, spillovers. 00:42:51.630 --> 00:42:56.520 align:middle line:84% Didn't come up, but it came up when we were discussing 00:42:56.520 --> 00:42:58.860 align:middle line:90% the SUTVA assumption. 00:42:58.860 --> 00:43:03.700 align:middle line:84% Cherry picking, and external validity, this will be with us. 00:43:03.700 --> 00:43:05.910 align:middle line:84% These are issues that are serious, that are real, 00:43:05.910 --> 00:43:10.360 align:middle line:84% and therefore that need to be addressed for each people. 00:43:10.360 --> 00:43:12.680 align:middle line:90% Now go back to graduation. 00:43:12.680 --> 00:43:15.670 align:middle line:90% And there are two recent papers. 00:43:15.670 --> 00:43:16.840 align:middle line:90% I wrote two studies. 00:43:16.840 --> 00:43:20.890 align:middle line:84% But in reality, the second paper is like six studies, 00:43:20.890 --> 00:43:25.970 align:middle line:84% and showing a remarkable effect of this program. 00:43:25.970 --> 00:43:29.455 align:middle line:84% The first one, by Oriana Bandiera, Robin Burgess, 00:43:29.455 --> 00:43:35.110 align:middle line:84% Imran Rasul, and other, looked directly at the BRAC pogrom 00:43:35.110 --> 00:43:37.780 align:middle line:84% in Bangladesh, the mothers of all program. 00:43:37.780 --> 00:43:40.740 align:middle line:90% And then there was a follow up. 00:43:40.740 --> 00:43:43.240 align:middle line:84% It was published before because it was published in Science. 00:43:43.240 --> 00:43:45.430 align:middle line:84% But in reality, this came after in terms 00:43:45.430 --> 00:43:50.830 align:middle line:84% of running the experiment by a bunch of people, 00:43:50.830 --> 00:43:53.350 align:middle line:84% including Abhijit and me, looking 00:43:53.350 --> 00:43:57.870 align:middle line:84% at a very similar intervention in six countries, which we have 00:43:57.870 --> 00:43:59.940 align:middle line:90% followed first for three years. 00:43:59.940 --> 00:44:04.080 align:middle line:84% And then in some of the sample, and I'm 00:44:04.080 --> 00:44:06.900 align:middle line:84% going to show you the data from for India, 00:44:06.900 --> 00:44:09.450 align:middle line:84% they were followed for another, until 10 years. 00:44:09.450 --> 00:44:11.220 align:middle line:84% They have been followed for 10 years. 00:44:11.220 --> 00:44:14.280 align:middle line:84% And these guys also, they lost the control after four years. 00:44:14.280 --> 00:44:16.770 align:middle line:84% But they continue to treat, to follow 00:44:16.770 --> 00:44:19.800 align:middle line:84% the x control and the x treatment until 10 years. 00:44:19.800 --> 00:44:21.540 align:middle line:84% In India, we never lost the control. 00:44:21.540 --> 00:44:24.250 align:middle line:90% We continue to follow them. 00:44:24.250 --> 00:44:26.550 align:middle line:90% So that gives you some-- 00:44:26.550 --> 00:44:29.670 align:middle line:84% So let's look a bit at the results first 00:44:29.670 --> 00:44:32.580 align:middle line:84% and then discuss about some of the features 00:44:32.580 --> 00:44:34.890 align:middle line:84% of this experiment, which address 00:44:34.890 --> 00:44:40.250 align:middle line:84% some extent some of the concerns that you guys might have. 00:44:40.250 --> 00:44:49.210 align:middle line:84% So these are the results for the Bangladesh study. 00:44:49.210 --> 00:44:51.940 align:middle line:84% The Bangladesh study was randomized 00:44:51.940 --> 00:44:56.310 align:middle line:90% at the geographic area level. 00:44:56.310 --> 00:44:58.470 align:middle line:84% So first, they decided where to go. 00:44:58.470 --> 00:45:01.810 align:middle line:84% And then they did the whole intervention. 00:45:01.810 --> 00:45:05.830 align:middle line:84% And they treated everyone who emerged from the process. 00:45:05.830 --> 00:45:07.560 align:middle line:84% So it's a good situation in terms 00:45:07.560 --> 00:45:09.540 align:middle line:90% of the spillover situation. 00:45:09.540 --> 00:45:12.420 align:middle line:84% Most of the studies in the Science paper 00:45:12.420 --> 00:45:14.340 align:middle line:84% were randomized at the individual level 00:45:14.340 --> 00:45:16.950 align:middle line:84% within village, although a couple 00:45:16.950 --> 00:45:18.450 align:middle line:90% were randomized at both levels. 00:45:18.450 --> 00:45:20.830 align:middle line:84% There was randomization first at the site level, 00:45:20.830 --> 00:45:23.590 align:middle line:84% and then within site, at the individual level. 00:45:23.590 --> 00:45:27.250 align:middle line:84% So what's the nice thing of having 00:45:27.250 --> 00:45:29.710 align:middle line:84% to randomize first at the site level 00:45:29.710 --> 00:45:33.370 align:middle line:84% and then individually within site? 00:45:33.370 --> 00:45:37.039 align:middle line:90% What does it allow to check? 00:45:37.039 --> 00:45:37.685 align:middle line:90% [INAUDIBLE] 00:45:37.685 --> 00:45:39.310 align:middle line:84% AUDIENCE: Whether there are spillovers. 00:45:39.310 --> 00:45:40.102 align:middle line:90% PROFESSOR: Exactly. 00:45:40.102 --> 00:45:42.160 align:middle line:84% So you can directly check for the spillovers 00:45:42.160 --> 00:45:44.080 align:middle line:84% because you can look at the untreated 00:45:44.080 --> 00:45:45.850 align:middle line:90% people in the treated villages. 00:45:45.850 --> 00:45:49.310 align:middle line:84% And you can see whether their outcomes are any different. 00:45:49.310 --> 00:45:51.455 align:middle line:84% So long and short of that, and that's 00:45:51.455 --> 00:45:53.830 align:middle line:84% why I'm going to treat all of these studies the same now, 00:45:53.830 --> 00:45:57.560 align:middle line:84% is that there is no evidence of spillover in this case. 00:45:57.560 --> 00:45:59.500 align:middle line:84% Doesn't mean that they don't exist in general. 00:45:59.500 --> 00:46:01.830 align:middle line:84% They're just not present in this case. 00:46:01.830 --> 00:46:05.190 align:middle line:84% So here is the result from the Bangladesh studies. 00:46:05.190 --> 00:46:10.250 align:middle line:84% So these are some of the few outcomes that are used-- 00:46:10.250 --> 00:46:16.455 align:middle line:84% below poverty line, consumption expenditure per capita 00:46:16.455 --> 00:46:19.430 align:middle line:84% or per adult equivalent, value of household assets, 00:46:19.430 --> 00:46:21.290 align:middle line:84% savings, whether or not they get loans, 00:46:21.290 --> 00:46:24.680 align:middle line:90% whether or not they give loans. 00:46:24.680 --> 00:46:30.050 align:middle line:84% Each row is a treatment effect, with the standard error 00:46:30.050 --> 00:46:34.553 align:middle line:84% of the treatment effect and stars to indicate significant. 00:46:34.553 --> 00:46:35.970 align:middle line:84% So that's for the first two lines. 00:46:35.970 --> 00:46:38.053 align:middle line:84% So each row in the first two line and first column 00:46:38.053 --> 00:46:40.592 align:middle line:90% is from a different analysis. 00:46:40.592 --> 00:46:41.300 align:middle line:90% That makes sense? 00:46:41.300 --> 00:46:44.310 align:middle line:84% We are every time comparing treatment versus control. 00:46:44.310 --> 00:46:49.670 align:middle line:84% So for example, if I read this, can someone-- 00:46:49.670 --> 00:46:52.476 align:middle line:90% Vishen, can you read that-- 00:46:52.476 --> 00:46:57.080 align:middle line:84% construct a sentence with that number, with that-- 00:46:57.080 --> 00:47:01.640 align:middle line:84% AUDIENCE: So four years after program impact, 00:47:01.640 --> 00:47:05.276 align:middle line:84% the treated households had on average-- 00:47:05.276 --> 00:47:07.540 align:middle line:90% what is this-- 00:47:07.540 --> 00:47:13.080 align:middle line:84% 8.4% lower people below the poverty line, percentage 00:47:13.080 --> 00:47:13.950 align:middle line:90% points. 00:47:13.950 --> 00:47:15.158 align:middle line:90% PROFESSOR: Percentage points. 00:47:15.158 --> 00:47:17.430 align:middle line:84% So they were 8.4 percentage points 00:47:17.430 --> 00:47:20.220 align:middle line:84% less likely to be below the poverty line. 00:47:20.220 --> 00:47:23.820 align:middle line:84% And then generally, this type of table, we 00:47:23.820 --> 00:47:26.460 align:middle line:84% are going to provide the control group in order 00:47:26.460 --> 00:47:28.780 align:middle line:90% to know what we are looking at. 00:47:28.780 --> 00:47:30.750 align:middle line:84% So here, sometimes set at baseline, 00:47:30.750 --> 00:47:33.540 align:middle line:84% here they show us the control group at the four-year 00:47:33.540 --> 00:47:34.410 align:middle line:90% follow up. 00:47:34.410 --> 00:47:36.220 align:middle line:90% That was 62%. 00:47:36.220 --> 00:47:40.690 align:middle line:84% So it's 8.4 percentage point on the basis of 62 percentage 00:47:40.690 --> 00:47:41.190 align:middle line:90% point. 00:47:41.190 --> 00:47:44.880 align:middle line:84% So that's a 13.5 percentage reduction. 00:47:44.880 --> 00:47:48.510 align:middle line:84% Then we are testing whether the impact are-- 00:47:48.510 --> 00:47:51.220 align:middle line:84% remember we are interested in long run. 00:47:51.220 --> 00:47:54.120 align:middle line:84% So we are testing whether our impact-- really, they 00:47:54.120 --> 00:47:59.550 align:middle line:84% could have done one-sided test to see whether the impact were 00:47:59.550 --> 00:48:01.440 align:middle line:90% decreasing. 00:48:01.440 --> 00:48:04.380 align:middle line:84% But they are testing whether they are different over time. 00:48:04.380 --> 00:48:06.930 align:middle line:84% In this particular instance, they seem to, if anything, 00:48:06.930 --> 00:48:10.510 align:middle line:90% increase over time. 00:48:10.510 --> 00:48:11.348 align:middle line:90% Then the R-square. 00:48:11.348 --> 00:48:13.140 align:middle line:84% I don't know what we do with R-square ever, 00:48:13.140 --> 00:48:14.190 align:middle line:90% but that's there. 00:48:14.190 --> 00:48:16.320 align:middle line:84% And then the number of women, the number 00:48:16.320 --> 00:48:18.030 align:middle line:84% of observations in their households, 00:48:18.030 --> 00:48:20.027 align:middle line:90% and the number of clusters. 00:48:20.027 --> 00:48:20.860 align:middle line:90% AUDIENCE: Professor. 00:48:20.860 --> 00:48:22.490 align:middle line:90% PROFESSOR: 00:48:22.490 --> 00:48:25.670 align:middle line:84% AUDIENCE: In reference to these, is there ever a decision 00:48:25.670 --> 00:48:27.620 align:middle line:84% when randomizing or choosing a sample 00:48:27.620 --> 00:48:30.800 align:middle line:84% to make sure that we don't treat more than a certain percentage 00:48:30.800 --> 00:48:34.010 align:middle line:84% of the village to make sure that the total capital stop does not 00:48:34.010 --> 00:48:35.260 align:middle line:90% change or something like that? 00:48:35.260 --> 00:48:38.350 align:middle line:84% I'm thinking about how you would argue that [? GB ?] effects are 00:48:38.350 --> 00:48:39.482 align:middle line:90% not that bad. 00:48:39.482 --> 00:48:40.690 align:middle line:90% PROFESSOR: So there could be. 00:48:40.690 --> 00:48:42.970 align:middle line:84% Or you, in fact, could go the opposite direction. 00:48:42.970 --> 00:48:46.450 align:middle line:90% There has been a lot of-- 00:48:46.450 --> 00:48:48.910 align:middle line:84% several studies in the past few years 00:48:48.910 --> 00:48:51.670 align:middle line:84% trying to vary the fraction of people 00:48:51.670 --> 00:48:58.560 align:middle line:84% who are treated in order to see whether you start seeing 00:48:58.560 --> 00:49:04.200 align:middle line:84% equilibrium-type of effect on prices and things 00:49:04.200 --> 00:49:07.090 align:middle line:84% like that by randomizing at that level. 00:49:07.090 --> 00:49:09.780 align:middle line:84% So sometimes you want to rule them out 00:49:09.780 --> 00:49:12.060 align:middle line:84% because you're interested in the individual effect. 00:49:12.060 --> 00:49:14.070 align:middle line:84% Sometimes you're very much interested 00:49:14.070 --> 00:49:15.280 align:middle line:90% in the market effect. 00:49:15.280 --> 00:49:17.340 align:middle line:84% So you want to make sure that you saturate 00:49:17.340 --> 00:49:19.240 align:middle line:90% enough on the contrary. 00:49:19.240 --> 00:49:22.530 align:middle line:84% So for example, Lauren, who is visiting here, 00:49:22.530 --> 00:49:27.010 align:middle line:84% she's doing a lot of work on making markets, in a way, 00:49:27.010 --> 00:49:32.760 align:middle line:84% creating a relationship between traders and potential sellers. 00:49:32.760 --> 00:49:35.470 align:middle line:84% And there she is looking almost for the opposite. 00:49:35.470 --> 00:49:37.890 align:middle line:84% She wants to make sure she has enough saturation. 00:49:37.890 --> 00:49:39.130 align:middle line:90% So yes, absolutely. 00:49:39.130 --> 00:49:42.190 align:middle line:84% It's something people have in mind to see where they are. 00:49:42.190 --> 00:49:46.190 align:middle line:84% So in fact, in the case of the ultra poor studies, 00:49:46.190 --> 00:49:48.680 align:middle line:84% the reason why I think we see no externalities 00:49:48.680 --> 00:49:53.470 align:middle line:84% is that in every village, very few people are treated, 00:49:53.470 --> 00:49:58.420 align:middle line:84% and not tremendously socially integrated people, either. 00:49:58.420 --> 00:50:00.850 align:middle line:84% So that's the research they find encouraging. 00:50:00.850 --> 00:50:03.620 align:middle line:84% Impact on pretty much everything, 00:50:03.620 --> 00:50:06.770 align:middle line:90% especially after four years. 00:50:06.770 --> 00:50:10.500 align:middle line:84% Then the Science study came about and, in a sense, 00:50:10.500 --> 00:50:13.580 align:middle line:84% was a response to the external validity 00:50:13.580 --> 00:50:17.210 align:middle line:84% point or a way of addressing the external validity point, 00:50:17.210 --> 00:50:19.820 align:middle line:84% given this super promising program. 00:50:19.820 --> 00:50:24.890 align:middle line:84% There was a lot of hope that this could be a strategy 00:50:24.890 --> 00:50:26.360 align:middle line:90% to address poverty. 00:50:26.360 --> 00:50:30.920 align:middle line:84% And but given how expensive and, in a sense, 00:50:30.920 --> 00:50:35.030 align:middle line:84% specific the program is, with all of these various component 00:50:35.030 --> 00:50:38.420 align:middle line:84% to it that you can't really easily untangle 00:50:38.420 --> 00:50:41.930 align:middle line:84% mechanism or understand why it works, 00:50:41.930 --> 00:50:47.890 align:middle line:84% there was really a desire to make sure 00:50:47.890 --> 00:50:53.800 align:middle line:84% that the program is replicated in as different context as 00:50:53.800 --> 00:50:57.760 align:middle line:84% possible in order to assess the external validity 00:50:57.760 --> 00:50:59.990 align:middle line:90% of the context dependent. 00:50:59.990 --> 00:51:06.550 align:middle line:84% So this is a project that was led by Dean Karlan, who 00:51:06.550 --> 00:51:10.030 align:middle line:84% is at Northwestern now, was at Yale now, and started 00:51:10.030 --> 00:51:14.170 align:middle line:84% IPA, which is an organization that 00:51:14.170 --> 00:51:16.090 align:middle line:84% conducts a lot of randomized controlled trials 00:51:16.090 --> 00:51:17.390 align:middle line:90% around the world. 00:51:17.390 --> 00:51:21.910 align:middle line:84% And he and the Ford Foundation and CGAP, the World Bank, 00:51:21.910 --> 00:51:25.240 align:middle line:84% microfinance, ultra poor arm, kind of 00:51:25.240 --> 00:51:26.560 align:middle line:90% got that project together. 00:51:26.560 --> 00:51:27.970 align:middle line:84% And I don't want to underestimate 00:51:27.970 --> 00:51:34.990 align:middle line:84% what this means to get six different organizations in six 00:51:34.990 --> 00:51:39.160 align:middle line:84% different countries with six different research teams 00:51:39.160 --> 00:51:44.140 align:middle line:84% to each conduct their project together. 00:51:44.140 --> 00:51:47.110 align:middle line:84% And what is interesting here is that there was really-- 00:51:47.110 --> 00:51:51.400 align:middle line:84% the goal was really to estimate the same project. 00:51:51.400 --> 00:51:53.350 align:middle line:84% But what does it mean to be the same project? 00:51:53.350 --> 00:51:56.080 align:middle line:84% You can't force people to do cows everywhere 00:51:56.080 --> 00:51:59.990 align:middle line:84% because the whole point is that people can choose their assets. 00:51:59.990 --> 00:52:02.350 align:middle line:84% So you need that to have a definition of same. 00:52:02.350 --> 00:52:03.640 align:middle line:90% What is same? 00:52:03.640 --> 00:52:07.000 align:middle line:84% And here, the way they enforced that sameness in a way 00:52:07.000 --> 00:52:09.910 align:middle line:84% is by having people from all over the program 00:52:09.910 --> 00:52:12.040 align:middle line:90% meet in Paris every year. 00:52:12.040 --> 00:52:14.117 align:middle line:84% I attended a few of these meetings, where 00:52:14.117 --> 00:52:15.700 align:middle line:84% they were discussing exactly what they 00:52:15.700 --> 00:52:18.760 align:middle line:84% were doing, the process they were following, et cetera. 00:52:18.760 --> 00:52:21.820 align:middle line:84% So same is like a sameness of spirit, 00:52:21.820 --> 00:52:26.860 align:middle line:84% and then embodied, in a sense, in different ways, 00:52:26.860 --> 00:52:30.460 align:middle line:84% in different places, with keeping, 00:52:30.460 --> 00:52:33.820 align:middle line:84% of course, the same element in place-- identification, 00:52:33.820 --> 00:52:37.660 align:middle line:84% the assets, the multifaceted support to go with it. 00:52:37.660 --> 00:52:39.855 align:middle line:84% But then people, for example, the staff 00:52:39.855 --> 00:52:41.980 align:middle line:84% is paid very different amounts in different places. 00:52:41.980 --> 00:52:44.450 align:middle line:84% The value of their assets is different in different places 00:52:44.450 --> 00:52:45.560 align:middle line:90% and so on and so forth. 00:52:45.560 --> 00:52:49.950 align:middle line:84% But to arrive at the same program given the context. 00:52:49.950 --> 00:52:53.780 align:middle line:84% So that's the first thing that is worth mentioning. 00:52:53.780 --> 00:52:56.060 align:middle line:84% And then the other thing that happened 00:52:56.060 --> 00:52:59.600 align:middle line:84% is that all of these different teams 00:52:59.600 --> 00:53:02.810 align:middle line:84% were conducting the project separately. 00:53:02.810 --> 00:53:07.310 align:middle line:84% And instead of each writing their own paper, 00:53:07.310 --> 00:53:09.650 align:middle line:84% a decision was made to write one paper, which 00:53:09.650 --> 00:53:13.280 align:middle line:84% would have all of the data together 00:53:13.280 --> 00:53:16.520 align:middle line:84% precisely because that was supposed 00:53:16.520 --> 00:53:19.475 align:middle line:84% to be looking at what are the effects of variety, 00:53:19.475 --> 00:53:21.210 align:middle line:90% heterogeneity, and so on. 00:53:21.210 --> 00:53:25.910 align:middle line:84% And therefore, you needed one kind of template. 00:53:25.910 --> 00:53:27.920 align:middle line:84% And the nice thing about that as well 00:53:27.920 --> 00:53:30.050 align:middle line:84% is that addresses another aspect, which is 00:53:30.050 --> 00:53:31.645 align:middle line:90% the cherry picking of results. 00:53:31.645 --> 00:53:33.770 align:middle line:84% So of course, there you also have a lot of results. 00:53:33.770 --> 00:53:35.885 align:middle line:90% But everybody has to-- 00:53:35.885 --> 00:53:39.080 align:middle line:84% everybody was running their own analysis separately, 00:53:39.080 --> 00:53:41.460 align:middle line:84% and they had to follow the same template. 00:53:41.460 --> 00:53:44.480 align:middle line:84% So the template is given in advance. 00:53:44.480 --> 00:53:47.240 align:middle line:84% So even though there are many outcomes, and here 00:53:47.240 --> 00:53:50.690 align:middle line:84% you have outcomes that are already there. 00:53:50.690 --> 00:53:52.790 align:middle line:84% For every one, they are pre-established. 00:53:52.790 --> 00:53:54.620 align:middle line:84% This is what the outcomes are going to be. 00:53:54.620 --> 00:54:00.090 align:middle line:84% And it leaves very little place for cherry picking. 00:54:00.090 --> 00:54:01.900 align:middle line:84% Despite that, you have a lot of outcomes. 00:54:01.900 --> 00:54:03.630 align:middle line:84% So to have something that is reliable, 00:54:03.630 --> 00:54:06.030 align:middle line:84% you end up doing something that is frequently 00:54:06.030 --> 00:54:08.340 align:middle line:84% done, in particular, in this type of project, 00:54:08.340 --> 00:54:13.110 align:middle line:84% where you have a various variable on food security, 00:54:13.110 --> 00:54:14.250 align:middle line:90% for example. 00:54:14.250 --> 00:54:16.050 align:middle line:90% Did everybody eat yesterday? 00:54:16.050 --> 00:54:18.000 align:middle line:90% Did kids miss any meal? 00:54:18.000 --> 00:54:20.010 align:middle line:90% Did adults miss any meal? 00:54:20.010 --> 00:54:21.780 align:middle line:90% Et cetera. 00:54:21.780 --> 00:54:26.160 align:middle line:84% So you form an index that indexed this component 00:54:26.160 --> 00:54:29.610 align:middle line:84% by standardizing each of the variable 00:54:29.610 --> 00:54:31.410 align:middle line:84% and then taking the average of them. 00:54:31.410 --> 00:54:33.000 align:middle line:84% And I'm not saying this is necessarily 00:54:33.000 --> 00:54:34.650 align:middle line:90% the best way to form an index. 00:54:34.650 --> 00:54:37.470 align:middle line:84% But this is how the literature have evolved. 00:54:37.470 --> 00:54:41.180 align:middle line:84% And this is the standard way of doing it. 00:54:41.180 --> 00:54:45.890 align:middle line:84% So here are the index that were used. 00:54:45.890 --> 00:54:48.550 align:middle line:84% This is a first in line, 15 month. 00:54:48.550 --> 00:54:50.140 align:middle line:84% Second in line is at three years. 00:54:50.140 --> 00:54:53.020 align:middle line:90% 00:54:53.020 --> 00:54:57.880 align:middle line:84% So now it's expressed in standard deviation of the mean 00:54:57.880 --> 00:55:03.510 align:middle line:84% directly, even for things that are in money to start with. 00:55:03.510 --> 00:55:08.880 align:middle line:84% And here again we find effects across the board on pretty 00:55:08.880 --> 00:55:12.180 align:middle line:84% much all of the outcome, with effect for consumption 00:55:12.180 --> 00:55:14.090 align:middle line:84% is about 0.12 standard deviation. 00:55:14.090 --> 00:55:19.020 align:middle line:84% So it doesn't make them Bill Gates, either. 00:55:19.020 --> 00:55:21.870 align:middle line:84% But it's an increase in consumption 00:55:21.870 --> 00:55:24.160 align:middle line:90% that's stable over time. 00:55:24.160 --> 00:55:25.570 align:middle line:90% This is a q-value. 00:55:25.570 --> 00:55:28.530 align:middle line:84% So what's the q-value to comparison to the p-value? 00:55:28.530 --> 00:55:32.230 align:middle line:90% 00:55:32.230 --> 00:55:32.730 align:middle line:90% Yep. 00:55:32.730 --> 00:55:35.222 align:middle line:84% AUDIENCE: It accounts for false discovery rate. 00:55:35.222 --> 00:55:35.930 align:middle line:90% PROFESSOR: Right. 00:55:35.930 --> 00:55:38.157 align:middle line:90% So it's one way to account. 00:55:38.157 --> 00:55:38.990 align:middle line:90% There are many ways. 00:55:38.990 --> 00:55:42.200 align:middle line:84% But it's one of the methods to account for the fact 00:55:42.200 --> 00:55:45.480 align:middle line:84% that you have more than one thing that is being tested. 00:55:45.480 --> 00:55:48.390 align:middle line:84% And each of the tests are connected to each other. 00:55:48.390 --> 00:55:53.630 align:middle line:84% So that's a false discovery rate adjusted p-value. 00:55:53.630 --> 00:55:59.450 align:middle line:84% And here it doesn't mess up too much with you 00:55:59.450 --> 00:56:04.760 align:middle line:84% because all of the outcomes tend to have similar impact. 00:56:04.760 --> 00:56:10.300 align:middle line:84% So that conforts the results of the BRAC study, 00:56:10.300 --> 00:56:14.830 align:middle line:84% showing impact at the 1.5 and at three years that are not-- 00:56:14.830 --> 00:56:18.940 align:middle line:84% the impact don't seem to be decreasing from one place 00:56:18.940 --> 00:56:21.560 align:middle line:90% to the other. 00:56:21.560 --> 00:56:24.220 align:middle line:90% So we already discussed that. 00:56:24.220 --> 00:56:26.170 align:middle line:84% It addresses some of the-- so that study 00:56:26.170 --> 00:56:28.810 align:middle line:84% addresses some of the shortcomings 00:56:28.810 --> 00:56:31.330 align:middle line:84% from the point of view of pure impact evaluation, 00:56:31.330 --> 00:56:34.660 align:middle line:84% of being confident that that program might 00:56:34.660 --> 00:56:39.320 align:middle line:84% work not only in those sites, but maybe in other sites. 00:56:39.320 --> 00:56:41.750 align:middle line:84% If we look at, then, since we have many countries, 00:56:41.750 --> 00:56:44.210 align:middle line:90% I showed you the effect for-- 00:56:44.210 --> 00:56:47.780 align:middle line:84% as if it was one giant regression here. 00:56:47.780 --> 00:56:50.580 align:middle line:84% But in reality, we have a lot of countries. 00:56:50.580 --> 00:56:54.080 align:middle line:84% So one thing we can do to start to look at external validity is 00:56:54.080 --> 00:56:56.300 align:middle line:84% ask ourselves, how do the results 00:56:56.300 --> 00:56:58.418 align:middle line:90% vary from country to country? 00:56:58.418 --> 00:57:00.710 align:middle line:84% And you can see that they are actually pretty variable. 00:57:00.710 --> 00:57:02.630 align:middle line:84% They appear to be pretty variable from country 00:57:02.630 --> 00:57:03.890 align:middle line:90% to country. 00:57:03.890 --> 00:57:07.310 align:middle line:84% In particular, in Honduras was a bit of a disaster. 00:57:07.310 --> 00:57:09.265 align:middle line:84% You can give an ex-post explanation for that. 00:57:09.265 --> 00:57:11.640 align:middle line:84% In fact, I can give you the ex-post explanation for that. 00:57:11.640 --> 00:57:12.860 align:middle line:90% The main asset was chicken. 00:57:12.860 --> 00:57:15.440 align:middle line:84% And all the chicken died because of bird flu. 00:57:15.440 --> 00:57:19.250 align:middle line:90% But that's ex-post. 00:57:19.250 --> 00:57:20.570 align:middle line:90% But that's true. 00:57:20.570 --> 00:57:22.040 align:middle line:90% So that's docile. 00:57:22.040 --> 00:57:23.990 align:middle line:84% But you see a variability in the-- 00:57:23.990 --> 00:57:26.820 align:middle line:84% But funny story about the assets. 00:57:26.820 --> 00:57:29.360 align:middle line:90% We were doing it in India. 00:57:29.360 --> 00:57:30.830 align:middle line:84% I was running the study in India. 00:57:30.830 --> 00:57:33.360 align:middle line:90% And my Indian co-author and I-- 00:57:33.360 --> 00:57:35.840 align:middle line:84% Ragha Chattopadhyay, his name is-- we went to-- 00:57:35.840 --> 00:57:37.460 align:middle line:84% I mean, I guess Abhijit is Indian too. 00:57:37.460 --> 00:57:40.130 align:middle line:84% But the guy who actually works in Kolkata. 00:57:40.130 --> 00:57:42.650 align:middle line:90% We went to the field together. 00:57:42.650 --> 00:57:45.650 align:middle line:84% And they were about to hand chicken. 00:57:45.650 --> 00:57:48.800 align:middle line:84% And he was like-- he went screaming, saying, you cannot-- 00:57:48.800 --> 00:57:51.290 align:middle line:84% this was before bird flu-- but you cannot hand chicken. 00:57:51.290 --> 00:57:53.310 align:middle line:90% They die. 00:57:53.310 --> 00:57:57.720 align:middle line:90% And he really convinced them. 00:57:57.720 --> 00:57:59.700 align:middle line:84% Almost went to the CEO of the organization 00:57:59.700 --> 00:58:02.320 align:middle line:84% to say that chicken should be ruled out. 00:58:02.320 --> 00:58:04.050 align:middle line:90% And so they didn't do chicken. 00:58:04.050 --> 00:58:05.855 align:middle line:84% They didn't really put it in the menu 00:58:05.855 --> 00:58:07.230 align:middle line:84% because they had been so popular, 00:58:07.230 --> 00:58:08.688 align:middle line:84% but he thought they were dangerous. 00:58:08.688 --> 00:58:10.830 align:middle line:84% And then this Honduras result came out 00:58:10.830 --> 00:58:12.650 align:middle line:84% with all of the chicken having died. 00:58:12.650 --> 00:58:13.690 align:middle line:90% Yep. 00:58:13.690 --> 00:58:18.320 align:middle line:84% AUDIENCE: I wonder how cash transfers and asset transfers 00:58:18.320 --> 00:58:19.745 align:middle line:84% differ conceptually in the center. 00:58:19.745 --> 00:58:21.483 align:middle line:84% Why would you expect [INAUDIBLE]?? 00:58:21.483 --> 00:58:22.150 align:middle line:90% PROFESSOR: Yeah. 00:58:22.150 --> 00:58:26.030 align:middle line:84% So that's a great point, which I will generalize a little bit. 00:58:26.030 --> 00:58:28.900 align:middle line:84% So the question is whether a cash transfer-- 00:58:28.900 --> 00:58:30.670 align:middle line:90% why do assets and not cash? 00:58:30.670 --> 00:58:31.510 align:middle line:90% How do they compare? 00:58:31.510 --> 00:58:34.060 align:middle line:90% Do we know that? 00:58:34.060 --> 00:58:36.918 align:middle line:84% This is, of course, this is a huge bundle thing. 00:58:36.918 --> 00:58:37.960 align:middle line:90% So I'm now I'm going to-- 00:58:37.960 --> 00:58:39.910 align:middle line:84% I'm showing you this reduced form effect 00:58:39.910 --> 00:58:43.300 align:middle line:84% and of doing everything together, and showing 00:58:43.300 --> 00:58:44.680 align:middle line:90% that there is an impact. 00:58:44.680 --> 00:58:47.860 align:middle line:84% And then we'll talk a bit about country heterogeneity. 00:58:47.860 --> 00:58:50.410 align:middle line:90% But then there is the-- 00:58:50.410 --> 00:58:53.320 align:middle line:84% once we satisfy ourselves that there is an impact 00:58:53.320 --> 00:58:56.110 align:middle line:84% and maybe it's doable is, where is it coming from? 00:58:56.110 --> 00:59:00.100 align:middle line:84% Could we take out some of the component of the program that 00:59:00.100 --> 00:59:01.030 align:middle line:90% are less effective? 00:59:01.030 --> 00:59:03.340 align:middle line:84% Could we go to cash and avoid this chicken issue 00:59:03.340 --> 00:59:06.590 align:middle line:84% or maybe increase this chicken issue? 00:59:06.590 --> 00:59:10.220 align:middle line:84% So we know some about that, which I'll come to in a bit. 00:59:10.220 --> 00:59:12.820 align:middle line:84% And then some is still not fully unpacked. 00:59:12.820 --> 00:59:14.650 align:middle line:90% But you're completely right. 00:59:14.650 --> 00:59:18.370 align:middle line:84% And it's not a criticism of the validity of experiment. 00:59:18.370 --> 00:59:22.870 align:middle line:84% But it's potentially a criticism of this kind of experiment, 00:59:22.870 --> 00:59:24.820 align:middle line:84% which are telling you, this is my treatment. 00:59:24.820 --> 00:59:25.900 align:middle line:90% That's its effect. 00:59:25.900 --> 00:59:29.880 align:middle line:84% Now, this is all I can tell you right now. 00:59:29.880 --> 00:59:35.070 align:middle line:84% And then it's kind of whet your appetite to say, well, 00:59:35.070 --> 00:59:36.240 align:middle line:90% what's the mechanism? 00:59:36.240 --> 00:59:38.280 align:middle line:84% Why does it work the way it works? 00:59:38.280 --> 00:59:40.720 align:middle line:90% And do we really need to do-- 00:59:40.720 --> 00:59:42.600 align:middle line:84% so from a policy point of view, do I really 00:59:42.600 --> 00:59:44.070 align:middle line:90% need to do all of this? 00:59:44.070 --> 00:59:47.040 align:middle line:84% From economics human behavior perspective 00:59:47.040 --> 00:59:48.780 align:middle line:84% is, what have I learned once you told me 00:59:48.780 --> 00:59:51.390 align:middle line:84% that all of these things matter together? 00:59:51.390 --> 00:59:51.990 align:middle line:90% Yeah. 00:59:51.990 --> 00:59:56.770 align:middle line:84% So that's the ultimate bundle experiment in that way. 00:59:56.770 --> 00:59:57.960 align:middle line:90% So that's the assets. 00:59:57.960 --> 01:00:01.050 align:middle line:84% That the income and revenue, a little bit more-- 01:00:01.050 --> 01:00:06.580 align:middle line:84% less diverse per capita consumption. 01:00:06.580 --> 01:00:08.700 align:middle line:84% So that's the question of interest. 01:00:08.700 --> 01:00:11.460 align:middle line:84% In the paper, we just showed some of these different-- some 01:00:11.460 --> 01:00:12.960 align:middle line:90% of these effects by country. 01:00:12.960 --> 01:00:14.520 align:middle line:84% And we present them in presentation. 01:00:14.520 --> 01:00:18.280 align:middle line:90% But we leave it at that. 01:00:18.280 --> 01:00:23.460 align:middle line:84% Now, the question we wanted to answer, 01:00:23.460 --> 01:00:26.680 align:middle line:84% if we want to answer Recca's question, which is, 01:00:26.680 --> 01:00:28.650 align:middle line:84% do I learn from one experiment what's going 01:00:28.650 --> 01:00:29.940 align:middle line:90% to happen in the next site? 01:00:29.940 --> 01:00:31.500 align:middle line:90% Or, in fact, your point as well. 01:00:31.500 --> 01:00:34.600 align:middle line:84% Well, your point adds this homogeneity in the experiment. 01:00:34.600 --> 01:00:40.330 align:middle line:84% But what's the predictive power of one site to another one? 01:00:40.330 --> 01:00:45.560 align:middle line:84% It's not really sufficient to look at those type of graphs 01:00:45.560 --> 01:00:49.720 align:middle line:84% because when you run an experiment in different places, 01:00:49.720 --> 01:00:51.320 align:middle line:90% so here we have six sites. 01:00:51.320 --> 01:00:54.330 align:middle line:84% Each site is different for two reasons. 01:00:54.330 --> 01:00:58.250 align:middle line:84% Number one is the treatment effect 01:00:58.250 --> 01:01:00.770 align:middle line:84% might be different fundamentally. 01:01:00.770 --> 01:01:03.410 align:middle line:84% It might be higher in India than in Honduras. 01:01:03.410 --> 01:01:06.400 align:middle line:84% Number two, there is noise in the world. 01:01:06.400 --> 01:01:11.020 align:middle line:84% And the sample that you happen to have found in India 01:01:11.020 --> 01:01:15.190 align:middle line:84% might be different than the sample in Honduras, 01:01:15.190 --> 01:01:16.450 align:middle line:90% both in treatment and control. 01:01:16.450 --> 01:01:23.060 align:middle line:84% And every estimate comes with its own standard error. 01:01:23.060 --> 01:01:26.600 align:middle line:84% So once I saw different treatment effect, 01:01:26.600 --> 01:01:29.500 align:middle line:84% this country-by-country treatment effect 01:01:29.500 --> 01:01:33.825 align:middle line:84% is conflating these two sources of uncertainty. 01:01:33.825 --> 01:01:35.200 align:middle line:84% So if someone shows you and said, 01:01:35.200 --> 01:01:37.100 align:middle line:84% those treatment effects are quite different. 01:01:37.100 --> 01:01:38.890 align:middle line:84% So maybe the next country would also 01:01:38.890 --> 01:01:40.150 align:middle line:90% have a very different effect. 01:01:40.150 --> 01:01:44.600 align:middle line:90% 01:01:44.600 --> 01:01:46.900 align:middle line:84% It's really attributing the entire difference 01:01:46.900 --> 01:01:48.880 align:middle line:84% to heterogeneity in treatment effect 01:01:48.880 --> 01:01:52.870 align:middle line:84% when, in fact, there is also the underlying noise in the world. 01:01:52.870 --> 01:01:55.620 align:middle line:90% So this is a bit-- 01:01:55.620 --> 01:01:58.850 align:middle line:84% that's not the answer you need or want. 01:01:58.850 --> 01:02:00.040 align:middle line:90% So the way to-- 01:02:00.040 --> 01:02:02.620 align:middle line:84% so you can't really solve this problem 01:02:02.620 --> 01:02:04.630 align:middle line:84% without being willing to put a little bit 01:02:04.630 --> 01:02:06.770 align:middle line:90% more structure to the problem. 01:02:06.770 --> 01:02:09.160 align:middle line:84% And one way to do this is by a method 01:02:09.160 --> 01:02:13.090 align:middle line:84% called Bayesian hierarchical analysis, which 01:02:13.090 --> 01:02:14.860 align:middle line:84% Rachael Meager, who was a student here 01:02:14.860 --> 01:02:18.820 align:middle line:84% and is now at the LSE, started to do first 01:02:18.820 --> 01:02:19.858 align:middle line:90% for microcredit studies. 01:02:19.858 --> 01:02:22.150 align:middle line:84% And then I'm going to show you her result that she just 01:02:22.150 --> 01:02:28.460 align:middle line:84% sent me off of the press to look at this very program. 01:02:28.460 --> 01:02:31.250 align:middle line:84% And the idea is to hear the structure-- 01:02:31.250 --> 01:02:32.810 align:middle line:84% the basic structure on the problem. 01:02:32.810 --> 01:02:34.700 align:middle line:84% And I'm not going to go to the detail of how 01:02:34.700 --> 01:02:36.410 align:middle line:90% you conduct the analysis. 01:02:36.410 --> 01:02:40.400 align:middle line:84% It's just computationally a little bit involved. 01:02:40.400 --> 01:02:45.080 align:middle line:84% But what's important for now is that we understand the concept. 01:02:45.080 --> 01:02:47.140 align:middle line:84% The concept is going to say, let's assume 01:02:47.140 --> 01:02:49.030 align:middle line:84% that the treatment-- that there is 01:02:49.030 --> 01:02:52.240 align:middle line:84% a distribution of the possible treatment effect, 01:02:52.240 --> 01:02:54.470 align:middle line:90% and that distribution is normal. 01:02:54.470 --> 01:02:56.770 align:middle line:84% So the treatment effects are themselves drawn 01:02:56.770 --> 01:02:59.530 align:middle line:90% from a normal distribution. 01:02:59.530 --> 01:03:04.830 align:middle line:84% And I'm trying to find out, for each site, the treatment 01:03:04.830 --> 01:03:08.550 align:middle line:84% effect for that site as well as the variance of the treatment 01:03:08.550 --> 01:03:12.910 align:middle line:84% effect across site in the world in general. 01:03:12.910 --> 01:03:16.280 align:middle line:90% Well, that's the idea. 01:03:16.280 --> 01:03:18.610 align:middle line:84% So if the variance in treatment effect 01:03:18.610 --> 01:03:23.470 align:middle line:84% is very, very small, of the distribution of the treatment 01:03:23.470 --> 01:03:26.830 align:middle line:84% effect, this suggests that the treatment effect itself 01:03:26.830 --> 01:03:28.540 align:middle line:90% is quite constant. 01:03:28.540 --> 01:03:31.300 align:middle line:84% And therefore it's, for example, it's 01:03:31.300 --> 01:03:34.720 align:middle line:84% legitimate to pull all the data and estimate one pool treatment 01:03:34.720 --> 01:03:35.800 align:middle line:90% effect. 01:03:35.800 --> 01:03:37.660 align:middle line:84% Once you know that, it's convenient 01:03:37.660 --> 01:03:41.800 align:middle line:84% because you can use information from all of the other countries 01:03:41.800 --> 01:03:43.840 align:middle line:84% to increase the precision of your estimate 01:03:43.840 --> 01:03:46.230 align:middle line:90% in any given site. 01:03:46.230 --> 01:03:48.210 align:middle line:84% So the precision of the estimate in India 01:03:48.210 --> 01:03:50.310 align:middle line:84% will increase a lot if you know that you can use 01:03:50.310 --> 01:03:52.260 align:middle line:90% Peru and Honduras, et cetera. 01:03:52.260 --> 01:03:56.708 align:middle line:84% And also, you can form credibility interval, 01:03:56.708 --> 01:03:58.750 align:middle line:84% which will be different from confidence interval, 01:03:58.750 --> 01:04:03.420 align:middle line:84% but are the same context, in prediction for the same-- 01:04:03.420 --> 01:04:05.100 align:middle line:90% for next effect. 01:04:05.100 --> 01:04:10.110 align:middle line:84% If, on the other hand, you find that the treatment effect are-- 01:04:10.110 --> 01:04:13.440 align:middle line:84% there is a huge variance in the treatment effect itself, 01:04:13.440 --> 01:04:15.840 align:middle line:84% and not just the outcome, then you know 01:04:15.840 --> 01:04:18.000 align:middle line:90% that there isn't heterogeneity. 01:04:18.000 --> 01:04:20.340 align:middle line:90% It's not very predictable. 01:04:20.340 --> 01:04:24.780 align:middle line:84% And maybe you can try to predict the heterogeneity itself. 01:04:24.780 --> 01:04:28.260 align:middle line:84% But otherwise, you know there is variation. 01:04:28.260 --> 01:04:30.150 align:middle line:90% So that's the idea. 01:04:30.150 --> 01:04:35.520 align:middle line:84% What she found is that both in terms of the-- 01:04:35.520 --> 01:04:38.460 align:middle line:90% so these are the effect using-- 01:04:38.460 --> 01:04:41.700 align:middle line:84% just the effect from the pooled regression 01:04:41.700 --> 01:04:43.060 align:middle line:90% using the linear model. 01:04:43.060 --> 01:04:45.270 align:middle line:84% So that's the result I already give you, 01:04:45.270 --> 01:04:48.450 align:middle line:84% assuming there is one treatment effect for the whole world. 01:04:48.450 --> 01:04:53.820 align:middle line:84% And this has a treatment effect in the sample 01:04:53.820 --> 01:04:58.410 align:middle line:84% with the hierarchical Bayesian model. 01:04:58.410 --> 01:05:01.980 align:middle line:84% And you can see in this instance, the confidence 01:05:01.980 --> 01:05:07.500 align:middle line:84% interval going up, reflecting heterogeneity in the site. 01:05:07.500 --> 01:05:09.840 align:middle line:84% This is making the assumption that everyone 01:05:09.840 --> 01:05:11.650 align:middle line:90% has the same treatment effect. 01:05:11.650 --> 01:05:13.770 align:middle line:84% This is not making that assumption anymore. 01:05:13.770 --> 01:05:19.890 align:middle line:84% And that decreases my confidence in the results. 01:05:19.890 --> 01:05:23.820 align:middle line:84% And now if I want to predict the next site because this 01:05:23.820 --> 01:05:26.170 align:middle line:90% is the predictive effect. 01:05:26.170 --> 01:05:30.480 align:middle line:84% So if I want to say, what's my best estimate of the effect, 01:05:30.480 --> 01:05:33.280 align:middle line:84% not in my sample, which is the effect here, 01:05:33.280 --> 01:05:38.580 align:middle line:84% but in any future sample, the predictive effects 01:05:38.580 --> 01:05:42.970 align:middle line:90% are quite wide. 01:05:42.970 --> 01:05:45.660 align:middle line:84% The credibility intervals are quite large, 01:05:45.660 --> 01:05:50.040 align:middle line:84% which reflect a large variability in the treatment 01:05:50.040 --> 01:05:51.850 align:middle line:90% effect from site to site. 01:05:51.850 --> 01:05:54.750 align:middle line:84% So here the fact that we saw already in the graph 01:05:54.750 --> 01:05:57.780 align:middle line:84% that the treatment effect seems to be going all over the place 01:05:57.780 --> 01:06:00.810 align:middle line:84% reflects some real heterogeneity, as opposed 01:06:00.810 --> 01:06:01.700 align:middle line:90% to noise in the data. 01:06:01.700 --> 01:06:03.900 align:middle line:84% In other words, each treatment effect 01:06:03.900 --> 01:06:06.720 align:middle line:84% is estimating relatively precisely for that country, 01:06:06.720 --> 01:06:08.970 align:middle line:84% but it happened to be different from site to site. 01:06:08.970 --> 01:06:13.110 align:middle line:84% So my predictability of the next site is not that good. 01:06:13.110 --> 01:06:16.230 align:middle line:84% That said, despite that, most of them 01:06:16.230 --> 01:06:20.080 align:middle line:84% are positive, even with that heterogeneity. 01:06:20.080 --> 01:06:22.330 align:middle line:84% So this result, by the way, are different from what 01:06:22.330 --> 01:06:23.980 align:middle line:84% she found for microcredit, where she 01:06:23.980 --> 01:06:28.120 align:middle line:84% found a lot of pooling, where she found that, basically, it's 01:06:28.120 --> 01:06:31.390 align:middle line:84% a nice big fat 0, regardless where you are. 01:06:31.390 --> 01:06:34.420 align:middle line:84% Here, we have positive effect, but viable. 01:06:34.420 --> 01:06:35.050 align:middle line:90% Yeah. 01:06:35.050 --> 01:06:37.175 align:middle line:84% AUDIENCE: Why does the [INAUDIBLE] estimate change? 01:06:37.175 --> 01:06:43.975 align:middle line:90% 01:06:43.975 --> 01:06:45.850 align:middle line:84% PROFESSOR: Here, you're making the assumption 01:06:45.850 --> 01:06:49.840 align:middle line:84% that the effect is the same everywhere. 01:06:49.840 --> 01:06:51.820 align:middle line:84% Therefore, one observation is one observation 01:06:51.820 --> 01:06:54.660 align:middle line:84% is one observation, whereas here, you're 01:06:54.660 --> 01:06:56.070 align:middle line:90% not making that assumption. 01:06:56.070 --> 01:06:58.980 align:middle line:90% 01:06:58.980 --> 01:07:00.437 align:middle line:84% So when you estimate your effect, 01:07:00.437 --> 01:07:02.520 align:middle line:84% you're not making the assumption that you can just 01:07:02.520 --> 01:07:06.240 align:middle line:90% run the regression all pooled. 01:07:06.240 --> 01:07:09.120 align:middle line:84% You are basically getting an average 01:07:09.120 --> 01:07:12.120 align:middle line:84% of the different treatment effects, where 01:07:12.120 --> 01:07:16.020 align:middle line:84% the weight don't become any more, 01:07:16.020 --> 01:07:17.908 align:middle line:90% like one paired observation. 01:07:17.908 --> 01:07:22.690 align:middle line:90% 01:07:22.690 --> 01:07:25.210 align:middle line:90% So in the rest of the-- 01:07:25.210 --> 01:07:28.480 align:middle line:84% so far, an existing paper, but I can send you the slide deck 01:07:28.480 --> 01:07:30.460 align:middle line:84% to start with, what they are doing 01:07:30.460 --> 01:07:33.640 align:middle line:84% is to try and see whether they can 01:07:33.640 --> 01:07:40.000 align:middle line:84% explain some fraction of that heterogeneity by covariates. 01:07:40.000 --> 01:07:43.120 align:middle line:90% 01:07:43.120 --> 01:07:45.200 align:middle line:84% Is the fact that-- in other words, 01:07:45.200 --> 01:07:48.280 align:middle line:84% can you use the heterogeneity in treatment effect 01:07:48.280 --> 01:07:51.370 align:middle line:84% by covariates to predict heterogeneity 01:07:51.370 --> 01:07:55.960 align:middle line:84% in treatment effect across sites based on the fact 01:07:55.960 --> 01:08:00.480 align:middle line:84% that the covariates of the sample was different? 01:08:00.480 --> 01:08:01.570 align:middle line:90% Does that make sense? 01:08:01.570 --> 01:08:04.740 align:middle line:84% So they are basically saying, well, the treatment effect, 01:08:04.740 --> 01:08:06.480 align:middle line:90% do they differ by people? 01:08:06.480 --> 01:08:08.400 align:middle line:90% To what extent do they? 01:08:08.400 --> 01:08:11.437 align:middle line:84% And then to what extent does that explain my difference 01:08:11.437 --> 01:08:12.270 align:middle line:90% in treatment effect? 01:08:12.270 --> 01:08:15.780 align:middle line:84% And the answer is not that much simply because we are not 01:08:15.780 --> 01:08:20.279 align:middle line:84% very good at using the covariate to predict different treatment 01:08:20.279 --> 01:08:23.220 align:middle line:84% effects in this case, even within countries. 01:08:23.220 --> 01:08:25.979 align:middle line:90% That I know for having tried. 01:08:25.979 --> 01:08:30.481 align:middle line:84% So if you're interested, with her permission, 01:08:30.481 --> 01:08:31.439 align:middle line:90% I'll diffuse the slide. 01:08:31.439 --> 01:08:33.606 align:middle line:84% I think she'll give me permission since she sends me 01:08:33.606 --> 01:08:34.950 align:middle line:90% the slide for this. 01:08:34.950 --> 01:08:37.830 align:middle line:84% And you can see a little bit what she does. 01:08:37.830 --> 01:08:40.979 align:middle line:84% It's not complete, but it will be at some point. 01:08:40.979 --> 01:08:44.700 align:middle line:90% 01:08:44.700 --> 01:08:48.540 align:middle line:84% So anyway, even despite-- so that's an interesting point, 01:08:48.540 --> 01:08:52.020 align:middle line:84% I think, for heterogeneity because this is really 01:08:52.020 --> 01:08:55.050 align:middle line:84% the first paper where I see a lot of heterogeneity 01:08:55.050 --> 01:08:57.330 align:middle line:84% being demonstrated, even in a program that, 01:08:57.330 --> 01:08:59.970 align:middle line:90% in general, works. 01:08:59.970 --> 01:09:05.439 align:middle line:84% In terms of cost-benefits, this program is super expensive. 01:09:05.439 --> 01:09:07.500 align:middle line:84% So it's only worth it if the benefits 01:09:07.500 --> 01:09:11.420 align:middle line:90% persist into the future. 01:09:11.420 --> 01:09:14.260 align:middle line:84% If you look at the first paper, the consumption increase 01:09:14.260 --> 01:09:17.270 align:middle line:84% is about 11% in Bandiera's paper. 01:09:17.270 --> 01:09:19.609 align:middle line:84% So in order to do cost-benefit analysis, 01:09:19.609 --> 01:09:21.324 align:middle line:84% they assume that then this effect lasts. 01:09:21.324 --> 01:09:24.670 align:middle line:90% 01:09:24.670 --> 01:09:27.439 align:middle line:84% Of course, the depression rate of benefits is key. 01:09:27.439 --> 01:09:29.890 align:middle line:84% And so whether or not this is a poverty trap 01:09:29.890 --> 01:09:32.080 align:middle line:84% becomes key also for whether it's 01:09:32.080 --> 01:09:35.020 align:middle line:84% a program worth doing, not just because we're interested in it. 01:09:35.020 --> 01:09:37.880 align:middle line:90% 01:09:37.880 --> 01:09:41.460 align:middle line:84% This is from the science paper, where 01:09:41.460 --> 01:09:43.770 align:middle line:84% we are looking at the sensitivity 01:09:43.770 --> 01:09:49.649 align:middle line:84% to different discount rate, time horizon, and benefits, 01:09:49.649 --> 01:09:55.320 align:middle line:84% and calculating for how long the benefits need 01:09:55.320 --> 01:10:02.060 align:middle line:84% to last in order for the program to be cost effective. 01:10:02.060 --> 01:10:04.190 align:middle line:84% And the point of that table, which 01:10:04.190 --> 01:10:06.110 align:middle line:84% you can look in more details at your leisure, 01:10:06.110 --> 01:10:09.110 align:middle line:84% is that if the benefits were to persist, 01:10:09.110 --> 01:10:13.760 align:middle line:84% despite the program being extremely expensive, 01:10:13.760 --> 01:10:16.920 align:middle line:90% it's very cost effective. 01:10:16.920 --> 01:10:19.490 align:middle line:84% But if they were to decline after four years, 01:10:19.490 --> 01:10:22.500 align:middle line:90% then it would not be, obviously. 01:10:22.500 --> 01:10:25.020 align:middle line:84% So then that becomes a very interesting empirical question, 01:10:25.020 --> 01:10:27.320 align:middle line:84% and it's tied to our poverty trap, is, do they persist? 01:10:27.320 --> 01:10:30.560 align:middle line:90% 01:10:30.560 --> 01:10:35.000 align:middle line:84% So we looked at this with Abhijit 01:10:35.000 --> 01:10:40.190 align:middle line:84% and with Garima, who is in the program here in West Bengal. 01:10:40.190 --> 01:10:42.650 align:middle line:84% After the three-year survey, we observed it 01:10:42.650 --> 01:10:46.020 align:middle line:84% at seven years and then at 10 years. 01:10:46.020 --> 01:10:50.420 align:middle line:84% And here is one of the graph for consumption, 01:10:50.420 --> 01:10:53.120 align:middle line:84% where you see that the treatment effect actually 01:10:53.120 --> 01:10:57.680 align:middle line:84% grows between year 3 and 7, and then 01:10:57.680 --> 01:11:01.820 align:middle line:84% is persistent-- mostly persistent between your year 01:11:01.820 --> 01:11:05.460 align:middle line:90% 7 and year 10. 01:11:05.460 --> 01:11:08.680 align:middle line:84% So this is in a context where there was economic growth. 01:11:08.680 --> 01:11:11.865 align:middle line:84% So the control grew where the treatment grew faster. 01:11:11.865 --> 01:11:16.070 align:middle line:84% And so the treatment become faster. 01:11:16.070 --> 01:11:19.010 align:middle line:84% So the control level grew from 1.8 to 2.9. 01:11:19.010 --> 01:11:22.310 align:middle line:84% And the treatment effect in terms of standard deviation 01:11:22.310 --> 01:11:25.010 align:middle line:84% actually jumped between year 3 and 7 01:11:25.010 --> 01:11:29.930 align:middle line:84% to 0.7 standard deviation, and is still 0.6 after 10 years. 01:11:29.930 --> 01:11:33.230 align:middle line:84% So that shows some persistence of effect, 01:11:33.230 --> 01:11:34.690 align:middle line:84% which means that when we actually 01:11:34.690 --> 01:11:39.430 align:middle line:84% reduce the cost-benefit analysis for India with this new data, 01:11:39.430 --> 01:11:41.920 align:middle line:84% it's actually much more favorable than it was done 01:11:41.920 --> 01:11:46.600 align:middle line:84% at year 3 because the treatment effect didn't vanish and didn't 01:11:46.600 --> 01:11:47.150 align:middle line:90% decline. 01:11:47.150 --> 01:11:50.020 align:middle line:84% So in fact, by year 10, even if the effect 01:11:50.020 --> 01:11:55.570 align:middle line:84% were going to disappear fully, the program is worth-- 01:11:55.570 --> 01:11:58.310 align:middle line:90% has paid off for itself. 01:11:58.310 --> 01:11:59.545 align:middle line:90% So this is for income. 01:11:59.545 --> 01:12:02.110 align:middle line:90% 01:12:02.110 --> 01:12:08.770 align:middle line:84% We're also looking across outcomes using, again, nicely. 01:12:08.770 --> 01:12:12.640 align:middle line:84% We have a nice pre-analysis plan given to us by the fact 01:12:12.640 --> 01:12:14.443 align:middle line:84% that we are following an existing paper. 01:12:14.443 --> 01:12:15.610 align:middle line:90% So we don't invent anything. 01:12:15.610 --> 01:12:18.610 align:middle line:90% We redo the same paper. 01:12:18.610 --> 01:12:21.310 align:middle line:84% So the paper also nicely writes itself. 01:12:21.310 --> 01:12:24.280 align:middle line:84% And you're finding the treatment effect 01:12:24.280 --> 01:12:28.690 align:middle line:84% to persist over time, with the exclusion 01:12:28.690 --> 01:12:32.140 align:middle line:84% of-- with the exception of financial inclusion, 01:12:32.140 --> 01:12:34.390 align:middle line:84% which is mainly whether people borrow or not, 01:12:34.390 --> 01:12:37.300 align:middle line:84% where we never really had much of an impact 01:12:37.300 --> 01:12:40.320 align:middle line:90% in the first place. 01:12:40.320 --> 01:12:43.320 align:middle line:90% In that paper, we discussed-- 01:12:43.320 --> 01:12:46.560 align:middle line:84% so interestingly, the effect on physical assets that 01:12:46.560 --> 01:12:51.060 align:middle line:84% people have, the clients have had, were first increasing, 01:12:51.060 --> 01:12:54.490 align:middle line:90% and then went back down. 01:12:54.490 --> 01:13:01.360 align:middle line:84% And another thing we see is that these people are getting quite 01:13:01.360 --> 01:13:04.520 align:middle line:90% old who got the first transfer. 01:13:04.520 --> 01:13:09.920 align:middle line:84% So a lot of them are also slowly retiring and winding down 01:13:09.920 --> 01:13:11.070 align:middle line:90% their business. 01:13:11.070 --> 01:13:17.572 align:middle line:84% What we saw in India between the first and the second in line, 01:13:17.572 --> 01:13:19.280 align:middle line:84% and then the second in line and the third 01:13:19.280 --> 01:13:22.850 align:middle line:84% is people diversifying out of cow, 01:13:22.850 --> 01:13:27.440 align:middle line:84% into nonagricultural businesses, like small business and stuff 01:13:27.440 --> 01:13:28.220 align:middle line:90% like that. 01:13:28.220 --> 01:13:31.940 align:middle line:84% What we saw them doing between this wave and that wave 01:13:31.940 --> 01:13:34.490 align:middle line:84% is winding down the business and sending migrants 01:13:34.490 --> 01:13:39.350 align:middle line:84% out there, and in particular, sending migrants faster. 01:13:39.350 --> 01:13:41.900 align:middle line:84% So to answer the question about what's the mechanism, 01:13:41.900 --> 01:13:43.010 align:middle line:90% we make an assumption. 01:13:43.010 --> 01:13:46.430 align:middle line:84% We make a guess here, a conjecture, 01:13:46.430 --> 01:13:49.280 align:middle line:84% which we are-- it seems to be borne out by the data we have, 01:13:49.280 --> 01:13:51.320 align:middle line:84% is that the way in which you actually 01:13:51.320 --> 01:13:54.350 align:middle line:84% have persistence of this program is 01:13:54.350 --> 01:13:57.260 align:middle line:84% first by diversification from agriculture 01:13:57.260 --> 01:14:00.860 align:middle line:84% to nonagricultural businesses, and then from small business 01:14:00.860 --> 01:14:03.770 align:middle line:84% to actually more stable wage work 01:14:03.770 --> 01:14:09.630 align:middle line:84% via the following generation, which moves for a longer 01:14:09.630 --> 01:14:14.340 align:middle line:84% period, for further away, and earns more money 01:14:14.340 --> 01:14:15.110 align:middle line:90% as they do that. 01:14:15.110 --> 01:14:22.440 align:middle line:90% 01:14:22.440 --> 01:14:25.230 align:middle line:84% Now, these programs have highly heterogeneous effect. 01:14:25.230 --> 01:14:27.920 align:middle line:90% We saw it by country. 01:14:27.920 --> 01:14:29.670 align:middle line:84% I already showed you that by country, they 01:14:29.670 --> 01:14:32.700 align:middle line:84% have super heterogeneous treatment effect. 01:14:32.700 --> 01:14:35.380 align:middle line:90% We also see it by people. 01:14:35.380 --> 01:14:38.100 align:middle line:90% So these are now within-- 01:14:38.100 --> 01:14:39.100 align:middle line:90% this is Bangladesh. 01:14:39.100 --> 01:14:42.890 align:middle line:84% So this is within a particular country, the quantile treatment 01:14:42.890 --> 01:14:43.390 align:middle line:90% effect. 01:14:43.390 --> 01:14:45.720 align:middle line:84% So remember, it's not the effect on the-- it's 01:14:45.720 --> 01:14:48.460 align:middle line:84% not the distribution of the treatment effect. 01:14:48.460 --> 01:14:52.740 align:middle line:84% But it's how the distribution affect 01:14:52.740 --> 01:14:58.170 align:middle line:84% the distribution of assets or consumption expenditure. 01:14:58.170 --> 01:15:01.590 align:middle line:84% And what you're seeing is that the effect at the low quantile 01:15:01.590 --> 01:15:06.450 align:middle line:84% are not very large, and they keep the effect on the highest 01:15:06.450 --> 01:15:09.780 align:middle line:90% quantile higher. 01:15:09.780 --> 01:15:13.800 align:middle line:84% We find the same thing in India, and people find the same thing 01:15:13.800 --> 01:15:15.930 align:middle line:84% everywhere, that the treatment effect tends 01:15:15.930 --> 01:15:19.770 align:middle line:84% to be towards the highest quantile, 01:15:19.770 --> 01:15:23.670 align:middle line:84% with less effect at the lowest quantile of the-- 01:15:23.670 --> 01:15:27.540 align:middle line:84% moving the lowest quantile of the distribution. 01:15:27.540 --> 01:15:29.490 align:middle line:84% On the other hand, as I said before, 01:15:29.490 --> 01:15:31.700 align:middle line:84% an effort to link that to observable 01:15:31.700 --> 01:15:33.330 align:middle line:90% is not very successful. 01:15:33.330 --> 01:15:34.940 align:middle line:84% So the effect of heterogenization 01:15:34.940 --> 01:15:37.040 align:middle line:90% is not very clear initially why. 01:15:37.040 --> 01:15:39.720 align:middle line:90% 01:15:39.720 --> 01:15:43.680 align:middle line:84% So a key question for the future from a program perspective is, 01:15:43.680 --> 01:15:46.510 align:middle line:84% how do you identify people which have very high return? 01:15:46.510 --> 01:15:49.890 align:middle line:84% And that's the same thing as in the microcredit business, 01:15:49.890 --> 01:15:52.260 align:middle line:84% where, even though they are, on average, as we are going 01:15:52.260 --> 01:15:55.650 align:middle line:84% to see when we study credit treatment effects of programs, 01:15:55.650 --> 01:15:58.800 align:middle line:84% are 0, they are quantile treatment effect 01:15:58.800 --> 01:16:00.870 align:middle line:84% that are very different at different quantile. 01:16:00.870 --> 01:16:02.340 align:middle line:90% Some people seem to benefit. 01:16:02.340 --> 01:16:04.890 align:middle line:84% How do you know who they are to target them? 01:16:04.890 --> 01:16:08.530 align:middle line:84% And in a way, the ultra poor have that as well. 01:16:08.530 --> 01:16:11.130 align:middle line:84% In our case, this is a project that 01:16:11.130 --> 01:16:13.620 align:middle line:84% is a question that is even made even more 01:16:13.620 --> 01:16:16.440 align:middle line:84% interesting by the fact that this 01:16:16.440 --> 01:16:18.420 align:middle line:90% is motivated by poverty trap. 01:16:18.420 --> 01:16:22.050 align:middle line:84% And we kind of have an idea of maybe something that 01:16:22.050 --> 01:16:23.730 align:middle line:84% could sell that you get huge effect 01:16:23.730 --> 01:16:29.040 align:middle line:84% or you don't, which is do you get out 01:16:29.040 --> 01:16:33.690 align:middle line:84% of the poverty trap for this persistent effect or not? 01:16:33.690 --> 01:16:40.850 align:middle line:84% So that crew who did the Bangladesh program, 01:16:40.850 --> 01:16:45.200 align:middle line:84% along with Clare Balboni, who is teaching in our department, 01:16:45.200 --> 01:16:50.000 align:middle line:84% as you know, and Maitreesh Ghatak, who joined them, 01:16:50.000 --> 01:16:53.850 align:middle line:84% tried to look at exactly that question. 01:16:53.850 --> 01:16:57.380 align:middle line:84% So it's really one of the first serious attempt 01:16:57.380 --> 01:17:00.170 align:middle line:84% that I know of to draw a kind of S-curve 01:17:00.170 --> 01:17:01.820 align:middle line:84% or to see whether there is S-curve 01:17:01.820 --> 01:17:05.310 align:middle line:90% at the individual level. 01:17:05.310 --> 01:17:09.090 align:middle line:84% And some people have been quite skeptical that those existed. 01:17:09.090 --> 01:17:12.120 align:middle line:84% There is a paper by McKenzie on small businesses showing, 01:17:12.120 --> 01:17:15.060 align:middle line:90% I don't see any of that. 01:17:15.060 --> 01:17:18.012 align:middle line:84% But with observational data, of course, 01:17:18.012 --> 01:17:19.470 align:middle line:84% what's the main problem with trying 01:17:19.470 --> 01:17:21.645 align:middle line:84% to draw an S-curve with observational data? 01:17:21.645 --> 01:17:24.810 align:middle line:90% 01:17:24.810 --> 01:17:27.770 align:middle line:84% And why is it that you might not find it in observational data? 01:17:27.770 --> 01:17:31.680 align:middle line:90% 01:17:31.680 --> 01:17:32.970 align:middle line:90% Yep. 01:17:32.970 --> 01:17:35.970 align:middle line:84% AUDIENCE: So if you do have this S-curve and the curve 01:17:35.970 --> 01:17:40.170 align:middle line:84% intersecting the 45-degree line, that steady state is unstable. 01:17:40.170 --> 01:17:43.250 align:middle line:84% So in a cross section, you actually 01:17:43.250 --> 01:17:44.880 align:middle line:84% had very few people who were there, 01:17:44.880 --> 01:17:46.170 align:middle line:90% so you wouldn't get them. 01:17:46.170 --> 01:17:46.962 align:middle line:90% PROFESSOR: Exactly. 01:17:46.962 --> 01:17:47.940 align:middle line:90% Nobody is there. 01:17:47.940 --> 01:17:49.590 align:middle line:84% It would be like an accident that you 01:17:49.590 --> 01:17:53.050 align:middle line:90% find even a few people there. 01:17:53.050 --> 01:17:56.320 align:middle line:84% And therefore, you shouldn't be able to capture it 01:17:56.320 --> 01:17:58.180 align:middle line:84% in observational data in the real world 01:17:58.180 --> 01:18:01.810 align:middle line:90% if you really believe the model. 01:18:01.810 --> 01:18:06.070 align:middle line:84% Unless the S-curve was actually individual specific, 01:18:06.070 --> 01:18:11.080 align:middle line:84% and people were moved, it seems that, in fact, you should never 01:18:11.080 --> 01:18:13.240 align:middle line:84% see people with a very, very high return range 01:18:13.240 --> 01:18:16.430 align:middle line:84% because they would have moved out of it by now. 01:18:16.430 --> 01:18:18.640 align:middle line:84% So in the real world, you will be 01:18:18.640 --> 01:18:21.280 align:middle line:84% able to draw a curve linking capital today 01:18:21.280 --> 01:18:22.840 align:middle line:90% to capital tomorrow. 01:18:22.840 --> 01:18:26.050 align:middle line:84% But that curve is going to reflect the heterogeneity 01:18:26.050 --> 01:18:30.670 align:middle line:84% of where those threshold might be located, as opposed to 01:18:30.670 --> 01:18:32.570 align:middle line:84% whether that threshold exist or not. 01:18:32.570 --> 01:18:35.120 align:middle line:90% That's pretty big problems. 01:18:35.120 --> 01:18:41.890 align:middle line:84% So why the ultra poor program helps us there pretty nicely? 01:18:41.890 --> 01:18:48.640 align:middle line:90% 01:18:48.640 --> 01:18:49.280 align:middle line:90% Yep. 01:18:49.280 --> 01:18:50.780 align:middle line:84% AUDIENCE: You follow them over time. 01:18:50.780 --> 01:18:52.113 align:middle line:90% So [INAUDIBLE]. 01:18:52.113 --> 01:18:53.280 align:middle line:90% PROFESSOR: So there is that. 01:18:53.280 --> 01:18:55.700 align:middle line:90% And in addition, so-- 01:18:55.700 --> 01:18:56.272 align:middle line:90% yeah. 01:18:56.272 --> 01:18:58.230 align:middle line:84% AUDIENCE: Since you start off with giving money 01:18:58.230 --> 01:19:00.210 align:middle line:84% to people who are ultra poor, you might put them 01:19:00.210 --> 01:19:01.135 align:middle line:90% into that middle area. 01:19:01.135 --> 01:19:03.510 align:middle line:84% And then you can watch them transition to the other side. 01:19:03.510 --> 01:19:03.870 align:middle line:90% PROFESSOR: Yeah. 01:19:03.870 --> 01:19:06.000 align:middle line:84% So what would have been the ideal experiment 01:19:06.000 --> 01:19:13.220 align:middle line:84% if testing this S-curve had been the ultimate goal? 01:19:13.220 --> 01:19:14.550 align:middle line:90% Yeah. 01:19:14.550 --> 01:19:17.362 align:middle line:84% AUDIENCE: Give different amounts of money or goods 01:19:17.362 --> 01:19:18.948 align:middle line:90% to different people. 01:19:18.948 --> 01:19:19.740 align:middle line:90% PROFESSOR: Exactly. 01:19:19.740 --> 01:19:21.840 align:middle line:84% So different amount of money to people 01:19:21.840 --> 01:19:24.840 align:middle line:84% who start from the same assets would have been the ideal thing 01:19:24.840 --> 01:19:25.920 align:middle line:90% to do. 01:19:25.920 --> 01:19:27.370 align:middle line:84% Or this is not what was done here, 01:19:27.370 --> 01:19:29.520 align:middle line:90% that there are two life program. 01:19:29.520 --> 01:19:31.980 align:middle line:84% And they were probably not about to do that. 01:19:31.980 --> 01:19:34.380 align:middle line:84% But you have something that is getting a little bit 01:19:34.380 --> 01:19:37.980 align:middle line:84% close to that, is people who are distributed whichever way. 01:19:37.980 --> 01:19:40.200 align:middle line:90% And then they got these assets. 01:19:40.200 --> 01:19:42.900 align:middle line:84% And then the asset would have been small enough of-- 01:19:42.900 --> 01:19:44.910 align:middle line:84% large enough or not, depending on how poor 01:19:44.910 --> 01:19:48.420 align:middle line:84% they were, to put them over the threshold or not. 01:19:48.420 --> 01:19:50.600 align:middle line:84% So in order to do this exercise, you 01:19:50.600 --> 01:19:53.120 align:middle line:84% have to really believe the model because you 01:19:53.120 --> 01:19:55.910 align:middle line:84% have to believe there is one threshold out of which-- 01:19:55.910 --> 01:19:58.040 align:middle line:84% that people can jump over or not. 01:19:58.040 --> 01:20:00.187 align:middle line:90% Otherwise, it'd be hard to find. 01:20:00.187 --> 01:20:00.687 align:middle line:90% Yeah. 01:20:00.687 --> 01:20:02.690 align:middle line:84% AUDIENCE: But then in the model, there's 01:20:02.690 --> 01:20:05.840 align:middle line:84% also one steady state and one threshold. 01:20:05.840 --> 01:20:08.240 align:middle line:84% And now we are allowing people to have 01:20:08.240 --> 01:20:10.730 align:middle line:84% different steady states, but assuming that they still 01:20:10.730 --> 01:20:12.362 align:middle line:90% have one threshold to pass. 01:20:12.362 --> 01:20:13.820 align:middle line:84% PROFESSOR: So I think in the model, 01:20:13.820 --> 01:20:17.780 align:middle line:84% there is two steady states-- one low one, one high one, 01:20:17.780 --> 01:20:20.150 align:middle line:84% one unstable steady state, or two or three, 01:20:20.150 --> 01:20:21.690 align:middle line:90% including an unstable one. 01:20:21.690 --> 01:20:23.940 align:middle line:84% And I think in the view of the real world, also, 01:20:23.940 --> 01:20:25.040 align:middle line:90% there are only-- 01:20:25.040 --> 01:20:26.960 align:middle line:90% every one faces the same thing. 01:20:26.960 --> 01:20:28.610 align:middle line:84% But then they are shocked, which means 01:20:28.610 --> 01:20:31.670 align:middle line:84% that at the given point in time, they end up-- 01:20:31.670 --> 01:20:34.430 align:middle line:84% they have end up one way or the other. 01:20:34.430 --> 01:20:40.220 align:middle line:84% Remember, in fact, they have a distribution. 01:20:40.220 --> 01:20:42.570 align:middle line:84% This is a distribution of assets. 01:20:42.570 --> 01:20:44.250 align:middle line:84% And I would say, no, no, they are two. 01:20:44.250 --> 01:20:47.190 align:middle line:84% They are in one unstable steady state and two stable 01:20:47.190 --> 01:20:49.570 align:middle line:90% steady state in the real world. 01:20:49.570 --> 01:20:51.510 align:middle line:84% So what I should see is that there 01:20:51.510 --> 01:20:54.420 align:middle line:84% are a lot of people at each of them, 01:20:54.420 --> 01:20:57.413 align:middle line:84% and then not too many people in the meantime. 01:20:57.413 --> 01:20:59.580 align:middle line:84% AUDIENCE: Well, if there are shops that are growing, 01:20:59.580 --> 01:21:01.793 align:middle line:84% I don't need to experiment to begin with. 01:21:01.793 --> 01:21:02.460 align:middle line:90% PROFESSOR: Yeah. 01:21:02.460 --> 01:21:04.918 align:middle line:84% But the point is that they are shock combined with the fact 01:21:04.918 --> 01:21:07.770 align:middle line:84% that-- so it's not saying that you'll never observe anyone 01:21:07.770 --> 01:21:09.270 align:middle line:90% in the unstable steady state. 01:21:09.270 --> 01:21:11.400 align:middle line:84% But you won't see very many people. 01:21:11.400 --> 01:21:15.000 align:middle line:84% So it's going to be very difficult to catch them. 01:21:15.000 --> 01:21:16.570 align:middle line:90% So here is the curve. 01:21:16.570 --> 01:21:19.170 align:middle line:84% And I think that that's really a beautiful graph, 01:21:19.170 --> 01:21:21.082 align:middle line:90% in a sense of-- 01:21:21.082 --> 01:21:23.290 align:middle line:84% I don't know if it would have occurred to me that you 01:21:23.290 --> 01:21:26.380 align:middle line:84% can even try to do that, which is to say, well, let me take 01:21:26.380 --> 01:21:27.760 align:middle line:90% the theory super seriously. 01:21:27.760 --> 01:21:29.350 align:middle line:90% There are two steady states. 01:21:29.350 --> 01:21:32.110 align:middle line:84% Therefore, I should see people with two levels 01:21:32.110 --> 01:21:34.630 align:middle line:84% of assets, the high level and the low level. 01:21:34.630 --> 01:21:36.540 align:middle line:84% And let me draw the distribution. 01:21:36.540 --> 01:21:40.710 align:middle line:84% And sure enough, they have this bimodal distribution. 01:21:40.710 --> 01:21:43.630 align:middle line:84% So they live in a world where all-- 01:21:43.630 --> 01:21:46.070 align:middle line:84% this seems to be a world where that thing actually 01:21:46.070 --> 01:21:50.817 align:middle line:84% exists and is relatively clear, what is the statistic. 01:21:50.817 --> 01:21:52.400 align:middle line:84% And then they are shocked, which means 01:21:52.400 --> 01:21:55.220 align:middle line:84% that you don't see nobody, but you don't see very many people 01:21:55.220 --> 01:21:56.360 align:middle line:90% there. 01:21:56.360 --> 01:21:58.820 align:middle line:84% And then once we have that, then I 01:21:58.820 --> 01:22:00.110 align:middle line:90% think we went through already. 01:22:00.110 --> 01:22:01.130 align:middle line:90% This is small. 01:22:01.130 --> 01:22:04.940 align:middle line:84% We can now add the value of the asset. 01:22:04.940 --> 01:22:06.510 align:middle line:90% This is quite small. 01:22:06.510 --> 01:22:08.840 align:middle line:84% So we add the medium value of a cow. 01:22:08.840 --> 01:22:12.500 align:middle line:84% And now, suddenly, we find a bunch of people in the middle. 01:22:12.500 --> 01:22:16.310 align:middle line:84% And in a sense, they were lucky because the transfer was such 01:22:16.310 --> 01:22:21.690 align:middle line:84% that given from the people who were very, very poor to start 01:22:21.690 --> 01:22:26.160 align:middle line:84% with, there are a lot of them who are just to the left 01:22:26.160 --> 01:22:29.340 align:middle line:84% or just to the right of the threshold. 01:22:29.340 --> 01:22:31.280 align:middle line:90% And now we can run that cow. 01:22:31.280 --> 01:22:33.080 align:middle line:90% So this is a treatment group. 01:22:33.080 --> 01:22:34.190 align:middle line:90% And this is saying-- 01:22:34.190 --> 01:22:36.600 align:middle line:84% that's the cow we already saw in the first class, exactly 01:22:36.600 --> 01:22:37.100 align:middle line:90% [INAUDIBLE]. 01:22:37.100 --> 01:22:40.970 align:middle line:84% This is pretty impressive, which is, let me show-- 01:22:40.970 --> 01:22:44.750 align:middle line:84% this is a nonparametric regression. 01:22:44.750 --> 01:22:46.400 align:middle line:90% There's no magic there. 01:22:46.400 --> 01:22:48.360 align:middle line:84% No rabbit has come out of the hat. 01:22:48.360 --> 01:22:51.230 align:middle line:84% That's the data showing that this 01:22:51.230 --> 01:22:55.760 align:middle line:84% is the baseline productive asset in the treatment group 01:22:55.760 --> 01:23:01.170 align:middle line:84% and in the baseline predictive asset in the control group. 01:23:01.170 --> 01:23:03.560 align:middle line:84% So the treatment is the asset they had at baseline 01:23:03.560 --> 01:23:05.960 align:middle line:90% plus the value of the transfer. 01:23:05.960 --> 01:23:07.830 align:middle line:84% And now you're able to draw this curve. 01:23:07.830 --> 01:23:09.440 align:middle line:84% And you can draw it because you have 01:23:09.440 --> 01:23:12.120 align:middle line:90% sufficient mass on both sides. 01:23:12.120 --> 01:23:16.120 align:middle line:84% And by contrast, if you try to do that in the control group, 01:23:16.120 --> 01:23:20.980 align:middle line:84% you only find one unique steady state. 01:23:20.980 --> 01:23:23.200 align:middle line:84% Maybe there is that other steady state 01:23:23.200 --> 01:23:27.670 align:middle line:84% that we can't catch it because there is nobody there. 01:23:27.670 --> 01:23:29.620 align:middle line:84% And then there is-- so you can turn that 01:23:29.620 --> 01:23:34.510 align:middle line:84% into a regression exercise and to say, define delta 01:23:34.510 --> 01:23:37.000 align:middle line:90% i as the difference in assets. 01:23:37.000 --> 01:23:39.100 align:middle line:84% And then k star is the threshold level 01:23:39.100 --> 01:23:42.690 align:middle line:84% of capital, which they estimate from that regression. 01:23:42.690 --> 01:23:45.090 align:middle line:84% And is it the case that people grow 01:23:45.090 --> 01:23:49.620 align:middle line:84% faster when their level plus treatment 01:23:49.620 --> 01:23:55.020 align:middle line:84% is above the threshold in the treatment villages 01:23:55.020 --> 01:23:56.580 align:middle line:84% compared to the control villages? 01:23:56.580 --> 01:23:59.580 align:middle line:84% Because the asset level plus treatment 01:23:59.580 --> 01:24:03.850 align:middle line:84% is just their asset level, so they don't benefit from that. 01:24:03.850 --> 01:24:09.100 align:middle line:84% So that gives you a regression, which exploits your experiment. 01:24:09.100 --> 01:24:11.070 align:middle line:84% So you're saying that in treatment, 01:24:11.070 --> 01:24:13.510 align:middle line:90% if you end up being-- 01:24:13.510 --> 01:24:17.080 align:middle line:84% if your own asset level plus the transfer 01:24:17.080 --> 01:24:19.180 align:middle line:84% puts you above the threshold, you 01:24:19.180 --> 01:24:23.188 align:middle line:84% have a fast growth in capital, but not in control 01:24:23.188 --> 01:24:24.730 align:middle line:84% because the treatment you didn't get. 01:24:24.730 --> 01:24:29.490 align:middle line:84% So you are above, but you didn't get it. 01:24:29.490 --> 01:24:31.860 align:middle line:84% And then we can put the two sample together 01:24:31.860 --> 01:24:35.190 align:middle line:90% and run this as an interaction. 01:24:35.190 --> 01:24:39.150 align:middle line:84% So that's where I'm going to leave it for today. 01:24:39.150 --> 01:24:43.450 align:middle line:84% The next question, of course, is, where does that come from? 01:24:43.450 --> 01:24:46.600 align:middle line:90% And we know it's not food. 01:24:46.600 --> 01:24:48.520 align:middle line:90% We know it's not credit. 01:24:48.520 --> 01:24:50.368 align:middle line:90% So the question is what it is. 01:24:50.368 --> 01:24:56.960 align:middle line:90% 01:24:56.960 --> 01:24:59.300 align:middle line:84% We're going to continue talking about that. 01:24:59.300 --> 01:25:02.200 align:middle line:84% In fact, we're going to continue talking about next week 01:25:02.200 --> 01:25:05.620 align:middle line:84% because you are going to talk to your-- 01:25:05.620 --> 01:25:09.230 align:middle line:84% Frank is going to come for two lecture to talk about behavior. 01:25:09.230 --> 01:25:11.260 align:middle line:84% And one of the potential explanation 01:25:11.260 --> 01:25:13.690 align:middle line:84% is that it has more to do with psychology, 01:25:13.690 --> 01:25:17.290 align:middle line:84% with various aspects of the psychology of poverty, which 01:25:17.290 --> 01:25:19.630 align:middle line:84% might have impacts that are functionally 01:25:19.630 --> 01:25:22.450 align:middle line:84% equivalent to the ideas that were 01:25:22.450 --> 01:25:24.190 align:middle line:84% in the nutrition-based poverty trap that 01:25:24.190 --> 01:25:29.420 align:middle line:84% comes from people's reasoning, as opposed to something else. 01:25:29.420 --> 01:25:31.180 align:middle line:84% And then we are going to see it again 01:25:31.180 --> 01:25:33.430 align:middle line:84% when you talk about labor markets because some of this 01:25:33.430 --> 01:25:36.400 align:middle line:90% could be productivity of labor. 01:25:36.400 --> 01:25:39.400 align:middle line:84% But we now know that, at least for the ultra poor, 01:25:39.400 --> 01:25:42.590 align:middle line:84% there might be something like that which is really at play. 01:25:42.590 --> 01:25:47.680 align:middle line:84% And so that should give us even more energy and motivation 01:25:47.680 --> 01:25:51.700 align:middle line:84% to find what it is that can be at play, hence the unpacking. 01:25:51.700 --> 01:25:55.660 align:middle line:84% That is also going to come up in other-- 01:25:55.660 --> 01:25:58.180 align:middle line:84% at several points, we're going to have a chance 01:25:58.180 --> 01:26:01.150 align:middle line:84% to unpack the ultra poor program. 01:26:01.150 --> 01:26:01.660 align:middle line:90% Thank you. 01:26:01.660 --> 01:26:03.000 align:middle line:90% That's it for today. 01:26:03.000 --> 01:26:06.170 align:middle line:84% So Monday-- next week will be Frank, and I'll see you again 01:26:06.170 --> 01:26:08.250 align:middle line:90% in the following week. 01:26:08.250 --> 01:26:20.000 align:middle line:90%