WEBVTT

00:00:00.000 --> 00:00:01.930
ANNOUNCER: The following content
is provided under a

00:00:01.930 --> 00:00:03.680
Creative Commons license.

00:00:03.680 --> 00:00:06.640
Your support will help MIT
OpenCourseWare continue to

00:00:06.640 --> 00:00:09.980
offer high quality educational
resources for free.

00:00:09.980 --> 00:00:12.820
To make a donation or to view
additional materials from

00:00:12.820 --> 00:00:16.750
hundreds of MIT courses, visit
MIT OpenCourseWare at

00:00:16.750 --> 00:00:18.000
ocw.mit.edu.

00:00:22.940 --> 00:00:24.100
PROFESSOR: Great, hi everyone.

00:00:24.100 --> 00:00:26.440
It's great to be here, and
I hope you guys are

00:00:26.440 --> 00:00:27.600
having fun so far.

00:00:27.600 --> 00:00:31.150
It's great to meet
all of you too.

00:00:31.150 --> 00:00:34.820
So this is the day where we talk
about how to randomize.

00:00:34.820 --> 00:00:37.350
A few other topics will come
up here and there, and

00:00:37.350 --> 00:00:40.540
hopefully you will stop me
whenever you have questions,

00:00:40.540 --> 00:00:43.840
and let me know.

00:00:43.840 --> 00:00:47.010
So don't hesitate to stop even
if it's just a clarifying

00:00:47.010 --> 00:00:51.200
point or a deeper, more
substantive question about

00:00:51.200 --> 00:00:52.450
what I'm saying.

00:00:58.410 --> 00:01:01.750
The three basic components of
this morning's lecture will be

00:01:01.750 --> 00:01:04.550
first about methods of
randomization, and that'll

00:01:04.550 --> 00:01:07.250
be-- the majority of what we'll
talk about is going

00:01:07.250 --> 00:01:10.590
through a few different ways
that we talk about

00:01:10.590 --> 00:01:12.240
randomization when we
talk randomization.

00:01:12.240 --> 00:01:15.990
One of the common misperceptions
that I've seen

00:01:15.990 --> 00:01:18.590
in the world when I am meeting
with organizations or

00:01:18.590 --> 00:01:21.310
governments for the first time,
and they've heard of a

00:01:21.310 --> 00:01:27.120
randomized trial as a concept,
they often have a very well

00:01:27.120 --> 00:01:32.290
defined and narrowly defined
concept of what that means.

00:01:32.290 --> 00:01:35.150
And the fact is, there's a lot
of creative ways that we go

00:01:35.150 --> 00:01:39.090
about doing randomized trials
that adapt to different

00:01:39.090 --> 00:01:41.450
settings, because there's a lot
of situations in which you

00:01:41.450 --> 00:01:46.760
can't do what might be
considered the most standard

00:01:46.760 --> 00:01:49.330
prescription drug type
randomized trial.

00:01:49.330 --> 00:01:51.280
And so we have to be a little
bit creative in settings and

00:01:51.280 --> 00:01:53.880
understanding, what are the
constraints we're facing, and

00:01:53.880 --> 00:01:58.420
how can we adapt the methodology
to fit in this

00:01:58.420 --> 00:01:59.020
particular setting?

00:01:59.020 --> 00:02:01.840
Or maybe not, right?

00:02:01.840 --> 00:02:03.960
So that's going to be methods
of randomization, we'll go

00:02:03.960 --> 00:02:07.550
through a few of the key
approaches that we use.

00:02:07.550 --> 00:02:10.820
The second, and this is
a topic which really--

00:02:10.820 --> 00:02:14.080
to say this is topic one, topic
two, topic three isn't

00:02:14.080 --> 00:02:15.210
quite exactly right.

00:02:15.210 --> 00:02:17.660
And by the time we finish one,
we're going to have talked a

00:02:17.660 --> 00:02:20.590
lot about number two and number
three, a lot of the

00:02:20.590 --> 00:02:22.760
number two being gathering
support for evaluation.

00:02:22.760 --> 00:02:27.770
And the point here is that one
of the reasons why we choose

00:02:27.770 --> 00:02:30.540
one method over another when
we're thinking about how to go

00:02:30.540 --> 00:02:32.710
about setting up the design is
because some methods are going

00:02:32.710 --> 00:02:35.900
to be easier for gathering
support for evaluation, and so

00:02:35.900 --> 00:02:38.640
that's part of a back and
forth process with

00:02:38.640 --> 00:02:40.970
organizations and situations.

00:02:40.970 --> 00:02:44.090
And then we're going to try to
walk through a typical plan,

00:02:44.090 --> 00:02:45.340
so to speak.

00:02:48.370 --> 00:02:50.980
Perhaps in my mind, one of the
single most important things

00:02:50.980 --> 00:02:53.990
to remember about doing an
evaluation is to remember that

00:02:53.990 --> 00:02:58.370
we're not trying to just
ask, how did we do?

00:02:58.370 --> 00:03:00.660
There's nothing wrong with
asking that, but it's very

00:03:00.660 --> 00:03:01.550
short-sighted.

00:03:01.550 --> 00:03:04.840
What we should be asking
is, what should we do?

00:03:04.840 --> 00:03:08.240
And that's the point of a good
evaluation, is to guide us in

00:03:08.240 --> 00:03:09.490
future decisions.

00:03:11.460 --> 00:03:16.180
If you're a donor, and you're
running a huge initiative, and

00:03:16.180 --> 00:03:18.000
you're spending $20 million
or something--

00:03:18.000 --> 00:03:19.510
I'm just picking a number--

00:03:19.510 --> 00:03:21.230
and you want to do an
evaluation of this.

00:03:21.230 --> 00:03:23.950
But because of whatever the
nature of your program is,

00:03:23.950 --> 00:03:25.810
this is it, this is the only
time you're ever going to do

00:03:25.810 --> 00:03:29.090
it, and it's a weird program
that you believe in, but it's

00:03:29.090 --> 00:03:31.510
weird, and no one else is
ever going to do it.

00:03:31.510 --> 00:03:34.950
I realize it's kind of a weird
example, just go with me.

00:03:34.950 --> 00:03:39.020
And so that's a situation
in which, I think, most

00:03:39.020 --> 00:03:40.870
reasonable people would say,
why are you doing an

00:03:40.870 --> 00:03:41.500
evaluation?

00:03:41.500 --> 00:03:42.920
What's the point?

00:03:42.920 --> 00:03:44.880
Is it just to pat yourself
on the back?

00:03:44.880 --> 00:03:46.650
Is that really the goal?

00:03:46.650 --> 00:03:49.980
Because if there's not future
money that's at stake, future

00:03:49.980 --> 00:03:53.600
money that we have to decide
how are we going to spend,

00:03:53.600 --> 00:03:55.880
what's the point of doing all
this, other than just to see

00:03:55.880 --> 00:03:59.220
whether I made a good decision
in the past or not?

00:03:59.220 --> 00:04:03.190
But that's not really useful,
that's not why we're here.

00:04:03.190 --> 00:04:05.510
We're here because we realize
that there are tons of future

00:04:05.510 --> 00:04:08.570
decisions being made, and we
need better information in

00:04:08.570 --> 00:04:09.680
order to make those decisions.

00:04:09.680 --> 00:04:11.660
And we need those as donors,
but we also need those as

00:04:11.660 --> 00:04:12.310
organizations.

00:04:12.310 --> 00:04:14.110
And that's one of the key things
going back to point

00:04:14.110 --> 00:04:15.730
number two in the outline.

00:04:15.730 --> 00:04:18.990
How do you get organizations
on board and excited and

00:04:18.990 --> 00:04:20.040
involved in evaluation?

00:04:20.040 --> 00:04:22.280
It's when the evaluation is
actually able to speak to

00:04:22.280 --> 00:04:24.110
questions that they have.

00:04:24.110 --> 00:04:29.230
And so good evaluations often
help to identify the key

00:04:29.230 --> 00:04:31.220
implementer's questions
and answer them.

00:04:31.220 --> 00:04:34.060
And I know I'm making that sound
really simple, like oh,

00:04:34.060 --> 00:04:36.110
that's all we have to do.

00:04:36.110 --> 00:04:38.480
But the key really is to taking
that type of approach

00:04:38.480 --> 00:04:39.950
when working with
organizations.

00:04:39.950 --> 00:04:43.750
How do you turn these things
into win win for operations?

00:04:43.750 --> 00:04:46.420
If you're a leader in an
organization, and you're not a

00:04:46.420 --> 00:04:48.780
researcher, so you kind of
understand the value of

00:04:48.780 --> 00:04:51.560
research, and it sounds like a
good thing, but you're hired

00:04:51.560 --> 00:04:54.420
because you need to go and
deliver these services, and

00:04:54.420 --> 00:04:58.060
you need to be efficient in
delivering your services.

00:04:58.060 --> 00:05:02.280
You want to know that the
research is nice and good and

00:05:02.280 --> 00:05:05.860
needs to be done, but isn't
going to get in your way.

00:05:05.860 --> 00:05:07.800
Or if it is going to get in your
way, you're going to get

00:05:07.800 --> 00:05:09.320
something for it.

00:05:09.320 --> 00:05:12.470
And that's a very common
attitude, and I can respect

00:05:12.470 --> 00:05:14.890
that attitude, if someone is
just really focused on

00:05:14.890 --> 00:05:16.240
operations.

00:05:16.240 --> 00:05:19.100
And so the question is, how can
we design research, how

00:05:19.100 --> 00:05:21.630
can we listen to what the
operations people are saying

00:05:21.630 --> 00:05:24.640
about what their challenges are,
what their struggles are,

00:05:24.640 --> 00:05:26.800
the choices that they're
making that are tough?

00:05:26.800 --> 00:05:29.190
And actually build into the
research ways of helping them

00:05:29.190 --> 00:05:30.010
answer those questions.

00:05:30.010 --> 00:05:32.920
So that's something that
we often aim to do.

00:05:38.740 --> 00:05:39.470
Methods of randomization.

00:05:39.470 --> 00:05:42.140
I'm going to walk through four
different methods that we will

00:05:42.140 --> 00:05:45.220
often use: basic lottery, a
phase in, a rotation, and

00:05:45.220 --> 00:05:46.200
encouragement.

00:05:46.200 --> 00:05:48.700
And we'll talk about
each of these.

00:05:48.700 --> 00:05:50.480
These are not all mutually
exclusive

00:05:50.480 --> 00:05:51.470
methods, to be clear.

00:05:51.470 --> 00:05:53.800
So I'm just going to point out
there's different ways and

00:05:53.800 --> 00:05:56.446
levers for doing things.

00:05:56.446 --> 00:05:57.600
Is that readable?

00:05:57.600 --> 00:05:58.850
OK.

00:06:06.840 --> 00:06:09.740
Sorry, let me skip that slide,
and we'll come back to it at

00:06:09.740 --> 00:06:11.570
the end to recap.

00:06:11.570 --> 00:06:15.530
So let's start with the
simplest, which is a lottery.

00:06:15.530 --> 00:06:19.230
A lottery is like a clinical
trial, where if we were

00:06:19.230 --> 00:06:21.840
running a test for prescription
drugs, it would

00:06:21.840 --> 00:06:23.700
be a very standard, regimented
process.

00:06:23.700 --> 00:06:26.340
We'd be perhaps in
some hospital.

00:06:26.340 --> 00:06:29.270
We'd have some sort of intake
process with people who are in

00:06:29.270 --> 00:06:32.510
the hospital and have certain
criteria, and then we'd

00:06:32.510 --> 00:06:35.140
approach them and say,
there's a new drug.

00:06:35.140 --> 00:06:37.660
It's experimental.

00:06:37.660 --> 00:06:39.420
There's risks, there's
potential

00:06:39.420 --> 00:06:40.960
rewards, it's up to you.

00:06:40.960 --> 00:06:43.410
You have the following disease,
these are the issues

00:06:43.410 --> 00:06:44.030
you're facing.

00:06:44.030 --> 00:06:45.710
What do you want to do?

00:06:45.710 --> 00:06:47.360
But that's a situation
where we would take

00:06:47.360 --> 00:06:49.200
1,000 of the people.

00:06:49.200 --> 00:06:51.710
They would all get informed
consent being told about what

00:06:51.710 --> 00:06:52.470
the risks were.

00:06:52.470 --> 00:06:56.850
There would be parameters set
up so that if the outcomes

00:06:56.850 --> 00:06:59.270
were proving very decisive
one way or another,

00:06:59.270 --> 00:07:00.520
the study would end.

00:07:04.480 --> 00:07:06.420
But it's fairly straightforward
from a

00:07:06.420 --> 00:07:08.770
statistical standpoint and
from a research design

00:07:08.770 --> 00:07:09.290
standpoint.

00:07:09.290 --> 00:07:15.250
You bring them in, 1,000 are
entered into the study, 500 of

00:07:15.250 --> 00:07:17.790
them are randomly chosen to get
a pill, 500 are randomly

00:07:17.790 --> 00:07:21.040
chosen to get a placebo, and
you measure the outcomes,

00:07:21.040 --> 00:07:24.900
whatever they may be,
from getting the

00:07:24.900 --> 00:07:26.030
pill versus the placebo.

00:07:26.030 --> 00:07:31.920
And so the question is, can we
apply this in social science,

00:07:31.920 --> 00:07:33.610
outside the laboratory
type setting?

00:07:36.360 --> 00:07:39.160
So some of the constraints that
we face when we try to

00:07:39.160 --> 00:07:42.120
take that really simplistic
way of doing things.

00:07:42.120 --> 00:07:46.700
So the first is that
we often can't--

00:07:46.700 --> 00:07:49.510
when you're doing a randomized
trial on a prescription drug,

00:07:49.510 --> 00:07:51.350
the whole point is the
research study.

00:07:51.350 --> 00:07:53.200
That is what it is.

00:07:53.200 --> 00:07:54.880
There is no program
around this.

00:07:54.880 --> 00:07:56.390
It's a study to see what
the effectiveness of a

00:07:56.390 --> 00:07:57.870
particular pill is.

00:07:57.870 --> 00:08:01.250
If we're working with an
organization that is trying to

00:08:01.250 --> 00:08:03.410
do teacher training in schools,
trying to issue

00:08:03.410 --> 00:08:06.760
loans, trying to promote
savings, trying to teach

00:08:06.760 --> 00:08:09.990
agricultural practices,
there's a program

00:08:09.990 --> 00:08:10.740
there that's involved.

00:08:10.740 --> 00:08:16.010
And you can't just go in and
change things around in the

00:08:16.010 --> 00:08:20.130
program without paying attention
to what this

00:08:20.130 --> 00:08:24.270
actually means for the goals
of the program has.

00:08:24.270 --> 00:08:27.130
So the second really important
element is that it must be

00:08:27.130 --> 00:08:28.440
perceived as fair.

00:08:28.440 --> 00:08:34.500
Now the thing that's the single
most common situation

00:08:34.500 --> 00:08:38.640
that I find myself in is when
I'm dealing with organizations

00:08:38.640 --> 00:08:40.350
that are capacity constrained.

00:08:40.350 --> 00:08:42.970
They only have enough money
to go to 200 schools.

00:08:42.970 --> 00:08:44.780
They only have enough money
to make 1,000 loans.

00:08:44.780 --> 00:08:47.480
They only have enough marketing
people to visit

00:08:47.480 --> 00:08:50.990
5,000 households and promote
savings, or whatever the

00:08:50.990 --> 00:08:52.950
activity is they're doing,
there's some capacity

00:08:52.950 --> 00:08:55.150
constraint.

00:08:55.150 --> 00:08:58.240
And so one principle that I've
actually taught my children,

00:08:58.240 --> 00:09:00.650
very young, they all know--

00:09:00.650 --> 00:09:04.680
so like last week, we had to
fly, and I hadn't seen them

00:09:04.680 --> 00:09:05.100
for the weekend.

00:09:05.100 --> 00:09:08.080
So the kids all wanted
to sit next to me.

00:09:08.080 --> 00:09:09.760
I have three kids.

00:09:09.760 --> 00:09:12.990
And it was resolved
very simply,

00:09:12.990 --> 00:09:14.150
no whining, no nothing.

00:09:14.150 --> 00:09:17.120
We just randomly chose one.

00:09:17.120 --> 00:09:19.380
And they all knew immediately.

00:09:19.380 --> 00:09:22.710
Now in fairness, I've
done this before.

00:09:22.710 --> 00:09:27.550
But they knew instinctfully just
that this is-- there's no

00:09:27.550 --> 00:09:28.140
complaining here.

00:09:28.140 --> 00:09:29.460
There's no favoritism.

00:09:29.460 --> 00:09:31.320
We took each of their
boarding passes, we

00:09:31.320 --> 00:09:33.490
flipped them upside down.

00:09:33.490 --> 00:09:36.430
We chose one.

00:09:36.430 --> 00:09:38.160
Actually I said it backwards.

00:09:38.160 --> 00:09:39.770
It was one that couldn't
sit with me.

00:09:39.770 --> 00:09:42.040
It was two with me and one
without, so we chose the one

00:09:42.040 --> 00:09:43.920
who didn't.

00:09:43.920 --> 00:09:46.480
Now ironically, the two that did
win ended up both falling

00:09:46.480 --> 00:09:51.260
asleep immediately, and so I
switched seats in the end.

00:09:51.260 --> 00:09:52.700
But that's a different issue.

00:09:58.220 --> 00:10:01.370
But the point hopefully is
clear, that there's really, in

00:10:01.370 --> 00:10:04.990
some respect, nothing more fair
than a random process.

00:10:04.990 --> 00:10:10.620
It is giving everybody who is
eligible, who's aware or not

00:10:10.620 --> 00:10:12.840
aware, depending on how the set
up is, but everybody who

00:10:12.840 --> 00:10:18.060
is within a setting of--

00:10:18.060 --> 00:10:20.860
who has access to a program, and
giving them all the same

00:10:20.860 --> 00:10:23.060
chance of participating.

00:10:23.060 --> 00:10:26.420
Now in a lot of settings, we
would actually suggest that

00:10:26.420 --> 00:10:29.550
this is actually more fair than,
for instance, letting

00:10:29.550 --> 00:10:32.890
politics and letting nepotism
and letting any sort of other

00:10:32.890 --> 00:10:36.220
favoritism play in to
deciding who gets

00:10:36.220 --> 00:10:37.070
things and who doesn't.

00:10:37.070 --> 00:10:40.040
Which we all know, in many
places are serious issues for

00:10:40.040 --> 00:10:45.110
the allocation of any sort of
resource, that depending on

00:10:45.110 --> 00:10:48.450
how it happens, then that could
actually be the worst

00:10:48.450 --> 00:10:51.350
possible outcome that we
would want to see as

00:10:51.350 --> 00:10:54.760
philanthropists, as utilitarian
individuals

00:10:54.760 --> 00:10:59.120
interested first and foremost
in alleviating poverty.

00:10:59.120 --> 00:11:02.710
But you could tell other stories
in which there is

00:11:02.710 --> 00:11:06.970
processes where you really want
to reach the very poorest

00:11:06.970 --> 00:11:07.830
of the poor.

00:11:07.830 --> 00:11:10.655
And so you have to--

00:11:16.250 --> 00:11:18.210
let's come back to that
issue towards the end.

00:11:18.210 --> 00:11:22.620
I'll use that as an example
later in the lecture.

00:11:22.620 --> 00:11:26.220
So it must be politically
feasible.

00:11:26.220 --> 00:11:30.950
Now, it might not be politically
feasible if people

00:11:30.950 --> 00:11:33.290
who have the power in the local
settings are not willing

00:11:33.290 --> 00:11:36.820
to do it, because they want to
be able to choose the people

00:11:36.820 --> 00:11:40.270
in their networks to provide
the services to them.

00:11:40.270 --> 00:11:43.100
So the politically feasible
could be for the exact reason

00:11:43.100 --> 00:11:45.130
that we want to do
it randomly.

00:11:45.130 --> 00:11:47.350
It also can just be politically
infeasible for

00:11:47.350 --> 00:11:50.480
other interpersonal reasons.

00:11:50.480 --> 00:11:52.770
The right people who need to be
part of the decision making

00:11:52.770 --> 00:11:56.620
process just haven't really
brought into the value of an

00:11:56.620 --> 00:11:58.260
evaluation.

00:11:58.260 --> 00:12:00.600
And there are situations we face
all the time like that,

00:12:00.600 --> 00:12:02.520
where there's nothing
we can do.

00:12:02.520 --> 00:12:05.630
There's just someone who is the
decision maker who needs

00:12:05.630 --> 00:12:12.330
to be on board with something,
and is simply not on board

00:12:12.330 --> 00:12:15.550
with doing a rigorous
evaluation, and perhaps

00:12:15.550 --> 00:12:19.380
concerned about the results that
are going to come out of

00:12:19.380 --> 00:12:24.450
the project, in terms of is
suspicious of external

00:12:24.450 --> 00:12:25.080
evaluators.

00:12:25.080 --> 00:12:29.840
They feel like we know what
works and what doesn't, and

00:12:29.840 --> 00:12:33.620
they don't want to relax
control to an outsider.

00:12:33.620 --> 00:12:35.800
There's also obviously
situations where people just

00:12:35.800 --> 00:12:36.850
feel like what do
I have to lose--

00:12:36.850 --> 00:12:39.780
I'm sorry, what do
I have to gain?

00:12:39.780 --> 00:12:43.690
Where organizations will be of
the ilk that, look, we have

00:12:43.690 --> 00:12:45.510
lots of media, lots
of attention

00:12:45.510 --> 00:12:46.690
for what we're doing.

00:12:46.690 --> 00:12:48.790
Everybody tells us it's
a great idea.

00:12:48.790 --> 00:12:50.680
We're executing, we're
implementing.

00:12:50.680 --> 00:12:52.410
What do I have to gain from
doing an evaluation.

00:12:52.410 --> 00:12:54.150
If we absorbed more money right
now, I wouldn't know how

00:12:54.150 --> 00:12:55.280
to spend it.

00:12:55.280 --> 00:12:57.760
And there are organizations that
I've interacted with that

00:12:57.760 --> 00:12:59.830
are more or less of that ilk.

00:12:59.830 --> 00:13:04.670
And so there's clearly
situations like that.

00:13:04.670 --> 00:13:06.950
Must be ethical.

00:13:06.950 --> 00:13:10.900
So this is something we're
always very concerned about

00:13:10.900 --> 00:13:11.820
and cautious about.

00:13:11.820 --> 00:13:16.420
And there are many situations
we find ourselves for the

00:13:16.420 --> 00:13:19.170
reasons I mentioned above, in
terms of fairness, that the

00:13:19.170 --> 00:13:21.760
random process is arguably
the more ethical process.

00:13:21.760 --> 00:13:24.490
But they're clearly situations
that someone could put forward

00:13:24.490 --> 00:13:25.840
where you could say,
well, wait a

00:13:25.840 --> 00:13:27.300
second, that's not good.

00:13:27.300 --> 00:13:28.200
That's not right.

00:13:28.200 --> 00:13:30.470
So these are issues that one
has to be sensitive to.

00:13:30.470 --> 00:13:32.670
I think one of the things that's
most important here is

00:13:32.670 --> 00:13:34.770
not necessarily whether
something's-- we might all be

00:13:34.770 --> 00:13:37.510
able to analytically agree on
the ethics, but that doesn't

00:13:37.510 --> 00:13:40.480
mean that everybody else will
agree and perceive things the

00:13:40.480 --> 00:13:42.670
way we perceive them.

00:13:42.670 --> 00:13:44.760
And so even if we can
analytically understand that a

00:13:44.760 --> 00:13:48.840
random process is fair, if
someone is just bringing a

00:13:48.840 --> 00:13:52.260
certain bias to the table in
terms of the way they think

00:13:52.260 --> 00:13:55.210
about a random processes, then
this can be a problem.

00:13:55.210 --> 00:13:57.760
And again, this is one of
persuasion and personality

00:13:57.760 --> 00:14:00.240
more so than logic
and philosophy.

00:14:04.130 --> 00:14:09.210
So, why resource constraints are
an evaluators best friend.

00:14:09.210 --> 00:14:12.830
I think I've actually now said
most of this, but basically,

00:14:12.830 --> 00:14:15.450
most programs have limited
resources.

00:14:15.450 --> 00:14:18.940
And so examples of where this
has been done is training

00:14:18.940 --> 00:14:21.120
programs for entrepreneurs
or farmers.

00:14:21.120 --> 00:14:23.330
School vouchers is perhaps one
of the single most common

00:14:23.330 --> 00:14:26.430
examples we've seen this done,
where there's a government

00:14:26.430 --> 00:14:29.150
program to provide school
vouchers for private school or

00:14:29.150 --> 00:14:33.120
secondary school, or even
college, and they just can't

00:14:33.120 --> 00:14:35.760
educate the entire population
of people who want to go to

00:14:35.760 --> 00:14:37.130
private school.

00:14:37.130 --> 00:14:39.020
And so there's a voucher
program, there's an enrollment

00:14:39.020 --> 00:14:40.700
process, you apply.

00:14:40.700 --> 00:14:44.890
There's a public lottery
literally done openly through

00:14:44.890 --> 00:14:48.040
the newspapers or on TV.

00:14:48.040 --> 00:14:51.790
And that type of transparency
is done in order to make it

00:14:51.790 --> 00:14:53.960
politically feasible, so
that everybody can see.

00:14:53.960 --> 00:14:57.510
If it's not done that way, then
the concerns become about

00:14:57.510 --> 00:14:59.790
whether is really truly random,
it was done behind

00:14:59.790 --> 00:15:00.970
closed doors.

00:15:00.970 --> 00:15:04.060
Was the politician really just
kind of pulling out his

00:15:04.060 --> 00:15:05.310
favorite people?

00:15:11.390 --> 00:15:15.230
So lotteries become, in the
simplest of case, it's often

00:15:15.230 --> 00:15:15.810
the starting point.

00:15:15.810 --> 00:15:19.740
If we can pull off a public
lottery in this way, or

00:15:19.740 --> 00:15:23.620
private, either way, then it
really does become the

00:15:23.620 --> 00:15:24.870
simplest approach.

00:15:28.080 --> 00:15:30.140
So when it's possible,
it's nice.

00:15:30.140 --> 00:15:32.640
But there are clearly situations
that come up where

00:15:32.640 --> 00:15:33.890
this is not going
to be possible.

00:15:45.950 --> 00:15:48.740
So there's also flexible ways
of doing the lottery.

00:15:48.740 --> 00:15:50.820
So let's go through a few
different scenarios here.

00:15:55.910 --> 00:16:00.840
So first of all, there's often a
question about the unit over

00:16:00.840 --> 00:16:02.700
which you randomize.

00:16:02.700 --> 00:16:05.320
What we mean by this is, if
you're going to do a lottery,

00:16:05.320 --> 00:16:07.690
do you do a lottery at
the individual level?

00:16:07.690 --> 00:16:10.790
Or do you do a lottery, for
instance, at the village, by

00:16:10.790 --> 00:16:13.750
offering an entire village
a package of services?

00:16:13.750 --> 00:16:18.210
Or if you're building wells, for
instance, this is actually

00:16:18.210 --> 00:16:20.260
a treatment that's really being
done at the village, not

00:16:20.260 --> 00:16:21.080
the individual.

00:16:21.080 --> 00:16:22.830
There might be some individuals
who live closer to

00:16:22.830 --> 00:16:24.480
the well than others further,
but you're really choosing a

00:16:24.480 --> 00:16:27.930
village and building wells.

00:16:27.930 --> 00:16:31.140
And so the lottery system
is flexible in this way.

00:16:31.140 --> 00:16:33.750
The general concept
is the same.

00:16:33.750 --> 00:16:38.330
We've done interventions in
Ghana, for instance, where the

00:16:38.330 --> 00:16:42.320
randomization was done at the
village level for receiving a

00:16:42.320 --> 00:16:45.080
community driven development
type program.

00:16:45.080 --> 00:16:48.440
And so different geographic
clusters were all put into a

00:16:48.440 --> 00:16:51.270
public lottery, and half of
them randomly chosen to

00:16:51.270 --> 00:16:55.150
receive financial resources
in order to help build an

00:16:55.150 --> 00:16:59.660
epicenter, and lots of labor
in terms of helping and

00:16:59.660 --> 00:17:03.850
guiding and facilitating the
process for building these

00:17:03.850 --> 00:17:05.100
quote, epicenters.

00:17:07.960 --> 00:17:11.400
So the second point is that
sometimes when we do

00:17:11.400 --> 00:17:13.940
lotteries, sometimes it's kind
of a full versus partial.

00:17:13.940 --> 00:17:20.810
What we mean by this is whether
it's done all at once

00:17:20.810 --> 00:17:23.240
in one kind of public way, or
whether it's done through an

00:17:23.240 --> 00:17:25.150
ongoing process.

00:17:25.150 --> 00:17:27.839
And depending on how things
are set up, sometimes it's

00:17:27.839 --> 00:17:28.820
done one way or the other.

00:17:28.820 --> 00:17:31.960
So the school vouchers is an
example where we would do it

00:17:31.960 --> 00:17:33.590
all at the very same time.

00:17:33.590 --> 00:17:35.440
There's a whole bunch of people
who apply for a school

00:17:35.440 --> 00:17:36.670
voucher program.

00:17:36.670 --> 00:17:39.190
There's 5,000 applicants,
there's 2,000 school vouchers,

00:17:39.190 --> 00:17:42.380
we randomly choose 2,000, they
get the school vouchers.

00:17:42.380 --> 00:17:45.610
In a situation that we've done
in South Africa and also the

00:17:45.610 --> 00:17:48.800
Philippines, when we build it
into a credit scoring process.

00:17:48.800 --> 00:17:53.570
And so every day, there's 10
applications for a loan.

00:17:53.570 --> 00:17:56.030
The bank is not going to
issue all the loans.

00:17:56.030 --> 00:17:57.770
They take those 10, they
put them in three

00:17:57.770 --> 00:18:00.730
piles, yes, no and maybe.

00:18:00.730 --> 00:18:03.250
And then what happens is they
take the maybes and they say,

00:18:03.250 --> 00:18:05.150
we only want to make
half of these.

00:18:05.150 --> 00:18:08.120
And then we randomize which half
of those get credit and

00:18:08.120 --> 00:18:09.270
which do not.

00:18:09.270 --> 00:18:11.430
And this is something we've done
now a few times in order

00:18:11.430 --> 00:18:12.630
to measure the impact
of credit.

00:18:12.630 --> 00:18:14.560
From the bank's perspective,
this is exactly one of the

00:18:14.560 --> 00:18:16.500
cases I was talking about before
where there's a win win

00:18:16.500 --> 00:18:17.900
from an operations standpoint.

00:18:17.900 --> 00:18:20.780
This is a bank who says, these
are genuine maybes.

00:18:20.780 --> 00:18:23.110
We don't know if we should be
lending to them or not.

00:18:23.110 --> 00:18:24.760
We're not sure if
it's profitable

00:18:24.760 --> 00:18:28.110
for us, we're a bank.

00:18:28.110 --> 00:18:30.740
So this is a method for them
that helps mitigate their

00:18:30.740 --> 00:18:34.750
risks in deciding what their
portfolio should

00:18:34.750 --> 00:18:36.710
look like as a whole.

00:18:36.710 --> 00:18:39.320
And from our perspective,
provides this nice lottery

00:18:39.320 --> 00:18:41.360
system where some people are
randomly assigned to get

00:18:41.360 --> 00:18:42.610
credit and others not.

00:18:47.300 --> 00:18:49.150
So let's go back, and now let's
think about some of the

00:18:49.150 --> 00:18:52.190
things that will often happen
when you do a lottery design.

00:18:52.190 --> 00:18:54.050
Suppose you have 500
applicants and

00:18:54.050 --> 00:18:57.370
you have 500 slots.

00:18:57.370 --> 00:18:59.800
At first glance, you might think
you're kind of screwed.

00:18:59.800 --> 00:19:02.760
You set up this nice, big
process, and lo and behold, it

00:19:02.760 --> 00:19:04.470
turns out you didn't have
over-subscription.

00:19:04.470 --> 00:19:05.710
You thought you did.

00:19:05.710 --> 00:19:07.810
You thought you were going to
use this over-subscription to

00:19:07.810 --> 00:19:10.620
randomize who gets in
and who does not.

00:19:10.620 --> 00:19:12.180
So what can you do in this
type of situation?

00:19:14.750 --> 00:19:18.140
So there's some low hanging
fruit type answers, and

00:19:18.140 --> 00:19:20.820
there's also a possibility
that this wouldn't work.

00:19:20.820 --> 00:19:25.550
But the first is, could you
increase the outreach

00:19:25.550 --> 00:19:27.690
activities?

00:19:27.690 --> 00:19:29.960
That a lot of times in this
situation, what this really

00:19:29.960 --> 00:19:32.570
means is that whatever was done
to market this program

00:19:32.570 --> 00:19:35.340
was not effective in getting in
the right number of people.

00:19:35.340 --> 00:19:38.230
Because the intent was to get
1,000 or 2,000 people

00:19:38.230 --> 00:19:40.830
applying, and only
500 applied.

00:19:40.830 --> 00:19:43.490
So now think about, from an
operations standpoint, what

00:19:43.490 --> 00:19:45.810
does that tell you
about applying?

00:19:45.810 --> 00:19:49.610
And that might be one of those
cases where there's actually

00:19:49.610 --> 00:19:51.370
useful, interesting research
for the organization.

00:19:51.370 --> 00:19:53.370
If the organization was saying,
we're going to get

00:19:53.370 --> 00:19:56.930
2,000 people applying for this,
and then only get 500,

00:19:56.930 --> 00:19:59.580
well, it tells you that maybe
you could do something to help

00:19:59.580 --> 00:20:02.530
them learn how they can get
their marketing up.

00:20:02.530 --> 00:20:04.500
And so then you can test out
different approaches for

00:20:04.500 --> 00:20:08.430
marketing that helps the
operations learn more about

00:20:08.430 --> 00:20:11.050
what is it that brings in people
to apply for these

00:20:11.050 --> 00:20:14.180
scholarships, or whatever it is
that's being done, and at

00:20:14.180 --> 00:20:17.385
the same time, benefits the
program from increasing--

00:20:17.385 --> 00:20:20.150
I'm sorry, benefits the
evaluation from getting a

00:20:20.150 --> 00:20:21.400
larger intake.

00:20:34.320 --> 00:20:41.110
The risk with this is that you
end up bringing people into

00:20:41.110 --> 00:20:44.280
the program who weren't really
part of the target program.

00:20:44.280 --> 00:20:47.590
So if that's the answer, then
this is actually a really bad

00:20:47.590 --> 00:20:50.690
idea to go out and do more
extensive marketing.

00:20:50.690 --> 00:20:53.260
If you have to go through extra
leaps and bounds in

00:20:53.260 --> 00:20:56.190
order to bring people in, so
much so that it changes the

00:20:56.190 --> 00:20:59.460
nature of what the program is,
that's a situation where you

00:20:59.460 --> 00:21:01.230
might want to go back to the
drawing board and think again

00:21:01.230 --> 00:21:04.830
about what the right
answer is.

00:21:10.330 --> 00:21:14.180
Suppose there's 2,000
applicants, and suppose in the

00:21:14.180 --> 00:21:16.950
process of doing this that the
organization that's doing

00:21:16.950 --> 00:21:18.200
things says, you know what?

00:21:20.790 --> 00:21:25.810
There's 500 worthy candidates
and there's 500 slots.

00:21:25.810 --> 00:21:28.680
So a simple lottery
would not work.

00:21:28.680 --> 00:21:33.530
So we have some sort of
screening process, and we

00:21:33.530 --> 00:21:34.220
ranked them.

00:21:34.220 --> 00:21:36.040
And we can rank them, and we
put numbers by everyone.

00:21:36.040 --> 00:21:40.030
So why should we do anything
other than just

00:21:40.030 --> 00:21:42.840
taking the top 500?

00:21:42.840 --> 00:21:46.450
So when that type of thing
happens, a lot of times the

00:21:46.450 --> 00:21:51.670
questions that we want to ask
then are about what was the

00:21:51.670 --> 00:21:54.240
screening process that
was being used here?

00:21:54.240 --> 00:21:57.300
So let's go back to the credit
scoring study that I was

00:21:57.300 --> 00:21:58.930
referring to.

00:21:58.930 --> 00:22:01.190
The thing that we're struck by
when we've done this credit

00:22:01.190 --> 00:22:03.820
scoring is really how wide that
maybe category is, when

00:22:03.820 --> 00:22:05.820
you really get into the nuts and
bolts in talking with the

00:22:05.820 --> 00:22:10.510
lenders about where their data
are coming from, and the

00:22:10.510 --> 00:22:14.260
quality of the data, and how
they're individual judgment

00:22:14.260 --> 00:22:17.510
weighs in to influence where
people fall within those three

00:22:17.510 --> 00:22:19.320
buckets, the yes, no, maybe.

00:22:19.320 --> 00:22:23.570
And the fact is that the
profitability of the person at

00:22:23.570 --> 00:22:25.750
the high end of the maybe is
really not much different, if

00:22:25.750 --> 00:22:28.700
at all, from the low
end of the maybe.

00:22:28.700 --> 00:22:33.100
And so the point is that when
you get inside and figure out,

00:22:33.100 --> 00:22:35.530
what is it that's really going
on that made them say, well,

00:22:35.530 --> 00:22:37.320
we had 2,000 applicants
and we had 500

00:22:37.320 --> 00:22:38.820
eligible, so we're done.

00:22:38.820 --> 00:22:40.610
If you got inside the box a
little bit more and talked

00:22:40.610 --> 00:22:43.760
with them about how they got
to those 500, things would

00:22:43.760 --> 00:22:46.750
start coming out of that process
that might be the

00:22:46.750 --> 00:22:49.400
exact areas were you can
say, why is that

00:22:49.400 --> 00:22:51.300
necessary as a criteria?

00:22:51.300 --> 00:22:54.010
Is that really something you
want to filter on to bring

00:22:54.010 --> 00:22:55.800
people in or out?

00:22:59.450 --> 00:23:00.700
Yeah?

00:23:03.490 --> 00:23:05.990
AUDIENCE: Let's say your
criteria are, one is gender

00:23:05.990 --> 00:23:10.468
and one is educational level
for the purpose here.

00:23:13.324 --> 00:23:16.850
And let's say that, for whatever
reason, you can't be

00:23:16.850 --> 00:23:19.300
sure that the educational
level is really

00:23:19.300 --> 00:23:20.994
what they say it is.

00:23:25.500 --> 00:23:28.453
So you could go ahead and
select on the basis of

00:23:28.453 --> 00:23:31.000
educational level, knowing
that maybe people are not

00:23:31.000 --> 00:23:34.370
telling the truth and
that would be OK.

00:23:34.370 --> 00:23:40.060
You could remove that as a
criteria entirely, or you

00:23:40.060 --> 00:23:42.690
could find some other
kind of proxy.

00:23:42.690 --> 00:23:43.670
That might work, right?

00:23:43.670 --> 00:23:44.080
PROFESSOR: Right.

00:23:44.080 --> 00:23:45.346
AUDIENCE: OK.

00:23:45.346 --> 00:23:48.200
Kind of random.

00:23:48.200 --> 00:23:50.040
PROFESSOR: But that's exactly
kind of one of the key things

00:23:50.040 --> 00:23:50.600
that would come out.

00:23:50.600 --> 00:23:53.140
So the point is, let's just go
with the middle example.

00:23:53.140 --> 00:23:57.500
If you really think that the
education was what you wanted

00:23:57.500 --> 00:23:59.620
to screen on, but you don't have
confidence in what you're

00:23:59.620 --> 00:24:03.660
looking at as actually being a
reliable measure of education.

00:24:03.660 --> 00:24:06.350
But yet that's causing
a filter to draw

00:24:06.350 --> 00:24:08.220
you down to the 500.

00:24:08.220 --> 00:24:10.840
But then when you get inside
and you realize this is a

00:24:10.840 --> 00:24:12.250
really bad measure
of education, why

00:24:12.250 --> 00:24:13.330
are we using this?

00:24:13.330 --> 00:24:15.210
And then all of the sudden you
relax that one rule and you're

00:24:15.210 --> 00:24:20.130
up to 900 people, it tells you
that maybe this isn't such a

00:24:20.130 --> 00:24:22.810
good way to be filtering, and
I should be doing the 900.

00:24:22.810 --> 00:24:24.610
And while I'm at it, maybe I
should try to think about how

00:24:24.610 --> 00:24:26.788
to measure education better
or something.

00:24:30.030 --> 00:24:39.530
So let's take another kind of
example, which is-- and this

00:24:39.530 --> 00:24:43.070
actually mimics the story I have
up here about training,

00:24:43.070 --> 00:24:44.750
but the credit scoring
example is another

00:24:44.750 --> 00:24:47.650
example exactly of this.

00:24:47.650 --> 00:24:49.350
Where you end up--

00:24:49.350 --> 00:24:52.110
and I said that right
here, sorry--

00:24:52.110 --> 00:24:55.230
you can think about having two
different piles of people

00:24:55.230 --> 00:24:58.050
that, when you say there's 500
that are eligible, really

00:24:58.050 --> 00:25:02.350
maybe what you have is not 500
that are eligible, but you had

00:25:02.350 --> 00:25:03.950
200 that are really eligible.

00:25:03.950 --> 00:25:06.550
You really want them in the
program, they must be there.

00:25:06.550 --> 00:25:09.990
And then the next 300, you're a
little more questionable on.

00:25:09.990 --> 00:25:12.310
And the next 300 really weren't
so different from the

00:25:12.310 --> 00:25:13.890
300 after those.

00:25:13.890 --> 00:25:17.530
And so what you can do in that
type of situation is set up a

00:25:17.530 --> 00:25:19.890
process where you say look, if
you have certain eligibility

00:25:19.890 --> 00:25:21.520
requirements, you're in.

00:25:21.520 --> 00:25:24.080
And then you're also not
part of the evaluation.

00:25:24.080 --> 00:25:27.340
And it's the next 300 that we're
going to combine with

00:25:27.340 --> 00:25:29.300
the following 300 after that,
we're going to put together a

00:25:29.300 --> 00:25:32.600
pool of 600, and we're going
to randomize those.

00:25:32.600 --> 00:25:36.660
Now this has a clear benefit
and a clear cost.

00:25:36.660 --> 00:25:41.670
The benefit is that you can
now get a very nice, clean

00:25:41.670 --> 00:25:43.900
estimate on the impact
of those 600.

00:25:43.900 --> 00:25:46.310
The cost is we've changed the
research question here.

00:25:46.310 --> 00:25:48.310
We've changed the evaluation
question.

00:25:48.310 --> 00:25:51.200
We're no longer answering the
question, what is the impact

00:25:51.200 --> 00:25:54.220
on everyone who receives
this service?

00:25:54.220 --> 00:25:55.180
And that's not a good thing.

00:25:55.180 --> 00:25:57.120
We don't want to lead
with methodology and

00:25:57.120 --> 00:25:59.610
then force fit questions.

00:25:59.610 --> 00:26:00.870
We want to set the research
questions.

00:26:00.870 --> 00:26:06.040
We have to ask ourselves, how
much are we losing by only

00:26:06.040 --> 00:26:07.210
studying those individuals?

00:26:07.210 --> 00:26:09.480
And in some settings, those are
the exact individuals you

00:26:09.480 --> 00:26:11.110
want to study.

00:26:11.110 --> 00:26:13.720
But in some, maybe that's
not so the case.

00:26:13.720 --> 00:26:16.360
So in the credit scoring, I
think of those as the exact

00:26:16.360 --> 00:26:17.130
people we want to study.

00:26:17.130 --> 00:26:19.610
Because when we think about
programs that expand access to

00:26:19.610 --> 00:26:22.100
credit, what we're doing is
we're talking about those

00:26:22.100 --> 00:26:24.080
people on the bubble, and we're
talking about ways of

00:26:24.080 --> 00:26:26.730
getting them access that they
didn't have otherwise.

00:26:26.730 --> 00:26:28.440
And the people who have really,
really good credit

00:26:28.440 --> 00:26:31.230
scores and are very credit
worthy, they're not the ones

00:26:31.230 --> 00:26:32.390
we're thinking about
when we think about

00:26:32.390 --> 00:26:35.980
expanding access to credit.

00:26:35.980 --> 00:26:38.720
So let me give you another
example of one where this

00:26:38.720 --> 00:26:40.360
would be bad.

00:26:40.360 --> 00:26:42.630
So we are doing these programs
about targeting the

00:26:42.630 --> 00:26:45.600
ultra-poor, where we
go into countries--

00:26:45.600 --> 00:26:47.200
I'm sorry, we do go
into countries,

00:26:47.200 --> 00:26:49.750
but we go into villages--

00:26:49.750 --> 00:26:55.270
and we first identify the 20
poorest people in the village.

00:26:55.270 --> 00:26:59.430
And then of those 20, we hold
a lottery, and 10 receive

00:26:59.430 --> 00:27:01.760
services and 10 do not.

00:27:01.760 --> 00:27:03.890
And then we do this in
about 30 villages.

00:27:03.890 --> 00:27:06.100
And the organizations we're
working with, it's exactly the

00:27:06.100 --> 00:27:07.170
setting I'm describing here.

00:27:07.170 --> 00:27:08.550
The organizations we're
working with

00:27:08.550 --> 00:27:09.750
have a fixed budget.

00:27:09.750 --> 00:27:12.970
They can provide services in
each situation to 1,200

00:27:12.970 --> 00:27:14.990
people, and that's it.

00:27:14.990 --> 00:27:20.280
And if we went into
120 villages, they

00:27:20.280 --> 00:27:21.940
can do 10 per village.

00:27:21.940 --> 00:27:24.650
If we went into 60 villages,
they could do 20.

00:27:24.650 --> 00:27:27.490
But either way they look at it,
they got 1,200 people that

00:27:27.490 --> 00:27:29.730
they're going to be able to
provide these services to.

00:27:29.730 --> 00:27:31.866
It's going to be an asset
transfer, they're going be on

00:27:31.866 --> 00:27:35.700
goats and training and
consumption bundles.

00:27:35.700 --> 00:27:40.820
Now, if in some these villages
people said,

00:27:40.820 --> 00:27:42.920
well, wait a second.

00:27:42.920 --> 00:27:46.800
We have the 20 poorest people
in the village, yes.

00:27:46.800 --> 00:27:50.100
But four of these people are
really just standout poor.

00:27:50.100 --> 00:27:53.140
They're much, much poorer
than everybody else.

00:27:53.140 --> 00:27:57.210
So we want to exclude the four
very, very, very poorest and

00:27:57.210 --> 00:27:59.440
make sure that they get it with
certainty, and then only

00:27:59.440 --> 00:28:01.450
evaluate the other 16.

00:28:01.450 --> 00:28:04.160
So this would actually be a bad
thing from the evaluation

00:28:04.160 --> 00:28:04.550
perspective.

00:28:04.550 --> 00:28:06.580
It might be the right thing to
do, and we can talk about what

00:28:06.580 --> 00:28:08.440
the trade offs are.

00:28:08.440 --> 00:28:11.100
We're not actually doing that,
and I'll explain why.

00:28:11.100 --> 00:28:13.280
But from an evaluation
perspective, that would be now

00:28:13.280 --> 00:28:15.650
changing the research question
in a bad way, as compared to

00:28:15.650 --> 00:28:16.870
the credit scoring, where
I was arguing

00:28:16.870 --> 00:28:18.460
that it's a good change.

00:28:18.460 --> 00:28:19.060
So why?

00:28:19.060 --> 00:28:20.770
Well, because this is a problem
where we really do

00:28:20.770 --> 00:28:23.830
actually want to know the impact
on those bottom four as

00:28:23.830 --> 00:28:25.800
well as the next 16.

00:28:25.800 --> 00:28:28.070
And so if we did something where
we just allowed the very

00:28:28.070 --> 00:28:31.010
bottom four to get the program
with certainty, and then only

00:28:31.010 --> 00:28:33.170
evaluated the next 16, we're
missing an important part of

00:28:33.170 --> 00:28:34.910
the sample frame.

00:28:34.910 --> 00:28:37.580
Now, the reason why, in these
settings, we've done it as the

00:28:37.580 --> 00:28:40.730
full 20 is because
realistically, when we've

00:28:40.730 --> 00:28:43.180
actually gone into these
villages and done this type of

00:28:43.180 --> 00:28:47.280
exercise, it's difficult, if
not impossible, to get

00:28:47.280 --> 00:28:49.950
consensus that there's really
four people that stand out.

00:28:49.950 --> 00:28:51.910
And the fact is, even when
you're measuring poverty--

00:28:51.910 --> 00:28:54.230
and we do have some objective
things, we all know that

00:28:54.230 --> 00:28:56.370
there's lots of components
to poverty.

00:28:56.370 --> 00:28:58.100
There's lots of ways
of measuring it.

00:28:58.100 --> 00:29:01.340
It's not even just consumption
this month, its vulnerability

00:29:01.340 --> 00:29:04.290
in general as a concept, which
could be about the variation

00:29:04.290 --> 00:29:06.630
over time, and how vulnerable
you are, and how your social

00:29:06.630 --> 00:29:09.300
networks are for helping
you absorb shocks.

00:29:09.300 --> 00:29:11.180
There's just many different ways
of measuring it, and it's

00:29:11.180 --> 00:29:14.140
unrealistic to think then we
can go into a village and

00:29:14.140 --> 00:29:17.140
really draw these amazingly fine
lines to say these people

00:29:17.140 --> 00:29:19.250
are standout different
than the rest.

00:29:19.250 --> 00:29:20.855
When you're going into a village
of this size and

00:29:20.855 --> 00:29:24.140
you're finding the 20 poorest,
those 20 are statistically

00:29:24.140 --> 00:29:26.410
indistinguishable from each
other, more or less.

00:29:26.410 --> 00:29:30.440
So that's the philosophy
of the organizations.

00:29:30.440 --> 00:29:33.016
And that's the reason
for doing it the

00:29:33.016 --> 00:29:34.266
way we're doing it.

00:29:43.150 --> 00:29:49.310
So sometimes, exclusion
is not desirable.

00:29:49.310 --> 00:29:52.990
Sometimes there's no way,
whatever the context is, of

00:29:52.990 --> 00:29:54.240
excluding individuals.

00:29:56.880 --> 00:30:00.840
So other points that we will
often use is the expansion of

00:30:00.840 --> 00:30:04.310
a program, or any sort of
program where there's some

00:30:04.310 --> 00:30:06.470
sort of initial stage, and
then what you're doing is

00:30:06.470 --> 00:30:08.690
you're simply randomizing where
that initial stage is,

00:30:08.690 --> 00:30:11.380
or where the program
expands into.

00:30:11.380 --> 00:30:16.040
A program is only going to go to
so many villages, and they

00:30:16.040 --> 00:30:19.380
can't exclude, but they can
control where their

00:30:19.380 --> 00:30:22.550
individuals, where their credit
officers, where their

00:30:22.550 --> 00:30:25.630
program officers, education
trainers, whatever it is that

00:30:25.630 --> 00:30:28.030
they're doing, they obviously
can control what villages they

00:30:28.030 --> 00:30:29.280
go to and what they do not.

00:30:31.670 --> 00:30:36.400
So when there is a process
like this, where there's

00:30:36.400 --> 00:30:39.690
gradual expansion, we often
will think about doing a

00:30:39.690 --> 00:30:42.620
program evaluation using
a phase in approach.

00:30:42.620 --> 00:30:49.040
A phase in approach basically
says, look, we're going to go

00:30:49.040 --> 00:30:54.780
to all 1,200 of these villages
over the next three years, but

00:30:54.780 --> 00:30:58.630
we can randomize which ones
we go to in each year.

00:30:58.630 --> 00:31:01.580
So everybody's going to get
services in the long run.

00:31:01.580 --> 00:31:04.560
Now one thing that's nice about
that is, particularly

00:31:04.560 --> 00:31:06.600
when there's community led
interventions, there's

00:31:06.600 --> 00:31:13.110
oftentimes a desire to have
some involvement from

00:31:13.110 --> 00:31:14.540
everybody in the program
at some point

00:31:14.540 --> 00:31:16.150
in time in the program.

00:31:16.150 --> 00:31:19.650
And this is often something that
organizations do ask for.

00:31:19.650 --> 00:31:22.790
So phase in approaches allow
for that more naturally,

00:31:22.790 --> 00:31:25.060
because everybody's going
to be receiving a

00:31:25.060 --> 00:31:26.650
service at some point.

00:31:26.650 --> 00:31:27.630
Similarly, the rotation--

00:31:27.630 --> 00:31:30.090
which I'll mention in a second--
is a very similar

00:31:30.090 --> 00:31:31.510
twist on the phase in.

00:31:31.510 --> 00:31:36.770
Where the rotation, instead of
having it be where you slowly

00:31:36.770 --> 00:31:39.120
phase in a process, it's a
process where everybody's

00:31:39.120 --> 00:31:41.510
actually receiving a service
at all points in time, and

00:31:41.510 --> 00:31:43.692
we're just randomizing
who gets what.

00:31:43.692 --> 00:31:45.900
So we'll talk about some
examples in a moment about

00:31:45.900 --> 00:31:47.150
when that can work.

00:31:53.880 --> 00:31:56.280
So some key advantages is that
everyone gets something

00:31:56.280 --> 00:31:59.905
eventually, and this provides
incentives to maintain contact

00:31:59.905 --> 00:32:03.570
as well with control villages.

00:32:03.570 --> 00:32:06.180
They're not just participating
in surveys, although we do do

00:32:06.180 --> 00:32:08.130
a lot of surveys sometimes where
there is no intervention

00:32:08.130 --> 00:32:10.640
related, and people are
often more than--

00:32:10.640 --> 00:32:12.710
at least in our experience--
more than happy to participate

00:32:12.710 --> 00:32:15.070
in this interesting, weird
thing, with these people

00:32:15.070 --> 00:32:17.210
coming and asking us all
these questions.

00:32:17.210 --> 00:32:19.320
But having said that, there are
situations where you want

00:32:19.320 --> 00:32:22.120
that continuous support and
continuous buy in to what

00:32:22.120 --> 00:32:25.070
we're doing, and so the
advantage is that it provides

00:32:25.070 --> 00:32:28.370
them some incentive to
maintain contact.

00:32:28.370 --> 00:32:30.510
Some of the concerns is that it
does complicate estimating

00:32:30.510 --> 00:32:31.720
long line effects.

00:32:31.720 --> 00:32:34.440
If the goal was to study
something over 10 years, but

00:32:34.440 --> 00:32:37.580
everybody got phased in at the
end of three, well, you can't

00:32:37.580 --> 00:32:39.160
study the 10 year effects.

00:32:39.160 --> 00:32:42.260
You can study the effect of
getting something for two more

00:32:42.260 --> 00:32:44.930
years 10 years later, but
that's not nearly as

00:32:44.930 --> 00:32:47.480
interesting as asking what the
10 year effect is from getting

00:32:47.480 --> 00:32:48.730
a particular service.

00:32:51.260 --> 00:32:53.690
So let me give you an example of
a situation where the first

00:32:53.690 --> 00:32:56.230
is actually perfectly
interesting.

00:32:56.230 --> 00:32:58.780
Take neonatal care.

00:32:58.780 --> 00:33:00.940
If you want to study the effect
of neonatal care, doing

00:33:00.940 --> 00:33:03.400
a phase in across villages is
perfectly fine, because this

00:33:03.400 --> 00:33:07.060
is something that is going
to be affecting infants.

00:33:07.060 --> 00:33:11.110
And so when you provide that
neonatal care, and you do this

00:33:11.110 --> 00:33:14.170
as a phase in, you're now
studying those children, and

00:33:14.170 --> 00:33:15.050
this is perfectly fine.

00:33:15.050 --> 00:33:18.640
You can study now the effect
of the neonatal care over a

00:33:18.640 --> 00:33:20.930
10, 15 year horizon,
even though it was

00:33:20.930 --> 00:33:21.640
phased in to everybody.

00:33:21.640 --> 00:33:24.160
Because once it's phased in, the
control area of those kids

00:33:24.160 --> 00:33:26.510
are three, four, five years old,
and so it doesn't apply.

00:33:31.630 --> 00:33:34.490
So I think the main thing to
talk about in terms of phase

00:33:34.490 --> 00:33:38.410
ins that does become an issue
about expectations.

00:33:38.410 --> 00:33:41.020
So let me give you the
simplest example.

00:33:41.020 --> 00:33:45.100
In the world of credit, this
is usually something that

00:33:45.100 --> 00:33:47.710
concerns me a lot when we're
talking about doing studies.

00:33:47.710 --> 00:33:50.580
And in fact, I've personally
been involved in studies where

00:33:50.580 --> 00:33:54.530
there was a proposal to do a
phased in credit program and

00:33:54.530 --> 00:33:55.860
we said no.

00:33:55.860 --> 00:33:59.060
We didn't do it, because we were
very concerned about what

00:33:59.060 --> 00:34:03.470
happens to a control group
individual who was told,

00:34:03.470 --> 00:34:05.880
you're going to get a loan,
but just please wait six

00:34:05.880 --> 00:34:07.850
months, or year, or year and a
half, or two years, whatever

00:34:07.850 --> 00:34:08.719
the amount is.

00:34:08.719 --> 00:34:10.889
Now the longer you have to wait,
the less it's an issue.

00:34:10.889 --> 00:34:15.710
But if it's a relatively short
period of time, like a year,

00:34:15.710 --> 00:34:17.360
then the question is, well,
what were they going to do

00:34:17.360 --> 00:34:19.260
with that money?

00:34:19.260 --> 00:34:22.600
And is it something they're
willing to wait a year for?

00:34:22.600 --> 00:34:26.659
And if the answer is yes, well
then this is a real problem

00:34:26.659 --> 00:34:29.270
for thinking that this is a
valid control group, because

00:34:29.270 --> 00:34:38.620
what it says is, we're going to
see a delay in an activity

00:34:38.620 --> 00:34:40.480
specifically because of
getting access to

00:34:40.480 --> 00:34:43.480
this loan in one year.

00:34:43.480 --> 00:34:45.980
So what it says is they did
have other options.

00:34:45.980 --> 00:34:47.929
They were a little bit more
expensive perhaps, a little

00:34:47.929 --> 00:34:51.719
more costly either in
time or interest.

00:34:51.719 --> 00:34:52.980
And they said, you know what?

00:34:52.980 --> 00:34:54.550
I don't need to build
that new roof now.

00:34:54.550 --> 00:34:56.360
I don't need to buy that new
sewing machine for my

00:34:56.360 --> 00:34:56.870
enterprise.

00:34:56.870 --> 00:34:59.860
I'll go ahead and just put it
off, and I'll do it in a year.

00:34:59.860 --> 00:35:01.940
And what it would do is it would
lead us to overestimate

00:35:01.940 --> 00:35:03.860
the impact of getting access
to credit, whereas if they

00:35:03.860 --> 00:35:06.350
weren't promised this good
loan in the year, they

00:35:06.350 --> 00:35:07.800
would've just borrowed at
a little bit of a higher

00:35:07.800 --> 00:35:09.240
interest rate right now.

00:35:09.240 --> 00:35:10.610
And they still would have made
the investment in their

00:35:10.610 --> 00:35:13.260
business, they just would have
had a higher interest cost

00:35:13.260 --> 00:35:15.500
that would cause us to
overestimate the impact of our

00:35:15.500 --> 00:35:18.230
program, because
we would only--

00:35:18.230 --> 00:35:20.890
the true impact is just really
a savings in interest, not

00:35:20.890 --> 00:35:23.650
about access to credit in
a binary sense, but

00:35:23.650 --> 00:35:24.960
just about the price.

00:35:24.960 --> 00:35:27.610
But yet what we would then find
ourselves measuring is

00:35:27.610 --> 00:35:31.770
seeing a treatment group of
people who fix their homes and

00:35:31.770 --> 00:35:34.310
bought sewing machines for
their businesses, and a

00:35:34.310 --> 00:35:36.850
control that didn't, and we
would think that aha, there

00:35:36.850 --> 00:35:39.020
was a real binary constraint
here where people were

00:35:39.020 --> 00:35:42.130
actually held back from getting
access to credit.

00:35:42.130 --> 00:35:44.245
And this program didn't have
that positive effect.

00:35:47.890 --> 00:35:50.330
So a rotation design.

00:35:50.330 --> 00:35:53.740
So a rotation design is
basically groups getting

00:35:53.740 --> 00:35:54.520
treatment in turns.

00:35:54.520 --> 00:35:56.790
Group A gets treatment in the
first period, group B gets

00:35:56.790 --> 00:35:59.110
treatment in the second.

00:35:59.110 --> 00:36:01.950
The main advantage is it's
perceived as fair and easier

00:36:01.950 --> 00:36:02.990
to get accepted.

00:36:02.990 --> 00:36:05.820
Everybody's getting something,
we're just randomizing what

00:36:05.820 --> 00:36:07.120
you get in a given round.

00:36:10.480 --> 00:36:13.590
So we have the same anticipation
issue that we

00:36:13.590 --> 00:36:17.690
just mentioned with phase
in as one concern here.

00:36:17.690 --> 00:36:19.490
It depends on what the two
treatments are, but if you

00:36:19.490 --> 00:36:22.420
really want that other treatment
that other people

00:36:22.420 --> 00:36:24.140
are getting, and you're told
you're going to get it in a

00:36:24.140 --> 00:36:26.390
year, this could affect
your behavior now

00:36:26.390 --> 00:36:28.400
for the same reason.

00:36:28.400 --> 00:36:33.800
Also it does have the same long
term problem of the phase

00:36:33.800 --> 00:36:35.180
in, in that everybody--

00:36:35.180 --> 00:36:36.540
if we're just rotating--
everybody

00:36:36.540 --> 00:36:38.440
is getting the treatment.

00:36:38.440 --> 00:36:44.520
Now, another twist on rotation
is not rotating just--

00:36:44.520 --> 00:36:46.870
call it like a placebo
design, so to speak.

00:36:46.870 --> 00:36:51.770
Suppose that you just have two
different treatments doing two

00:36:51.770 --> 00:36:55.160
totally different things,
everybody gets something.

00:36:55.160 --> 00:36:57.170
And you use one to measure
the impact of the

00:36:57.170 --> 00:36:59.070
other and vice versa.

00:36:59.070 --> 00:37:02.360
So it's similar to rotation,
except not going full circle.

00:37:02.360 --> 00:37:04.220
It just means everybody
gets something.

00:37:04.220 --> 00:37:10.130
So in something like that, you
don't have this long term

00:37:10.130 --> 00:37:13.500
impact issue, or this issue.

00:37:13.500 --> 00:37:16.780
But you do have a problem where
it's not so clear what

00:37:16.780 --> 00:37:18.460
you're comparing anymore.

00:37:18.460 --> 00:37:20.050
So you'd have to be really
careful and think about what

00:37:20.050 --> 00:37:21.300
it is you're trying to do.

00:37:26.610 --> 00:37:32.760
We tend to think that a lot of
interventions have indirect

00:37:32.760 --> 00:37:35.180
effects in many facets
of our life.

00:37:35.180 --> 00:37:37.740
So if you're providing
training about

00:37:37.740 --> 00:37:41.800
entrepreneurship to one group,
and another group you're

00:37:41.800 --> 00:37:44.580
providing some health service,
and you think, well, this is

00:37:44.580 --> 00:37:48.290
great, because what does
entrepreneurship training have

00:37:48.290 --> 00:37:49.680
to do with health?

00:37:49.680 --> 00:37:50.800
And what does health
have to do with

00:37:50.800 --> 00:37:51.540
entrepreneurship training?

00:37:51.540 --> 00:37:55.160
So I can provide my health
services over here and measure

00:37:55.160 --> 00:37:58.780
the health outcomes for my other
group and compare them.

00:37:58.780 --> 00:38:00.970
And the same thing with
business activity.

00:38:00.970 --> 00:38:02.810
But it's not too hard to tell
stories where these are going

00:38:02.810 --> 00:38:03.860
to interact with each other.

00:38:03.860 --> 00:38:07.120
So you're healthier, and this
makes you more able to work,

00:38:07.120 --> 00:38:08.780
and your business does better.

00:38:08.780 --> 00:38:11.350
Your business does better, this
makes you richer, and you

00:38:11.350 --> 00:38:12.410
spend more money on health.

00:38:12.410 --> 00:38:15.590
It's not hard to tell stories
across two seemingly unrelated

00:38:15.590 --> 00:38:19.570
sectors where you will have that
type of effect on each.

00:38:19.570 --> 00:38:21.380
You have to think about
those types of issues.

00:38:21.380 --> 00:38:23.855
Yeah?

00:38:23.855 --> 00:38:27.395
AUDIENCE: Can a rotation work
well in agriculture?

00:38:30.185 --> 00:38:33.340
A given country would have--

00:38:33.340 --> 00:38:35.300
so you'd like to give
the farmers--

00:38:35.300 --> 00:38:36.110
everybody gets something.

00:38:36.110 --> 00:38:39.030
Some people get credit, some
people get seeds, some people

00:38:39.030 --> 00:38:41.430
get a variety of things.

00:38:41.430 --> 00:38:49.080
But obviously some places have
good soil, some places the

00:38:49.080 --> 00:38:56.580
farmers are near a road, and the
data that you need to look

00:38:56.580 --> 00:38:59.780
at your sampling design is kind
of tricky, because you

00:38:59.780 --> 00:39:03.910
may not know that much about all
the variables that would

00:39:03.910 --> 00:39:06.195
impact a farmer's success.

00:39:12.580 --> 00:39:14.500
Rotation seems a nice way to--

00:39:14.500 --> 00:39:16.580
because there are so many
different treatments that

00:39:16.580 --> 00:39:20.260
people can get, that it would
seem pretty tricky to

00:39:20.260 --> 00:39:21.790
implement if there were--?

00:39:24.450 --> 00:39:26.020
PROFESSOR: So I think, in that
type of setting what you're

00:39:26.020 --> 00:39:27.490
describing, we'll come
to one in the end.

00:39:27.490 --> 00:39:32.380
But let me just say an example
here, which is the question

00:39:32.380 --> 00:39:34.690
you're proposing about
agriculture is about how

00:39:34.690 --> 00:39:36.360
different treatments will
interact with other

00:39:36.360 --> 00:39:39.950
treatments, and with
underlying context.

00:39:39.950 --> 00:39:41.670
So there's two things going
on in your question.

00:39:41.670 --> 00:39:44.620
One is how does soil quality
affect whether a certain

00:39:44.620 --> 00:39:46.130
treatment is effective or not?

00:39:46.130 --> 00:39:49.880
And the second is maybe credit
alone is bad, and maybe seeds

00:39:49.880 --> 00:39:51.840
alone is bad, but the two
together is good, and things

00:39:51.840 --> 00:39:53.210
of this nature.

00:39:53.210 --> 00:39:56.190
So that's not a setting where
we would think instinctively

00:39:56.190 --> 00:39:58.570
about a rotation design.

00:39:58.570 --> 00:40:00.730
That's a study where we would
think about two things.

00:40:00.730 --> 00:40:04.320
One is making sure that our
study is being done in a wide

00:40:04.320 --> 00:40:07.340
enough variety of soil, to use
that example, so that we can

00:40:07.340 --> 00:40:11.200
actually study the effect on one
soil quality and another.

00:40:11.200 --> 00:40:12.890
And then the second thing is we
were thinking about having

00:40:12.890 --> 00:40:17.480
multiple treatments, but not in
a rotation style, but in a

00:40:17.480 --> 00:40:19.470
way that you have some people,
they get seed.

00:40:19.470 --> 00:40:20.770
Some people, they
get training.

00:40:20.770 --> 00:40:22.200
Some people, they get
seed plus training.

00:40:22.200 --> 00:40:25.280
Some people, they get nothing.

00:40:25.280 --> 00:40:27.590
So we'll get to an example like
that hopefully towards

00:40:27.590 --> 00:40:28.950
the end, but that's
that design.

00:40:28.950 --> 00:40:34.200
A rotation design is really more
about when you're doing

00:40:34.200 --> 00:40:37.690
something that realistically
will not have an indirect

00:40:37.690 --> 00:40:38.610
effect on the other group.

00:40:38.610 --> 00:40:40.540
So the example I'm going to give
you is a rotation study

00:40:40.540 --> 00:40:42.965
that was done, the Balsakhi Case
that's done, I think it's

00:40:42.965 --> 00:40:44.290
one of the cases in
your reading.

00:40:44.290 --> 00:40:46.290
It's what what you do
this morning, right?

00:40:46.290 --> 00:40:51.090
So that's a classic rotation
example, because what we're

00:40:51.090 --> 00:40:53.590
doing is, some schools got third
grade, and some schools

00:40:53.590 --> 00:40:55.830
got fourth grade, and
then they rotate.

00:40:55.830 --> 00:40:57.670
And the idea is as long as the
third graders don't affect the

00:40:57.670 --> 00:41:00.750
fourth, and the fourth graders
don't affect the third, then

00:41:00.750 --> 00:41:02.790
this is good, and every
school got something.

00:41:02.790 --> 00:41:05.290
And then we just rotate around
what they're getting.

00:41:05.290 --> 00:41:07.930
And that's what we mean more
by rotation design.

00:41:17.480 --> 00:41:20.980
The key here is that this is a
great example of where the

00:41:20.980 --> 00:41:24.640
rotation design was a useful way
of getting the support of

00:41:24.640 --> 00:41:26.040
the schools.

00:41:26.040 --> 00:41:28.290
It's going to the schools and
get them to agree to do all

00:41:28.290 --> 00:41:29.480
these tests with the children.

00:41:29.480 --> 00:41:31.890
And it would have been hard to
just go in and get them to do

00:41:31.890 --> 00:41:35.920
tests without being offered some
service along with that.

00:41:35.920 --> 00:41:37.880
Now they're willing to accept
that the service only went to

00:41:37.880 --> 00:41:39.730
one grade, not both.

00:41:39.730 --> 00:41:42.930
They understood it was a phase
in within their school, one

00:41:42.930 --> 00:41:45.290
gets it one year, that
other, the next.

00:41:45.290 --> 00:41:48.000
But that was the way of
getting the schools to

00:41:48.000 --> 00:41:50.500
cooperate was by offering
it through this

00:41:50.500 --> 00:41:51.750
type of rotation design.

00:41:54.250 --> 00:41:56.180
So next is encouragement
design.

00:41:56.180 --> 00:41:58.770
Now encouragement
designs are--

00:41:58.770 --> 00:42:01.250
first of all, this is orthogonal
to everything else

00:42:01.250 --> 00:42:02.670
I've been saying.

00:42:02.670 --> 00:42:05.410
Encouragement design can be done
on top of a phase in, on

00:42:05.410 --> 00:42:09.670
top of a rotation, on
top of a lottery.

00:42:09.670 --> 00:42:11.790
This is not mutually
exclusive with the

00:42:11.790 --> 00:42:12.600
others that I've discussed.

00:42:12.600 --> 00:42:12.910
Yeah?

00:42:12.910 --> 00:42:14.218
AUDIENCE: I just had
a question on

00:42:14.218 --> 00:42:14.874
the phase in approach.

00:42:14.874 --> 00:42:15.856
PROFESSOR: Yeah?

00:42:15.856 --> 00:42:19.293
AUDIENCE: So suppose you wanted
to roll out packaging

00:42:19.293 --> 00:42:22.730
group one, and then you'll roll
out packaging group two.

00:42:22.730 --> 00:42:25.430
When you roll out packaging, you
notice that something is

00:42:25.430 --> 00:42:27.934
not working that well, and then
you wanted to tweak it a

00:42:27.934 --> 00:42:29.604
little bit, you wanted
to change the spec.

00:42:29.604 --> 00:42:32.304
And for the sake of the
experiment, are you not

00:42:32.304 --> 00:42:34.415
supposed to tweak it when you
roll it out to the second

00:42:34.415 --> 00:42:35.005
group, you just keep
it the same?

00:42:35.005 --> 00:42:37.970
Because [UNINTELLIGIBLE PHRASE]?

00:42:37.970 --> 00:42:38.300
PROFESSOR: Right.

00:42:38.300 --> 00:42:39.340
So great question.

00:42:39.340 --> 00:42:42.360
I think the key here is to
think about the timeline.

00:42:42.360 --> 00:42:43.610
I don't really have
a chalkboard.

00:42:45.730 --> 00:42:48.490
The key is to remember that with
the phase in-- so let's

00:42:48.490 --> 00:42:50.810
go with a really simple
phase in, two waves.

00:42:50.810 --> 00:42:54.930
So in that setting, the second
group, when you do the

00:42:54.930 --> 00:42:56.430
treatment with them, that's
actually after

00:42:56.430 --> 00:42:58.910
the study is over.

00:42:58.910 --> 00:43:01.710
So the idea is that they're
really your control group, but

00:43:01.710 --> 00:43:03.560
they're participating with you
because they know they're

00:43:03.560 --> 00:43:06.480
going to get it in the future,
or whatever the

00:43:06.480 --> 00:43:07.040
circumstance is.

00:43:07.040 --> 00:43:10.370
So that's a situation in which
the answer is yeah, you can do

00:43:10.370 --> 00:43:11.970
whatever you want with them.

00:43:11.970 --> 00:43:15.660
But if you did know from
operational observations that

00:43:15.660 --> 00:43:19.920
the treatment itself wasn't
working so well, then just

00:43:19.920 --> 00:43:22.040
remember that when you're
evaluating something, what

00:43:22.040 --> 00:43:24.590
you're evaluating was a program
which you already

00:43:24.590 --> 00:43:27.750
think from operational reasons
was less than effective.

00:43:27.750 --> 00:43:30.790
And so that should perhaps
inform you a little bit about

00:43:30.790 --> 00:43:35.310
things like what to measure in
terms of what you want to put

00:43:35.310 --> 00:43:36.560
in the followup surveys.

00:43:42.010 --> 00:43:44.090
I suppose we could complicate
your question a little bit and

00:43:44.090 --> 00:43:46.120
add a third wave.

00:43:46.120 --> 00:43:48.630
So you have three waves,
one for each year.

00:43:48.630 --> 00:43:53.140
And after the first year you
learn, oh, turns out we

00:43:53.140 --> 00:43:54.130
shouldn't have done
it like this.

00:43:54.130 --> 00:43:56.140
We should have done it
differently, and so you want

00:43:56.140 --> 00:43:58.090
to change things for
the second wave.

00:43:58.090 --> 00:43:59.810
And that's perfectly fine.

00:43:59.810 --> 00:44:02.620
It does mean now when you're
doing your analysis, you

00:44:02.620 --> 00:44:04.040
should think about this
as two studies.

00:44:07.030 --> 00:44:11.140
You have your first wave, and
you can compare that to your

00:44:11.140 --> 00:44:15.450
wave three, that is control
for the entire study.

00:44:15.450 --> 00:44:17.530
And then you have your second
wave, and you can look at them

00:44:17.530 --> 00:44:20.490
for one year and compare them to
wave three, and you really

00:44:20.490 --> 00:44:24.310
have two different studies
in that setting.

00:44:24.310 --> 00:44:29.210
So encouragement designs, like
I said, this is not mutually

00:44:29.210 --> 00:44:30.740
exclusive to the others.

00:44:30.740 --> 00:44:32.450
And encouragement design,
just think about

00:44:32.450 --> 00:44:33.330
what the word means.

00:44:33.330 --> 00:44:35.560
It means we're encouraging
people to do something.

00:44:35.560 --> 00:44:39.070
We're not forcing, we're
not mandating.

00:44:39.070 --> 00:44:42.380
That means the control group
does not necessarily have

00:44:42.380 --> 00:44:44.530
nobody getting services, and
a treatment group does not

00:44:44.530 --> 00:44:47.380
necessarily have everybody
getting the service.

00:44:47.380 --> 00:44:50.320
There's simply something done
to encourage people to do

00:44:50.320 --> 00:44:52.850
something, to participate.

00:44:52.850 --> 00:44:55.870
Now the key here is to think
about what we're really

00:44:55.870 --> 00:45:00.230
saying, is the control in the
phrase randomized control

00:45:00.230 --> 00:45:05.170
trial, the reason for the word
control is this idea that the

00:45:05.170 --> 00:45:08.280
researcher has some control over
the process in deciding

00:45:08.280 --> 00:45:10.470
who gets a service
and who doesn't.

00:45:10.470 --> 00:45:13.500
So now we're just moving the
control, and it's no longer

00:45:13.500 --> 00:45:15.510
over who gets the service
and who doesn't.

00:45:15.510 --> 00:45:19.130
It's over who's offered the
service and who's not, or who

00:45:19.130 --> 00:45:21.190
has some encouragement to
get the service or not.

00:45:21.190 --> 00:45:24.090
And we still have perfect
control if it's executed

00:45:24.090 --> 00:45:27.320
properly over that offer, over
that suggestion, that

00:45:27.320 --> 00:45:28.400
encouragement.

00:45:28.400 --> 00:45:30.990
But we don't have perfect
control over who actually gets

00:45:30.990 --> 00:45:33.070
the service.

00:45:33.070 --> 00:45:38.920
So a very simple example of this
is suppose that I gave

00:45:38.920 --> 00:45:45.910
each of you a marketing brochure
to go to Au Bon Pain

00:45:45.910 --> 00:45:50.990
during lunch and go because of
their delicious scones, and I

00:45:50.990 --> 00:45:52.240
only gave it to half of you.

00:45:55.210 --> 00:45:57.980
I am now encouraging half of you
to go, the other half not.

00:45:57.980 --> 00:46:01.650
Anybody can go to Au Bon Pain,
I'm not controlling that.

00:46:01.650 --> 00:46:06.730
And if I wanted to then, for
some reason, study the effect

00:46:06.730 --> 00:46:08.300
of going to Au Bon Pain,
I could do that.

00:46:08.300 --> 00:46:10.730
I'm not sure what the point
of that would be.

00:46:10.730 --> 00:46:13.000
But the point is, I'm only
controlling who receives this

00:46:13.000 --> 00:46:14.450
offer and who doesn't.

00:46:14.450 --> 00:46:16.790
I'm not controlling who actually
goes to Au Bon Pain

00:46:16.790 --> 00:46:18.780
and who does not.

00:46:18.780 --> 00:46:22.330
And so this is often the easiest
thing to control in

00:46:22.330 --> 00:46:24.070
the process.

00:46:24.070 --> 00:46:28.880
And the entire key here from
a statistical perspective--

00:46:28.880 --> 00:46:31.600
not the entire, we'll go into
some other issues-- is about

00:46:31.600 --> 00:46:34.470
what that differential usage
rate will be among those who

00:46:34.470 --> 00:46:36.720
were encouraged and those
who were not.

00:46:36.720 --> 00:46:38.920
And when you get into power
calculations later in this

00:46:38.920 --> 00:46:40.890
week, that's going to be a
very important element to

00:46:40.890 --> 00:46:42.010
think about.

00:46:42.010 --> 00:46:45.300
Because if that encouragement
is really, really weak and

00:46:45.300 --> 00:46:47.930
just barely changes people's
behavior, it means you need a

00:46:47.930 --> 00:46:50.360
huge sample.

00:46:50.360 --> 00:46:53.070
In an extreme, an encouragement
design is

00:46:53.070 --> 00:46:56.910
exactly a perfectly controlled
randomized controlled trial.

00:46:56.910 --> 00:46:59.760
An encouragement that gets
people who get the marketing,

00:46:59.760 --> 00:47:02.010
every single one of you goes
to Au Bon Pain, and if you

00:47:02.010 --> 00:47:04.410
didn't receive the marketing,
nobody goes.

00:47:04.410 --> 00:47:07.050
Statistically it's
the same now as a

00:47:07.050 --> 00:47:09.490
perfect lottery system.

00:47:09.490 --> 00:47:11.780
But usually when we're doing
encouragement design is when

00:47:11.780 --> 00:47:14.590
we have some expectation for it
not to be perfect, and so

00:47:14.590 --> 00:47:15.840
we're using that.

00:47:26.570 --> 00:47:30.380
So what makes something
a good encouragement?

00:47:30.380 --> 00:47:32.840
So I think there's two
things to think

00:47:32.840 --> 00:47:34.610
about that are important.

00:47:34.610 --> 00:47:40.830
One is that it's not
itself a treatment.

00:47:40.830 --> 00:47:43.090
The minute the encouragement
design itself becomes a

00:47:43.090 --> 00:47:46.900
treatment, then we have to think
about what is it that

00:47:46.900 --> 00:47:48.260
you're actually evaluating
here.

00:47:48.260 --> 00:47:50.880
Your goal is for your
encouragement to be totally

00:47:50.880 --> 00:47:54.790
innocuous, to just by chance,
by randomness, some people

00:47:54.790 --> 00:47:56.970
will be more likely to use
a service than others.

00:47:56.970 --> 00:47:59.880
So you want it to be as
innocuous as possible.

00:47:59.880 --> 00:48:03.000
So a good idea is typically
marketing.

00:48:03.000 --> 00:48:06.000
We typically think of marketing
as a good approach,

00:48:06.000 --> 00:48:08.450
just making people aware of
a service makes them more

00:48:08.450 --> 00:48:10.840
likely to use it.

00:48:10.840 --> 00:48:13.820
So we've done marketing
experiments, for instance, in

00:48:13.820 --> 00:48:15.920
the Philippines a lot where
we're doing some sort of door

00:48:15.920 --> 00:48:17.990
to door marketing of
a savings product

00:48:17.990 --> 00:48:19.620
offering people savings.

00:48:19.620 --> 00:48:21.880
Anybody in the village could
walk into the bank and open a

00:48:21.880 --> 00:48:22.710
bank account.

00:48:22.710 --> 00:48:27.040
But realistically, only those
who get a knock on their door

00:48:27.040 --> 00:48:28.690
become aware enough of
it to actually go and

00:48:28.690 --> 00:48:30.890
open up a bank account.

00:48:30.890 --> 00:48:32.630
Here's a bad idea.

00:48:32.630 --> 00:48:36.810
Let's provide training to people
that encourages them to

00:48:36.810 --> 00:48:39.380
use credit.

00:48:39.380 --> 00:48:42.740
So let's bring them in, let's
give them a big course about

00:48:42.740 --> 00:48:49.500
business management and how to
use credit in order to take

00:48:49.500 --> 00:48:50.030
out a loan.

00:48:50.030 --> 00:48:53.410
And let's use that as an
encouragement tool for

00:48:53.410 --> 00:48:55.350
measuring the effect of credit,
because after doing

00:48:55.350 --> 00:48:57.950
this course, they'll be
more likely to borrow.

00:48:57.950 --> 00:48:59.750
So the problem with this, if
we want to look at business

00:48:59.750 --> 00:49:02.290
outcomes, is we just gave them
a month long course in

00:49:02.290 --> 00:49:04.060
management of an enterprise.

00:49:04.060 --> 00:49:06.470
And that alone is going to
have an impact on their

00:49:06.470 --> 00:49:09.630
enterprise, we think, we hope.

00:49:09.630 --> 00:49:12.360
And so if it does, well
then, what are you

00:49:12.360 --> 00:49:13.420
measuring the impact on?

00:49:13.420 --> 00:49:15.170
Was it an impact of the
training program?

00:49:15.170 --> 00:49:17.190
Or was it an impact of getting
access to credit?

00:49:17.190 --> 00:49:21.120
And you can't separate
these out at all.

00:49:21.120 --> 00:49:23.580
So this first thing to think
about is just making sure that

00:49:23.580 --> 00:49:26.620
treatment is really innocuous.

00:49:26.620 --> 00:49:28.780
In econometrics, we refer
to this as the exclusion

00:49:28.780 --> 00:49:34.810
restriction, in that what it's
saying is that we want to make

00:49:34.810 --> 00:49:36.950
sure that the only--

00:49:36.950 --> 00:49:41.600
if we're going to draw a link
from the encouragement to the

00:49:41.600 --> 00:49:45.190
take up decision to the outcome
measure we care about,

00:49:45.190 --> 00:49:48.590
that the only path through
which the encouragement

00:49:48.590 --> 00:49:53.630
affects the outcome is as it
generates higher take up.

00:49:53.630 --> 00:49:56.150
If it has its own effect outside
of the decision to

00:49:56.150 --> 00:50:00.610
take up, now it's a problem
econometrically, and we can't

00:50:00.610 --> 00:50:04.650
really claim that we're
measuring the impact of using

00:50:04.650 --> 00:50:05.670
the service.

00:50:05.670 --> 00:50:10.080
We could only measure the net
effect of the two together.

00:50:10.080 --> 00:50:11.590
So the second issue
is for whom are we

00:50:11.590 --> 00:50:12.830
estimating the treatment.

00:50:12.830 --> 00:50:15.780
So here's my favorite tongue
in cheek example for this.

00:50:15.780 --> 00:50:20.460
Suppose we went into a village
and we offered free alcohol to

00:50:20.460 --> 00:50:24.360
anybody who takes out a loan.

00:50:24.360 --> 00:50:27.170
Might be great in the first
stage in the sense that it

00:50:27.170 --> 00:50:30.710
generates lots of higher
borrowing.

00:50:30.710 --> 00:50:33.280
But what are we measuring
here in terms

00:50:33.280 --> 00:50:34.310
of who we're studying?

00:50:34.310 --> 00:50:36.940
We're studying people who
respond to this particular

00:50:36.940 --> 00:50:39.160
incentive of free alcohol.

00:50:39.160 --> 00:50:41.460
That's certainly not the program
that we're typically

00:50:41.460 --> 00:50:43.460
trying to evaluate when we're
trying to do an evaluation of

00:50:43.460 --> 00:50:44.860
microcredit.

00:50:44.860 --> 00:50:47.510
And we want to make sure that
we're getting the people in

00:50:47.510 --> 00:50:50.950
the study who are the right
people, who are the types of

00:50:50.950 --> 00:50:56.550
people that are thought about
as the target audience for a

00:50:56.550 --> 00:50:58.390
microcredit program.

00:50:58.390 --> 00:51:01.980
That means not drunkards.

00:51:01.980 --> 00:51:05.060
And so you want to make sure,
you do want whatever that

00:51:05.060 --> 00:51:09.470
approach is to be something that
is sensible, that seems

00:51:09.470 --> 00:51:12.650
somewhat in the scope
of normal.

00:51:12.650 --> 00:51:15.470
Or at least doesn't create a
sample selection bias in the

00:51:15.470 --> 00:51:18.980
sense that it doesn't make the
people who take up the program

00:51:18.980 --> 00:51:22.360
be particularly different in
a way that is not useful.

00:51:22.360 --> 00:51:22.848
Yeah?

00:51:22.848 --> 00:51:25.776
AUDIENCE: So an example that I'm
thinking about is access

00:51:25.776 --> 00:51:32.960
or information about a
microcredit program to

00:51:32.960 --> 00:51:36.500
participants who are
typically very

00:51:36.500 --> 00:51:38.320
uninformed about these things.

00:51:38.320 --> 00:51:41.070
So the information
is out there.

00:51:41.070 --> 00:51:44.530
Theoretically, it's
really accessible.

00:51:44.530 --> 00:51:48.420
But we know that unless we tell
them that this program is

00:51:48.420 --> 00:51:51.850
out there for them, chances are
very good that they would

00:51:51.850 --> 00:51:54.100
never think of it
on their own.

00:51:54.100 --> 00:52:01.232
So that would not then be a good
situation for this kind

00:52:01.232 --> 00:52:04.676
of a thing, because we know that
we are in effect offering

00:52:04.676 --> 00:52:08.080
them a sort of special in by
the very effect of offering

00:52:08.080 --> 00:52:12.460
it, even though theoretically
it's accessible.

00:52:12.460 --> 00:52:13.640
PROFESSOR: I would actually
says that's

00:52:13.640 --> 00:52:17.110
actually a perfect setting.

00:52:17.110 --> 00:52:21.690
To do this, let me rephrase the
question, which is suppose

00:52:21.690 --> 00:52:25.120
you have a program, and only the
highly informed normally

00:52:25.120 --> 00:52:26.560
are going to come in.

00:52:26.560 --> 00:52:28.300
And so in order to do an
encouragement design, what

00:52:28.300 --> 00:52:30.540
you're doing is you're going out
and you're only going to

00:52:30.540 --> 00:52:33.940
move the people who are
not highly informed.

00:52:33.940 --> 00:52:35.130
The highly informed already
know about you.

00:52:35.130 --> 00:52:36.730
They're either coming
in or they're not.

00:52:36.730 --> 00:52:38.370
You give them information, it
doesn't matter, I already knew

00:52:38.370 --> 00:52:40.550
about this.

00:52:40.550 --> 00:52:42.590
So what you're doing is you're
moving the less informed

00:52:42.590 --> 00:52:44.960
people, you're informing them
about the service you're

00:52:44.960 --> 00:52:49.680
offering, and now they're coming
in or not as they wish.

00:52:49.680 --> 00:52:51.500
But they're more likely to come
in now than the people

00:52:51.500 --> 00:52:52.730
who are not informed.

00:52:52.730 --> 00:52:56.380
So this is a perfectly relevant
approach if it's the

00:52:56.380 --> 00:52:58.640
case that this is an
organization that does aspire

00:52:58.640 --> 00:53:03.100
to grow, and they're going to
grow through informing people.

00:53:03.100 --> 00:53:05.680
In most of the settings I've
been involved in, at least the

00:53:05.680 --> 00:53:08.160
type of information we're
dealing with is usually not

00:53:08.160 --> 00:53:10.740
much different than what
they do normally.

00:53:10.740 --> 00:53:12.900
It's just marketing.

00:53:12.900 --> 00:53:15.300
It's just targeted and
controlled marketing, where we

00:53:15.300 --> 00:53:20.050
control what villages they go
to do the marketing, or what

00:53:20.050 --> 00:53:22.350
household's doors
they knock on.

00:53:22.350 --> 00:53:23.970
But in a lot of situations,
the encouragement design

00:53:23.970 --> 00:53:27.000
literally has them doing exactly
the same operation's

00:53:27.000 --> 00:53:28.710
that they normally would do.

00:53:28.710 --> 00:53:29.840
But it's just recognizing
that it's

00:53:29.840 --> 00:53:30.770
still a voluntary decision.

00:53:30.770 --> 00:53:32.010
They can't make someone
borrow.

00:53:32.010 --> 00:53:34.090
They're going to a village,
they're holding meetings,

00:53:34.090 --> 00:53:35.730
they're presenting what
they do, and some

00:53:35.730 --> 00:53:36.920
borrow and some don't.

00:53:36.920 --> 00:53:38.566
AUDIENCE: I think I'm saying
something slightly different,

00:53:38.566 --> 00:53:40.760
but that might
[UNINTELLIGIBLE PHRASE].

00:53:40.760 --> 00:53:45.940
So among the group who would not
normally know about this,

00:53:45.940 --> 00:53:48.010
it's not that I'm going to--

00:53:48.010 --> 00:53:52.286
I'm not saying the group who
knows, forget about them.

00:53:52.286 --> 00:53:55.100
I don't know, I'm confusing
myself.

00:53:55.100 --> 00:53:57.585
We're assuming that the group of
people that we work with to

00:53:57.585 --> 00:54:00.570
provide our program, we would
provide a precursor program,

00:54:00.570 --> 00:54:04.530
and we would say among people
that we work with, half of

00:54:04.530 --> 00:54:07.420
them we would tell, and half
of them we won't tell.

00:54:07.420 --> 00:54:10.070
Is that what you're
saying too?

00:54:10.070 --> 00:54:12.570
PROFESSOR: You would go out
of your way to give them

00:54:12.570 --> 00:54:13.930
information about the program.

00:54:13.930 --> 00:54:16.270
Everyone can get in, but you go
out of your way to approach

00:54:16.270 --> 00:54:18.250
half and tell them about
the services.

00:54:18.250 --> 00:54:21.310
AUDIENCE: Understanding that
chances are that if we don't

00:54:21.310 --> 00:54:23.034
tell them, they won't go,
because they're just

00:54:23.034 --> 00:54:24.435
uninformed?

00:54:24.435 --> 00:54:24.902
OK.

00:54:24.902 --> 00:54:27.240
PROFESSOR: Right.

00:54:27.240 --> 00:54:27.560
Yeah?

00:54:27.560 --> 00:54:30.302
AUDIENCE: I don't want to
interrupt if there was more to

00:54:30.302 --> 00:54:30.948
this exchange.

00:54:30.948 --> 00:54:34.210
My question is about
distiguishing thing between

00:54:34.210 --> 00:54:35.260
marketing and training.

00:54:35.260 --> 00:54:38.740
If the treatment is something
like a financial product or

00:54:38.740 --> 00:54:43.140
service that's poorly
understood, and you don't want

00:54:43.140 --> 00:54:46.530
to-- do you think that maybe
financial literacy is an

00:54:46.530 --> 00:54:49.480
important determinant, but you
want to isolate just access to

00:54:49.480 --> 00:54:52.226
the product or service and keep

00:54:52.226 --> 00:54:53.654
financial liberty separate?

00:54:53.654 --> 00:54:55.695
How do you draw the distinction
between marketing

00:54:55.695 --> 00:54:57.610
and training?

00:54:57.610 --> 00:54:59.680
PROFESSOR: I can tell you in
one setting, here's what we

00:54:59.680 --> 00:55:02.590
did to try to understand
this better.

00:55:02.590 --> 00:55:04.150
Let me restate the question,
which is how do you

00:55:04.150 --> 00:55:06.370
distinguish between marketing
and training?

00:55:06.370 --> 00:55:08.130
This is really a spectrum.

00:55:08.130 --> 00:55:10.760
So an example I gave that was
bad was a month long training

00:55:10.760 --> 00:55:13.390
program, and I said it's fine
to just knock on a door.

00:55:13.390 --> 00:55:15.620
Why am I drawing
the line there?

00:55:15.620 --> 00:55:17.910
And it's a perfect question,
and I can tell you that the

00:55:17.910 --> 00:55:23.010
first time I actually ever did
this type of design, we

00:55:23.010 --> 00:55:25.870
actually had an entire treatment
group that was just

00:55:25.870 --> 00:55:28.730
knocking on doors, but
with no product.

00:55:28.730 --> 00:55:31.770
It was just to test out whether
the knocking on the

00:55:31.770 --> 00:55:34.540
door had an effect on savings.

00:55:34.540 --> 00:55:35.880
So we had two treatment
groups.

00:55:35.880 --> 00:55:37.020
We had a treatment
group that got a

00:55:37.020 --> 00:55:39.200
commitment savings account.

00:55:39.200 --> 00:55:41.980
A bank officer went to the door,
knocked on it, gave them

00:55:41.980 --> 00:55:44.670
a pitch about why they need to
save, and savings is good, and

00:55:44.670 --> 00:55:45.760
here's a goal.

00:55:45.760 --> 00:55:47.330
You should have a goal
for savings.

00:55:47.330 --> 00:55:49.660
And here's an account that we'll
offer you to help you

00:55:49.660 --> 00:55:50.320
reach your goal.

00:55:50.320 --> 00:55:55.080
It wasn't a very long pitch, but
it was a marketing visit.

00:55:55.080 --> 00:55:58.540
And we had a pure control that
got no contact from the bank.

00:55:58.540 --> 00:56:00.420
And then we had a second
treatment group that we called

00:56:00.420 --> 00:56:01.530
the marketing treatment group.

00:56:01.530 --> 00:56:04.020
And this group got the knock
on the door, got the same

00:56:04.020 --> 00:56:06.020
pitch for about 5, 10 minutes
about why it's important to

00:56:06.020 --> 00:56:09.050
save, and how the bank is there
to help them save, but

00:56:09.050 --> 00:56:11.640
didn't get offered that special
savings account that

00:56:11.640 --> 00:56:14.230
had special rules to it.

00:56:14.230 --> 00:56:17.150
And that's done exactly to
try to understand where

00:56:17.150 --> 00:56:18.100
to draw that line.

00:56:18.100 --> 00:56:21.550
So if it's a situation where
you're particularly concerned,

00:56:21.550 --> 00:56:24.040
then you could actually think
about having treatments

00:56:24.040 --> 00:56:26.510
designed specifically to test
whether there is a direct

00:56:26.510 --> 00:56:29.360
effect without the treatment
you really care about.

00:56:29.360 --> 00:56:29.630
Yeah?

00:56:29.630 --> 00:56:32.630
AUDIENCE: I'm kind of jumping
onto the last point as well.

00:56:32.630 --> 00:56:36.200
But the one encouragement design
that I'm familiar with

00:56:36.200 --> 00:56:40.750
is one where a subsidy was
actually utilized, but then it

00:56:40.750 --> 00:56:43.200
was a random distribution of who
was offered the subsidy.

00:56:43.200 --> 00:56:47.130
And, for that matter, because
they're trying to determine a

00:56:47.130 --> 00:56:48.560
demand curve, the
subsidy varied.

00:56:48.560 --> 00:56:52.490
So maybe you'd be offered
35% off, maybe you'd

00:56:52.490 --> 00:56:54.270
be offered 75% off.

00:56:54.270 --> 00:56:57.900
Still random in who was given
the offer, but then they had

00:56:57.900 --> 00:56:59.650
the encouragement to take
off based on how

00:56:59.650 --> 00:57:00.650
big the subsidy was.

00:57:00.650 --> 00:57:03.520
But it seem my initial-- not
knowing enough about the

00:57:03.520 --> 00:57:05.150
details of the program--

00:57:05.150 --> 00:57:13.280
is that there would be a problem
based on the economic

00:57:13.280 --> 00:57:15.590
status of those who were offered
a program in the first

00:57:15.590 --> 00:57:17.810
thing if they were not
very, very similar.

00:57:17.810 --> 00:57:22.820
If I am marginally wealthy and
I'm offered a 35% discount,

00:57:22.820 --> 00:57:25.847
I'm more likely to take out than
someone who is broke and

00:57:25.847 --> 00:57:27.600
is offered a 35% discount.

00:57:27.600 --> 00:57:30.960
Then that would affect
your sample.

00:57:30.960 --> 00:57:32.450
PROFESSOR: With one twist.

00:57:32.450 --> 00:57:34.940
So it's not the levels that
matter, but it's actually the

00:57:34.940 --> 00:57:36.950
slope that would
have to matter.

00:57:36.950 --> 00:57:41.800
It has to be not that the
wealthy is more likely to take

00:57:41.800 --> 00:57:44.740
up with any given level.

00:57:44.740 --> 00:57:48.010
It's that they have to be more
elastic or less elastic than

00:57:48.010 --> 00:57:50.470
the poor in order for
that to be an issue.

00:57:50.470 --> 00:57:52.600
And then you're absolutely
right, that is an issue.

00:57:52.600 --> 00:57:56.530
And then what you're studying
is when you do that subsidy,

00:57:56.530 --> 00:58:00.980
you're studying your treatment
effect on those people who are

00:58:00.980 --> 00:58:04.770
going to be more responsive
to that subsidy.

00:58:04.770 --> 00:58:06.908
Is there another hand?

00:58:06.908 --> 00:58:09.490
No.

00:58:09.490 --> 00:58:11.520
Wendy, now we're to your
question, multiple treatments.

00:58:14.460 --> 00:58:15.550
So let me just say
one more thing on

00:58:15.550 --> 00:58:16.350
encouragement designs.

00:58:16.350 --> 00:58:19.040
So one of the key things to
remember with encouragement

00:58:19.040 --> 00:58:24.370
designs is that in a lot of
situations, the encouragement

00:58:24.370 --> 00:58:25.690
design is--

00:58:25.690 --> 00:58:28.990
in some situations, it is set
up where the control group

00:58:28.990 --> 00:58:30.820
does get into a program.

00:58:30.820 --> 00:58:33.760
So where you're dealing with a
10% take up rate in control,

00:58:33.760 --> 00:58:36.670
and a 30% take up rate in the
treatment, In a lot of the

00:58:36.670 --> 00:58:38.450
setting, though, it's really
more that you just have

00:58:38.450 --> 00:58:41.870
incomplete take up in the
treatment group, that

00:58:41.870 --> 00:58:43.460
participation is voluntary.

00:58:43.460 --> 00:58:48.800
And so by encouragement, all we
really mean here is that a

00:58:48.800 --> 00:58:50.470
treatment is being offered
to people.

00:58:50.470 --> 00:58:53.660
They can say yes or no, and
they're not being offered to

00:58:53.660 --> 00:58:55.890
the control group.

00:58:55.890 --> 00:58:58.400
And so we end up with take
up of some percent in the

00:58:58.400 --> 00:59:01.050
treatment group, and
zero in control.

00:59:01.050 --> 00:59:03.000
Like the savings experiment
that I just referred to a

00:59:03.000 --> 00:59:06.090
moment ago, we had a 28% take up
in the treatment group, we

00:59:06.090 --> 00:59:08.190
can't make people open
a savings account.

00:59:08.190 --> 00:59:11.660
All we can do is offer it to
them, and we had a 0% take up

00:59:11.660 --> 00:59:12.840
rate in the control.

00:59:12.840 --> 00:59:14.780
We did a similar thing in the
same place in the Philippines

00:59:14.780 --> 00:59:17.730
on a commitment account
to stop smoking.

00:59:17.730 --> 00:59:19.630
Again, 11% take up rate in
the treatment group.

00:59:19.630 --> 00:59:22.140
We can't make people want to
stop smoking and sign accounts

00:59:22.140 --> 00:59:24.780
and contracts to do this.

00:59:24.780 --> 00:59:27.540
But we can prevent the
control group.

00:59:27.540 --> 00:59:30.020
So there was perfect control in
the control, in the sense

00:59:30.020 --> 00:59:31.150
that they were not offered the

00:59:31.150 --> 00:59:32.780
opportunity to open the account.

00:59:32.780 --> 00:59:35.940
But the treatment group
has to be voluntary.

00:59:35.940 --> 00:59:38.860
That is what it is.

00:59:38.860 --> 00:59:41.760
And so it's an encouragement
design, with 11% percent take

00:59:41.760 --> 00:59:44.730
up rate in the treatment
and 0% in the control.

00:59:44.730 --> 00:59:46.830
So sometimes you do have control
on one half but not

00:59:46.830 --> 00:59:47.840
the other for who uses.

00:59:47.840 --> 00:59:48.330
Yeah?

00:59:48.330 --> 00:59:51.270
AUDIENCE: And so just the main
point is that what constraint

00:59:51.270 --> 00:59:53.720
are you going around by an
encouragement design?

00:59:53.720 --> 00:59:56.170
Just an ethical problem
if you can't afford

00:59:56.170 --> 00:59:58.420
treatment to everybody?

00:59:58.420 --> 01:00:02.446
PROFESSOR: In that situation,
I don't know that I'd pose

01:00:02.446 --> 01:00:03.350
that as an ethical issue.

01:00:03.350 --> 01:00:05.690
But the point to be made is
just that you can't force

01:00:05.690 --> 01:00:08.040
people into a program.

01:00:08.040 --> 01:00:12.020
It's a voluntary participation,
and that's OK.

01:00:12.020 --> 01:00:18.340
So one of the things that I've
often read or heard is when

01:00:18.340 --> 01:00:19.890
someone says, well,
wait a second.

01:00:19.890 --> 01:00:20.750
This is voluntary

01:00:20.750 --> 01:00:24.150
participation, so how can you--

01:00:24.150 --> 01:00:25.780
doesn't that introduce
selection bias?

01:00:25.780 --> 01:00:28.360
And the answer is no, because
what we're going to do when we

01:00:28.360 --> 01:00:30.330
do the analysis of that, is
we're going to compare

01:00:30.330 --> 01:00:32.990
everybody who was offered the
account, everybody who was not

01:00:32.990 --> 01:00:34.780
offered the account.

01:00:34.780 --> 01:00:36.210
And so there's no selection
bias there.

01:00:36.210 --> 01:00:39.720
There would be a selection
bias if what we did is we

01:00:39.720 --> 01:00:41.770
analyzed everybody who took up
in the treatment group, and

01:00:41.770 --> 01:00:43.440
compared them to everybody
in control.

01:00:43.440 --> 01:00:44.840
And that would be a
flawed analysis.

01:00:44.840 --> 01:00:46.870
But that's not what we do.

01:00:46.870 --> 01:00:48.950
So if you ever hear someone say,
ah, encouragement design,

01:00:48.950 --> 01:00:51.970
doesn't that introduce a
selection bias, because

01:00:51.970 --> 01:00:52.830
participation is voluntary?

01:00:52.830 --> 01:00:53.930
The answer is no.

01:00:53.930 --> 01:00:55.340
That only introduces selection
bias if you do

01:00:55.340 --> 01:00:56.770
the analysis wrong.

01:00:56.770 --> 01:00:59.260
What you want to do is compare
what's refereed to as the

01:00:59.260 --> 01:01:02.490
intent to treat analysis, and
it means comparing everybody

01:01:02.490 --> 01:01:04.560
who's offered to everybody
who's not offered.

01:01:08.880 --> 01:01:09.680
Multiple treatments.

01:01:09.680 --> 01:01:12.200
So this goes back to
Wendy's question.

01:01:12.200 --> 01:01:17.080
This is one of the areas where
I tend to think is most ripe

01:01:17.080 --> 01:01:17.920
for helping--

01:01:17.920 --> 01:01:21.120
going back to one of the first
points I was making about

01:01:21.120 --> 01:01:24.470
making sure that the evaluation
speaks nicely and

01:01:24.470 --> 01:01:27.920
informatively to needs of the
implementers, needs of the

01:01:27.920 --> 01:01:29.230
organization.

01:01:29.230 --> 01:01:32.090
That a lot of times, there's
very specific operational

01:01:32.090 --> 01:01:33.060
questions that they have.

01:01:33.060 --> 01:01:34.150
Should we really
do it this way?

01:01:34.150 --> 01:01:36.020
Or should we do it this way?

01:01:36.020 --> 01:01:38.410
I really made some tough choices
here, and I just went

01:01:38.410 --> 01:01:40.310
with what I thought was best.

01:01:40.310 --> 01:01:43.140
But gosh, if the research can
actually help guide me and

01:01:43.140 --> 01:01:45.100
tell me whether this particular
component is

01:01:45.100 --> 01:01:47.940
necessary or not, that
would be great.

01:01:47.940 --> 01:01:49.560
So imagine you're doing--

01:01:49.560 --> 01:01:53.180
let's go with Wendy's example
of an agricultural program.

01:01:53.180 --> 01:01:56.730
And suppose that you're trying
to decide, how important is

01:01:56.730 --> 01:01:57.780
this training component?

01:01:57.780 --> 01:02:00.830
I'm going to provide seeds,
and introduce people to

01:02:00.830 --> 01:02:02.380
marketplaces.

01:02:02.380 --> 01:02:03.650
I'm just making something up.

01:02:03.650 --> 01:02:05.930
Let's not get into the details
too much, but let's just say

01:02:05.930 --> 01:02:07.480
there's a training component
alongside of it.

01:02:07.480 --> 01:02:09.770
And that training component is
really expensive, it takes a

01:02:09.770 --> 01:02:11.190
lot of time.

01:02:11.190 --> 01:02:13.960
And I'm thinking to myself, OK,
I can help twice as many

01:02:13.960 --> 01:02:17.940
people and drop the training, or
keep it my current program

01:02:17.940 --> 01:02:19.400
size and have training.

01:02:19.400 --> 01:02:21.450
What's better?

01:02:21.450 --> 01:02:24.780
Well, the research can help
answer that question by having

01:02:24.780 --> 01:02:27.420
an evaluation which evaluates
the overall program, but then

01:02:27.420 --> 01:02:30.130
also randomizes whether or not
there's training involved.

01:02:41.210 --> 01:02:44.140
And so this is one of those key
areas where it's a win win

01:02:44.140 --> 01:02:44.750
for operations.

01:02:44.750 --> 01:02:46.950
Where you can help answer
questions for them beyond the

01:02:46.950 --> 01:02:49.220
simple impact question.

01:02:49.220 --> 01:02:51.400
There are situations in multiple
treatments that we've

01:02:51.400 --> 01:02:55.430
been in where there's
no pure control.

01:02:55.430 --> 01:02:59.410
And there's nothing invalid
about doing that.

01:02:59.410 --> 01:03:02.460
It does validate the study, but
we just have to remember

01:03:02.460 --> 01:03:05.260
that you're no longer saying,
what is the impact of the

01:03:05.260 --> 01:03:07.620
program compared to not
doing the program?

01:03:07.620 --> 01:03:11.020
You're now comparing it one
option relative to another

01:03:11.020 --> 01:03:12.290
option relative to
another option.

01:03:12.290 --> 01:03:15.130
And hopefully in the design
there, you have one option

01:03:15.130 --> 01:03:19.750
which is kind of like a placebo,
so that you have some

01:03:19.750 --> 01:03:21.700
group that you really don't--

01:03:21.700 --> 01:03:24.750
it was in a very extensive way,
but you have some method

01:03:24.750 --> 01:03:26.340
of being able to say what
the overall effect is.

01:03:26.340 --> 01:03:29.060
But there's many situations
we're in where that's actually

01:03:29.060 --> 01:03:32.070
just not part of what
the study's about.

01:03:32.070 --> 01:03:34.570
So we've done savings product
designs are a perfect example

01:03:34.570 --> 01:03:37.030
of this, where we're dealing
with people who opened a

01:03:37.030 --> 01:03:38.630
savings account.

01:03:38.630 --> 01:03:40.860
There's no control group of
people who were not offered a

01:03:40.860 --> 01:03:41.360
savings account.

01:03:41.360 --> 01:03:43.920
We're just dealing with a bank
and they take people in.

01:03:43.920 --> 01:03:46.200
And the question to us was,
well, how can you help our

01:03:46.200 --> 01:03:49.290
existing savings people
save more?

01:03:49.290 --> 01:03:51.220
So we tested something out in
three different countries

01:03:51.220 --> 01:03:53.990
where we sent people
reminders to save.

01:03:53.990 --> 01:03:56.190
So half the people basically
got a little text message

01:03:56.190 --> 01:03:58.080
saying, hey, don't forget
to save this month,

01:03:58.080 --> 01:03:59.910
and half did not.

01:03:59.910 --> 01:04:03.270
So we have no control group
here of people who got no

01:04:03.270 --> 01:04:04.340
savings account.

01:04:04.340 --> 01:04:06.860
So we're not measuring the
impact of savings on things.

01:04:06.860 --> 01:04:08.450
We're just measuring the
impact of getting this

01:04:08.450 --> 01:04:11.500
reminder on how much you save.

01:04:11.500 --> 01:04:14.460
And we've done similar things
with loan repayments.

01:04:14.460 --> 01:04:16.670
There's no study on the impact
of the credit, we're just

01:04:16.670 --> 01:04:19.970
testing out operational
questions about how to run the

01:04:19.970 --> 01:04:21.751
program better.

01:04:21.751 --> 01:04:25.140
And in those types of designs,
we'll often test out five

01:04:25.140 --> 01:04:26.730
different messages all
at the same time.

01:04:37.380 --> 01:04:41.030
I think I said that slide.

01:04:41.030 --> 01:04:42.280
Oh, maybe not.

01:04:45.954 --> 01:04:48.910
Yeah, we talked about
randomization in the bubble.

01:04:53.100 --> 01:04:54.670
So this is the list of
the various things

01:04:54.670 --> 01:04:57.470
that we've now described.

01:04:57.470 --> 01:05:01.110
And just remember that these
are not mutually exclusive.

01:05:01.110 --> 01:05:03.010
Multiple treatments and
encouragement design in

01:05:03.010 --> 01:05:05.890
particular kind of fit within
almost any of these other

01:05:05.890 --> 01:05:07.140
things going on here.

01:05:11.190 --> 01:05:13.410
Any questions so far?

01:05:13.410 --> 01:05:15.470
OK, part two, gathering
support.

01:05:23.300 --> 01:05:27.510
So here are some things
that we commonly hear.

01:05:27.510 --> 01:05:30.560
So this part of the lecture is
really all about how we deal

01:05:30.560 --> 01:05:34.460
with this kind of introductory
conversations, exploratory

01:05:34.460 --> 01:05:36.330
conversations where we're trying
to work with partners

01:05:36.330 --> 01:05:42.505
to figure out how to go about
doing a randomized trial.

01:05:42.505 --> 01:05:47.990
So one answer which is always
a tough one to get, but I

01:05:47.990 --> 01:05:49.350
already know the answer.

01:05:49.350 --> 01:05:50.850
And I don't want to risk
learning that we

01:05:50.850 --> 01:05:52.100
do not have an impact.

01:05:56.750 --> 01:05:59.350
There are situations that we'll
be in-- and I don't mean

01:05:59.350 --> 01:06:01.400
to sound like a pessimist-- but
there are situations we'll

01:06:01.400 --> 01:06:03.740
be in where you just realize
this is not a

01:06:03.740 --> 01:06:05.210
good setting for it.

01:06:05.210 --> 01:06:07.300
You have to work with
people who actually

01:06:07.300 --> 01:06:09.600
want to know the answer.

01:06:09.600 --> 01:06:14.060
And you can recognize that
merely observing that their

01:06:14.060 --> 01:06:16.740
program has grown is not
necessarily a sufficient

01:06:16.740 --> 01:06:19.620
measure to say whether they've
had an impact.

01:06:19.620 --> 01:06:21.430
And it's certainly not a
sufficient measure to say

01:06:21.430 --> 01:06:24.810
whether their program is a good
allocation of resources

01:06:24.810 --> 01:06:27.810
compared to other programs that
have also had similar

01:06:27.810 --> 01:06:29.440
operational success.

01:06:29.440 --> 01:06:31.200
And so when we have to make
the tough choices, this is

01:06:31.200 --> 01:06:35.070
where we need the evidence.

01:06:35.070 --> 01:06:38.560
Listening is probably the
single most trite but

01:06:38.560 --> 01:06:41.910
important thing I have to say on
how to have these types of

01:06:41.910 --> 01:06:44.390
conversations.

01:06:44.390 --> 01:06:46.160
Trying to understand the
perspectives and the

01:06:46.160 --> 01:06:47.830
objectives of the people
in the table.

01:06:47.830 --> 01:06:49.280
What is it that's making
them tick?

01:06:49.280 --> 01:06:52.530
What is it that's making them
have this conversation in the

01:06:52.530 --> 01:06:54.280
first place?

01:06:54.280 --> 01:06:57.740
And finding ways to make the
research operationally useful

01:06:57.740 --> 01:07:00.820
is perhaps the single most
useful and important thing to

01:07:00.820 --> 01:07:04.870
do when working in the field.

01:07:04.870 --> 01:07:10.510
One thing that I've often found
too is that often in

01:07:10.510 --> 01:07:19.660
practice, there's a caution.

01:07:19.660 --> 01:07:22.690
There's almost a mistrust that
some might have if they're not

01:07:22.690 --> 01:07:24.505
familiar with what is
that's going on.

01:07:24.505 --> 01:07:26.300
And it's one of the most
important things that the

01:07:26.300 --> 01:07:29.500
field staff can do in working
with the organization is to

01:07:29.500 --> 01:07:33.870
just gain the trust of the
people in the field who are

01:07:33.870 --> 01:07:35.610
working for that organization.

01:07:35.610 --> 01:07:38.460
And some of this comes about in
getting their feedback and

01:07:38.460 --> 01:07:41.510
input into things like
survey design.

01:07:41.510 --> 01:07:44.380
Having it so they feel part of
the process, and their input

01:07:44.380 --> 01:07:46.810
is received and incorporated
into what we're doing.

01:07:46.810 --> 01:07:48.490
And that's good for the
program, good for the

01:07:48.490 --> 01:07:49.820
evaluation to get
their feedback.

01:07:49.820 --> 01:07:54.990
It's also good in a purely
interpersonal way, in terms of

01:07:54.990 --> 01:07:59.020
helping to have a relationship
that's good by making sure

01:07:59.020 --> 01:08:02.800
that people feel that they
are part of that process.

01:08:07.600 --> 01:08:09.420
So some other specific
things that come up.

01:08:09.420 --> 01:08:12.620
The first, one of the most
common things is gossip.

01:08:12.620 --> 01:08:13.870
People will talk.

01:08:15.960 --> 01:08:19.060
So what do we do if the control
group finds out about

01:08:19.060 --> 01:08:20.310
the program?

01:08:22.819 --> 01:08:27.910
So I think the thing to think
about is to try to separate

01:08:27.910 --> 01:08:29.370
out these types of issues.

01:08:29.370 --> 01:08:31.510
Let's just put this into a more
general category called

01:08:31.510 --> 01:08:33.470
spillovers.

01:08:33.470 --> 01:08:37.720
So spillovers meaning there's
any sort of indirect effects

01:08:37.720 --> 01:08:40.510
that are going to occur, from
those who were treated to

01:08:40.510 --> 01:08:42.649
those who are untreated.

01:08:42.649 --> 01:08:44.710
I think it's really important
to separate these into two

01:08:44.710 --> 01:08:45.910
categories.

01:08:45.910 --> 01:08:49.300
There's natural spillovers, and
let's call them research

01:08:49.300 --> 01:08:51.470
spillovers.

01:08:51.470 --> 01:08:54.359
Now, by a natural spillover,
what do I mean here?

01:08:54.359 --> 01:08:57.279
I mean a spillover that is
naturally occurring.

01:08:57.279 --> 01:09:00.279
That is, if you go and you
provide a service to 100

01:09:00.279 --> 01:09:03.130
people, the fact is this is
going to affect those 100

01:09:03.130 --> 01:09:05.450
people and 200 more.

01:09:05.450 --> 01:09:06.920
And that's the nature
of the intervention.

01:09:06.920 --> 01:09:09.899
It has nothing to do
with the research.

01:09:09.899 --> 01:09:12.680
The example that you have
a case on is deworming.

01:09:12.680 --> 01:09:13.930
We're going to deworm
half of you.

01:09:13.930 --> 01:09:16.180
The other half will benefit
from that, because you're

01:09:16.180 --> 01:09:17.630
going to be less likely
to catch the worms

01:09:17.630 --> 01:09:19.880
from the first half.

01:09:19.880 --> 01:09:22.779
Let's say I took half of you
right now, and I went into the

01:09:22.779 --> 01:09:24.680
other room, and I gave you a
whole big lesson in power

01:09:24.680 --> 01:09:28.359
calculations, and I ignored the
other half, and I didn't

01:09:28.359 --> 01:09:29.050
give that to you.

01:09:29.050 --> 01:09:30.160
Well, there'd be a spillover.

01:09:30.160 --> 01:09:32.069
You'd come back, you're in
a group, you'd talk.

01:09:32.069 --> 01:09:34.430
Oh no, no, I just learned about
power calculations, let

01:09:34.430 --> 01:09:35.770
me show you.

01:09:35.770 --> 01:09:37.020
Hopefully it'd be
a positive one.

01:09:39.930 --> 01:09:43.290
So these are all natural
spillovers though.

01:09:43.290 --> 01:09:44.850
There's learning that
takes place.

01:09:44.850 --> 01:09:46.920
You teach some people,
they teach others.

01:09:46.920 --> 01:09:50.600
You deworm schoolchildren, other
schoolchildren benefit

01:09:50.600 --> 01:09:53.020
because they're less likely
to catch the worms.

01:09:53.020 --> 01:09:54.840
There could be negative
spillovers.

01:09:54.840 --> 01:09:57.840
We go and we offered really,
really cheap credit to some

01:09:57.840 --> 01:10:01.400
people, or we only offer it
to some because we're

01:10:01.400 --> 01:10:01.810
constrained.

01:10:01.810 --> 01:10:02.670
That's the organization.

01:10:02.670 --> 01:10:04.740
That's just how many
loans we make.

01:10:04.740 --> 01:10:06.490
It gives them a competitive
advantage--

01:10:06.490 --> 01:10:07.920
I'm not saying this is right.

01:10:07.920 --> 01:10:10.960
But this is an argument that
people will make when they are

01:10:10.960 --> 01:10:14.650
arguing against subsidized
microcredit.

01:10:14.650 --> 01:10:17.190
And what does this do to the
people who don't get access to

01:10:17.190 --> 01:10:19.480
the microcredit loans?

01:10:19.480 --> 01:10:20.330
It shuts them out of business.

01:10:20.330 --> 01:10:22.590
It makes it so they can't
operate their enterprise

01:10:22.590 --> 01:10:24.220
because they're competing
against someone who's getting

01:10:24.220 --> 01:10:26.300
subsidized credit.

01:10:26.300 --> 01:10:28.900
So that has a negative
spillover.

01:10:28.900 --> 01:10:30.390
These are natural, though.

01:10:30.390 --> 01:10:33.760
So a good study is one that
helps to measure these things.

01:10:33.760 --> 01:10:36.110
And there are ways that we can
design experiments to measure

01:10:36.110 --> 01:10:37.850
those types of spillovers.

01:10:37.850 --> 01:10:41.560
So a very simple example of
one that measures this is

01:10:41.560 --> 01:10:44.490
suppose we have villages.

01:10:44.490 --> 01:10:51.410
And what we're going to do is
take 90 villages, and instead

01:10:51.410 --> 01:10:54.330
of just dividing them up
treatment, control, what we're

01:10:54.330 --> 01:10:57.810
going to do is we're going to
divide them into three piles.

01:10:57.810 --> 01:10:59.780
We're going to divide
them first into two,

01:10:59.780 --> 01:11:01.190
treatment and control.

01:11:01.190 --> 01:11:03.850
So we'll have 60 of those
villages being treatment and

01:11:03.850 --> 01:11:05.270
30 being control.

01:11:05.270 --> 01:11:07.340
And then within the 60 that
are treatment, we're only

01:11:07.340 --> 01:11:09.650
going to go and deliver services
to half the people in

01:11:09.650 --> 01:11:12.080
those villages.

01:11:12.080 --> 01:11:13.160
So what do we have?

01:11:13.160 --> 01:11:16.140
We have a treatment village
that's half treated, half

01:11:16.140 --> 01:11:19.125
untreated, and we have
control villages.

01:11:19.125 --> 01:11:20.540
Now throw away the people
that got treated.

01:11:20.540 --> 01:11:22.190
Just ignore them.

01:11:22.190 --> 01:11:25.600
What's an interesting analysis
to do here is to compare the

01:11:25.600 --> 01:11:28.590
untreated children or people,
or whatever the intervention

01:11:28.590 --> 01:11:31.590
is in the treatment
villages, and

01:11:31.590 --> 01:11:33.950
compare them to the control.

01:11:33.950 --> 01:11:36.100
These are two people that
didn't get services.

01:11:36.100 --> 01:11:37.700
Neither one got treated.

01:11:37.700 --> 01:11:40.510
But some of them live near
people who got treated and

01:11:40.510 --> 01:11:43.170
some of them do not.

01:11:43.170 --> 01:11:44.790
So that measures the
indirect effect.

01:11:44.790 --> 01:11:47.630
That measures the natural
spillover.

01:11:47.630 --> 01:11:50.210
So if a natural spillover is
something that one was

01:11:50.210 --> 01:11:52.960
concerned with, this is exactly
the way you would

01:11:52.960 --> 01:11:56.090
think ahead of time about
setting up the research design

01:11:56.090 --> 01:11:58.630
to measure that.

01:11:58.630 --> 01:12:01.080
But then there's unnatural
spillover, what I was

01:12:01.080 --> 01:12:02.430
referring to as research
spillovers.

01:12:02.430 --> 01:12:02.920
Yeah?

01:12:02.920 --> 01:12:03.788
AUDIENCE: Just a
quick question.

01:12:03.788 --> 01:12:06.910
Does the fact that now the
treatment is half the size

01:12:06.910 --> 01:12:09.030
compared to the entire control
group, does it matter?

01:12:12.150 --> 01:12:13.090
PROFESSOR: Yes, it matters.

01:12:13.090 --> 01:12:16.570
And that's a question of
power calculations.

01:12:16.570 --> 01:12:22.900
And so you have to trade off
your measurement of the

01:12:22.900 --> 01:12:24.120
spillover versus your
measurement

01:12:24.120 --> 01:12:25.050
of the direct effect.

01:12:25.050 --> 01:12:27.460
But that's a mathematical
problem that can be solved

01:12:27.460 --> 01:12:28.710
analytically.

01:12:36.950 --> 01:12:38.740
So research spillovers, those
are the bad ones.

01:12:38.740 --> 01:12:41.580
These are the ones we don't
like, because it's not

01:12:41.580 --> 01:12:43.120
interesting, it's not useful.

01:12:43.120 --> 01:12:45.400
It's not representative of what
happens in the real world

01:12:45.400 --> 01:12:46.480
when you do an intervention.

01:12:46.480 --> 01:12:50.880
It's just an artifact of
the research process.

01:12:50.880 --> 01:12:54.990
The simplest example is the
control group person who says,

01:12:54.990 --> 01:12:58.300
I don't like the fact that I
am in the control group.

01:12:58.300 --> 01:13:01.580
Maybe they don't believe it was
truly random, or they just

01:13:01.580 --> 01:13:04.612
don't like the fact that they
didn't win the lottery.

01:13:04.612 --> 01:13:06.470
And so they actually change
their behavior

01:13:06.470 --> 01:13:07.720
now because of this.

01:13:10.360 --> 01:13:13.310
Let's use a very simple example
of being in a bank,

01:13:13.310 --> 01:13:16.600
and let's say you're doing
a lottery across existing

01:13:16.600 --> 01:13:19.770
borrowers, and half of them
got an extra service to go

01:13:19.770 --> 01:13:22.930
along with their loan,
and others did not.

01:13:22.930 --> 01:13:24.070
The ones who didn't
get the extra

01:13:24.070 --> 01:13:25.430
service, they're now upset.

01:13:25.430 --> 01:13:28.210
I didn't get the extra service,
I'm not happy.

01:13:28.210 --> 01:13:28.920
Why did they get it?

01:13:28.920 --> 01:13:30.480
I didn't get it.

01:13:30.480 --> 01:13:33.900
And you can explain, well,
it was random.

01:13:33.900 --> 01:13:35.740
They don't accept that that.

01:13:35.740 --> 01:13:36.500
And now what do they do?

01:13:36.500 --> 01:13:39.080
Maybe they don't pay
back their loan.

01:13:39.080 --> 01:13:40.460
Maybe they leave the
program altogether

01:13:40.460 --> 01:13:42.670
because they're mad.

01:13:42.670 --> 01:13:45.200
Now in studies we've had, I can
honestly tell you we've

01:13:45.200 --> 01:13:48.360
not had this happen yet in a
microcredit setting, but

01:13:48.360 --> 01:13:50.250
there's certainly things
that we will do to try

01:13:50.250 --> 01:13:51.550
to avoid the problem.

01:13:51.550 --> 01:13:54.680
So for instance, one of the
studies we had where this was

01:13:54.680 --> 01:13:58.130
a bigger concern than others was
we were testing out group

01:13:58.130 --> 01:13:59.960
versus individual liability.

01:13:59.960 --> 01:14:03.220
Now most borrowers really like
the idea of individual

01:14:03.220 --> 01:14:04.910
liability if they're
given a choice.

01:14:04.910 --> 01:14:07.290
They don't want to be on the
hook with other people in

01:14:07.290 --> 01:14:10.000
their community, they much
prefer to have a loan that's

01:14:10.000 --> 01:14:13.100
just to them and them alone.

01:14:13.100 --> 01:14:15.840
So when we were randomizing
whether people got offered

01:14:15.840 --> 01:14:19.680
group or individual
liability--

01:14:19.680 --> 01:14:22.510
and it was existing borrowers
who were already borrowing

01:14:22.510 --> 01:14:23.240
from a bank--

01:14:23.240 --> 01:14:26.080
what we had to do is take
villages that were really

01:14:26.080 --> 01:14:30.030
right next to each other
and put them together.

01:14:30.030 --> 01:14:31.790
Because we couldn't have it
that we had these little

01:14:31.790 --> 01:14:34.270
sister villages where there
was lots of interaction

01:14:34.270 --> 01:14:38.250
across, and one got switched
and the other did not.

01:14:38.250 --> 01:14:41.500
So we put them together and
treated them like one.

01:14:41.500 --> 01:14:43.600
And so basically, it's another
way of saying this is when you

01:14:43.600 --> 01:14:47.210
think this is an issue, you
just need to think about

01:14:47.210 --> 01:14:50.880
making sure that you have some
sort of boundaries separating

01:14:50.880 --> 01:14:54.120
out your treatment and
your control areas.

01:14:54.120 --> 01:14:57.700
Now in an urban setting, it
could be a little bit harder

01:14:57.700 --> 01:15:00.790
if you don't have clearly
defined boundaries.

01:15:00.790 --> 01:15:04.440
But it's still feasible to
do this type of process.

01:15:04.440 --> 01:15:06.770
You just have to think a little
bit about how to do the

01:15:06.770 --> 01:15:11.180
boundary, and also what to do
if you have control group

01:15:11.180 --> 01:15:12.280
people who do come in.

01:15:12.280 --> 01:15:14.740
And so this is an area where
encouragement designs might

01:15:14.740 --> 01:15:18.550
actually be a useful way of
dealing with it, is to allow

01:15:18.550 --> 01:15:21.090
some control group people in,
for instance, if they come in.

01:15:21.090 --> 01:15:25.150
But otherwise not unless they
actually come on their own.

01:15:25.150 --> 01:15:28.200
So this is another way of saying
take an urban area, and

01:15:28.200 --> 01:15:31.080
you just encourage some blocks
to borrow and not others.

01:15:31.080 --> 01:15:33.980
Encourage some blocks to go to
school and get some extra

01:15:33.980 --> 01:15:38.580
service, and not others,
whatever the program is.

01:15:38.580 --> 01:15:43.570
And that's a way of trying to
make sure that if the groups

01:15:43.570 --> 01:15:45.220
talk to each other, it's OK.

01:15:45.220 --> 01:15:48.130
It's not going to ruin the
study, and there's no

01:15:48.130 --> 01:15:49.740
jealousy, it's just a matter
of some receiving

01:15:49.740 --> 01:15:50.990
encouragement and others not.

01:15:58.110 --> 01:15:59.960
So fairness, hopefully this is--
the one point, if I had

01:15:59.960 --> 01:16:02.610
to leave you with one simple
thought of this lecture, it's

01:16:02.610 --> 01:16:03.680
the fairness point.

01:16:03.680 --> 01:16:06.990
It's perhaps the single most
commonly raised issue, and

01:16:06.990 --> 01:16:11.620
it's the easiest of the issues
to explain in 99.9% of the

01:16:11.620 --> 01:16:13.350
settings we're in.

01:16:13.350 --> 01:16:18.440
And it's this fairness issue of,
oh, but gosh, I don't want

01:16:18.440 --> 01:16:21.650
to do it by lottery, I want to
do it by some other process.

01:16:21.650 --> 01:16:24.430
And the answer--

01:16:24.430 --> 01:16:28.030
or gosh, I can't imagine
restricting access to people.

01:16:28.030 --> 01:16:31.580
And the answer is always just
about the same, which is how

01:16:31.580 --> 01:16:33.460
many people can you deliver
in this program?

01:16:33.460 --> 01:16:35.800
What's your budget?

01:16:35.800 --> 01:16:37.440
Now let's divide by the
cost per person.

01:16:37.440 --> 01:16:39.860
And so you can do this for
1,000 people, or 2,000,

01:16:39.860 --> 01:16:42.190
10,000, whatever your constraint
is, you have a

01:16:42.190 --> 01:16:43.440
constraint.

01:16:43.440 --> 01:16:46.800
Now all we're going to do is
use that constraint to then

01:16:46.800 --> 01:16:49.900
find a way to do this
randomization, and that's it.

01:16:49.900 --> 01:16:52.660
So we're not restricting
access to anyone.

01:16:52.660 --> 01:16:55.790
The only sense in which we're
restricting access is perhaps

01:16:55.790 --> 01:16:59.540
a bigger picture or thought,
which is if half a million

01:16:59.540 --> 01:17:02.180
dollars is being spent on the
evaluation, that's half a

01:17:02.180 --> 01:17:04.280
million dollars that's not
being spent on services.

01:17:04.280 --> 01:17:07.740
That's the only sense in which
the randomization is costly in

01:17:07.740 --> 01:17:08.690
terms of delivering services.

01:17:08.690 --> 01:17:11.090
But this is a different
calculation all together.

01:17:11.090 --> 01:17:14.600
This is now asking the question
of whether it's worth

01:17:14.600 --> 01:17:16.360
half a million dollars to
find out the impact of

01:17:16.360 --> 01:17:18.680
this program or not.

01:17:18.680 --> 01:17:20.430
And that's just a very
different question.

01:17:20.430 --> 01:17:22.430
It's not about the resources
and the fairness to those

01:17:22.430 --> 01:17:26.110
individuals, it's about the
question of whether this

01:17:26.110 --> 01:17:28.705
program will be done enough
times in the future, and will

01:17:28.705 --> 01:17:30.920
the marginal value in terms of
our knowledge about what's

01:17:30.920 --> 01:17:35.250
being learned from this study
be high enough to warrant

01:17:35.250 --> 01:17:37.140
spending the money on
the research period.

01:17:37.140 --> 01:17:38.390
It's a different question.

01:17:57.150 --> 01:17:59.840
So on ethics, some of the things
that we'll often hear

01:17:59.840 --> 01:18:01.790
are statements like, well, it's
wrong to use people as

01:18:01.790 --> 01:18:02.840
guinea pigs.

01:18:02.840 --> 01:18:07.660
Or if it works, then it's wrong
not to treat everyone.

01:18:07.660 --> 01:18:11.830
So the first thing to think
about is that first

01:18:11.830 --> 01:18:14.120
of all, it's not--

01:18:14.120 --> 01:18:19.410
I think it's a very leading
question first of all.

01:18:24.520 --> 01:18:26.810
One thing to often
ask yourself--

01:18:26.810 --> 01:18:30.050
or ask people in this type
of conversation--

01:18:30.050 --> 01:18:34.240
is why is this different than
prescription drugs?

01:18:34.240 --> 01:18:40.280
Why should we be more willing
to proceed and deliver

01:18:40.280 --> 01:18:42.400
interventions and deliver
services to people without

01:18:42.400 --> 01:18:47.760
knowing their impact then we
are to prescribe drugs?

01:18:47.760 --> 01:18:49.990
We ourselves, for instance,
would never take a

01:18:49.990 --> 01:18:52.360
prescription drug if it hadn't
gone through a randomized

01:18:52.360 --> 01:18:54.350
trial, or more than one.

01:18:54.350 --> 01:18:58.220
And so why should we be using
a different set of standards

01:18:58.220 --> 01:19:01.640
in terms of the ethics of the
two, in terms of the bar, the

01:19:01.640 --> 01:19:04.080
rigor that we want to use in
order to decide how to

01:19:04.080 --> 01:19:05.785
allocate our resources.

01:19:08.910 --> 01:19:11.080
The second thing to think
about in terms of, if it

01:19:11.080 --> 01:19:14.220
works, then it's wrong not to
treat everyone, I think

01:19:14.220 --> 01:19:18.030
there's an important point to
note, that there's lots of

01:19:18.030 --> 01:19:19.260
ideas that--

01:19:19.260 --> 01:19:20.810
there's two issues that often
come up in this setting.

01:19:20.810 --> 01:19:22.600
First of all, there's lots of
ideas that sound good, but

01:19:22.600 --> 01:19:25.282
then when evaluated, turn
out to not work.

01:19:25.282 --> 01:19:27.970
And even when something works,
the question is how

01:19:27.970 --> 01:19:28.850
well does it work?

01:19:28.850 --> 01:19:30.730
We have other ideas that work.

01:19:30.730 --> 01:19:33.920
So even if something is good,
even if everyone around the

01:19:33.920 --> 01:19:36.380
table is totally confident that
it's going to work in

01:19:36.380 --> 01:19:39.390
some respect, we don't know how
well it's going to work.

01:19:39.390 --> 01:19:41.790
And when we're allocating
resources, we're not just

01:19:41.790 --> 01:19:44.860
trying to beat zero, although
that's always good.

01:19:44.860 --> 01:19:47.710
We're actually trying to
do the best we can.

01:19:47.710 --> 01:19:49.750
And so we're choosing across
five ideas, and

01:19:49.750 --> 01:19:51.450
they all sound good.

01:19:51.450 --> 01:19:53.850
No one's throwing out ideas
that-- well, I shouldn't say

01:19:53.850 --> 01:19:54.860
that, that's probably
not true.

01:19:54.860 --> 01:19:56.750
We can probably think of some
that don't sound good.

01:19:56.750 --> 01:19:59.460
But for the most part, we like
to think that we're sitting

01:19:59.460 --> 01:20:02.060
around the table thinking across
choices that sound

01:20:02.060 --> 01:20:04.480
good, and we have to choose.

01:20:04.480 --> 01:20:09.570
And so that's the most important
thing to remember in

01:20:09.570 --> 01:20:11.390
that type of conversation.

01:20:11.390 --> 01:20:12.640
Cost.

01:20:14.980 --> 01:20:19.890
So there is two things that
often come up when people talk

01:20:19.890 --> 01:20:21.420
about cost.

01:20:21.420 --> 01:20:25.080
And there is a common perception
and argument that

01:20:25.080 --> 01:20:27.630
randomized trials are much
more expensive than other

01:20:27.630 --> 01:20:28.990
approaches.

01:20:28.990 --> 01:20:31.270
So I think there's two things to
think about when this type

01:20:31.270 --> 01:20:34.330
of conversation or
point is made.

01:20:34.330 --> 01:20:39.140
The first is thinking about
what the cost--

01:20:39.140 --> 01:20:41.830
it's about doing a cost
benefit analysis.

01:20:41.830 --> 01:20:43.090
So let's assume for a
second randomized

01:20:43.090 --> 01:20:44.020
trials are more expensive.

01:20:44.020 --> 01:20:45.660
And I'll point out some
examples in a

01:20:45.660 --> 01:20:47.070
moment where it's not.

01:20:47.070 --> 01:20:49.680
But let's say it is in
a given setting.

01:20:49.680 --> 01:20:51.090
Well, that's only part
of the equation.

01:20:51.090 --> 01:20:53.480
You have to say, well, what's
the cost and the benefit?

01:20:53.480 --> 01:20:55.230
The whole point of doing
randomized trial is to think

01:20:55.230 --> 01:20:56.810
about costs and benefits too.

01:20:56.810 --> 01:20:58.700
There's no reason why we should
think any differently

01:20:58.700 --> 01:21:01.320
when we think about
how to evaluate.

01:21:01.320 --> 01:21:03.400
So what's the benefit we're
going to get them from doing a

01:21:03.400 --> 01:21:05.380
randomized trial versus a
non-randonized trial, and

01:21:05.380 --> 01:21:07.030
what's the cost difference?

01:21:07.030 --> 01:21:09.150
And if the benefit that we're
going to get in terms of the

01:21:09.150 --> 01:21:13.990
reliability in results is high
enough to then make more

01:21:13.990 --> 01:21:17.250
impact on our ability to make
future decisions, well then,

01:21:17.250 --> 01:21:19.570
it's probably worth spending a
little bit more money on the

01:21:19.570 --> 01:21:21.260
evaluation itself.

01:21:21.260 --> 01:21:23.110
Now obviously, that's relative
to our existing

01:21:23.110 --> 01:21:24.980
knowledge in the space.

01:21:24.980 --> 01:21:28.710
If something's already been
tested 15, 20 times, then this

01:21:28.710 --> 01:21:31.820
might become a situation in
which you would argue that no,

01:21:31.820 --> 01:21:33.670
the benefits don't outweigh
it, because the marginal

01:21:33.670 --> 01:21:36.620
impact from the research on one
more study is not going to

01:21:36.620 --> 01:21:38.030
be that high.

01:21:38.030 --> 01:21:39.950
And so the costs are
not worth it.

01:21:39.950 --> 01:21:42.550
I'm not aware of situations
I would say fit that, but

01:21:42.550 --> 01:21:45.370
hopefully we will be
there someday.

01:21:45.370 --> 01:21:48.810
The second is that the cost of
doing randomized trials is

01:21:48.810 --> 01:21:50.900
often not actually more
expensive than

01:21:50.900 --> 01:21:51.890
non-randomized methods.

01:21:51.890 --> 01:21:54.840
But it's really key to state
clearly what the

01:21:54.840 --> 01:21:56.100
counter-factual is here.

01:21:56.100 --> 01:21:58.460
What's the alternative method
one's describing?

01:21:58.460 --> 01:22:00.940
So it's clearly cheaper
than doing nothing.

01:22:00.940 --> 01:22:03.460
And there are situations that
I've been in where my best

01:22:03.460 --> 01:22:04.990
advice is don't evaluate.

01:22:08.850 --> 01:22:11.520
For whatever reason,
the setting is--

01:22:11.520 --> 01:22:14.640
you're not going to get a
reliable result, and the best

01:22:14.640 --> 01:22:19.930
thing one can do is to not
do the evaluation.

01:22:19.930 --> 01:22:22.980
Let's compare now to the most
common comparison one makes,

01:22:22.980 --> 01:22:28.210
which is to a non-experimental
quantitative method.

01:22:28.210 --> 01:22:32.230
So suppose the alternative
approach is to survey a bunch

01:22:32.230 --> 01:22:35.760
of people who received a
service, and to survey a bunch

01:22:35.760 --> 01:22:37.660
of people who didn't
receive a service.

01:22:37.660 --> 01:22:38.455
It wasn't done randomly.

01:22:38.455 --> 01:22:41.140
It was some people chose
to be borrowers from

01:22:41.140 --> 01:22:44.270
a microcredit program.

01:22:44.270 --> 01:22:46.750
For those of you who know me
know I do a lot of work in

01:22:46.750 --> 01:22:47.860
microcredit, this is why I keep

01:22:47.860 --> 01:22:50.560
using microcredit examples.

01:22:50.560 --> 01:22:54.160
So I'm going to survey a whole
bunch of people in microcredit

01:22:54.160 --> 01:22:55.530
that are part of a program.

01:22:55.530 --> 01:22:57.550
And then I'm going to go into
the same community so I can

01:22:57.550 --> 01:23:00.300
find people who seem very
similar, have the same

01:23:00.300 --> 01:23:03.430
macroeconomic conditions, but
are not participating in the

01:23:03.430 --> 01:23:05.115
program and going
to survey them.

01:23:05.115 --> 01:23:07.150
I'm going to follow everybody
before and after.

01:23:07.150 --> 01:23:10.000
So this is actually a more
expensive study.

01:23:10.000 --> 01:23:11.010
Why?

01:23:11.010 --> 01:23:13.980
Well, I need a larger
sample size here.

01:23:13.980 --> 01:23:16.100
I need a larger sample size
because I actually have to

01:23:16.100 --> 01:23:19.190
really understand something
deeper now about who's opting

01:23:19.190 --> 01:23:20.310
in and who's not.

01:23:20.310 --> 01:23:24.960
And I have to try to use my
econometric tools to correct

01:23:24.960 --> 01:23:26.150
for selection biases.

01:23:26.150 --> 01:23:29.600
And this costs me sample size.

01:23:29.600 --> 01:23:32.210
And so this study would actually
cost more money,

01:23:32.210 --> 01:23:34.980
because I would want a larger
number of observations in the

01:23:34.980 --> 01:23:38.780
analysis in order to try
to get it right.

01:23:38.780 --> 01:23:40.480
Now it's clearly going
to be more expensive.

01:23:40.480 --> 01:23:42.200
Now let's flip to another one.

01:23:42.200 --> 01:23:46.570
If our counter-factual approach
is to not even do any

01:23:46.570 --> 01:23:50.570
surveys, like there's no big
econometrics, but instead to

01:23:50.570 --> 01:23:53.830
do a simple before after.

01:23:53.830 --> 01:23:54.790
I'm just going to compare
before after.

01:23:54.790 --> 01:23:57.120
So you've studied yesterday, you
talked about before after,

01:23:57.120 --> 01:23:59.960
you went through some of the
issues that you have that you

01:23:59.960 --> 01:24:02.910
don't know what else is changing
the environment.

01:24:02.910 --> 01:24:04.930
But if that's the comparison
you're going to be using, well

01:24:04.930 --> 01:24:06.610
then yeah, this is the more
expensive, because yopu've got

01:24:06.610 --> 01:24:10.540
to survey control group
people too.

01:24:10.540 --> 01:24:13.930
That's an example where it's
hard to come up with settings

01:24:13.930 --> 01:24:16.180
in which there aren't
outside factors--

01:24:16.180 --> 01:24:19.030
economic, social, environmental,
health--

01:24:19.030 --> 01:24:24.040
that cause changes over time in
outcomes for people, such

01:24:24.040 --> 01:24:27.440
that a simple before after
analysis is in anyway

01:24:27.440 --> 01:24:30.440
informative at all about the
impact of a program.

01:24:36.380 --> 01:24:40.003
Timing, OK.

01:24:40.003 --> 01:24:41.520
I'm going to try to
wrap up quickly.

01:24:41.520 --> 01:24:44.360
The one thing to say about
timing is it's very common to

01:24:44.360 --> 01:24:47.610
have a constraint where the
organization is like, but we

01:24:47.610 --> 01:24:48.860
need the answers now.

01:24:53.670 --> 01:24:57.670
Randomized trials are no
different than non-randomized

01:24:57.670 --> 01:25:00.660
trials that follow people before
after, but they're

01:25:00.660 --> 01:25:03.090
certainly going to be a lot
longer than things that simply

01:25:03.090 --> 01:25:05.070
look retrospectively at people
that have already received

01:25:05.070 --> 01:25:07.820
services and hold focus groups
and discussions to try to

01:25:07.820 --> 01:25:09.130
assess impact.

01:25:09.130 --> 01:25:10.690
And there's no way
around that.

01:25:10.690 --> 01:25:15.040
So this is a question of just
being patient, and working

01:25:15.040 --> 01:25:17.710
with organizations that are able
to be patient in order to

01:25:17.710 --> 01:25:18.960
have those answers.

01:25:28.190 --> 01:25:31.220
I'm just going to run through
this initial slide so you can

01:25:31.220 --> 01:25:36.560
have the basic key points of
the overall plan when we're

01:25:36.560 --> 01:25:39.840
doing an evaluation.

01:25:39.840 --> 01:25:42.650
So the three steps we have
listed here are plan, pilot,

01:25:42.650 --> 01:25:44.020
and implement.

01:25:44.020 --> 01:25:45.900
I think it is important to note
that there are situations

01:25:45.900 --> 01:25:47.130
where we don't actually
do a pilot.

01:25:47.130 --> 01:25:50.920
Depending on the circumstances,
the situations

01:25:50.920 --> 01:25:53.000
in which we do pilots are
typically when there's a lot

01:25:53.000 --> 01:25:55.320
of uncertainty about what the
intervention is in the first

01:25:55.320 --> 01:25:57.580
place, and so we're actually
working with the organization

01:25:57.580 --> 01:25:59.350
to figure that out.

01:25:59.350 --> 01:26:02.180
Or if there's some uncertainty
about the way the process is

01:26:02.180 --> 01:26:02.960
going to play out.

01:26:02.960 --> 01:26:05.420
Maybe we're uncertain about
the encouragement design.

01:26:05.420 --> 01:26:07.270
We're not sure if it's
going to work.

01:26:07.270 --> 01:26:09.160
We're not really sure, will this
actually encourage people

01:26:09.160 --> 01:26:11.320
to come in and use
a service more so

01:26:11.320 --> 01:26:12.560
than they would otherwise?

01:26:12.560 --> 01:26:16.270
So we need a pilot to test out
whether that approach will

01:26:16.270 --> 01:26:17.550
have an effect or not.

01:26:17.550 --> 01:26:20.230
We just need a smaller sample
just to gauge whether we're

01:26:20.230 --> 01:26:23.310
dealing with 60% take up rate
in our treatment group as a

01:26:23.310 --> 01:26:26.240
result of encouragement and 10
in the control, or are we

01:26:26.240 --> 01:26:28.750
dealing with 12 and 10.

01:26:28.750 --> 01:26:31.380
What's our range?

01:26:31.380 --> 01:26:34.970
So the five steps we've laid
out here is identify the

01:26:34.970 --> 01:26:37.030
problem and proposed solution.

01:26:37.030 --> 01:26:41.130
So I think one of the things
that should never escape us is

01:26:41.130 --> 01:26:44.480
that you don't take a
randomized trial.

01:26:44.480 --> 01:26:48.810
You don't start off saying we
have a tool, now what research

01:26:48.810 --> 01:26:50.040
questions can we ask?

01:26:50.040 --> 01:26:51.090
You go the other way around.

01:26:51.090 --> 01:26:52.910
You want to think, well, what's
the research question

01:26:52.910 --> 01:26:54.020
we're asking here?

01:26:54.020 --> 01:26:57.225
What is the problem that we see
in the market we're in, in

01:26:57.225 --> 01:26:58.505
the society we're in?

01:27:01.640 --> 01:27:04.810
What's the market failure that
this intervention is trying to

01:27:04.810 --> 01:27:07.350
solve or measure or test,
and then what's

01:27:07.350 --> 01:27:09.990
the proposed solution?

01:27:09.990 --> 01:27:12.780
Think totally abstractly, don't
even get into what the

01:27:12.780 --> 01:27:14.830
randomized trial is and how
it will be designed.

01:27:14.830 --> 01:27:17.740
Just think first order about
what the market failure is,

01:27:17.740 --> 01:27:21.640
and what the logic is behind
the proposed solution.

01:27:21.640 --> 01:27:23.770
Second, and this goes back
somewhat when we're talking

01:27:23.770 --> 01:27:25.860
about in terms of identifying
the key players.

01:27:25.860 --> 01:27:28.930
There's nothing more frustrating
than a really good

01:27:28.930 --> 01:27:32.030
project, where you just don't
have the right players on

01:27:32.030 --> 01:27:36.620
board participating and
collaborating in a cooperative

01:27:36.620 --> 01:27:37.915
way to make the project work.

01:27:41.110 --> 01:27:43.110
Identify the key operations
questions to

01:27:43.110 --> 01:27:44.170
include in the study.

01:27:44.170 --> 01:27:47.850
This goes back to hopefully the
other theme of my lecture

01:27:47.850 --> 01:27:50.810
this morning, is about making
the research into win win

01:27:50.810 --> 01:27:54.040
opportunities, finding those
operation questions and trying

01:27:54.040 --> 01:27:57.620
to build them into the
research as well.

01:27:57.620 --> 01:28:02.070
Then design the randomization
strategy, and define the data

01:28:02.070 --> 01:28:03.550
collection plan.

01:28:03.550 --> 01:28:06.850
Data collection can be done
continuously lots of waves,

01:28:06.850 --> 01:28:07.810
one wave at the end.

01:28:07.810 --> 01:28:10.250
There's lots of other tools
that we can use in data

01:28:10.250 --> 01:28:15.140
collection, both qualitative and
quantitative approaches.

01:28:15.140 --> 01:28:18.460
One of the other common
misperceptions that I've heard

01:28:18.460 --> 01:28:20.765
is people saying that there's a
spectrum between qualitative

01:28:20.765 --> 01:28:22.015
and randomized trials.

01:28:25.380 --> 01:28:27.950
That's kind of mixing
apples and oranges.

01:28:27.950 --> 01:28:31.350
Qualitative versus quantitative
is about how you

01:28:31.350 --> 01:28:34.350
go about measuring things
and what you measure.

01:28:34.350 --> 01:28:37.670
A randomized trial is just about
identification of the

01:28:37.670 --> 01:28:39.420
effect of an intervention.

01:28:39.420 --> 01:28:41.050
It's about random assignments of
treatment and control, but

01:28:41.050 --> 01:28:43.640
it has nothing to do with
whether the measurement is

01:28:43.640 --> 01:28:45.290
going to be done through
a qualitative or

01:28:45.290 --> 01:28:46.690
quantitative process.

01:28:46.690 --> 01:28:49.130
And there's a lot of examples
of studies that we have that

01:28:49.130 --> 01:28:52.680
use mixed methods and creative
approaches for measuring

01:28:52.680 --> 01:28:54.250
things, and there's a lot of
studies we have where it's

01:28:54.250 --> 01:28:59.220
very cut and dry, normal
quantitative, how many

01:28:59.220 --> 01:29:00.755
potatoes did you eat
type questions.

01:29:09.730 --> 01:29:13.750
So pilots vary in
size and rigor.

01:29:13.750 --> 01:29:16.080
The pilots and the qualitative
steps that often go into them

01:29:16.080 --> 01:29:18.930
are very important for helping
to understand the intervention

01:29:18.930 --> 01:29:22.580
and design it, particularly when
we get into designs that

01:29:22.580 --> 01:29:24.760
are doing sub-treatments.

01:29:24.760 --> 01:29:27.570
A lot of times those come out of
the qualitative process in

01:29:27.570 --> 01:29:28.820
the design of a study.

01:29:37.220 --> 01:29:40.460
And then for the actual
implementation--

01:29:40.460 --> 01:29:41.790
oh, I skipped something.

01:29:56.780 --> 01:29:58.920
Identifying the actual target
population is going to be

01:29:58.920 --> 01:30:01.690
covered later in the day,
in the second lecture.

01:30:01.690 --> 01:30:03.920
And then collecting the
baseline data will be

01:30:03.920 --> 01:30:06.020
discussed later on.

01:30:06.020 --> 01:30:09.350
When we do it, we usually
do it, but not always.

01:30:09.350 --> 01:30:15.310
The actual randomization, there
is various times and

01:30:15.310 --> 01:30:16.030
points in which you do it.

01:30:16.030 --> 01:30:17.630
This is what we were talking
about in the beginning of the

01:30:17.630 --> 01:30:20.700
class, real time randomization
like the credit scoring all at

01:30:20.700 --> 01:30:25.000
once, villages known up front,
and you randomize them in or

01:30:25.000 --> 01:30:26.250
not to a program.

01:30:31.600 --> 01:30:35.300
Then the next phase is
implementation intervention to

01:30:35.300 --> 01:30:38.320
the treatment groups, and this
is where internal controls can

01:30:38.320 --> 01:30:39.700
be really critical.

01:30:39.700 --> 01:30:44.750
There's nothing worse than doing
all of this work, doing

01:30:44.750 --> 01:30:48.060
all these surveys, and then not
having the right control

01:30:48.060 --> 01:30:52.270
in the field to be working with
the individuals from the

01:30:52.270 --> 01:30:54.050
organizations that are
delivering services to make

01:30:54.050 --> 01:30:55.310
sure that things happen
the way they're

01:30:55.310 --> 01:30:56.950
actually supposed to happen.

01:30:56.950 --> 01:31:00.370
And I've had projects go bust,
where we're working with

01:31:00.370 --> 01:31:03.970
organizations that thought they
had the right internal

01:31:03.970 --> 01:31:05.260
controls in place.

01:31:05.260 --> 01:31:08.730
And when we go in to do spot
checks to see, and we go to

01:31:08.730 --> 01:31:12.110
some villages to see, are they
getting services or not?

01:31:12.110 --> 01:31:13.670
And lo and behold,
they were not.

01:31:13.670 --> 01:31:15.516
Or they were when they
shouldn't be.

01:31:15.516 --> 01:31:17.610
And we go back and we try
to work with them.

01:31:17.610 --> 01:31:20.490
I've had at least one project I
can point to that literally

01:31:20.490 --> 01:31:22.680
we just canceled after
a year and a half.

01:31:22.680 --> 01:31:25.720
It was very unfortunate, but
this is what happens when

01:31:25.720 --> 01:31:28.570
there wasn't the right level of
internal controls in place.

01:31:28.570 --> 01:31:29.820
And I learned.

01:31:33.560 --> 01:31:34.870
And then measuring
the questions.

01:31:34.870 --> 01:31:37.320
One of the most common
questions we get with

01:31:37.320 --> 01:31:39.000
measuring is, how long
should we wait?

01:31:39.000 --> 01:31:43.930
And there's really no
one answer to this.

01:31:43.930 --> 01:31:46.420
There's often a trade
off with operations.

01:31:46.420 --> 01:31:48.590
If there is any sort of holding
back of a control

01:31:48.590 --> 01:31:51.200
area, then this is going to be
something that has to be

01:31:51.200 --> 01:31:53.560
negotiated and discussed
with operations.

01:31:53.560 --> 01:31:56.050
In a lot of situations we're in
though, it's not that the

01:31:56.050 --> 01:31:57.540
two sides are actually
differing--

01:31:57.540 --> 01:32:00.650
I mean, the operations maybe--
but the head of the

01:32:00.650 --> 01:32:03.090
organization might have
incentives that are perfectly

01:32:03.090 --> 01:32:04.010
aligned with the researchers.

01:32:04.010 --> 01:32:06.090
They want to wait long enough
in order to make sure that

01:32:06.090 --> 01:32:07.510
they've given their
program a full

01:32:07.510 --> 01:32:09.860
chance to have its impact.

01:32:09.860 --> 01:32:12.790
And so usually when I am posed
with this question by an

01:32:12.790 --> 01:32:15.220
organization, I usually just ask
right back to them, well,

01:32:15.220 --> 01:32:16.500
you tell me.

01:32:16.500 --> 01:32:19.310
What do you think you need in
order to see the impact of

01:32:19.310 --> 01:32:20.350
your program?

01:32:20.350 --> 01:32:22.790
If you're telling me a story
about it being a 5, 10 year

01:32:22.790 --> 01:32:25.220
program in order to see
everything flourish, well then

01:32:25.220 --> 01:32:26.950
that's your answer.

01:32:26.950 --> 01:32:29.570
If you're telling me that this
is like an amazing thing that

01:32:29.570 --> 01:32:34.250
just transforms people's lives
within six months, well then

01:32:34.250 --> 01:32:36.140
we can go in six months
and see that amazing

01:32:36.140 --> 01:32:38.010
transformation.

01:32:38.010 --> 01:32:40.150
We might also want to see the
two year impacts, but that

01:32:40.150 --> 01:32:42.620
would be something that could
happen with the organization,

01:32:42.620 --> 01:32:44.540
and they could say, yes, we
think it's transformation in

01:32:44.540 --> 01:32:45.080
six months.

01:32:45.080 --> 01:32:47.560
And two years is just beyond--

01:32:47.560 --> 01:32:49.280
I mean, I don't know what the
word is to say beyond

01:32:49.280 --> 01:32:50.530
transferring.

01:32:52.450 --> 01:32:54.440
Lastly, analyse and
assess results.

01:32:54.440 --> 01:32:56.910
And obviously, there's a lot
more in the class that will be

01:32:56.910 --> 01:32:58.160
discussing that.