ANNOUNCER: The following content is provided under a
Creative Commons license.
Your support will help MIT OpenCourseWare continue to
offer high quality educational resources for free.
To make a donation or to view additional materials from
hundreds of MIT courses, visit MIT OpenCourseWare at
ocw.mit.edu.
PROFESSOR: Great, hi everyone.
It's great to be here, and I hope you guys are
having fun so far.
It's great to meet all of you too.
So this is the day where we talk about how to randomize.
A few other topics will come up here and there, and
hopefully you will stop me whenever you have questions,
and let me know.
So don't hesitate to stop even if it's just a clarifying
point or a deeper, more substantive question about
what I'm saying.
The three basic components of this morning's lecture will be
first about methods of randomization, and that'll
be-- the majority of what we'll talk about is going
through a few different ways that we talk about
randomization when we talk randomization.
One of the common misperceptions that I've seen
in the world when I am meeting with organizations or
governments for the first time, and they've heard of a
randomized trial as a concept, they often have a very well
defined and narrowly defined concept of what that means.
And the fact is, there's a lot of creative ways that we go
about doing randomized trials that adapt to different
settings, because there's a lot of situations in which you
can't do what might be considered the most standard
prescription drug type randomized trial.
And so we have to be a little bit creative in settings and
understanding, what are the constraints we're facing, and
how can we adapt the methodology to fit in this
particular setting?
Or maybe not, right?
So that's going to be methods of randomization, we'll go
through a few of the key approaches that we use.
The second, and this is a topic which really--
to say this is topic one, topic two, topic three isn't
quite exactly right.
And by the time we finish one, we're going to have talked a
lot about number two and number three, a lot of the
number two being gathering support for evaluation.
And the point here is that one of the reasons why we choose
one method over another when we're thinking about how to go
about setting up the design is because some methods are going
to be easier for gathering support for evaluation, and so
that's part of a back and forth process with
organizations and situations.
And then we're going to try to walk through a typical plan,
so to speak.
Perhaps in my mind, one of the single most important things
to remember about doing an evaluation is to remember that
we're not trying to just ask, how did we do?
There's nothing wrong with asking that, but it's very
short-sighted.
What we should be asking is, what should we do?
And that's the point of a good evaluation, is to guide us in
future decisions.
If you're a donor, and you're running a huge initiative, and
you're spending $20 million or something--
I'm just picking a number--
and you want to do an evaluation of this.
But because of whatever the nature of your program is,
this is it, this is the only time you're ever going to do
it, and it's a weird program that you believe in, but it's
weird, and no one else is ever going to do it.
I realize it's kind of a weird example, just go with me.
And so that's a situation in which, I think, most
reasonable people would say, why are you doing an
evaluation?
What's the point?
Is it just to pat yourself on the back?
Is that really the goal?
Because if there's not future money that's at stake, future
money that we have to decide how are we going to spend,
what's the point of doing all this, other than just to see
whether I made a good decision in the past or not?
But that's not really useful, that's not why we're here.
We're here because we realize that there are tons of future
decisions being made, and we need better information in
order to make those decisions.
And we need those as donors, but we also need those as
organizations.
And that's one of the key things going back to point
number two in the outline.
How do you get organizations on board and excited and
involved in evaluation?
It's when the evaluation is actually able to speak to
questions that they have.
And so good evaluations often help to identify the key
implementer's questions and answer them.
And I know I'm making that sound really simple, like oh,
that's all we have to do.
But the key really is to taking that type of approach
when working with organizations.
How do you turn these things into win win for operations?
If you're a leader in an organization, and you're not a
researcher, so you kind of understand the value of
research, and it sounds like a good thing, but you're hired
because you need to go and deliver these services, and
you need to be efficient in delivering your services.
You want to know that the research is nice and good and
needs to be done, but isn't going to get in your way.
Or if it is going to get in your way, you're going to get
something for it.
And that's a very common attitude, and I can respect
that attitude, if someone is just really focused on
operations.
And so the question is, how can we design research, how
can we listen to what the operations people are saying
about what their challenges are, what their struggles are,
the choices that they're making that are tough?
And actually build into the research ways of helping them
answer those questions.
So that's something that we often aim to do.
Methods of randomization.
I'm going to walk through four different methods that we will
often use: basic lottery, a phase in, a rotation, and
encouragement.
And we'll talk about each of these.
These are not all mutually exclusive
methods, to be clear.
So I'm just going to point out there's different ways and
levers for doing things.
Is that readable?
OK.
Sorry, let me skip that slide, and we'll come back to it at
the end to recap.
So let's start with the simplest, which is a lottery.
A lottery is like a clinical trial, where if we were
running a test for prescription drugs, it would
be a very standard, regimented process.
We'd be perhaps in some hospital.
We'd have some sort of intake process with people who are in
the hospital and have certain criteria, and then we'd
approach them and say, there's a new drug.
It's experimental.
There's risks, there's potential
rewards, it's up to you.
You have the following disease, these are the issues
you're facing.
What do you want to do?
But that's a situation where we would take
1,000 of the people.
They would all get informed consent being told about what
the risks were.
There would be parameters set up so that if the outcomes
were proving very decisive one way or another,
the study would end.
But it's fairly straightforward from a
statistical standpoint and from a research design
standpoint.
You bring them in, 1,000 are entered into the study, 500 of
them are randomly chosen to get a pill, 500 are randomly
chosen to get a placebo, and you measure the outcomes,
whatever they may be, from getting the
pill versus the placebo.
And so the question is, can we apply this in social science,
outside the laboratory type setting?
So some of the constraints that we face when we try to
take that really simplistic way of doing things.
So the first is that we often can't--
when you're doing a randomized trial on a prescription drug,
the whole point is the research study.
That is what it is.
There is no program around this.
It's a study to see what the effectiveness of a
particular pill is.
If we're working with an organization that is trying to
do teacher training in schools, trying to issue
loans, trying to promote savings, trying to teach
agricultural practices, there's a program
there that's involved.
And you can't just go in and change things around in the
program without paying attention to what this
actually means for the goals of the program has.
So the second really important element is that it must be
perceived as fair.
Now the thing that's the single most common situation
that I find myself in is when I'm dealing with organizations
that are capacity constrained.
They only have enough money to go to 200 schools.
They only have enough money to make 1,000 loans.
They only have enough marketing people to visit
5,000 households and promote savings, or whatever the
activity is they're doing, there's some capacity
constraint.
And so one principle that I've actually taught my children,
very young, they all know--
so like last week, we had to fly, and I hadn't seen them
for the weekend.
So the kids all wanted to sit next to me.
I have three kids.
And it was resolved very simply,
no whining, no nothing.
We just randomly chose one.
And they all knew immediately.
Now in fairness, I've done this before.
But they knew instinctfully just that this is-- there's no
complaining here.
There's no favoritism.
We took each of their boarding passes, we
flipped them upside down.
We chose one.
Actually I said it backwards.
It was one that couldn't sit with me.
It was two with me and one without, so we chose the one
who didn't.
Now ironically, the two that did win ended up both falling
asleep immediately, and so I switched seats in the end.
But that's a different issue.
But the point hopefully is clear, that there's really, in
some respect, nothing more fair than a random process.
It is giving everybody who is eligible, who's aware or not
aware, depending on how the set up is, but everybody who
is within a setting of--
who has access to a program, and giving them all the same
chance of participating.
Now in a lot of settings, we would actually suggest that
this is actually more fair than, for instance, letting
politics and letting nepotism and letting any sort of other
favoritism play in to deciding who gets
things and who doesn't.
Which we all know, in many places are serious issues for
the allocation of any sort of resource, that depending on
how it happens, then that could actually be the worst
possible outcome that we would want to see as
philanthropists, as utilitarian individuals
interested first and foremost in alleviating poverty.
But you could tell other stories in which there is
processes where you really want to reach the very poorest
of the poor.
And so you have to--
let's come back to that issue towards the end.
I'll use that as an example later in the lecture.
So it must be politically feasible.
Now, it might not be politically feasible if people
who have the power in the local settings are not willing
to do it, because they want to be able to choose the people
in their networks to provide the services to them.
So the politically feasible could be for the exact reason
that we want to do it randomly.
It also can just be politically infeasible for
other interpersonal reasons.
The right people who need to be part of the decision making
process just haven't really brought into the value of an
evaluation.
And there are situations we face all the time like that,
where there's nothing we can do.
There's just someone who is the decision maker who needs
to be on board with something, and is simply not on board
with doing a rigorous evaluation, and perhaps
concerned about the results that are going to come out of
the project, in terms of is suspicious of external
evaluators.
They feel like we know what works and what doesn't, and
they don't want to relax control to an outsider.
There's also obviously situations where people just
feel like what do I have to lose--
I'm sorry, what do I have to gain?
Where organizations will be of the ilk that, look, we have
lots of media, lots of attention
for what we're doing.
Everybody tells us it's a great idea.
We're executing, we're implementing.
What do I have to gain from doing an evaluation.
If we absorbed more money right now, I wouldn't know how
to spend it.
And there are organizations that I've interacted with that
are more or less of that ilk.
And so there's clearly situations like that.
Must be ethical.
So this is something we're always very concerned about
and cautious about.
And there are many situations we find ourselves for the
reasons I mentioned above, in terms of fairness, that the
random process is arguably the more ethical process.
But they're clearly situations that someone could put forward
where you could say, well, wait a
second, that's not good.
That's not right.
So these are issues that one has to be sensitive to.
I think one of the things that's most important here is
not necessarily whether something's-- we might all be
able to analytically agree on the ethics, but that doesn't
mean that everybody else will agree and perceive things the
way we perceive them.
And so even if we can analytically understand that a
random process is fair, if someone is just bringing a
certain bias to the table in terms of the way they think
about a random processes, then this can be a problem.
And again, this is one of persuasion and personality
more so than logic and philosophy.
So, why resource constraints are an evaluators best friend.
I think I've actually now said most of this, but basically,
most programs have limited resources.
And so examples of where this has been done is training
programs for entrepreneurs or farmers.
School vouchers is perhaps one of the single most common
examples we've seen this done, where there's a government
program to provide school vouchers for private school or
secondary school, or even college, and they just can't
educate the entire population of people who want to go to
private school.
And so there's a voucher program, there's an enrollment
process, you apply.
There's a public lottery literally done openly through
the newspapers or on TV.
And that type of transparency is done in order to make it
politically feasible, so that everybody can see.
If it's not done that way, then the concerns become about
whether is really truly random, it was done behind
closed doors.
Was the politician really just kind of pulling out his
favorite people?
So lotteries become, in the simplest of case, it's often
the starting point.
If we can pull off a public lottery in this way, or
private, either way, then it really does become the
simplest approach.
So when it's possible, it's nice.
But there are clearly situations that come up where
this is not going to be possible.
So there's also flexible ways of doing the lottery.
So let's go through a few different scenarios here.
So first of all, there's often a question about the unit over
which you randomize.
What we mean by this is, if you're going to do a lottery,
do you do a lottery at the individual level?
Or do you do a lottery, for instance, at the village, by
offering an entire village a package of services?
Or if you're building wells, for instance, this is actually
a treatment that's really being done at the village, not
the individual.
There might be some individuals who live closer to
the well than others further, but you're really choosing a
village and building wells.
And so the lottery system is flexible in this way.
The general concept is the same.
We've done interventions in Ghana, for instance, where the
randomization was done at the village level for receiving a
community driven development type program.
And so different geographic clusters were all put into a
public lottery, and half of them randomly chosen to
receive financial resources in order to help build an
epicenter, and lots of labor in terms of helping and
guiding and facilitating the process for building these
quote, epicenters.
So the second point is that sometimes when we do
lotteries, sometimes it's kind of a full versus partial.
What we mean by this is whether it's done all at once
in one kind of public way, or whether it's done through an
ongoing process.
And depending on how things are set up, sometimes it's
done one way or the other.
So the school vouchers is an example where we would do it
all at the very same time.
There's a whole bunch of people who apply for a school
voucher program.
There's 5,000 applicants, there's 2,000 school vouchers,
we randomly choose 2,000, they get the school vouchers.
In a situation that we've done in South Africa and also the
Philippines, when we build it into a credit scoring process.
And so every day, there's 10 applications for a loan.
The bank is not going to issue all the loans.
They take those 10, they put them in three
piles, yes, no and maybe.
And then what happens is they take the maybes and they say,
we only want to make half of these.
And then we randomize which half of those get credit and
which do not.
And this is something we've done now a few times in order
to measure the impact of credit.
From the bank's perspective, this is exactly one of the
cases I was talking about before where there's a win win
from an operations standpoint.
This is a bank who says, these are genuine maybes.
We don't know if we should be lending to them or not.
We're not sure if it's profitable
for us, we're a bank.
So this is a method for them that helps mitigate their
risks in deciding what their portfolio should
look like as a whole.
And from our perspective, provides this nice lottery
system where some people are randomly assigned to get
credit and others not.
So let's go back, and now let's think about some of the
things that will often happen when you do a lottery design.
Suppose you have 500 applicants and
you have 500 slots.
At first glance, you might think you're kind of screwed.
You set up this nice, big process, and lo and behold, it
turns out you didn't have over-subscription.
You thought you did.
You thought you were going to use this over-subscription to
randomize who gets in and who does not.
So what can you do in this type of situation?
So there's some low hanging fruit type answers, and
there's also a possibility that this wouldn't work.
But the first is, could you increase the outreach
activities?
That a lot of times in this situation, what this really
means is that whatever was done to market this program
was not effective in getting in the right number of people.
Because the intent was to get 1,000 or 2,000 people
applying, and only 500 applied.
So now think about, from an operations standpoint, what
does that tell you about applying?
And that might be one of those cases where there's actually
useful, interesting research for the organization.
If the organization was saying, we're going to get
2,000 people applying for this, and then only get 500,
well, it tells you that maybe you could do something to help
them learn how they can get their marketing up.
And so then you can test out different approaches for
marketing that helps the operations learn more about
what is it that brings in people to apply for these
scholarships, or whatever it is that's being done, and at
the same time, benefits the program from increasing--
I'm sorry, benefits the evaluation from getting a
larger intake.
The risk with this is that you end up bringing people into
the program who weren't really part of the target program.
So if that's the answer, then this is actually a really bad
idea to go out and do more extensive marketing.
If you have to go through extra leaps and bounds in
order to bring people in, so much so that it changes the
nature of what the program is, that's a situation where you
might want to go back to the drawing board and think again
about what the right answer is.
Suppose there's 2,000 applicants, and suppose in the
process of doing this that the organization that's doing
things says, you know what?
There's 500 worthy candidates and there's 500 slots.
So a simple lottery would not work.
So we have some sort of screening process, and we
ranked them.
And we can rank them, and we put numbers by everyone.
So why should we do anything other than just
taking the top 500?
So when that type of thing happens, a lot of times the
questions that we want to ask then are about what was the
screening process that was being used here?
So let's go back to the credit scoring study that I was
referring to.
The thing that we're struck by when we've done this credit
scoring is really how wide that maybe category is, when
you really get into the nuts and bolts in talking with the
lenders about where their data are coming from, and the
quality of the data, and how they're individual judgment
weighs in to influence where people fall within those three
buckets, the yes, no, maybe.
And the fact is that the profitability of the person at
the high end of the maybe is really not much different, if
at all, from the low end of the maybe.
And so the point is that when you get inside and figure out,
what is it that's really going on that made them say, well,
we had 2,000 applicants and we had 500
eligible, so we're done.
If you got inside the box a little bit more and talked
with them about how they got to those 500, things would
start coming out of that process that might be the
exact areas were you can say, why is that
necessary as a criteria?
Is that really something you want to filter on to bring
people in or out?
Yeah?
AUDIENCE: Let's say your criteria are, one is gender
and one is educational level for the purpose here.
And let's say that, for whatever reason, you can't be
sure that the educational level is really
what they say it is.
So you could go ahead and select on the basis of
educational level, knowing that maybe people are not
telling the truth and that would be OK.
You could remove that as a criteria entirely, or you
could find some other kind of proxy.
That might work, right?
PROFESSOR: Right.
AUDIENCE: OK.
Kind of random.
PROFESSOR: But that's exactly kind of one of the key things
that would come out.
So the point is, let's just go with the middle example.
If you really think that the education was what you wanted
to screen on, but you don't have confidence in what you're
looking at as actually being a reliable measure of education.
But yet that's causing a filter to draw
you down to the 500.
But then when you get inside and you realize this is a
really bad measure of education, why
are we using this?
And then all of the sudden you relax that one rule and you're
up to 900 people, it tells you that maybe this isn't such a
good way to be filtering, and I should be doing the 900.
And while I'm at it, maybe I should try to think about how
to measure education better or something.
So let's take another kind of example, which is-- and this
actually mimics the story I have up here about training,
but the credit scoring example is another
example exactly of this.
Where you end up--
and I said that right here, sorry--
you can think about having two different piles of people
that, when you say there's 500 that are eligible, really
maybe what you have is not 500 that are eligible, but you had
200 that are really eligible.
You really want them in the program, they must be there.
And then the next 300, you're a little more questionable on.
And the next 300 really weren't so different from the
300 after those.
And so what you can do in that type of situation is set up a
process where you say look, if you have certain eligibility
requirements, you're in.
And then you're also not part of the evaluation.
And it's the next 300 that we're going to combine with
the following 300 after that, we're going to put together a
pool of 600, and we're going to randomize those.
Now this has a clear benefit and a clear cost.
The benefit is that you can now get a very nice, clean
estimate on the impact of those 600.
The cost is we've changed the research question here.
We've changed the evaluation question.
We're no longer answering the question, what is the impact
on everyone who receives this service?
And that's not a good thing.
We don't want to lead with methodology and
then force fit questions.
We want to set the research questions.
We have to ask ourselves, how much are we losing by only
studying those individuals?
And in some settings, those are the exact individuals you
want to study.
But in some, maybe that's not so the case.
So in the credit scoring, I think of those as the exact
people we want to study.
Because when we think about programs that expand access to
credit, what we're doing is we're talking about those
people on the bubble, and we're talking about ways of
getting them access that they didn't have otherwise.
And the people who have really, really good credit
scores and are very credit worthy, they're not the ones
we're thinking about when we think about
expanding access to credit.
So let me give you another example of one where this
would be bad.
So we are doing these programs about targeting the
ultra-poor, where we go into countries--
I'm sorry, we do go into countries,
but we go into villages--
and we first identify the 20 poorest people in the village.
And then of those 20, we hold a lottery, and 10 receive
services and 10 do not.
And then we do this in about 30 villages.
And the organizations we're working with, it's exactly the
setting I'm describing here.
The organizations we're working with
have a fixed budget.
They can provide services in each situation to 1,200
people, and that's it.
And if we went into 120 villages, they
can do 10 per village.
If we went into 60 villages, they could do 20.
But either way they look at it, they got 1,200 people that
they're going to be able to provide these services to.
It's going to be an asset transfer, they're going be on
goats and training and consumption bundles.
Now, if in some these villages people said,
well, wait a second.
We have the 20 poorest people in the village, yes.
But four of these people are really just standout poor.
They're much, much poorer than everybody else.
So we want to exclude the four very, very, very poorest and
make sure that they get it with certainty, and then only
evaluate the other 16.
So this would actually be a bad thing from the evaluation
perspective.
It might be the right thing to do, and we can talk about what
the trade offs are.
We're not actually doing that, and I'll explain why.
But from an evaluation perspective, that would be now
changing the research question in a bad way, as compared to
the credit scoring, where I was arguing
that it's a good change.
So why?
Well, because this is a problem where we really do
actually want to know the impact on those bottom four as
well as the next 16.
And so if we did something where we just allowed the very
bottom four to get the program with certainty, and then only
evaluated the next 16, we're missing an important part of
the sample frame.
Now, the reason why, in these settings, we've done it as the
full 20 is because realistically, when we've
actually gone into these villages and done this type of
exercise, it's difficult, if not impossible, to get
consensus that there's really four people that stand out.
And the fact is, even when you're measuring poverty--
and we do have some objective things, we all know that
there's lots of components to poverty.
There's lots of ways of measuring it.
It's not even just consumption this month, its vulnerability
in general as a concept, which could be about the variation
over time, and how vulnerable you are, and how your social
networks are for helping you absorb shocks.
There's just many different ways of measuring it, and it's
unrealistic to think then we can go into a village and
really draw these amazingly fine lines to say these people
are standout different than the rest.
When you're going into a village of this size and
you're finding the 20 poorest, those 20 are statistically
indistinguishable from each other, more or less.
So that's the philosophy of the organizations.
And that's the reason for doing it the
way we're doing it.
So sometimes, exclusion is not desirable.
Sometimes there's no way, whatever the context is, of
excluding individuals.
So other points that we will often use is the expansion of
a program, or any sort of program where there's some
sort of initial stage, and then what you're doing is
you're simply randomizing where that initial stage is,
or where the program expands into.
A program is only going to go to so many villages, and they
can't exclude, but they can control where their
individuals, where their credit officers, where their
program officers, education trainers, whatever it is that
they're doing, they obviously can control what villages they
go to and what they do not.
So when there is a process like this, where there's
gradual expansion, we often will think about doing a
program evaluation using a phase in approach.
A phase in approach basically says, look, we're going to go
to all 1,200 of these villages over the next three years, but
we can randomize which ones we go to in each year.
So everybody's going to get services in the long run.
Now one thing that's nice about that is, particularly
when there's community led interventions, there's
oftentimes a desire to have some involvement from
everybody in the program at some point
in time in the program.
And this is often something that organizations do ask for.
So phase in approaches allow for that more naturally,
because everybody's going to be receiving a
service at some point.
Similarly, the rotation--
which I'll mention in a second-- is a very similar
twist on the phase in.
Where the rotation, instead of having it be where you slowly
phase in a process, it's a process where everybody's
actually receiving a service at all points in time, and
we're just randomizing who gets what.
So we'll talk about some examples in a moment about
when that can work.
So some key advantages is that everyone gets something
eventually, and this provides incentives to maintain contact
as well with control villages.
They're not just participating in surveys, although we do do
a lot of surveys sometimes where there is no intervention
related, and people are often more than--
at least in our experience-- more than happy to participate
in this interesting, weird thing, with these people
coming and asking us all these questions.
But having said that, there are situations where you want
that continuous support and continuous buy in to what
we're doing, and so the advantage is that it provides
them some incentive to maintain contact.
Some of the concerns is that it does complicate estimating
long line effects.
If the goal was to study something over 10 years, but
everybody got phased in at the end of three, well, you can't
study the 10 year effects.
You can study the effect of getting something for two more
years 10 years later, but that's not nearly as
interesting as asking what the 10 year effect is from getting
a particular service.
So let me give you an example of a situation where the first
is actually perfectly interesting.
Take neonatal care.
If you want to study the effect of neonatal care, doing
a phase in across villages is perfectly fine, because this
is something that is going to be affecting infants.
And so when you provide that neonatal care, and you do this
as a phase in, you're now studying those children, and
this is perfectly fine.
You can study now the effect of the neonatal care over a
10, 15 year horizon, even though it was
phased in to everybody.
Because once it's phased in, the control area of those kids
are three, four, five years old, and so it doesn't apply.
So I think the main thing to talk about in terms of phase
ins that does become an issue about expectations.
So let me give you the simplest example.
In the world of credit, this is usually something that
concerns me a lot when we're talking about doing studies.
And in fact, I've personally been involved in studies where
there was a proposal to do a phased in credit program and
we said no.
We didn't do it, because we were very concerned about what
happens to a control group individual who was told,
you're going to get a loan, but just please wait six
months, or year, or year and a half, or two years, whatever
the amount is.
Now the longer you have to wait, the less it's an issue.
But if it's a relatively short period of time, like a year,
then the question is, well, what were they going to do
with that money?
And is it something they're willing to wait a year for?
And if the answer is yes, well then this is a real problem
for thinking that this is a valid control group, because
what it says is, we're going to see a delay in an activity
specifically because of getting access to
this loan in one year.
So what it says is they did have other options.
They were a little bit more expensive perhaps, a little
more costly either in time or interest.
And they said, you know what?
I don't need to build that new roof now.
I don't need to buy that new sewing machine for my
enterprise.
I'll go ahead and just put it off, and I'll do it in a year.
And what it would do is it would lead us to overestimate
the impact of getting access to credit, whereas if they
weren't promised this good loan in the year, they
would've just borrowed at a little bit of a higher
interest rate right now.
And they still would have made the investment in their
business, they just would have had a higher interest cost
that would cause us to overestimate the impact of our
program, because we would only--
the true impact is just really a savings in interest, not
about access to credit in a binary sense, but
just about the price.
But yet what we would then find ourselves measuring is
seeing a treatment group of people who fix their homes and
bought sewing machines for their businesses, and a
control that didn't, and we would think that aha, there
was a real binary constraint here where people were
actually held back from getting access to credit.
And this program didn't have that positive effect.
So a rotation design.
So a rotation design is basically groups getting
treatment in turns.
Group A gets treatment in the first period, group B gets
treatment in the second.
The main advantage is it's perceived as fair and easier
to get accepted.
Everybody's getting something, we're just randomizing what
you get in a given round.
So we have the same anticipation issue that we
just mentioned with phase in as one concern here.
It depends on what the two treatments are, but if you
really want that other treatment that other people
are getting, and you're told you're going to get it in a
year, this could affect your behavior now
for the same reason.
Also it does have the same long term problem of the phase
in, in that everybody--
if we're just rotating-- everybody
is getting the treatment.
Now, another twist on rotation is not rotating just--
call it like a placebo design, so to speak.
Suppose that you just have two different treatments doing two
totally different things, everybody gets something.
And you use one to measure the impact of the
other and vice versa.
So it's similar to rotation, except not going full circle.
It just means everybody gets something.
So in something like that, you don't have this long term
impact issue, or this issue.
But you do have a problem where it's not so clear what
you're comparing anymore.
So you'd have to be really careful and think about what
it is you're trying to do.
We tend to think that a lot of interventions have indirect
effects in many facets of our life.
So if you're providing training about
entrepreneurship to one group, and another group you're
providing some health service, and you think, well, this is
great, because what does entrepreneurship training have
to do with health?
And what does health have to do with
entrepreneurship training?
So I can provide my health services over here and measure
the health outcomes for my other group and compare them.
And the same thing with business activity.
But it's not too hard to tell stories where these are going
to interact with each other.
So you're healthier, and this makes you more able to work,
and your business does better.
Your business does better, this makes you richer, and you
spend more money on health.
It's not hard to tell stories across two seemingly unrelated
sectors where you will have that type of effect on each.
You have to think about those types of issues.
Yeah?
AUDIENCE: Can a rotation work well in agriculture?
A given country would have--
so you'd like to give the farmers--
everybody gets something.
Some people get credit, some people get seeds, some people
get a variety of things.
But obviously some places have good soil, some places the
farmers are near a road, and the data that you need to look
at your sampling design is kind of tricky, because you
may not know that much about all the variables that would
impact a farmer's success.
Rotation seems a nice way to--
because there are so many different treatments that
people can get, that it would seem pretty tricky to
implement if there were--?
PROFESSOR: So I think, in that type of setting what you're
describing, we'll come to one in the end.
But let me just say an example here, which is the question
you're proposing about agriculture is about how
different treatments will interact with other
treatments, and with underlying context.
So there's two things going on in your question.
One is how does soil quality affect whether a certain
treatment is effective or not?
And the second is maybe credit alone is bad, and maybe seeds
alone is bad, but the two together is good, and things
of this nature.
So that's not a setting where we would think instinctively
about a rotation design.
That's a study where we would think about two things.
One is making sure that our study is being done in a wide
enough variety of soil, to use that example, so that we can
actually study the effect on one soil quality and another.
And then the second thing is we were thinking about having
multiple treatments, but not in a rotation style, but in a
way that you have some people, they get seed.
Some people, they get training.
Some people, they get seed plus training.
Some people, they get nothing.
So we'll get to an example like that hopefully towards
the end, but that's that design.
A rotation design is really more about when you're doing
something that realistically will not have an indirect
effect on the other group.
So the example I'm going to give you is a rotation study
that was done, the Balsakhi Case that's done, I think it's
one of the cases in your reading.
It's what what you do this morning, right?
So that's a classic rotation example, because what we're
doing is, some schools got third grade, and some schools
got fourth grade, and then they rotate.
And the idea is as long as the third graders don't affect the
fourth, and the fourth graders don't affect the third, then
this is good, and every school got something.
And then we just rotate around what they're getting.
And that's what we mean more by rotation design.
The key here is that this is a great example of where the
rotation design was a useful way of getting the support of
the schools.
It's going to the schools and get them to agree to do all
these tests with the children.
And it would have been hard to just go in and get them to do
tests without being offered some service along with that.
Now they're willing to accept that the service only went to
one grade, not both.
They understood it was a phase in within their school, one
gets it one year, that other, the next.
But that was the way of getting the schools to
cooperate was by offering it through this
type of rotation design.
So next is encouragement design.
Now encouragement designs are--
first of all, this is orthogonal to everything else
I've been saying.
Encouragement design can be done on top of a phase in, on
top of a rotation, on top of a lottery.
This is not mutually exclusive with the
others that I've discussed.
Yeah?
AUDIENCE: I just had a question on
the phase in approach.
PROFESSOR: Yeah?
AUDIENCE: So suppose you wanted to roll out packaging
group one, and then you'll roll out packaging group two.
When you roll out packaging, you notice that something is
not working that well, and then you wanted to tweak it a
little bit, you wanted to change the spec.
And for the sake of the experiment, are you not
supposed to tweak it when you roll it out to the second
group, you just keep it the same?
Because [UNINTELLIGIBLE PHRASE]?
PROFESSOR: Right.
So great question.
I think the key here is to think about the timeline.
I don't really have a chalkboard.
The key is to remember that with the phase in-- so let's
go with a really simple phase in, two waves.
So in that setting, the second group, when you do the
treatment with them, that's actually after
the study is over.
So the idea is that they're really your control group, but
they're participating with you because they know they're
going to get it in the future, or whatever the
circumstance is.
So that's a situation in which the answer is yeah, you can do
whatever you want with them.
But if you did know from operational observations that
the treatment itself wasn't working so well, then just
remember that when you're evaluating something, what
you're evaluating was a program which you already
think from operational reasons was less than effective.
And so that should perhaps inform you a little bit about
things like what to measure in terms of what you want to put
in the followup surveys.
I suppose we could complicate your question a little bit and
add a third wave.
So you have three waves, one for each year.
And after the first year you learn, oh, turns out we
shouldn't have done it like this.
We should have done it differently, and so you want
to change things for the second wave.
And that's perfectly fine.
It does mean now when you're doing your analysis, you
should think about this as two studies.
You have your first wave, and you can compare that to your
wave three, that is control for the entire study.
And then you have your second wave, and you can look at them
for one year and compare them to wave three, and you really
have two different studies in that setting.
So encouragement designs, like I said, this is not mutually
exclusive to the others.
And encouragement design, just think about
what the word means.
It means we're encouraging people to do something.
We're not forcing, we're not mandating.
That means the control group does not necessarily have
nobody getting services, and a treatment group does not
necessarily have everybody getting the service.
There's simply something done to encourage people to do
something, to participate.
Now the key here is to think about what we're really
saying, is the control in the phrase randomized control
trial, the reason for the word control is this idea that the
researcher has some control over the process in deciding
who gets a service and who doesn't.
So now we're just moving the control, and it's no longer
over who gets the service and who doesn't.
It's over who's offered the service and who's not, or who
has some encouragement to get the service or not.
And we still have perfect control if it's executed
properly over that offer, over that suggestion, that
encouragement.
But we don't have perfect control over who actually gets
the service.
So a very simple example of this is suppose that I gave
each of you a marketing brochure to go to Au Bon Pain
during lunch and go because of their delicious scones, and I
only gave it to half of you.
I am now encouraging half of you to go, the other half not.
Anybody can go to Au Bon Pain, I'm not controlling that.
And if I wanted to then, for some reason, study the effect
of going to Au Bon Pain, I could do that.
I'm not sure what the point of that would be.
But the point is, I'm only controlling who receives this
offer and who doesn't.
I'm not controlling who actually goes to Au Bon Pain
and who does not.
And so this is often the easiest thing to control in
the process.
And the entire key here from a statistical perspective--
not the entire, we'll go into some other issues-- is about
what that differential usage rate will be among those who
were encouraged and those who were not.
And when you get into power calculations later in this
week, that's going to be a very important element to
think about.
Because if that encouragement is really, really weak and
just barely changes people's behavior, it means you need a
huge sample.
In an extreme, an encouragement design is
exactly a perfectly controlled randomized controlled trial.
An encouragement that gets people who get the marketing,
every single one of you goes to Au Bon Pain, and if you
didn't receive the marketing, nobody goes.
Statistically it's the same now as a
perfect lottery system.
But usually when we're doing encouragement design is when
we have some expectation for it not to be perfect, and so
we're using that.
So what makes something a good encouragement?
So I think there's two things to think
about that are important.
One is that it's not itself a treatment.
The minute the encouragement design itself becomes a
treatment, then we have to think about what is it that
you're actually evaluating here.
Your goal is for your encouragement to be totally
innocuous, to just by chance, by randomness, some people
will be more likely to use a service than others.
So you want it to be as innocuous as possible.
So a good idea is typically marketing.
We typically think of marketing as a good approach,
just making people aware of a service makes them more
likely to use it.
So we've done marketing experiments, for instance, in
the Philippines a lot where we're doing some sort of door
to door marketing of a savings product
offering people savings.
Anybody in the village could walk into the bank and open a
bank account.
But realistically, only those who get a knock on their door
become aware enough of it to actually go and
open up a bank account.
Here's a bad idea.
Let's provide training to people that encourages them to
use credit.
So let's bring them in, let's give them a big course about
business management and how to use credit in order to take
out a loan.
And let's use that as an encouragement tool for
measuring the effect of credit, because after doing
this course, they'll be more likely to borrow.
So the problem with this, if we want to look at business
outcomes, is we just gave them a month long course in
management of an enterprise.
And that alone is going to have an impact on their
enterprise, we think, we hope.
And so if it does, well then, what are you
measuring the impact on?
Was it an impact of the training program?
Or was it an impact of getting access to credit?
And you can't separate these out at all.
So this first thing to think about is just making sure that
treatment is really innocuous.
In econometrics, we refer to this as the exclusion
restriction, in that what it's saying is that we want to make
sure that the only--
if we're going to draw a link from the encouragement to the
take up decision to the outcome measure we care about,
that the only path through which the encouragement
affects the outcome is as it generates higher take up.
If it has its own effect outside of the decision to
take up, now it's a problem econometrically, and we can't
really claim that we're measuring the impact of using
the service.
We could only measure the net effect of the two together.
So the second issue is for whom are we
estimating the treatment.
So here's my favorite tongue in cheek example for this.
Suppose we went into a village and we offered free alcohol to
anybody who takes out a loan.
Might be great in the first stage in the sense that it
generates lots of higher borrowing.
But what are we measuring here in terms
of who we're studying?
We're studying people who respond to this particular
incentive of free alcohol.
That's certainly not the program that we're typically
trying to evaluate when we're trying to do an evaluation of
microcredit.
And we want to make sure that we're getting the people in
the study who are the right people, who are the types of
people that are thought about as the target audience for a
microcredit program.
That means not drunkards.
And so you want to make sure, you do want whatever that
approach is to be something that is sensible, that seems
somewhat in the scope of normal.
Or at least doesn't create a sample selection bias in the
sense that it doesn't make the people who take up the program
be particularly different in a way that is not useful.
Yeah?
AUDIENCE: So an example that I'm thinking about is access
or information about a microcredit program to
participants who are typically very
uninformed about these things.
So the information is out there.
Theoretically, it's really accessible.
But we know that unless we tell them that this program is
out there for them, chances are very good that they would
never think of it on their own.
So that would not then be a good situation for this kind
of a thing, because we know that we are in effect offering
them a sort of special in by the very effect of offering
it, even though theoretically it's accessible.
PROFESSOR: I would actually says that's
actually a perfect setting.
To do this, let me rephrase the question, which is suppose
you have a program, and only the highly informed normally
are going to come in.
And so in order to do an encouragement design, what
you're doing is you're going out and you're only going to
move the people who are not highly informed.
The highly informed already know about you.
They're either coming in or they're not.
You give them information, it doesn't matter, I already knew
about this.
So what you're doing is you're moving the less informed
people, you're informing them about the service you're
offering, and now they're coming in or not as they wish.
But they're more likely to come in now than the people
who are not informed.
So this is a perfectly relevant approach if it's the
case that this is an organization that does aspire
to grow, and they're going to grow through informing people.
In most of the settings I've been involved in, at least the
type of information we're dealing with is usually not
much different than what they do normally.
It's just marketing.
It's just targeted and controlled marketing, where we
control what villages they go to do the marketing, or what
household's doors they knock on.
But in a lot of situations, the encouragement design
literally has them doing exactly the same operation's
that they normally would do.
But it's just recognizing that it's
still a voluntary decision.
They can't make someone borrow.
They're going to a village, they're holding meetings,
they're presenting what they do, and some
borrow and some don't.
AUDIENCE: I think I'm saying something slightly different,
but that might [UNINTELLIGIBLE PHRASE].
So among the group who would not normally know about this,
it's not that I'm going to--
I'm not saying the group who knows, forget about them.
I don't know, I'm confusing myself.
We're assuming that the group of people that we work with to
provide our program, we would provide a precursor program,
and we would say among people that we work with, half of
them we would tell, and half of them we won't tell.
Is that what you're saying too?
PROFESSOR: You would go out of your way to give them
information about the program.
Everyone can get in, but you go out of your way to approach
half and tell them about the services.
AUDIENCE: Understanding that chances are that if we don't
tell them, they won't go, because they're just
uninformed?
OK.
PROFESSOR: Right.
Yeah?
AUDIENCE: I don't want to interrupt if there was more to
this exchange.
My question is about distiguishing thing between
marketing and training.
If the treatment is something like a financial product or
service that's poorly understood, and you don't want
to-- do you think that maybe financial literacy is an
important determinant, but you want to isolate just access to
the product or service and keep
financial liberty separate?
How do you draw the distinction between marketing
and training?
PROFESSOR: I can tell you in one setting, here's what we
did to try to understand this better.
Let me restate the question, which is how do you
distinguish between marketing and training?
This is really a spectrum.
So an example I gave that was bad was a month long training
program, and I said it's fine to just knock on a door.
Why am I drawing the line there?
And it's a perfect question, and I can tell you that the
first time I actually ever did this type of design, we
actually had an entire treatment group that was just
knocking on doors, but with no product.
It was just to test out whether the knocking on the
door had an effect on savings.
So we had two treatment groups.
We had a treatment group that got a
commitment savings account.
A bank officer went to the door, knocked on it, gave them
a pitch about why they need to save, and savings is good, and
here's a goal.
You should have a goal for savings.
And here's an account that we'll offer you to help you
reach your goal.
It wasn't a very long pitch, but it was a marketing visit.
And we had a pure control that got no contact from the bank.
And then we had a second treatment group that we called
the marketing treatment group.
And this group got the knock on the door, got the same
pitch for about 5, 10 minutes about why it's important to
save, and how the bank is there to help them save, but
didn't get offered that special savings account that
had special rules to it.
And that's done exactly to try to understand where
to draw that line.
So if it's a situation where you're particularly concerned,
then you could actually think about having treatments
designed specifically to test whether there is a direct
effect without the treatment you really care about.
Yeah?
AUDIENCE: I'm kind of jumping onto the last point as well.
But the one encouragement design that I'm familiar with
is one where a subsidy was actually utilized, but then it
was a random distribution of who was offered the subsidy.
And, for that matter, because they're trying to determine a
demand curve, the subsidy varied.
So maybe you'd be offered 35% off, maybe you'd
be offered 75% off.
Still random in who was given the offer, but then they had
the encouragement to take off based on how
big the subsidy was.
But it seem my initial-- not knowing enough about the
details of the program--
is that there would be a problem based on the economic
status of those who were offered a program in the first
thing if they were not very, very similar.
If I am marginally wealthy and I'm offered a 35% discount,
I'm more likely to take out than someone who is broke and
is offered a 35% discount.
Then that would affect your sample.
PROFESSOR: With one twist.
So it's not the levels that matter, but it's actually the
slope that would have to matter.
It has to be not that the wealthy is more likely to take
up with any given level.
It's that they have to be more elastic or less elastic than
the poor in order for that to be an issue.
And then you're absolutely right, that is an issue.
And then what you're studying is when you do that subsidy,
you're studying your treatment effect on those people who are
going to be more responsive to that subsidy.
Is there another hand?
No.
Wendy, now we're to your question, multiple treatments.
So let me just say one more thing on
encouragement designs.
So one of the key things to remember with encouragement
designs is that in a lot of situations, the encouragement
design is--
in some situations, it is set up where the control group
does get into a program.
So where you're dealing with a 10% take up rate in control,
and a 30% take up rate in the treatment, In a lot of the
setting, though, it's really more that you just have
incomplete take up in the treatment group, that
participation is voluntary.
And so by encouragement, all we really mean here is that a
treatment is being offered to people.
They can say yes or no, and they're not being offered to
the control group.
And so we end up with take up of some percent in the
treatment group, and zero in control.
Like the savings experiment that I just referred to a
moment ago, we had a 28% take up in the treatment group, we
can't make people open a savings account.
All we can do is offer it to them, and we had a 0% take up
rate in the control.
We did a similar thing in the same place in the Philippines
on a commitment account to stop smoking.
Again, 11% take up rate in the treatment group.
We can't make people want to stop smoking and sign accounts
and contracts to do this.
But we can prevent the control group.
So there was perfect control in the control, in the sense
that they were not offered the
opportunity to open the account.
But the treatment group has to be voluntary.
That is what it is.
And so it's an encouragement design, with 11% percent take
up rate in the treatment and 0% in the control.
So sometimes you do have control on one half but not
the other for who uses.
Yeah?
AUDIENCE: And so just the main point is that what constraint
are you going around by an encouragement design?
Just an ethical problem if you can't afford
treatment to everybody?
PROFESSOR: In that situation, I don't know that I'd pose
that as an ethical issue.
But the point to be made is just that you can't force
people into a program.
It's a voluntary participation, and that's OK.
So one of the things that I've often read or heard is when
someone says, well, wait a second.
This is voluntary
participation, so how can you--
doesn't that introduce selection bias?
And the answer is no, because what we're going to do when we
do the analysis of that, is we're going to compare
everybody who was offered the account, everybody who was not
offered the account.
And so there's no selection bias there.
There would be a selection bias if what we did is we
analyzed everybody who took up in the treatment group, and
compared them to everybody in control.
And that would be a flawed analysis.
But that's not what we do.
So if you ever hear someone say, ah, encouragement design,
doesn't that introduce a selection bias, because
participation is voluntary?
The answer is no.
That only introduces selection bias if you do
the analysis wrong.
What you want to do is compare what's refereed to as the
intent to treat analysis, and it means comparing everybody
who's offered to everybody who's not offered.
Multiple treatments.
So this goes back to Wendy's question.
This is one of the areas where I tend to think is most ripe
for helping--
going back to one of the first points I was making about
making sure that the evaluation speaks nicely and
informatively to needs of the implementers, needs of the
organization.
That a lot of times, there's very specific operational
questions that they have.
Should we really do it this way?
Or should we do it this way?
I really made some tough choices here, and I just went
with what I thought was best.
But gosh, if the research can actually help guide me and
tell me whether this particular component is
necessary or not, that would be great.
So imagine you're doing--
let's go with Wendy's example of an agricultural program.
And suppose that you're trying to decide, how important is
this training component?
I'm going to provide seeds, and introduce people to
marketplaces.
I'm just making something up.
Let's not get into the details too much, but let's just say
there's a training component alongside of it.
And that training component is really expensive, it takes a
lot of time.
And I'm thinking to myself, OK, I can help twice as many
people and drop the training, or keep it my current program
size and have training.
What's better?
Well, the research can help answer that question by having
an evaluation which evaluates the overall program, but then
also randomizes whether or not there's training involved.
And so this is one of those key areas where it's a win win
for operations.
Where you can help answer questions for them beyond the
simple impact question.
There are situations in multiple treatments that we've
been in where there's no pure control.
And there's nothing invalid about doing that.
It does validate the study, but we just have to remember
that you're no longer saying, what is the impact of the
program compared to not doing the program?
You're now comparing it one option relative to another
option relative to another option.
And hopefully in the design there, you have one option
which is kind of like a placebo, so that you have some
group that you really don't--
it was in a very extensive way, but you have some method
of being able to say what the overall effect is.
But there's many situations we're in where that's actually
just not part of what the study's about.
So we've done savings product designs are a perfect example
of this, where we're dealing with people who opened a
savings account.
There's no control group of people who were not offered a
savings account.
We're just dealing with a bank and they take people in.
And the question to us was, well, how can you help our
existing savings people save more?
So we tested something out in three different countries
where we sent people reminders to save.
So half the people basically got a little text message
saying, hey, don't forget to save this month,
and half did not.
So we have no control group here of people who got no
savings account.
So we're not measuring the impact of savings on things.
We're just measuring the impact of getting this
reminder on how much you save.
And we've done similar things with loan repayments.
There's no study on the impact of the credit, we're just
testing out operational questions about how to run the
program better.
And in those types of designs, we'll often test out five
different messages all at the same time.
I think I said that slide.
Oh, maybe not.
Yeah, we talked about randomization in the bubble.
So this is the list of the various things
that we've now described.
And just remember that these are not mutually exclusive.
Multiple treatments and encouragement design in
particular kind of fit within almost any of these other
things going on here.
Any questions so far?
OK, part two, gathering support.
So here are some things that we commonly hear.
So this part of the lecture is really all about how we deal
with this kind of introductory conversations, exploratory
conversations where we're trying to work with partners
to figure out how to go about doing a randomized trial.
So one answer which is always a tough one to get, but I
already know the answer.
And I don't want to risk learning that we
do not have an impact.
There are situations that we'll be in-- and I don't mean
to sound like a pessimist-- but there are situations we'll
be in where you just realize this is not a
good setting for it.
You have to work with people who actually
want to know the answer.
And you can recognize that merely observing that their
program has grown is not necessarily a sufficient
measure to say whether they've had an impact.
And it's certainly not a sufficient measure to say
whether their program is a good allocation of resources
compared to other programs that have also had similar
operational success.
And so when we have to make the tough choices, this is
where we need the evidence.
Listening is probably the single most trite but
important thing I have to say on how to have these types of
conversations.
Trying to understand the perspectives and the
objectives of the people in the table.
What is it that's making them tick?
What is it that's making them have this conversation in the
first place?
And finding ways to make the research operationally useful
is perhaps the single most useful and important thing to
do when working in the field.
One thing that I've often found too is that often in
practice, there's a caution.
There's almost a mistrust that some might have if they're not
familiar with what is that's going on.
And it's one of the most important things that the
field staff can do in working with the organization is to
just gain the trust of the people in the field who are
working for that organization.
And some of this comes about in getting their feedback and
input into things like survey design.
Having it so they feel part of the process, and their input
is received and incorporated into what we're doing.
And that's good for the program, good for the
evaluation to get their feedback.
It's also good in a purely interpersonal way, in terms of
helping to have a relationship that's good by making sure
that people feel that they are part of that process.
So some other specific things that come up.
The first, one of the most common things is gossip.
People will talk.
So what do we do if the control group finds out about
the program?
So I think the thing to think about is to try to separate
out these types of issues.
Let's just put this into a more general category called
spillovers.
So spillovers meaning there's any sort of indirect effects
that are going to occur, from those who were treated to
those who are untreated.
I think it's really important to separate these into two
categories.
There's natural spillovers, and let's call them research
spillovers.
Now, by a natural spillover, what do I mean here?
I mean a spillover that is naturally occurring.
That is, if you go and you provide a service to 100
people, the fact is this is going to affect those 100
people and 200 more.
And that's the nature of the intervention.
It has nothing to do with the research.
The example that you have a case on is deworming.
We're going to deworm half of you.
The other half will benefit from that, because you're
going to be less likely to catch the worms
from the first half.
Let's say I took half of you right now, and I went into the
other room, and I gave you a whole big lesson in power
calculations, and I ignored the other half, and I didn't
give that to you.
Well, there'd be a spillover.
You'd come back, you're in a group, you'd talk.
Oh no, no, I just learned about power calculations, let
me show you.
Hopefully it'd be a positive one.
So these are all natural spillovers though.
There's learning that takes place.
You teach some people, they teach others.
You deworm schoolchildren, other schoolchildren benefit
because they're less likely to catch the worms.
There could be negative spillovers.
We go and we offered really, really cheap credit to some
people, or we only offer it to some because we're
constrained.
That's the organization.
That's just how many loans we make.
It gives them a competitive advantage--
I'm not saying this is right.
But this is an argument that people will make when they are
arguing against subsidized microcredit.
And what does this do to the people who don't get access to
the microcredit loans?
It shuts them out of business.
It makes it so they can't operate their enterprise
because they're competing against someone who's getting
subsidized credit.
So that has a negative spillover.
These are natural, though.
So a good study is one that helps to measure these things.
And there are ways that we can design experiments to measure
those types of spillovers.
So a very simple example of one that measures this is
suppose we have villages.
And what we're going to do is take 90 villages, and instead
of just dividing them up treatment, control, what we're
going to do is we're going to divide them into three piles.
We're going to divide them first into two,
treatment and control.
So we'll have 60 of those villages being treatment and
30 being control.
And then within the 60 that are treatment, we're only
going to go and deliver services to half the people in
those villages.
So what do we have?
We have a treatment village that's half treated, half
untreated, and we have control villages.
Now throw away the people that got treated.
Just ignore them.
What's an interesting analysis to do here is to compare the
untreated children or people, or whatever the intervention
is in the treatment villages, and
compare them to the control.
These are two people that didn't get services.
Neither one got treated.
But some of them live near people who got treated and
some of them do not.
So that measures the indirect effect.
That measures the natural spillover.
So if a natural spillover is something that one was
concerned with, this is exactly the way you would
think ahead of time about setting up the research design
to measure that.
But then there's unnatural spillover, what I was
referring to as research spillovers.
Yeah?
AUDIENCE: Just a quick question.
Does the fact that now the treatment is half the size
compared to the entire control group, does it matter?
PROFESSOR: Yes, it matters.
And that's a question of power calculations.
And so you have to trade off your measurement of the
spillover versus your measurement
of the direct effect.
But that's a mathematical problem that can be solved
analytically.
So research spillovers, those are the bad ones.
These are the ones we don't like, because it's not
interesting, it's not useful.
It's not representative of what happens in the real world
when you do an intervention.
It's just an artifact of the research process.
The simplest example is the control group person who says,
I don't like the fact that I am in the control group.
Maybe they don't believe it was truly random, or they just
don't like the fact that they didn't win the lottery.
And so they actually change their behavior
now because of this.
Let's use a very simple example of being in a bank,
and let's say you're doing a lottery across existing
borrowers, and half of them got an extra service to go
along with their loan, and others did not.
The ones who didn't get the extra
service, they're now upset.
I didn't get the extra service, I'm not happy.
Why did they get it?
I didn't get it.
And you can explain, well, it was random.
They don't accept that that.
And now what do they do?
Maybe they don't pay back their loan.
Maybe they leave the program altogether
because they're mad.
Now in studies we've had, I can honestly tell you we've
not had this happen yet in a microcredit setting, but
there's certainly things that we will do to try
to avoid the problem.
So for instance, one of the studies we had where this was
a bigger concern than others was we were testing out group
versus individual liability.
Now most borrowers really like the idea of individual
liability if they're given a choice.
They don't want to be on the hook with other people in
their community, they much prefer to have a loan that's
just to them and them alone.
So when we were randomizing whether people got offered
group or individual liability--
and it was existing borrowers who were already borrowing
from a bank--
what we had to do is take villages that were really
right next to each other and put them together.
Because we couldn't have it that we had these little
sister villages where there was lots of interaction
across, and one got switched and the other did not.
So we put them together and treated them like one.
And so basically, it's another way of saying this is when you
think this is an issue, you just need to think about
making sure that you have some sort of boundaries separating
out your treatment and your control areas.
Now in an urban setting, it could be a little bit harder
if you don't have clearly defined boundaries.
But it's still feasible to do this type of process.
You just have to think a little bit about how to do the
boundary, and also what to do if you have control group
people who do come in.
And so this is an area where encouragement designs might
actually be a useful way of dealing with it, is to allow
some control group people in, for instance, if they come in.
But otherwise not unless they actually come on their own.
So this is another way of saying take an urban area, and
you just encourage some blocks to borrow and not others.
Encourage some blocks to go to school and get some extra
service, and not others, whatever the program is.
And that's a way of trying to make sure that if the groups
talk to each other, it's OK.
It's not going to ruin the study, and there's no
jealousy, it's just a matter of some receiving
encouragement and others not.
So fairness, hopefully this is-- the one point, if I had
to leave you with one simple thought of this lecture, it's
the fairness point.
It's perhaps the single most commonly raised issue, and
it's the easiest of the issues to explain in 99.9% of the
settings we're in.
And it's this fairness issue of, oh, but gosh, I don't want
to do it by lottery, I want to do it by some other process.
And the answer--
or gosh, I can't imagine restricting access to people.
And the answer is always just about the same, which is how
many people can you deliver in this program?
What's your budget?
Now let's divide by the cost per person.
And so you can do this for 1,000 people, or 2,000,
10,000, whatever your constraint is, you have a
constraint.
Now all we're going to do is use that constraint to then
find a way to do this randomization, and that's it.
So we're not restricting access to anyone.
The only sense in which we're restricting access is perhaps
a bigger picture or thought, which is if half a million
dollars is being spent on the evaluation, that's half a
million dollars that's not being spent on services.
That's the only sense in which the randomization is costly in
terms of delivering services.
But this is a different calculation all together.
This is now asking the question of whether it's worth
half a million dollars to find out the impact of
this program or not.
And that's just a very different question.
It's not about the resources and the fairness to those
individuals, it's about the question of whether this
program will be done enough times in the future, and will
the marginal value in terms of our knowledge about what's
being learned from this study be high enough to warrant
spending the money on the research period.
It's a different question.
So on ethics, some of the things that we'll often hear
are statements like, well, it's wrong to use people as
guinea pigs.
Or if it works, then it's wrong not to treat everyone.
So the first thing to think about is that first
of all, it's not--
I think it's a very leading question first of all.
One thing to often ask yourself--
or ask people in this type of conversation--
is why is this different than prescription drugs?
Why should we be more willing to proceed and deliver
interventions and deliver services to people without
knowing their impact then we are to prescribe drugs?
We ourselves, for instance, would never take a
prescription drug if it hadn't gone through a randomized
trial, or more than one.
And so why should we be using a different set of standards
in terms of the ethics of the two, in terms of the bar, the
rigor that we want to use in order to decide how to
allocate our resources.
The second thing to think about in terms of, if it
works, then it's wrong not to treat everyone, I think
there's an important point to note, that there's lots of
ideas that--
there's two issues that often come up in this setting.
First of all, there's lots of ideas that sound good, but
then when evaluated, turn out to not work.
And even when something works, the question is how
well does it work?
We have other ideas that work.
So even if something is good, even if everyone around the
table is totally confident that it's going to work in
some respect, we don't know how well it's going to work.
And when we're allocating resources, we're not just
trying to beat zero, although that's always good.
We're actually trying to do the best we can.
And so we're choosing across five ideas, and
they all sound good.
No one's throwing out ideas that-- well, I shouldn't say
that, that's probably not true.
We can probably think of some that don't sound good.
But for the most part, we like to think that we're sitting
around the table thinking across choices that sound
good, and we have to choose.
And so that's the most important thing to remember in
that type of conversation.
Cost.
So there is two things that often come up when people talk
about cost.
And there is a common perception and argument that
randomized trials are much more expensive than other
approaches.
So I think there's two things to think about when this type
of conversation or point is made.
The first is thinking about what the cost--
it's about doing a cost benefit analysis.
So let's assume for a second randomized
trials are more expensive.
And I'll point out some examples in a
moment where it's not.
But let's say it is in a given setting.
Well, that's only part of the equation.
You have to say, well, what's the cost and the benefit?
The whole point of doing randomized trial is to think
about costs and benefits too.
There's no reason why we should think any differently
when we think about how to evaluate.
So what's the benefit we're going to get them from doing a
randomized trial versus a non-randonized trial, and
what's the cost difference?
And if the benefit that we're going to get in terms of the
reliability in results is high enough to then make more
impact on our ability to make future decisions, well then,
it's probably worth spending a little bit more money on the
evaluation itself.
Now obviously, that's relative to our existing
knowledge in the space.
If something's already been tested 15, 20 times, then this
might become a situation in which you would argue that no,
the benefits don't outweigh it, because the marginal
impact from the research on one more study is not going to
be that high.
And so the costs are not worth it.
I'm not aware of situations I would say fit that, but
hopefully we will be there someday.
The second is that the cost of doing randomized trials is
often not actually more expensive than
non-randomized methods.
But it's really key to state clearly what the
counter-factual is here.
What's the alternative method one's describing?
So it's clearly cheaper than doing nothing.
And there are situations that I've been in where my best
advice is don't evaluate.
For whatever reason, the setting is--
you're not going to get a reliable result, and the best
thing one can do is to not do the evaluation.
Let's compare now to the most common comparison one makes,
which is to a non-experimental quantitative method.
So suppose the alternative approach is to survey a bunch
of people who received a service, and to survey a bunch
of people who didn't receive a service.
It wasn't done randomly.
It was some people chose to be borrowers from
a microcredit program.
For those of you who know me know I do a lot of work in
microcredit, this is why I keep
using microcredit examples.
So I'm going to survey a whole bunch of people in microcredit
that are part of a program.
And then I'm going to go into the same community so I can
find people who seem very similar, have the same
macroeconomic conditions, but are not participating in the
program and going to survey them.
I'm going to follow everybody before and after.
So this is actually a more expensive study.
Why?
Well, I need a larger sample size here.
I need a larger sample size because I actually have to
really understand something deeper now about who's opting
in and who's not.
And I have to try to use my econometric tools to correct
for selection biases.
And this costs me sample size.
And so this study would actually cost more money,
because I would want a larger number of observations in the
analysis in order to try to get it right.
Now it's clearly going to be more expensive.
Now let's flip to another one.
If our counter-factual approach is to not even do any
surveys, like there's no big econometrics, but instead to
do a simple before after.
I'm just going to compare before after.
So you've studied yesterday, you talked about before after,
you went through some of the issues that you have that you
don't know what else is changing the environment.
But if that's the comparison you're going to be using, well
then yeah, this is the more expensive, because yopu've got
to survey control group people too.
That's an example where it's hard to come up with settings
in which there aren't outside factors--
economic, social, environmental, health--
that cause changes over time in outcomes for people, such
that a simple before after analysis is in anyway
informative at all about the impact of a program.
Timing, OK.
I'm going to try to wrap up quickly.
The one thing to say about timing is it's very common to
have a constraint where the organization is like, but we
need the answers now.
Randomized trials are no different than non-randomized
trials that follow people before after, but they're
certainly going to be a lot longer than things that simply
look retrospectively at people that have already received
services and hold focus groups and discussions to try to
assess impact.
And there's no way around that.
So this is a question of just being patient, and working
with organizations that are able to be patient in order to
have those answers.
I'm just going to run through this initial slide so you can
have the basic key points of the overall plan when we're
doing an evaluation.
So the three steps we have listed here are plan, pilot,
and implement.
I think it is important to note that there are situations
where we don't actually do a pilot.
Depending on the circumstances, the situations
in which we do pilots are typically when there's a lot
of uncertainty about what the intervention is in the first
place, and so we're actually working with the organization
to figure that out.
Or if there's some uncertainty about the way the process is
going to play out.
Maybe we're uncertain about the encouragement design.
We're not sure if it's going to work.
We're not really sure, will this actually encourage people
to come in and use a service more so
than they would otherwise?
So we need a pilot to test out whether that approach will
have an effect or not.
We just need a smaller sample just to gauge whether we're
dealing with 60% take up rate in our treatment group as a
result of encouragement and 10 in the control, or are we
dealing with 12 and 10.
What's our range?
So the five steps we've laid out here is identify the
problem and proposed solution.
So I think one of the things that should never escape us is
that you don't take a randomized trial.
You don't start off saying we have a tool, now what research
questions can we ask?
You go the other way around.
You want to think, well, what's the research question
we're asking here?
What is the problem that we see in the market we're in, in
the society we're in?
What's the market failure that this intervention is trying to
solve or measure or test, and then what's
the proposed solution?
Think totally abstractly, don't even get into what the
randomized trial is and how it will be designed.
Just think first order about what the market failure is,
and what the logic is behind the proposed solution.
Second, and this goes back somewhat when we're talking
about in terms of identifying the key players.
There's nothing more frustrating than a really good
project, where you just don't have the right players on
board participating and collaborating in a cooperative
way to make the project work.
Identify the key operations questions to
include in the study.
This goes back to hopefully the other theme of my lecture
this morning, is about making the research into win win
opportunities, finding those operation questions and trying
to build them into the research as well.
Then design the randomization strategy, and define the data
collection plan.
Data collection can be done continuously lots of waves,
one wave at the end.
There's lots of other tools that we can use in data
collection, both qualitative and quantitative approaches.
One of the other common misperceptions that I've heard
is people saying that there's a spectrum between qualitative
and randomized trials.
That's kind of mixing apples and oranges.
Qualitative versus quantitative is about how you
go about measuring things and what you measure.
A randomized trial is just about identification of the
effect of an intervention.
It's about random assignments of treatment and control, but
it has nothing to do with whether the measurement is
going to be done through a qualitative or
quantitative process.
And there's a lot of examples of studies that we have that
use mixed methods and creative approaches for measuring
things, and there's a lot of studies we have where it's
very cut and dry, normal quantitative, how many
potatoes did you eat type questions.
So pilots vary in size and rigor.
The pilots and the qualitative steps that often go into them
are very important for helping to understand the intervention
and design it, particularly when we get into designs that
are doing sub-treatments.
A lot of times those come out of the qualitative process in
the design of a study.
And then for the actual implementation--
oh, I skipped something.
Identifying the actual target population is going to be
covered later in the day, in the second lecture.
And then collecting the baseline data will be
discussed later on.
When we do it, we usually do it, but not always.
The actual randomization, there is various times and
points in which you do it.
This is what we were talking about in the beginning of the
class, real time randomization like the credit scoring all at
once, villages known up front, and you randomize them in or
not to a program.
Then the next phase is implementation intervention to
the treatment groups, and this is where internal controls can
be really critical.
There's nothing worse than doing all of this work, doing
all these surveys, and then not having the right control
in the field to be working with the individuals from the
organizations that are delivering services to make
sure that things happen the way they're
actually supposed to happen.
And I've had projects go bust, where we're working with
organizations that thought they had the right internal
controls in place.
And when we go in to do spot checks to see, and we go to
some villages to see, are they getting services or not?
And lo and behold, they were not.
Or they were when they shouldn't be.
And we go back and we try to work with them.
I've had at least one project I can point to that literally
we just canceled after a year and a half.
It was very unfortunate, but this is what happens when
there wasn't the right level of internal controls in place.
And I learned.
And then measuring the questions.
One of the most common questions we get with
measuring is, how long should we wait?
And there's really no one answer to this.
There's often a trade off with operations.
If there is any sort of holding back of a control
area, then this is going to be something that has to be
negotiated and discussed with operations.
In a lot of situations we're in though, it's not that the
two sides are actually differing--
I mean, the operations maybe-- but the head of the
organization might have incentives that are perfectly
aligned with the researchers.
They want to wait long enough in order to make sure that
they've given their program a full
chance to have its impact.
And so usually when I am posed with this question by an
organization, I usually just ask right back to them, well,
you tell me.
What do you think you need in order to see the impact of
your program?
If you're telling me a story about it being a 5, 10 year
program in order to see everything flourish, well then
that's your answer.
If you're telling me that this is like an amazing thing that
just transforms people's lives within six months, well then
we can go in six months and see that amazing
transformation.
We might also want to see the two year impacts, but that
would be something that could happen with the organization,
and they could say, yes, we think it's transformation in
six months.
And two years is just beyond--
I mean, I don't know what the word is to say beyond
transferring.
Lastly, analyse and assess results.
And obviously, there's a lot more in the class that will be
discussing that.