WEBVTT

00:00:00.040 --> 00:00:02.500
The following content is
provided under a Creative

00:00:02.500 --> 00:00:04.010
Commons license.

00:00:04.010 --> 00:00:06.360
Your support will help
MIT OpenCourseWare

00:00:06.360 --> 00:00:10.710
continue to offer high quality
educational resources for free.

00:00:10.710 --> 00:00:13.340
To make a donation or
view additional materials

00:00:13.340 --> 00:00:17.245
from hundreds of MIT courses,
visit MIT OpenCourseWare

00:00:17.245 --> 00:00:17.870
at ocw.mit.edu.

00:00:21.150 --> 00:00:23.530
PROFESSOR 1: We've
revised 18.05.

00:00:23.530 --> 00:00:26.770
At MIT, 18.05 is the
introductory probability

00:00:26.770 --> 00:00:32.299
and statistics class for
mostly non-math majors.

00:00:32.299 --> 00:00:35.719
For most of them, it's the
only probability and statistics

00:00:35.719 --> 00:00:36.510
class they'll take.

00:00:36.510 --> 00:00:39.500
A lot of them will
go on to be doctors

00:00:39.500 --> 00:00:43.700
and so our goal is to have
at least statistically

00:00:43.700 --> 00:00:47.590
literate doctors out there.

00:00:47.590 --> 00:00:50.110
That's important.

00:00:50.110 --> 00:00:53.780
Now let's keep it back there.

00:00:53.780 --> 00:00:55.570
We've been supported
for two years

00:00:55.570 --> 00:00:57.160
by a Davis Foundation grant.

00:00:57.160 --> 00:00:59.860
The PI on the grant
is Haynes Miller.

00:00:59.860 --> 00:01:03.100
And I classify him as
our PI and visionary,

00:01:03.100 --> 00:01:06.500
because the vision of bringing
active learning into the math

00:01:06.500 --> 00:01:08.970
department was his, and
it takes some effort.

00:01:08.970 --> 00:01:15.550
So thank you for that, and
it's working in some way.

00:01:15.550 --> 00:01:16.889
You'll see.

00:01:16.889 --> 00:01:18.180
Here's the outline of the talk.

00:01:18.180 --> 00:01:19.638
We'll talk about
what we inherited,

00:01:19.638 --> 00:01:24.120
but we came into with
18.05, what we've done,

00:01:24.120 --> 00:01:25.720
and what we've learned.

00:01:25.720 --> 00:01:27.855
And along the way, we'll
do a little demonstration.

00:01:30.360 --> 00:01:35.610
So what we found when we-- the
state of 1805 three years ago,

00:01:35.610 --> 00:01:37.670
was it was a traditional
lecture class

00:01:37.670 --> 00:01:43.390
taught at MIT as a lecture
with a recitation component.

00:01:43.390 --> 00:01:45.740
And the enrollment
was dwindling.

00:01:45.740 --> 00:01:47.240
Students were not taking it.

00:01:47.240 --> 00:01:48.960
They migrated to another class.

00:01:48.960 --> 00:01:52.110
Part of it was that
the faculty who

00:01:52.110 --> 00:01:55.220
were teaching it
were not-- how do

00:01:55.220 --> 00:02:00.130
I say it gently-- were not
so interested in this section

00:02:00.130 --> 00:02:02.090
of undergraduate education.

00:02:02.090 --> 00:02:04.095
And in fact, the
year we took it over,

00:02:04.095 --> 00:02:05.970
I think we were faced--
Haynes can correct me

00:02:05.970 --> 00:02:08.759
if I'm wrong-- with the
choice of either us taking it

00:02:08.759 --> 00:02:12.300
over or not teaching the class.

00:02:12.300 --> 00:02:15.330
So the first year we did
teach it as a lecture class.

00:02:15.330 --> 00:02:17.600
John was the
recitation instructor.

00:02:17.600 --> 00:02:18.690
I was the lecturer.

00:02:18.690 --> 00:02:21.140
And then in the
next two years we

00:02:21.140 --> 00:02:25.900
converted to a fully flipped
classroom, active learning,

00:02:25.900 --> 00:02:28.990
and we co-teach it.

00:02:28.990 --> 00:02:30.570
So what we found is just that.

00:02:30.570 --> 00:02:33.150
Haynes had a big interest
in active learning,

00:02:33.150 --> 00:02:35.630
he's been pushing
them in recitations

00:02:35.630 --> 00:02:40.810
for years with some success I
would say now, it's built up,

00:02:40.810 --> 00:02:45.960
and Davis and everyone is
interested in the world

00:02:45.960 --> 00:02:49.110
of online learning and
what can happen there.

00:02:49.110 --> 00:02:52.310
So we have a component
of the class online.

00:02:52.310 --> 00:02:57.370
The class itself will be put
on OpenCourseWare Educator

00:02:57.370 --> 00:02:58.380
at the end of this year.

00:03:01.090 --> 00:03:02.170
So what did we do?

00:03:02.170 --> 00:03:04.100
We changed everything at once.

00:03:04.100 --> 00:03:07.900
In retrospect, I would
say I don't recommend it.

00:03:07.900 --> 00:03:09.549
But having gotten
through it, you're

00:03:09.549 --> 00:03:12.090
always glad that you did it all
at once, because now we don't

00:03:12.090 --> 00:03:13.400
have to do any of these pieces.

00:03:13.400 --> 00:03:16.107
We can just keep refining them.

00:03:16.107 --> 00:03:17.690
It's a lot to talk
about, so we're not

00:03:17.690 --> 00:03:19.510
going to focus much
on the new curriculum.

00:03:19.510 --> 00:03:21.320
I'll just say a few
words, that we're

00:03:21.320 --> 00:03:25.110
really proud of the
curriculum that we have.

00:03:25.110 --> 00:03:28.600
We've introduced a unit
on Bayesian inference.

00:03:28.600 --> 00:03:35.234
[INAUDIBLE] here is our
Bayesian, can I say zealot?

00:03:35.234 --> 00:03:37.725
Does it say that's your
religion on your Facebook site?

00:03:37.725 --> 00:03:38.391
PROFESSOR 2: No.

00:03:38.391 --> 00:03:38.870
That's me.

00:03:38.870 --> 00:03:40.078
PROFESSOR 1: Oh, that's John.

00:03:42.010 --> 00:03:43.680
That's on John's site.

00:03:43.680 --> 00:03:45.184
John is a convert.

00:03:45.184 --> 00:03:47.660
A Bayesian unit.

00:03:47.660 --> 00:03:49.360
What that has done
for us is allows

00:03:49.360 --> 00:03:53.800
us to bridge the gap between
the probability and statistics

00:03:53.800 --> 00:03:55.200
part, which is
always a hard gap.

00:03:55.200 --> 00:03:58.270
Probability I'll say is
a mathematical science.

00:03:58.270 --> 00:04:00.530
Statistic is that
statistics are at least as

00:04:00.530 --> 00:04:02.792
much art as science.

00:04:02.792 --> 00:04:04.250
And so it allows
us to bridge that.

00:04:04.250 --> 00:04:06.970
Bayesian inference
and Bayesian learning

00:04:06.970 --> 00:04:10.930
are becoming more and
more important today.

00:04:10.930 --> 00:04:12.550
So it's good that they see it.

00:04:12.550 --> 00:04:14.670
And finally, even
if they're just

00:04:14.670 --> 00:04:17.130
doctors who need to
know what a p-value is,

00:04:17.130 --> 00:04:19.040
we feel we can explain
the p-value better

00:04:19.040 --> 00:04:20.500
by having introduced this unit.

00:04:20.500 --> 00:04:23.700
So we actually added three weeks
worth of material to the class,

00:04:23.700 --> 00:04:26.340
and we'll talk a little
bit about that later.

00:04:26.340 --> 00:04:29.574
Then we'll speak more
about the pedagogy,

00:04:29.574 --> 00:04:31.740
the importance of the
classroom, and what technology

00:04:31.740 --> 00:04:34.820
we have in the class.

00:04:34.820 --> 00:04:37.190
And now we-- actually
we're going to show you--

00:04:37.190 --> 00:04:40.734
this is a video-- a little bit,
where you should pay attention

00:04:40.734 --> 00:04:42.150
while you're
watching the video is

00:04:42.150 --> 00:04:45.420
the energy in the classroom
and particularly the comments

00:04:45.420 --> 00:04:48.151
and the oohs and aahs that come
at the end when you'll see John

00:04:48.151 --> 00:04:49.400
talking some students on this.

00:04:49.400 --> 00:04:53.000
Today we're going to do
Bayesian updating, which

00:04:53.000 --> 00:04:55.200
is to say we're going to
be using Bayes' theorem

00:04:55.200 --> 00:04:56.370
to learn from data.

00:04:56.370 --> 00:04:58.840
Which treatment would you choose
if you needed a treatment?

00:04:58.840 --> 00:05:02.590
Treatment one cured
100% of the patients.

00:05:02.590 --> 00:05:05.800
Treatment two cured
95% in a trial.

00:05:05.800 --> 00:05:07.820
And treatment three cured 90%.

00:05:07.820 --> 00:05:08.410
All right.

00:05:08.410 --> 00:05:11.420
There's some holdouts
for the 95% cure.

00:05:11.420 --> 00:05:12.420
I'm not sure why.

00:05:12.420 --> 00:05:15.910
What if I gave you
this information.

00:05:15.910 --> 00:05:17.910
Treatment one cured three
out of three patients.

00:05:17.910 --> 00:05:19.740
That's 100%.

00:05:19.740 --> 00:05:22.310
Treatment two cured
19 of 20 patients.

00:05:22.310 --> 00:05:24.450
That's 95%.

00:05:24.450 --> 00:05:26.050
Or you have a standard
treatment which

00:05:26.050 --> 00:05:30.700
has cured 90,000 out of
100,000 patients-- 90%.

00:05:30.700 --> 00:05:35.520
PROFESSOR 2: So suppose,
in this at MIT mug,

00:05:35.520 --> 00:05:39.890
I have dice of two types--
4-sided and 20-sided.

00:05:39.890 --> 00:05:43.608
I'm going to randomly pull
out a die and roll it.

00:05:43.608 --> 00:05:46.020
All right, I got a one.

00:05:46.020 --> 00:05:49.560
Which type of die do you
think I randomly chose?

00:05:49.560 --> 00:05:52.090
OK that was actually
what I had in this cup.

00:05:52.090 --> 00:05:53.860
Does this change at
all of your analysis

00:05:53.860 --> 00:05:56.190
of which die you think I
rolled and got a one with?

00:05:56.190 --> 00:05:57.770
So this is the first
board question.

00:05:57.770 --> 00:05:58.270
All right.

00:05:58.270 --> 00:06:01.190
I want you to practice making
the table Jerry showed you.

00:06:01.190 --> 00:06:03.120
Make one for if it were 13.

00:06:03.120 --> 00:06:05.730
Make one if I rolled
a five instead.

00:06:05.730 --> 00:06:07.850
Or do the same question
if I had rolled a nine.

00:06:17.300 --> 00:06:20.627
MALE VOICE 1: He knew
it was a 20-sided die.

00:06:20.627 --> 00:06:22.210
PROFESSOR 2: Your
beliefs have changed

00:06:22.210 --> 00:06:24.200
based on that about the
probability that you

00:06:24.200 --> 00:06:26.040
have each of the five dice.

00:06:26.040 --> 00:06:29.660
So if you take that and
you use that as your prior

00:06:29.660 --> 00:06:34.130
here, then the likelihood,
multiply and sum,

00:06:34.130 --> 00:06:36.040
that should give you--

00:06:36.040 --> 00:06:38.569
FEMALE VOICE: I Like that.

00:06:38.569 --> 00:06:39.360
PROFESSOR 1: Right?

00:06:39.360 --> 00:06:41.570
So I've lectured for years.

00:06:41.570 --> 00:06:43.500
I don't know that
I've ever heard

00:06:43.500 --> 00:06:45.760
of that many oohs and
aahs or a statement,

00:06:45.760 --> 00:06:47.370
I like that, at the end.

00:06:47.370 --> 00:06:49.906
Sometimes there's a student
who nods very enthusiastically.

00:06:53.531 --> 00:06:54.030
OK.

00:06:54.030 --> 00:06:57.220
So what I'm going to go through
here is just what we do.

00:06:57.220 --> 00:06:58.920
I'm going to resist
the urge to talk

00:06:58.920 --> 00:07:00.570
a lot about why
we like doing it,

00:07:00.570 --> 00:07:01.960
because that will come later.

00:07:01.960 --> 00:07:04.230
And John will talk about that.

00:07:04.230 --> 00:07:07.780
The structure is we need
three times 80 minutes.

00:07:07.780 --> 00:07:11.650
So that's 240 minutes,
which matches the five

00:07:11.650 --> 00:07:14.050
times 50 minutes that a
traditional class meets.

00:07:14.050 --> 00:07:16.690
Three lecture hours,
two recitation.

00:07:16.690 --> 00:07:20.440
So it's not more
time, but at all times

00:07:20.440 --> 00:07:25.540
we have two teachers this year
and three assistant teachers

00:07:25.540 --> 00:07:26.230
in the room.

00:07:26.230 --> 00:07:31.350
So there's five instructors
in the room at all times.

00:07:31.350 --> 00:07:32.280
What we have them do.

00:07:32.280 --> 00:07:34.697
Before class, they
do some reading.

00:07:34.697 --> 00:07:36.280
To get them to do
the reading, there's

00:07:36.280 --> 00:07:38.120
reading questions which
they get credit for.

00:07:38.120 --> 00:07:41.970
These are all online,
it's hosted on MITx,

00:07:41.970 --> 00:07:43.310
and they get immediate feedback.

00:07:43.310 --> 00:07:46.290
They know whether they got
the question right or wrong.

00:07:46.290 --> 00:07:49.480
It's possible to ask lots
and lots of different types

00:07:49.480 --> 00:07:52.190
of questions, although we
restrict ourselves mostly

00:07:52.190 --> 00:07:57.480
to multiple choice and fill in
the number type of question.

00:07:57.480 --> 00:08:00.880
Jenny French, who's here, has
done the same thing for 18.03,

00:08:00.880 --> 00:08:05.530
and she's been more ambitious
with the types of questions.

00:08:05.530 --> 00:08:08.260
They come to class and we do
a minimal amount of lecturing.

00:08:08.260 --> 00:08:11.650
What gets left out of a
math class, of our class,

00:08:11.650 --> 00:08:14.590
as opposed to a traditional
class, lecture class, is we

00:08:14.590 --> 00:08:15.892
don't do examples.

00:08:15.892 --> 00:08:17.850
We're going to have the
students work examples,

00:08:17.850 --> 00:08:20.410
and it's much more better--
much more better?--

00:08:20.410 --> 00:08:25.194
much better for students
to work one example than

00:08:25.194 --> 00:08:26.360
for the teacher to work two.

00:08:26.360 --> 00:08:29.320
We feel this strongly.

00:08:29.320 --> 00:08:30.184
You saw that.

00:08:30.184 --> 00:08:31.600
You saw a little
bit of lecturing.

00:08:31.600 --> 00:08:32.200
It was cut.

00:08:32.200 --> 00:08:35.270
I maybe lectured a little more
in the video than you saw,

00:08:35.270 --> 00:08:36.710
but very quickly,
we have them up

00:08:36.710 --> 00:08:38.650
at the boards doing
group problems

00:08:38.650 --> 00:08:43.690
as you saw in the video, while
the teachers and the assistants

00:08:43.690 --> 00:08:45.940
are circulating and
talking to people

00:08:45.940 --> 00:08:48.420
and helping them get
past stumbling points

00:08:48.420 --> 00:08:52.147
and assessing what they're
understanding and correcting

00:08:52.147 --> 00:08:52.772
misconceptions.

00:08:57.300 --> 00:08:59.960
Other components of the class
which you didn't see as much:

00:08:59.960 --> 00:09:02.120
we have whole-class
discussions; we'll

00:09:02.120 --> 00:09:04.950
have them discuss things that
they're tables in groups;

00:09:04.950 --> 00:09:08.220
we can operate on many
levels of interaction

00:09:08.220 --> 00:09:11.410
between students and students,
and students and teachers.

00:09:11.410 --> 00:09:15.090
We also use a little bit of
technology, clicker questions,

00:09:15.090 --> 00:09:17.160
where we'll ask a
multiple choice,

00:09:17.160 --> 00:09:18.920
usually a kind of
concept question

00:09:18.920 --> 00:09:23.800
to see if they've
understood one idea,

00:09:23.800 --> 00:09:26.850
and it also allows us
to take attendance.

00:09:26.850 --> 00:09:29.460
And the students
see it that way.

00:09:29.460 --> 00:09:31.620
Once a week, on
Fridays, we actually

00:09:31.620 --> 00:09:33.750
do a computer
based studio, where

00:09:33.750 --> 00:09:34.850
they have their computers.

00:09:34.850 --> 00:09:38.000
It's the one time we let
them take their computers out

00:09:38.000 --> 00:09:41.560
and we do it in our--
typically we're not

00:09:41.560 --> 00:09:42.860
introducing new material.

00:09:42.860 --> 00:09:45.940
We're doing more advanced
work or simulations

00:09:45.940 --> 00:09:48.360
that you can do on the computer.

00:09:48.360 --> 00:09:51.740
And finally, we do have
traditional problem sets.

00:09:51.740 --> 00:09:55.160
They have to turn in the problem
set on paper once a week,

00:09:55.160 --> 00:09:59.270
and it's graded
in the usual way.

00:09:59.270 --> 00:10:01.190
The pset checker
is not traditional.

00:10:01.190 --> 00:10:04.695
This is something we borrowed
from the physics people.

00:10:04.695 --> 00:10:06.030
Safe is here, I think.

00:10:06.030 --> 00:10:08.105
I think you guys pioneered this.

00:10:08.105 --> 00:10:11.080
It allows the students
to check their answers

00:10:11.080 --> 00:10:16.320
to their homework ahead of time,
which allows us to tell them

00:10:16.320 --> 00:10:18.900
we care about how
you present it,

00:10:18.900 --> 00:10:22.120
that your logic and your
thinking is correct,

00:10:22.120 --> 00:10:24.239
it allows the students--
I can't resist saying

00:10:24.239 --> 00:10:25.530
this one thing's on, I'm sorry.

00:10:25.530 --> 00:10:28.370
It allows the students to
reflect on the piece sets

00:10:28.370 --> 00:10:29.670
as they're doing it.

00:10:29.670 --> 00:10:32.170
One of the things we know is
when we turn back a pset a week

00:10:32.170 --> 00:10:35.319
later, it's gone cold,
they've barely looked at it

00:10:35.319 --> 00:10:36.610
to see what they've done wrong.

00:10:36.610 --> 00:10:41.000
Here they see immediately it's
wrong, and they can reflect.

00:10:41.000 --> 00:10:41.500
OK.

00:10:41.500 --> 00:10:46.331
I think now we have an
active part of this.

00:10:46.331 --> 00:10:47.080
John is going to--

00:10:47.080 --> 00:10:49.246
PROFESSOR 2: We believe
strongly in active learning,

00:10:49.246 --> 00:10:51.211
so we got to do some
active learning.

00:10:51.211 --> 00:10:52.960
But really what I want
to take you through

00:10:52.960 --> 00:10:57.200
is how we might envision a
particular new concept being

00:10:57.200 --> 00:10:58.629
introduced to the students.

00:10:58.629 --> 00:11:01.170
What are the various components
and how do they fit together?

00:11:01.170 --> 00:11:03.642
So Jerry mentioned
that we have this unit,

00:11:03.642 --> 00:11:05.850
the middle unit of our
course, on Bayesian inference.

00:11:05.850 --> 00:11:08.475
And the first day we do a number
of problems that involve dice.

00:11:08.475 --> 00:11:10.308
In particular, at the
beginning of the term,

00:11:10.308 --> 00:11:12.080
we actually give
students dice that we

00:11:12.080 --> 00:11:14.210
expect them to keep with
them throughout the term,

00:11:14.210 --> 00:11:15.680
and we use them
to generate data,

00:11:15.680 --> 00:11:18.150
and to do probability
experiments,

00:11:18.150 --> 00:11:20.240
but also just to make
something tangible.

00:11:20.240 --> 00:11:22.701
And so, in particular
on this first day,

00:11:22.701 --> 00:11:25.200
the students will have come in
having done reading, maybe we

00:11:25.200 --> 00:11:27.760
had about 10 pages, they
read, they answer some reading

00:11:27.760 --> 00:11:28.790
questions.

00:11:28.790 --> 00:11:31.610
And this introduces them
to some new concepts,

00:11:31.610 --> 00:11:33.640
but doesn't expect them
to have mastered it.

00:11:33.640 --> 00:11:36.520
And then, in the beginning of
that class that you saw filmed,

00:11:36.520 --> 00:11:39.090
we had some discussions
surrounding these dice.

00:11:39.090 --> 00:11:41.470
So what you see here
is I have five dice.

00:11:41.470 --> 00:11:43.000
This four-sided.

00:11:43.000 --> 00:11:45.590
This is a six-sided
regular old die.

00:11:45.590 --> 00:11:47.220
Eight, 12, and 20.

00:11:47.220 --> 00:11:48.950
So these are platonic dice.

00:11:48.950 --> 00:11:52.250
And the question I have
for you is as follows.

00:11:52.250 --> 00:11:53.870
So I actually have
smaller versions.

00:11:53.870 --> 00:11:56.585
I'm going to take them
and put them in this cup.

00:11:56.585 --> 00:11:57.360
All right.

00:11:57.360 --> 00:12:00.310
I'll shake it around.

00:12:00.310 --> 00:12:00.810
All right.

00:12:00.810 --> 00:12:03.070
Allison, reach in and grab
whatever and don't look.

00:12:06.651 --> 00:12:07.150
All right.

00:12:07.150 --> 00:12:08.530
Hand it to me.

00:12:08.530 --> 00:12:09.280
Thank you.

00:12:09.280 --> 00:12:13.901
And then Lourdes can you come
up and be our verifier here.

00:12:13.901 --> 00:12:14.400
All right.

00:12:14.400 --> 00:12:17.350
So four dice are
still in this cup.

00:12:17.350 --> 00:12:18.547
Lourdes is going tell--

00:12:18.547 --> 00:12:19.297
PROFESSOR 1: John.

00:12:19.297 --> 00:12:19.890
John.

00:12:19.890 --> 00:12:20.556
PROFESSOR 2: No.

00:12:20.556 --> 00:12:21.730
I'll do it in the cup.

00:12:21.730 --> 00:12:23.310
So you're going to
say what I roll.

00:12:23.310 --> 00:12:23.810
OK.

00:12:23.810 --> 00:12:25.900
I'm going to drop it
in, shake it around.

00:12:25.900 --> 00:12:26.627
All right.

00:12:26.627 --> 00:12:27.460
Just say the number.

00:12:27.460 --> 00:12:29.166
Do not say which die.

00:12:29.166 --> 00:12:29.832
OK?

00:12:29.832 --> 00:12:30.790
What number do you see?

00:12:30.790 --> 00:12:31.430
AUDIENCE: Four.

00:12:31.430 --> 00:12:32.850
PROFESSOR 2: A four.

00:12:32.850 --> 00:12:33.650
Great?

00:12:33.650 --> 00:12:37.232
So which die do
you think I chose?

00:12:37.232 --> 00:12:39.440
Raise your hand if you think
it's the four-sided die.

00:12:39.440 --> 00:12:41.130
Let's say which is
the most likely.

00:12:41.130 --> 00:12:44.150
Raise your hand if you
think it's a six-sided.

00:12:44.150 --> 00:12:45.260
OK.

00:12:45.260 --> 00:12:46.658
Eight?

00:12:46.658 --> 00:12:47.590
12?

00:12:47.590 --> 00:12:48.320
20?

00:12:48.320 --> 00:12:50.361
Raise your hand if you
think actually they're all

00:12:50.361 --> 00:12:52.610
equally likely at this point.

00:12:52.610 --> 00:12:53.380
Great.

00:12:53.380 --> 00:12:55.440
So this is an example of
an excellent opportunity

00:12:55.440 --> 00:12:57.648
to have a whole-class
discussion, because clearly you

00:12:57.648 --> 00:12:58.460
don't all agree.

00:12:58.460 --> 00:13:00.918
So in our class, since they
sit around these tables of nine

00:13:00.918 --> 00:13:03.000
people, we might have them
say, OK, well, discuss

00:13:03.000 --> 00:13:05.459
at the level of your table--
you're all facing each other--

00:13:05.459 --> 00:13:07.208
and see if you can
come to some consensus.

00:13:07.208 --> 00:13:08.870
So let's try to
convince each other.

00:13:08.870 --> 00:13:09.369
All right.

00:13:09.369 --> 00:13:11.830
Well, I'm not going to tell
you the answer just yet.

00:13:11.830 --> 00:13:13.030
Let's do another roll.

00:13:13.030 --> 00:13:13.855
Maybe that'll help.

00:13:13.855 --> 00:13:15.160
AUDIENCE: Is that the same die?

00:13:15.160 --> 00:13:15.970
PROFESSOR 2: It's the same die.

00:13:15.970 --> 00:13:17.140
I didn't, You saw.

00:13:17.140 --> 00:13:19.940
It's not magic.

00:13:19.940 --> 00:13:20.440
All right.

00:13:20.440 --> 00:13:21.152
What I just roll?

00:13:21.152 --> 00:13:21.860
AUDIENCE: A five.

00:13:21.860 --> 00:13:23.080
PROFESSOR 2: A five.

00:13:23.080 --> 00:13:23.580
Ooh.

00:13:23.580 --> 00:13:24.220
OK.

00:13:24.220 --> 00:13:29.320
So now raise your hand,
is it the four-sided die?

00:13:29.320 --> 00:13:30.300
Very good.

00:13:30.300 --> 00:13:32.290
Six-sided die?

00:13:32.290 --> 00:13:32.790
OK.

00:13:32.790 --> 00:13:35.210
Eight?

00:13:35.210 --> 00:13:36.180
12?

00:13:36.180 --> 00:13:36.770
20?

00:13:36.770 --> 00:13:37.760
Are they all equally likely?

00:13:37.760 --> 00:13:38.660
Raise your hand
if you still think

00:13:38.660 --> 00:13:39.857
they're all equally likely.

00:13:39.857 --> 00:13:40.940
That's interesting, right?

00:13:40.940 --> 00:13:42.917
So there's been
some progress here.

00:13:42.917 --> 00:13:44.500
Maybe discuss with
your neighbors now.

00:13:44.500 --> 00:13:46.462
I'll give you 10 seconds.

00:13:46.462 --> 00:13:47.420
PROFESSOR 1: All right.

00:13:47.420 --> 00:13:48.330
This is great.

00:13:48.330 --> 00:13:50.270
Usually it takes us
about a week and a half

00:13:50.270 --> 00:13:53.420
before we get people to
discuss this animatedly when

00:13:53.420 --> 00:13:54.630
we do things.

00:13:54.630 --> 00:13:56.010
They have to learn to trust us.

00:13:56.010 --> 00:13:56.310
All right.

00:13:56.310 --> 00:13:57.330
So you've had some discussions.

00:13:57.330 --> 00:13:59.590
I might call on people to
give their explanations,

00:13:59.590 --> 00:14:02.180
but in the interest of time, let
me just put a few more times.

00:14:02.180 --> 00:14:03.450
Now I got a six.

00:14:03.450 --> 00:14:04.560
Now I got a three.

00:14:04.560 --> 00:14:05.700
Now I got a two.

00:14:05.700 --> 00:14:06.820
Now I got a one.

00:14:06.820 --> 00:14:07.780
Now I got a six.

00:14:07.780 --> 00:14:08.280
OK.

00:14:08.280 --> 00:14:11.450
Raise your hand if you
think it a six-sided die?

00:14:11.450 --> 00:14:12.260
Wonderful.

00:14:12.260 --> 00:14:12.760
All right.

00:14:12.760 --> 00:14:14.160
FEMALE VOICE: It is.

00:14:14.160 --> 00:14:16.927
PROFESSOR 2: You never know in
science whether you're right.

00:14:16.927 --> 00:14:18.260
PROFESSOR 1: This is statistics.

00:14:18.260 --> 00:14:20.046
I'm not going to tell you.

00:14:20.046 --> 00:14:20.545
Sorry.

00:14:24.362 --> 00:14:26.320
No, that's the difference
between probabilities

00:14:26.320 --> 00:14:26.945
and statistics.

00:14:26.945 --> 00:14:28.110
You never know.

00:14:28.110 --> 00:14:30.820
All right, so, so-- great.

00:14:30.820 --> 00:14:32.590
We've gotten the
students to appreciate

00:14:32.590 --> 00:14:35.020
there's some interesting
question here,

00:14:35.020 --> 00:14:36.010
maybe some intuition.

00:14:36.010 --> 00:14:37.170
They've discussed it.

00:14:37.170 --> 00:14:39.881
Next, what you saw on the video
was the students at the boards,

00:14:39.881 --> 00:14:41.630
working on problems
where they're actually

00:14:41.630 --> 00:14:44.622
doing computations
meant to take them

00:14:44.622 --> 00:14:46.330
to the next level,
where they're not just

00:14:46.330 --> 00:14:48.413
beginning to appreciate
that something's going on,

00:14:48.413 --> 00:14:51.030
but to actually rigorously
mathematically do

00:14:51.030 --> 00:14:51.980
some computations.

00:14:51.980 --> 00:14:53.855
And they've seen examples
of this computation

00:14:53.855 --> 00:14:54.481
in the reading.

00:14:54.481 --> 00:14:55.980
And we're all going
around the room,

00:14:55.980 --> 00:14:58.000
and we're helping,
interacting with the groups.

00:14:58.000 --> 00:15:00.780
And after they do this,
they come back together,

00:15:00.780 --> 00:15:02.597
we might discuss
a solution, maybe

00:15:02.597 --> 00:15:04.430
there'll be some clicker
questions involved.

00:15:04.430 --> 00:15:07.680
And they'll still have,
on Friday, the studio.

00:15:07.680 --> 00:15:09.990
In the studio they
might use R. Perhaps

00:15:09.990 --> 00:15:12.470
we've written a
script that simulates

00:15:12.470 --> 00:15:14.130
this rolling of a
die, and updates

00:15:14.130 --> 00:15:17.550
the prior to the posterior
about which die it is, and then

00:15:17.550 --> 00:15:20.259
they'll experiment with
different situations

00:15:20.259 --> 00:15:22.550
with this simulation, or
changing some component of it.

00:15:22.550 --> 00:15:23.890
We don't expect them
to be experts in R,

00:15:23.890 --> 00:15:25.681
but we expect them to
be able to understand

00:15:25.681 --> 00:15:28.389
code we've written and
modified in some way to build

00:15:28.389 --> 00:15:29.680
their conceptual understanding.

00:15:29.680 --> 00:15:30.890
And then they'll
have a problem set.

00:15:30.890 --> 00:15:32.723
The problem set will
ask them some questions

00:15:32.723 --> 00:15:33.650
that build further.

00:15:33.650 --> 00:15:35.120
And there will be
exams, and so on.

00:15:35.120 --> 00:15:39.650
So that's the sort of
trajectory of seeing

00:15:39.650 --> 00:15:44.140
a topic, sort of building up
your level of understanding.

00:15:44.140 --> 00:15:48.180
So now, I'm going to say a
little bit now about the why.

00:15:48.180 --> 00:15:50.130
So Jerry told you of
the what of what we do.

00:15:50.130 --> 00:15:51.425
Here's a little bit
about why we do it

00:15:51.425 --> 00:15:53.800
and what we've learned from
the experience of doing it,

00:15:53.800 --> 00:15:55.210
and what we like.

00:15:55.210 --> 00:15:59.470
So in terms of active learning
versus traditional lecture.

00:15:59.470 --> 00:16:01.820
The first thing to note is
that just the physical act

00:16:01.820 --> 00:16:05.330
of standing up serves
many very useful purposes.

00:16:05.330 --> 00:16:07.700
The first is that in a
given 80-minute session,

00:16:07.700 --> 00:16:10.380
students might be
up on their feet

00:16:10.380 --> 00:16:12.820
two or three times
for 10 to 15 minutes.

00:16:12.820 --> 00:16:15.680
And when they're up,
they're not on the phone,

00:16:15.680 --> 00:16:18.770
they're not on the computer,
and they're not asleep.

00:16:18.770 --> 00:16:20.150
So those are all good things.

00:16:20.150 --> 00:16:22.500
But even more, just
the act of standing,

00:16:22.500 --> 00:16:24.490
it sort of in some ways
activates your focus.

00:16:24.490 --> 00:16:26.448
You're at the board,
there aren't distractions,

00:16:26.448 --> 00:16:29.400
you're with your group members,
and you have a task at hand.

00:16:29.400 --> 00:16:32.680
And so just the physical
act of standing we find

00:16:32.680 --> 00:16:36.070
is a way to raise the
energy in the room.

00:16:36.070 --> 00:16:38.790
Now I'll not make all you
stand while we talk about this,

00:16:38.790 --> 00:16:41.960
but I think you would agree
that standing wakes you up.

00:16:41.960 --> 00:16:43.060
The physical space.

00:16:43.060 --> 00:16:44.660
This TEAL room is
kind of amazing,

00:16:44.660 --> 00:16:47.420
but what's most
amazing about it is not

00:16:47.420 --> 00:16:49.830
that it has all these
computers at the tables, which

00:16:49.830 --> 00:16:52.621
at this stage isn't so useful,
because everyone has laptops.

00:16:52.621 --> 00:16:54.870
It's not even really the
video cameras and projectors.

00:16:54.870 --> 00:16:58.720
It's just the physical space is
such that students can rapidly

00:16:58.720 --> 00:17:01.750
get out of the seats, find room
on the board, work on a problem

00:17:01.750 --> 00:17:03.857
comfortably, come back,
sit at these tables.

00:17:03.857 --> 00:17:05.690
Their chairs swivel
without them getting up,

00:17:05.690 --> 00:17:07.065
because they have
little rollers.

00:17:07.065 --> 00:17:08.680
It's carpeted, so
it's not too noisy.

00:17:08.680 --> 00:17:11.329
There's just lots of things
about the physical space that

00:17:11.329 --> 00:17:13.589
are really nice and
make the class possible.

00:17:13.589 --> 00:17:15.849
So if we had a choice
between technology

00:17:15.849 --> 00:17:19.260
versus just physical space,
we would choose the latter,

00:17:19.260 --> 00:17:20.212
in a heartbeat.

00:17:20.212 --> 00:17:22.420
And that means that in
classrooms where you might not

00:17:22.420 --> 00:17:23.819
have whiteboards
everywhere, you might want

00:17:23.819 --> 00:17:25.110
to think about putting some up.

00:17:25.110 --> 00:17:28.720
In Haiti we actually put
up paper on the walls

00:17:28.720 --> 00:17:31.622
in order to do simulations
with professors there,

00:17:31.622 --> 00:17:34.080
to get them to feel what active
learning was like, at least

00:17:34.080 --> 00:17:35.710
the way we were trying it.

00:17:35.710 --> 00:17:37.540
Peer and teacher instruction.

00:17:37.540 --> 00:17:40.270
So here, there's a lot of
talk about peer instruction,

00:17:40.270 --> 00:17:42.674
about how students can
help teach one another,

00:17:42.674 --> 00:17:43.840
and that's really important.

00:17:43.840 --> 00:17:45.330
And there's also
teacher instruction,

00:17:45.330 --> 00:17:47.090
where the teacher's
interacting with the student,

00:17:47.090 --> 00:17:49.216
and helping them move along
in their understanding.

00:17:49.216 --> 00:17:51.506
And what we find, in the
classroom, is really happening

00:17:51.506 --> 00:17:52.300
is a lot of both.

00:17:52.300 --> 00:17:53.510
They're both important.

00:17:53.510 --> 00:17:56.154
So the peer instruction happens
because they have discussions

00:17:56.154 --> 00:17:58.195
at the table, or because
they're working together

00:17:58.195 --> 00:17:59.490
at groups in the board.

00:17:59.490 --> 00:18:02.180
In particular, at
the board, you might

00:18:02.180 --> 00:18:04.300
have one student
who's very strong,

00:18:04.300 --> 00:18:07.220
a couple others who aren't as
strong on a particular problem,

00:18:07.220 --> 00:18:09.017
and us going around
can very much

00:18:09.017 --> 00:18:11.225
encourage the student who
seems to be taking the lead

00:18:11.225 --> 00:18:13.550
to make sure the other
group members understand,

00:18:13.550 --> 00:18:15.980
or to try to vary
who's taking that lead.

00:18:15.980 --> 00:18:17.880
But you see a lot of
students explaining

00:18:17.880 --> 00:18:20.080
to each other, the students
being able to look over

00:18:20.080 --> 00:18:22.840
at the next group if
they're stuck, all of this

00:18:22.840 --> 00:18:24.542
is a kind of peer instruction.

00:18:24.542 --> 00:18:26.250
But there's also the
teacher instruction,

00:18:26.250 --> 00:18:28.830
which is the fact that
Jerry and I, and then

00:18:28.830 --> 00:18:30.730
a graduate student
and two undergrads

00:18:30.730 --> 00:18:34.250
are in the room, able to
basically distribute ourselves

00:18:34.250 --> 00:18:38.420
like zone defense, and monitor
a certain number of groups,

00:18:38.420 --> 00:18:40.842
and really keep tabs on the
progress they're making.

00:18:40.842 --> 00:18:42.300
Because you're
going to have groups

00:18:42.300 --> 00:18:45.302
that are faster and slower, and
you want to kind of bring them

00:18:45.302 --> 00:18:47.010
to about the same
point at the same time,

00:18:47.010 --> 00:18:48.730
so they can get the
most out of the time

00:18:48.730 --> 00:18:50.770
when you come to
the explanation.

00:18:50.770 --> 00:18:54.330
So that sort of interaction
between the teachers

00:18:54.330 --> 00:18:55.920
and the students is
also really huge,

00:18:55.920 --> 00:18:58.378
because the students actually
feel like they get to know us

00:18:58.378 --> 00:19:00.440
and we get to feel
like we know them.

00:19:00.440 --> 00:19:04.160
The student self-assessment,
that's referring to the idea

00:19:04.160 --> 00:19:07.270
that there's a really rich
formative assessment happening

00:19:07.270 --> 00:19:10.410
in the classroom, when the
students are asked especially

00:19:10.410 --> 00:19:12.370
to get up and solve
problems at the board.

00:19:12.370 --> 00:19:13.790
When you listen
to teacher lecture

00:19:13.790 --> 00:19:18.094
at you, doing an example,
you may nod your head

00:19:18.094 --> 00:19:19.260
and think you understand it.

00:19:19.260 --> 00:19:21.880
But when you're at a board, or
when you're using a clicker,

00:19:21.880 --> 00:19:23.965
you actually have
to do the problem.

00:19:23.965 --> 00:19:26.090
You figure out really
quickly if you understand it,

00:19:26.090 --> 00:19:26.980
because if you're
standing there,

00:19:26.980 --> 00:19:29.150
and you have no idea
where to get started,

00:19:29.150 --> 00:19:30.300
you don't understand it.

00:19:30.300 --> 00:19:30.944
And you know.

00:19:30.944 --> 00:19:32.485
But you want to
create an environment

00:19:32.485 --> 00:19:34.760
where that's OK, because
maybe your group mates know.

00:19:34.760 --> 00:19:35.610
Or maybe we know.

00:19:35.610 --> 00:19:38.570
And they're comfortable
saying, hey John, Jerry, can

00:19:38.570 --> 00:19:39.570
you help us get started.

00:19:39.570 --> 00:19:41.262
Or even better,
we're on top of them

00:19:41.262 --> 00:19:42.720
even before they
get to that point,

00:19:42.720 --> 00:19:44.130
because we have a sense
at this point which

00:19:44.130 --> 00:19:45.800
groups are going to have the
most trouble getting started.

00:19:45.800 --> 00:19:47.990
And there are some groups where
once you get them started,

00:19:47.990 --> 00:19:48.540
they fly.

00:19:48.540 --> 00:19:50.520
And it's just a matter of
being there at the beginning.

00:19:50.520 --> 00:19:52.395
There are other groups
which are the reverse.

00:19:52.395 --> 00:19:55.301
So you get to learn that
through this rich interaction.

00:19:55.301 --> 00:19:57.300
And that self-assessment
the students say really

00:19:57.300 --> 00:19:59.800
helps them know if they know.

00:19:59.800 --> 00:20:00.940
Right?

00:20:00.940 --> 00:20:02.200
Teacher formative assessment.

00:20:02.200 --> 00:20:03.450
I guess I just mentioned that.

00:20:03.450 --> 00:20:06.930
That's us being able to see
where these groups are at.

00:20:06.930 --> 00:20:09.156
So the advantage of them
standing up at the board

00:20:09.156 --> 00:20:11.030
is that from a distance
you can scan the room

00:20:11.030 --> 00:20:14.690
and see very quickly how far
each student or group has

00:20:14.690 --> 00:20:17.230
gotten on these problems, which
student is holding the pen

00:20:17.230 --> 00:20:19.860
or explaining.

00:20:19.860 --> 00:20:21.790
By the time this
period has ended

00:20:21.790 --> 00:20:23.700
where they've been
working on these problems,

00:20:23.700 --> 00:20:27.667
you've helped several groups get
past several stumbling points.

00:20:27.667 --> 00:20:29.750
So now comes time to bring
the class back together

00:20:29.750 --> 00:20:31.940
and go through an
explanation of those

00:20:31.940 --> 00:20:34.090
parts which are most
helpful, and you know,

00:20:34.090 --> 00:20:37.110
because you actually
already helped them.

00:20:37.110 --> 00:20:39.860
It's like you had an office
hour where everyone attended,

00:20:39.860 --> 00:20:42.030
and now you get to
explain the answer, right?

00:20:42.030 --> 00:20:46.380
And so that's been
really exciting for us.

00:20:46.380 --> 00:20:50.180
So those are some of the active
learning benefits we found.

00:20:50.180 --> 00:20:52.750
And then, in terms of
technology, there are, I guess,

00:20:52.750 --> 00:20:54.280
three ways we're
using technology

00:20:54.280 --> 00:20:55.490
to enhance the class.

00:20:55.490 --> 00:20:57.440
The reading questions,
as Jerry said,

00:20:57.440 --> 00:20:59.560
these are, by and
large, a way to get

00:20:59.560 --> 00:21:03.005
students to at least open
the PDF that has the reading.

00:21:03.005 --> 00:21:05.980
The students at MIT are going
to play some game theory

00:21:05.980 --> 00:21:08.660
and manage their time to
balance a lot of obligations.

00:21:12.779 --> 00:21:14.570
You can't expect them
to say your course is

00:21:14.570 --> 00:21:16.403
the most important
course, and of course I'm

00:21:16.403 --> 00:21:18.300
going to do this before
I do anything else.

00:21:18.300 --> 00:21:19.850
But at least if they
get to the stage

00:21:19.850 --> 00:21:21.600
where they open the
document, because they

00:21:21.600 --> 00:21:23.620
know they're going to get
graded, 5% of the grade,

00:21:23.620 --> 00:21:24.800
on answering these
reading questions,

00:21:24.800 --> 00:21:26.740
you have a good chance
that they're going

00:21:26.740 --> 00:21:28.259
to have some first exposure.

00:21:28.259 --> 00:21:30.800
And if you create an expectation
at the beginning of the term

00:21:30.800 --> 00:21:33.175
that we're going to assume
that they've done the reading,

00:21:33.175 --> 00:21:35.640
and not go over it as if
they didn't, then that also

00:21:35.640 --> 00:21:38.570
helps to enforce and get
students to actually do it.

00:21:38.570 --> 00:21:41.129
So they are actually
graded on these questions

00:21:41.129 --> 00:21:42.920
in terms of whether
they're right or wrong,

00:21:42.920 --> 00:21:45.070
and they're not meant to be
particularly hard questions.

00:21:45.070 --> 00:21:46.903
They're more like
questions that are exactly

00:21:46.903 --> 00:21:48.650
parallel to an example,
so at least they

00:21:48.650 --> 00:21:50.858
have to look through the
example, change the numbers,

00:21:50.858 --> 00:21:52.620
plug in an answer.

00:21:52.620 --> 00:21:55.087
Clickers and attendance.

00:21:55.087 --> 00:21:56.670
When you do an active
learning course,

00:21:56.670 --> 00:21:58.640
it's even more important
that students come to class.

00:21:58.640 --> 00:22:00.723
You can't watch an active
learning course on video

00:22:00.723 --> 00:22:02.250
like you can watch a lecture.

00:22:02.250 --> 00:22:06.030
And so in some sense, it's a way
to bring value back to campus

00:22:06.030 --> 00:22:09.100
in the face of MOOCs.

00:22:09.100 --> 00:22:10.639
Well, one way to
force attendance

00:22:10.639 --> 00:22:11.930
is to actually take attendance.

00:22:11.930 --> 00:22:13.940
But we can do that
implicitly with the clickers.

00:22:13.940 --> 00:22:15.773
So students know that
if they don't show up,

00:22:15.773 --> 00:22:17.648
they're not clicking in
to certain questions,

00:22:17.648 --> 00:22:19.106
they don't get any
points for that.

00:22:19.106 --> 00:22:20.180
That's 5% of their grade.

00:22:20.180 --> 00:22:22.580
Now we definitely have a
few students who are still

00:22:22.580 --> 00:22:26.570
not going to show up at all,
because they ace every exam,

00:22:26.570 --> 00:22:28.490
they learn it on
their own, great.

00:22:28.490 --> 00:22:30.501
They'll get 95%, they'll
get an A in the course.

00:22:30.501 --> 00:22:31.000
Right?

00:22:31.000 --> 00:22:32.247
And that's OK.

00:22:32.247 --> 00:22:33.830
It's not that
necessarily everyone has

00:22:33.830 --> 00:22:35.935
to learn the same
way, but it's really

00:22:35.935 --> 00:22:37.560
important for those
students who aren't

00:22:37.560 --> 00:22:39.660
able to do everything on their
own that they come to class.

00:22:39.660 --> 00:22:41.570
And we get a pretty
good attendance rate.

00:22:41.570 --> 00:22:43.870
And we think the clickers
help incentivize that.

00:22:43.870 --> 00:22:46.620
The pset checker,
Jerry mentioned,

00:22:46.620 --> 00:22:49.670
is a tool to get
students to know ahead

00:22:49.670 --> 00:22:53.470
of time whether or not they
got their answer correct, so

00:22:53.470 --> 00:22:56.427
that they still have time
to reflect and ask a friend

00:22:56.427 --> 00:22:58.510
or think about it some
more while they might still

00:22:58.510 --> 00:22:59.176
get some credit.

00:22:59.176 --> 00:23:00.310
So there's still incentive.

00:23:00.310 --> 00:23:01.870
As opposed to a
couple weeks later,

00:23:01.870 --> 00:23:04.453
when the problem set comes back,
they have three other problem

00:23:04.453 --> 00:23:06.574
sets, they have an
exam, it's not in RAM,

00:23:06.574 --> 00:23:08.490
might as well not give
it back to them at all.

00:23:08.490 --> 00:23:10.320
I can tell you how big
the stack of problem

00:23:10.320 --> 00:23:12.510
sets I have that
students-- I mean,

00:23:12.510 --> 00:23:14.510
every day, every week I
bring them to the class.

00:23:14.510 --> 00:23:15.770
I'm like, please, unburden me.

00:23:15.770 --> 00:23:17.550
Take back these problem sets.

00:23:17.550 --> 00:23:19.880
Most students don't
even want them back.

00:23:19.880 --> 00:23:21.820
It just tells you
how little they're

00:23:21.820 --> 00:23:23.850
benefiting from the grading
of the problem sets.

00:23:23.850 --> 00:23:26.330
And I think this tool
in particular transfers

00:23:26.330 --> 00:23:27.500
very broadly very easily.

00:23:27.500 --> 00:23:29.480
It doesn't have to be a math
class or a physics course.

00:23:29.480 --> 00:23:31.521
It just has to be a course
where there's some way

00:23:31.521 --> 00:23:34.030
to do some assessment,
or some way

00:23:34.030 --> 00:23:36.991
to evaluate if a student
has made progress.

00:23:36.991 --> 00:23:38.740
It might be there's a
formula and you just

00:23:38.740 --> 00:23:40.695
plug in some number and
that's how you know.

00:23:40.695 --> 00:23:42.570
Granted, they don't
learn what the answer is.

00:23:42.570 --> 00:23:45.780
They just get told that 3.58
is not the right answer.

00:23:45.780 --> 00:23:49.470
It's not very efficient for
them to then try 3.59, 3.57, you

00:23:49.470 --> 00:23:50.890
know, it won't help them.

00:23:50.890 --> 00:23:54.505
So they use it in a way that's
very satisfying if you've ever

00:23:54.505 --> 00:23:57.404
taken a MOOC, which is you
get those green checks when

00:23:57.404 --> 00:23:58.320
your answer's correct.

00:23:58.320 --> 00:23:58.980
It feels great.

00:23:58.980 --> 00:24:01.099
And so they know, great,
I figure these out.

00:24:01.099 --> 00:24:01.890
I got green checks.

00:24:01.890 --> 00:24:02.830
I'm turning this in.

00:24:02.830 --> 00:24:04.180
It's going to be correct.

00:24:04.180 --> 00:24:05.860
But they still get
graded on their logic

00:24:05.860 --> 00:24:07.130
and on their writing.

00:24:07.130 --> 00:24:08.754
That's the only part
of the problem set

00:24:08.754 --> 00:24:12.480
that actually counts
toward the grade.

00:24:12.480 --> 00:24:14.080
So I think those
are those points.

00:24:14.080 --> 00:24:14.871
PROFESSOR 1: Right.

00:24:14.871 --> 00:24:18.640
I can add that since it's
on MITx, we have analytics.

00:24:18.640 --> 00:24:21.790
They use the pset checker a lot.

00:24:21.790 --> 00:24:24.560
Most students are using it.

00:24:24.560 --> 00:24:26.550
I'd say well over 2/3.

00:24:26.550 --> 00:24:29.480
And they go through,
you can see how many

00:24:29.480 --> 00:24:33.450
attempts they're making
for the various questions.

00:24:33.450 --> 00:24:34.540
Next slide.

00:24:36.780 --> 00:24:37.280
Is it me?

00:24:37.280 --> 00:24:38.800
PROFESSOR 2: I think
it's both of us.

00:24:38.800 --> 00:24:39.030
PROFESSOR 1: Ah yeah.

00:24:39.030 --> 00:24:41.090
So we'll give our
own takes on this.

00:24:41.090 --> 00:24:44.980
So when you talk
about active learning,

00:24:44.980 --> 00:24:46.230
there's some common questions.

00:24:46.230 --> 00:24:48.390
So how much work was all this?

00:24:48.390 --> 00:24:52.310
The answer was an unbelievably
large amount of work for us.

00:24:52.310 --> 00:24:55.670
But part of that was
we changed everything.

00:24:55.670 --> 00:24:57.520
So it's hard to say
what it would be,

00:24:57.520 --> 00:25:01.250
had we taught this as a lecture
course for three or four years,

00:25:01.250 --> 00:25:04.570
had we changed our curriculum
and added the Bayesian

00:25:04.570 --> 00:25:09.040
unit before converting it
into an active learning class,

00:25:09.040 --> 00:25:13.849
had we not been so ambitious
in using the MITx platform,

00:25:13.849 --> 00:25:15.890
all of those things would
have lessened the work.

00:25:15.890 --> 00:25:19.100
But I think you could say
it's still a lot of work

00:25:19.100 --> 00:25:24.000
to convert from a lecture
to a active learning format.

00:25:24.000 --> 00:25:27.970
For one thing,
there's no faking what

00:25:27.970 --> 00:25:29.970
you're going to do in an
active learning format.

00:25:29.970 --> 00:25:33.330
We have slides, we have to
have questions, well thought

00:25:33.330 --> 00:25:34.710
out questions, ahead of time.

00:25:34.710 --> 00:25:39.020
You can't pull examples kind
of out of your hat the way

00:25:39.020 --> 00:25:41.930
I would never do, but I've
heard that some people do

00:25:41.930 --> 00:25:44.489
when they lecture.

00:25:44.489 --> 00:25:45.780
How much are you able to cover?

00:25:45.780 --> 00:25:48.227
This is also a typical question.

00:25:48.227 --> 00:25:49.310
How can you cover as much?

00:25:49.310 --> 00:25:52.970
The answers is we find we
can cover at least as much.

00:25:52.970 --> 00:25:56.140
I'll say a little
more about that.

00:25:56.140 --> 00:25:58.510
You're cutting out examples,
you spend a lot of time

00:25:58.510 --> 00:26:01.820
in a math class working
through examples.

00:26:01.820 --> 00:26:03.710
The students are
doing the examples

00:26:03.710 --> 00:26:07.820
so they're learning
the material better

00:26:07.820 --> 00:26:10.410
than you're doing the examples.

00:26:10.410 --> 00:26:12.420
And that time from
your doing the examples

00:26:12.420 --> 00:26:14.830
is saved for them to work.

00:26:14.830 --> 00:26:19.500
In terms of 18.05, it's
a little hard to measure.

00:26:19.500 --> 00:26:21.690
We know we've added three
weeks worth of material

00:26:21.690 --> 00:26:23.720
to the class we inherited.

00:26:23.720 --> 00:26:28.580
It's possible that the class we
inherited was underambitious.

00:26:28.580 --> 00:26:31.650
And we could've added
that in a lecture class.

00:26:31.650 --> 00:26:34.349
We know we cover the first
unit, the probability unit in--

00:26:34.349 --> 00:26:36.390
PROFESSOR 2: I think five
weeks instead of seven.

00:26:36.390 --> 00:26:38.490
PROFESSOR 1: Five
weeks instead of seven.

00:26:38.490 --> 00:26:42.430
And they do just as well
on comparable exams.

00:26:42.430 --> 00:26:45.092
So I think I feel safe
in saying that we cover

00:26:45.092 --> 00:26:46.175
at least as much material.

00:26:50.100 --> 00:26:52.240
You spend more time
in the classroom

00:26:52.240 --> 00:26:56.250
because, as the teachers, we're
in there for all 240 minutes.

00:26:56.250 --> 00:26:59.340
None of it is pawned off
on recitation leaders.

00:26:59.340 --> 00:27:01.800
It might be interesting
to know whether that's

00:27:01.800 --> 00:27:05.070
really necessary
for all the teachers

00:27:05.070 --> 00:27:06.990
to be there all the time.

00:27:06.990 --> 00:27:08.780
Do you have anything
to add to that, John?

00:27:08.780 --> 00:27:10.780
PROFESSOR 2: I think for us, one
thing that added a lot of work

00:27:10.780 --> 00:27:12.863
was that when we decided
to change the curriculum,

00:27:12.863 --> 00:27:14.720
we realized there
was no textbook that

00:27:14.720 --> 00:27:17.890
taught what we wanted to teach,
so we wrote the textbook.

00:27:17.890 --> 00:27:21.514
And that takes a lot of time.

00:27:21.514 --> 00:27:23.430
PROFESSOR 1: Yeah, I
said that was in November

00:27:23.430 --> 00:27:26.680
for a class that was starting
in January that we finally

00:27:26.680 --> 00:27:28.590
decided we had to throw
away the textbook.

00:27:28.590 --> 00:27:29.381
PROFESSOR 2: Right.

00:27:29.381 --> 00:27:31.550
So in the end, we
wrote these materials,

00:27:31.550 --> 00:27:33.650
but I can say that
the students really

00:27:33.650 --> 00:27:36.300
appreciated that the
reading they were doing

00:27:36.300 --> 00:27:37.282
was very targeted.

00:27:37.282 --> 00:27:38.990
It told them the
objectives, exactly what

00:27:38.990 --> 00:27:42.340
we expected of them, what kinds
of problems, at what level,

00:27:42.340 --> 00:27:44.460
and so it turned out to
have other benefits we

00:27:44.460 --> 00:27:47.140
didn't foresee going in.

00:27:47.140 --> 00:27:49.390
And then in terms of how
much we're able to cover,

00:27:49.390 --> 00:27:53.506
like Jerry said, we gave
exactly the same first midterm

00:27:53.506 --> 00:27:54.380
three years in a row.

00:27:54.380 --> 00:27:55.844
We're not going to give
it a fourth year in a row,

00:27:55.844 --> 00:27:55.960
so don't--

00:27:55.960 --> 00:27:56.626
PROFESSOR 1: No.

00:27:56.626 --> 00:27:57.660
We added one question.

00:27:57.660 --> 00:27:58.951
PROFESSOR 2: Plus one question.

00:27:58.951 --> 00:28:02.555
So because we gave
exam at seven weeks,

00:28:02.555 --> 00:28:03.930
the first time we
taught it, that

00:28:03.930 --> 00:28:05.201
was just the probability unit.

00:28:05.201 --> 00:28:07.450
The second and third time
we taught it-- so third time

00:28:07.450 --> 00:28:09.350
is right now-- that
was the probability

00:28:09.350 --> 00:28:13.540
unit plus like a week
or two of more stuff.

00:28:13.540 --> 00:28:14.040
Right?

00:28:14.040 --> 00:28:15.206
So it had one more question.

00:28:15.206 --> 00:28:17.000
Like six instead
of five questions.

00:28:17.000 --> 00:28:19.250
And if you remove the
one more question,

00:28:19.250 --> 00:28:23.036
they actually did as
well or slightly better

00:28:23.036 --> 00:28:24.160
compared to the first unit.

00:28:24.160 --> 00:28:26.326
If you include it, they did
it pretty much the same.

00:28:26.326 --> 00:28:29.150
But you have the same amount
of time to take the exam,

00:28:29.150 --> 00:28:31.530
so it's a pretty concrete
measure that they did better.

00:28:31.530 --> 00:28:33.767
Now we don't know that
that's because in part we

00:28:33.767 --> 00:28:35.600
actually understood a
little more statistics

00:28:35.600 --> 00:28:37.050
and could explain better
the second time around.

00:28:37.050 --> 00:28:38.430
PROFESSOR 1: No, this
is just probability.

00:28:38.430 --> 00:28:39.010
We understood probability.

00:28:39.010 --> 00:28:40.590
PROFESSOR 2: We did
understand probability.

00:28:40.590 --> 00:28:41.090
OK.

00:28:41.090 --> 00:28:43.267
That was good.

00:28:43.267 --> 00:28:45.100
PROFESSOR 1: We should
say, so neither of us

00:28:45.100 --> 00:28:47.100
are statisticians going in.

00:28:47.100 --> 00:28:49.030
We've learned a
lot of statistics,

00:28:49.030 --> 00:28:50.800
and particularly
how to teach it.

00:28:50.800 --> 00:28:53.120
I don't think, at a
mathematical level,

00:28:53.120 --> 00:28:57.640
the statistics we're
doing is terribly hard,

00:28:57.640 --> 00:28:59.930
but we all believe,
I think, as teachers

00:28:59.930 --> 00:29:01.330
that the more
expertise you have.

00:29:01.330 --> 00:29:03.770
So Sansa, who I
mentioned earlier,

00:29:03.770 --> 00:29:06.710
does know a lot of statistics
and has been consulting with us

00:29:06.710 --> 00:29:12.344
and been a tremendous asset in
getting us up to speed on this.

00:29:12.344 --> 00:29:14.312
AUDIENCE: Question.

00:29:14.312 --> 00:29:16.280
How did you know that
the students didn't

00:29:16.280 --> 00:29:18.248
cheat, for example,
and have access

00:29:18.248 --> 00:29:20.387
to the previous years' exams?

00:29:20.387 --> 00:29:22.220
PROFESSOR 1: You ask a
really good question,

00:29:22.220 --> 00:29:28.910
but I think you can infer that
from reading their papers.

00:29:28.910 --> 00:29:30.810
If you read the papers,
students are not

00:29:30.810 --> 00:29:33.870
getting everything right.

00:29:33.870 --> 00:29:36.480
PROFESSOR 2: By the way, we did
change the wording, the topic,

00:29:36.480 --> 00:29:37.164
and the numbers.

00:29:37.164 --> 00:29:39.830
I mean we didn't leave it so you
could just put the same answer,

00:29:39.830 --> 00:29:41.450
certainly.

00:29:41.450 --> 00:29:43.470
And doing it a
third year was kind

00:29:43.470 --> 00:29:46.484
of stretching the limits of
what you're exactly asking.

00:29:46.484 --> 00:29:47.900
PROFESSOR 1: We
won't do it again.

00:29:47.900 --> 00:29:50.850
I will say this, and I
often get into a discussion

00:29:50.850 --> 00:29:54.410
about cheating, which is I
could give-- I know from lecture

00:29:54.410 --> 00:29:56.990
classes and maybe Arthur
will verify this who's

00:29:56.990 --> 00:30:00.670
sitting there, Arthur Mattuck,
whose done a lot of lecturing--

00:30:00.670 --> 00:30:03.180
I've seen when I've teach
recitation that I can tell

00:30:03.180 --> 00:30:06.730
my students, the teacher has
given you practice questions.

00:30:06.730 --> 00:30:09.020
Three of them will
be on the exam.

00:30:09.020 --> 00:30:11.810
Two of the questions on the
exam will be just small changes

00:30:11.810 --> 00:30:15.500
of this, and I still know that
many students won't find time

00:30:15.500 --> 00:30:17.740
to do the practice exam.

00:30:17.740 --> 00:30:20.020
I think it's kind of a myth.

00:30:20.020 --> 00:30:23.530
If they found my exam in
the copier-- well actually,

00:30:23.530 --> 00:30:27.990
most of them would return it
to me, but someone might cheat.

00:30:27.990 --> 00:30:34.020
But the effort of going to
try to do something like this

00:30:34.020 --> 00:30:37.450
without being certain that
you're going to be getting

00:30:37.450 --> 00:30:39.590
something that's going
to be on the exam,

00:30:39.590 --> 00:30:41.950
it just doesn't
happen that often.

00:30:41.950 --> 00:30:45.860
So I think now if it
comes out that we'd

00:30:45.860 --> 00:30:48.150
give the same exam
year in and year out,

00:30:48.150 --> 00:30:49.590
people will look at it.

00:30:49.590 --> 00:30:52.369
So we won't do that again.

00:30:52.369 --> 00:30:53.160
Other observations.

00:30:56.980 --> 00:30:59.390
This is kind of the
psychological observations.

00:30:59.390 --> 00:31:02.060
Active learning is a lot
more fun for the students.

00:31:02.060 --> 00:31:04.872
You could see even in that
video when they stand up,

00:31:04.872 --> 00:31:06.580
there's a lot of energy
in the classroom.

00:31:06.580 --> 00:31:08.200
You could hear the buzz.

00:31:08.200 --> 00:31:10.480
People are not sleeping.

00:31:10.480 --> 00:31:13.560
Co-teaching, if you can
do it, is tremendous fun.

00:31:13.560 --> 00:31:17.210
I can't get away with anything
with John standing there.

00:31:17.210 --> 00:31:19.200
I'm a little more
generous to him.

00:31:19.200 --> 00:31:20.270
I let him make mistakes.

00:31:22.986 --> 00:31:24.860
Students like getting
to know their teachers.

00:31:24.860 --> 00:31:29.370
This is a really
important value at MIT.

00:31:29.370 --> 00:31:32.010
A number of students
graduate from MIT

00:31:32.010 --> 00:31:34.710
without knowing enough
teachers well enough

00:31:34.710 --> 00:31:37.654
to get three letters
of recommendation.

00:31:37.654 --> 00:31:39.570
Here, we're walking
around, were meeting them,

00:31:39.570 --> 00:31:43.330
both of us like students,
and like talking to students.

00:31:43.330 --> 00:31:45.840
And they appreciate it,
they do get to know us,

00:31:45.840 --> 00:31:48.780
we have to write a lot more
letters of recommendation

00:31:48.780 --> 00:31:50.400
than we ever did before.

00:31:50.400 --> 00:31:53.670
There's a downside
to everything.

00:31:53.670 --> 00:31:56.090
The students like
the targeted reading.

00:31:56.090 --> 00:31:59.250
In fact, Glenda Stump, who is
with the teaching and learning

00:31:59.250 --> 00:32:03.320
laboratory here and has
done surveys of our students

00:32:03.320 --> 00:32:05.920
and interviews with our
students, the report-- and this

00:32:05.920 --> 00:32:08.715
was interesting to us-- they
like the reading more than they

00:32:08.715 --> 00:32:10.840
like watching videos.

00:32:10.840 --> 00:32:11.860
Videos take longer.

00:32:11.860 --> 00:32:15.100
The targeted reading is
just what they need to know.

00:32:15.100 --> 00:32:20.660
We're telling them, this is
what we expect you to learn,

00:32:20.660 --> 00:32:23.865
and it doesn't take as
much time as a video.

00:32:23.865 --> 00:32:25.990
And finally, students really
love the pset checker.

00:32:25.990 --> 00:32:27.790
It gives them a very
comfortable feeling.

00:32:32.300 --> 00:32:34.510
So finally, looking
forward, Glenda

00:32:34.510 --> 00:32:38.010
is continuing to study
what we're doing.

00:32:38.010 --> 00:32:41.440
At some point, there
will be a paper report

00:32:41.440 --> 00:32:43.140
that comes out of this.

00:32:43.140 --> 00:32:46.820
The course will be on
OpenCourseWare and the OCW

00:32:46.820 --> 00:32:50.320
Educator, where we'll give--
it's being videotaped here,

00:32:50.320 --> 00:32:55.310
so this talk might show up
on OpenCourseWare Educator,

00:32:55.310 --> 00:32:57.660
to tell other teachers
about what we're doing,

00:32:57.660 --> 00:33:00.810
as well as present
the course materials.

00:33:00.810 --> 00:33:03.970
And finally, and
this is important,

00:33:03.970 --> 00:33:07.040
we've overstaffed the
class in the sense of what

00:33:07.040 --> 00:33:08.840
the budgets will really allow.

00:33:08.840 --> 00:33:11.950
We've had a Davis foundation
grant that helps with it.

00:33:11.950 --> 00:33:14.030
The math department
has supported it.

00:33:14.030 --> 00:33:16.070
We have to move to a more
traditional staffing.

00:33:16.070 --> 00:33:20.810
So actually next year, I'll
teach the class by myself,

00:33:20.810 --> 00:33:22.276
without John's help.

00:33:22.276 --> 00:33:23.150
We'll see what we do.

00:33:23.150 --> 00:33:25.066
I'll have to learn how
do these sort of demos,

00:33:25.066 --> 00:33:28.610
because we leave that to John.

00:33:28.610 --> 00:33:33.270
And with just undergraduates
and maybe a graduate student

00:33:33.270 --> 00:33:34.211
in the class.

00:33:34.211 --> 00:33:35.710
This year John
didn't say it, but we

00:33:35.710 --> 00:33:37.980
have actually another
instructor who's

00:33:37.980 --> 00:33:40.290
a statistician in
the class with us--

00:33:40.290 --> 00:33:42.640
Peter Kempthorne-- at all times.

00:33:42.640 --> 00:33:45.400
And then, equally important
for a project like this

00:33:45.400 --> 00:33:48.550
is making a transition
to the next teacher.

00:33:48.550 --> 00:33:50.930
We're going to have to
convince someone else

00:33:50.930 --> 00:33:53.660
that they want to teach like
this, and they want to do it.

00:33:53.660 --> 00:33:56.230
Part of our strategy will
be, after three years,

00:33:56.230 --> 00:33:58.925
to have such a compelling
set of materials

00:33:58.925 --> 00:34:01.570
that is so easy for
them to use, that it

00:34:01.570 --> 00:34:04.200
won't feel like as much work.

00:34:04.200 --> 00:34:07.710
So that's what we're looking
forward to in the future.

00:34:07.710 --> 00:34:09.500
Finally, come visit.

00:34:09.500 --> 00:34:11.620
We have had visitors.

00:34:11.620 --> 00:34:13.300
We like it when
people come to visit.

00:34:13.300 --> 00:34:15.380
If you send us a
note, we'll tell you

00:34:15.380 --> 00:34:16.810
what's a good day to visit.

00:34:16.810 --> 00:34:19.260
If not, you can take your
chances and just show up.

00:34:19.260 --> 00:34:21.710
We're in the basement
of the Stata Center

00:34:21.710 --> 00:34:23.880
in the TEAL room, which
we should have mentioned.

00:34:23.880 --> 00:34:27.480
This is in part an outgrowth
of the physics TEAL project.

00:34:27.480 --> 00:34:28.730
PROFESSOR 2: It's a huge room.

00:34:28.730 --> 00:34:30.540
There's plenty of space.

00:34:30.540 --> 00:34:32.460
So you don't have to
worry it'll be full.

00:34:32.460 --> 00:34:35.810
PROFESSOR 1: You saw in
the video it's a big room.

00:34:35.810 --> 00:34:38.880
Stata, Tuesday, Thursday
and Friday at 1:00 to 2:30.

00:34:38.880 --> 00:34:40.170
Come for all or part of it.

00:34:40.170 --> 00:34:41.909
PROFESSOR 2: And I should say
that Tuesday and Thursday are

00:34:41.909 --> 00:34:44.370
really the days, our sort
of standard flipped days.

00:34:44.370 --> 00:34:46.964
And then Fridays are when
we do that studio with R,

00:34:46.964 --> 00:34:48.130
and they have the computers.

00:34:48.130 --> 00:34:49.132
So it's a bit different.

00:34:49.132 --> 00:34:50.840
And of course some
weeks they have exams.

00:34:50.840 --> 00:34:52.715
So you might not want
to come on an exam day.

00:34:52.715 --> 00:34:54.730
That won't be as interesting.

00:34:54.730 --> 00:34:57.229
So you might just let us know
when you're going to show up.

00:34:57.229 --> 00:34:59.190
AUDIENCE: What's the
size of your [INAUDIBLE]?

00:34:59.190 --> 00:35:01.170
PROFESSOR 1: We have
about 60 students.

00:35:01.170 --> 00:35:04.010
Right now the room could
comfortably hold, I think,

00:35:04.010 --> 00:35:06.120
90 people.

00:35:06.120 --> 00:35:08.840
So the class actually
had about 31,

00:35:08.840 --> 00:35:11.030
32 students two years
ago when we took it over

00:35:11.030 --> 00:35:12.350
as a lecture course.

00:35:12.350 --> 00:35:15.172
Last year that bumped
up to the low 40s.

00:35:15.172 --> 00:35:16.630
The first time
there was a flipped,

00:35:16.630 --> 00:35:19.970
and now we have 60, or
maybe 58 the second time

00:35:19.970 --> 00:35:24.145
through, so we're hoping
that we can keep that going.

00:35:29.520 --> 00:35:34.144
AUDIENCE: Do you find that it's
more work to teach the class--

00:35:34.144 --> 00:35:34.810
PROFESSOR 1: No.

00:35:34.810 --> 00:35:38.330
I find it's less work
once you're in the class.

00:35:38.330 --> 00:35:43.425
If you have the slides
prepared and-- you

00:35:43.425 --> 00:35:45.050
spend a lot of time
walking around just

00:35:45.050 --> 00:35:47.920
schmoozing with students.

00:35:47.920 --> 00:35:50.040
And working-- no, it's not.

00:35:50.040 --> 00:35:52.850
It's easier, it's
more energizing,

00:35:52.850 --> 00:35:55.370
I'm never tired when
we're done, I never

00:35:55.370 --> 00:35:58.660
feel that I've just
droned on for 40 minutes

00:35:58.660 --> 00:36:01.940
and not modulated
my voice at all.

00:36:01.940 --> 00:36:04.580
So it's not more work to teach.

00:36:04.580 --> 00:36:07.340
It's more work to
prepare, I should say.

00:36:07.340 --> 00:36:09.470
You have to be fully prepared.

00:36:09.470 --> 00:36:11.330
There's no thinking
about, yeah what

00:36:11.330 --> 00:36:14.070
is this topic on the
way to the lecture,

00:36:14.070 --> 00:36:15.660
and giving the
example, and being

00:36:15.660 --> 00:36:18.420
confident that you'll be able
to work it out the next line

00:36:18.420 --> 00:36:21.546
before you get there.

00:36:21.546 --> 00:36:23.129
AUDIENCE: You might
have heard that we

00:36:23.129 --> 00:36:27.650
don't have the bodies to
staff like this, knowing

00:36:27.650 --> 00:36:28.650
how much money we had.

00:36:28.650 --> 00:36:31.150
So I guess I'm
wondering, how does

00:36:31.150 --> 00:36:34.151
the need for staffing scale with
enrollment, if you 90 students

00:36:34.151 --> 00:36:34.650
show up?

00:36:34.650 --> 00:36:36.108
PROFESSOR 1: The
physics department

00:36:36.108 --> 00:36:38.600
has been doing this
for a lot of years.

00:36:38.600 --> 00:36:40.810
They have one
instructor in the room

00:36:40.810 --> 00:36:42.710
and typically a lot
of graduate students

00:36:42.710 --> 00:36:45.480
who would be teaching
a recitation otherwise.

00:36:45.480 --> 00:36:48.900
So you can staff it in that
way, with one instructor

00:36:48.900 --> 00:36:50.400
and graduate students.

00:36:50.400 --> 00:36:52.410
Undergraduates,
good undergraduates,

00:36:52.410 --> 00:36:53.880
are also very good.

00:36:53.880 --> 00:36:56.000
It's a little harder
to find students

00:36:56.000 --> 00:36:59.160
who are free Tuesday,
Thursday and Friday.

00:36:59.160 --> 00:37:02.740
I think maybe next year, because
Friday is just the studio,

00:37:02.740 --> 00:37:05.180
and there's less of a need,
they do less on Friday.

00:37:05.180 --> 00:37:07.801
If I get students just
on Tuesday and Thursday,

00:37:07.801 --> 00:37:08.300
we'll do it.

00:37:08.300 --> 00:37:10.880
But there's a lot of
very good undergraduates

00:37:10.880 --> 00:37:11.945
who are quite helpful.

00:37:11.945 --> 00:37:14.715
AUDIENCE: [INAUDIBLE] a lot
of the grading is automatic.

00:37:14.715 --> 00:37:15.700
Is that the only
thing [INAUDIBLE]?

00:37:15.700 --> 00:37:15.940
PROFESSOR 1: No.

00:37:15.940 --> 00:37:16.120
The good--

00:37:16.120 --> 00:37:16.340
PROFESSOR 2: No.

00:37:16.340 --> 00:37:17.760
It's not automatic at all.

00:37:17.760 --> 00:37:20.650
PROFESSOR 1: The psets are
graded by graders on paper.

00:37:20.650 --> 00:37:21.030
PROFESSOR 2: Undergraduates.

00:37:21.030 --> 00:37:22.488
PROFESSOR 1:
Undergraduate graders.

00:37:22.488 --> 00:37:24.870
That's the same as if we
taught it as a lecture.

00:37:24.870 --> 00:37:26.220
You have to hire them.

00:37:26.220 --> 00:37:28.140
It's easier to find
graders because they

00:37:28.140 --> 00:37:30.750
don't commit to being
in a certain place

00:37:30.750 --> 00:37:31.530
at a certain time.

00:37:34.248 --> 00:37:36.060
AUDIENCE: I asked
you this at lunch.

00:37:36.060 --> 00:37:39.570
Did you get feedback
from students

00:37:39.570 --> 00:37:44.932
that help you refine it or make
you think about the things that

00:37:44.932 --> 00:37:47.390
maybe weren't so good and things
that were especially good?

00:37:47.390 --> 00:37:49.090
PROFESSOR 1: Yes.

00:37:49.090 --> 00:37:53.070
So the first year we did
the studio in MATLAB,

00:37:53.070 --> 00:37:54.287
and it's not MATLAB's fault.

00:37:54.287 --> 00:37:55.870
PROFESSOR 2: It
wasn't MATLAB's fault.

00:37:55.870 --> 00:37:57.510
PROFESSOR 1: It's not MATLAB's
fault. It was our fault.

00:37:57.510 --> 00:37:58.980
It was really tedious.

00:37:58.980 --> 00:38:03.447
We did a lot of tutorials
on using the software,

00:38:03.447 --> 00:38:05.030
and so it was click
here, click there.

00:38:05.030 --> 00:38:06.821
You have students at
very different levels.

00:38:06.821 --> 00:38:08.550
MIT students are very good.

00:38:08.550 --> 00:38:11.740
If you show them a
program, a script--

00:38:11.740 --> 00:38:14.710
you don't write complicated
scripts with a lot of logic.

00:38:14.710 --> 00:38:16.470
It's typically just
a series of commands

00:38:16.470 --> 00:38:18.640
that you can get out
of this-- they're

00:38:18.640 --> 00:38:20.500
able to see what's
going on and use it.

00:38:20.500 --> 00:38:23.430
So it's based on
feedback from them.

00:38:23.430 --> 00:38:25.060
Part of it is just
being in the room

00:38:25.060 --> 00:38:27.790
and realizing how dull it was.

00:38:27.790 --> 00:38:30.570
But also from
talking to students.

00:38:30.570 --> 00:38:33.620
We changed that.

00:38:33.620 --> 00:38:34.420
What else?

00:38:34.420 --> 00:38:36.359
We did get feedback
on others but I'm

00:38:36.359 --> 00:38:37.400
forgetting what they say.

00:38:37.400 --> 00:38:38.140
PROFESSOR 2: But we
got feedback fact

00:38:38.140 --> 00:38:40.810
that it's good to have the
full week's materials posted

00:38:40.810 --> 00:38:42.440
before the week
starts, which was very

00:38:42.440 --> 00:38:44.910
challenging the first year,
because we were often writing

00:38:44.910 --> 00:38:48.120
it like at that exact moment.

00:38:48.120 --> 00:38:51.280
This year we've corrected that.

00:38:51.280 --> 00:38:54.900
As far as using the software, I
think exactly what Jerry said,

00:38:54.900 --> 00:38:56.990
plus I'll add that
we had to think,

00:38:56.990 --> 00:38:58.790
we thought more of
what our goals were,

00:38:58.790 --> 00:39:00.130
with them interacting
with the software.

00:39:00.130 --> 00:39:01.588
And really it was
less about making

00:39:01.588 --> 00:39:03.562
them experts in
using MATLAB or R,

00:39:03.562 --> 00:39:05.520
and more about leveraging
the software as a way

00:39:05.520 --> 00:39:07.680
to increase their
understanding of the material.

00:39:07.680 --> 00:39:09.140
And from that
point of view, it's

00:39:09.140 --> 00:39:10.973
not quite as necessary
to have them build up

00:39:10.973 --> 00:39:13.510
complicated things from
scratch, but rather

00:39:13.510 --> 00:39:15.150
to be able to
understand something

00:39:15.150 --> 00:39:17.180
we've written well
enough that they

00:39:17.180 --> 00:39:20.620
can change components, play
with it, visualize things,

00:39:20.620 --> 00:39:22.460
and so we've really
moved in that direction.

00:39:22.460 --> 00:39:24.440
And also, another piece
of feedback we got

00:39:24.440 --> 00:39:25.970
was that students
really appreciate

00:39:25.970 --> 00:39:27.940
real-world applications,
especially lot of them

00:39:27.940 --> 00:39:30.330
are pre-med, they want to
see medical research papers,

00:39:30.330 --> 00:39:33.300
they want to see data
that speaks to them.

00:39:33.300 --> 00:39:36.080
And so Peter
Kempthorne, for example,

00:39:36.080 --> 00:39:39.230
has a background in
the financial world,

00:39:39.230 --> 00:39:42.770
so he's helped us create some
studios centered around stock

00:39:42.770 --> 00:39:45.250
prices moving around.

00:39:45.250 --> 00:39:49.240
The one we created last week was
a studio where, basically, they

00:39:49.240 --> 00:39:51.050
used a two-dimensional
Bayesian updating

00:39:51.050 --> 00:39:54.130
to simulate looking
for Malaysia flight 370

00:39:54.130 --> 00:39:56.420
under the Indian Ocean
by putting a sensor grid.

00:39:56.420 --> 00:39:58.211
PROFESSOR 1: And it
was found the next day.

00:39:58.211 --> 00:39:59.315
Or they got that ping.

00:39:59.315 --> 00:40:00.590
PROFESSOR 2: That's right.

00:40:00.590 --> 00:40:02.930
Right?

00:40:02.930 --> 00:40:06.080
They're not going to know how
to make this sort of heat map

00:40:06.080 --> 00:40:10.480
representation or a contour plot
representation of the posterior

00:40:10.480 --> 00:40:13.840
of updating based on what
sensors detected, and so on,

00:40:13.840 --> 00:40:15.896
but that's OK.

00:40:15.896 --> 00:40:17.520
We can instead give
them the framework,

00:40:17.520 --> 00:40:18.220
they can play with it.

00:40:18.220 --> 00:40:20.050
We can say, well, what
happens if instead

00:40:20.050 --> 00:40:22.660
of the black box being right
in the middle of this grid

00:40:22.660 --> 00:40:23.727
of sensors, it's up here.

00:40:23.727 --> 00:40:25.560
And then they realize
after playing with it,

00:40:25.560 --> 00:40:27.226
oh, actually it takes
longer to converge

00:40:27.226 --> 00:40:29.795
and at first, the probability
sort of spread out this way,

00:40:29.795 --> 00:40:31.170
and why is that,
oh, because it's

00:40:31.170 --> 00:40:33.710
about the same
distance left to right.

00:40:33.710 --> 00:40:37.344
Those things were coming
naturally to them.

00:40:37.344 --> 00:40:38.760
Didn't mean they
had to understand

00:40:38.760 --> 00:40:40.410
how to do this
complicated syntax,

00:40:40.410 --> 00:40:43.043
but it was a benefit of getting
these visuals right away,

00:40:43.043 --> 00:40:44.435
being able to play around.

00:40:44.435 --> 00:40:45.930
AUDIENCE: And
relevant to what was

00:40:45.930 --> 00:40:47.820
of interest to them because
their field or something--

00:40:47.820 --> 00:40:49.980
PROFESSOR 2: Well in this case,
it was relevant to the New York

00:40:49.980 --> 00:40:51.170
Times report the
night before, which

00:40:51.170 --> 00:40:53.290
said that they were deploying
sensors behind ships.

00:40:53.290 --> 00:40:54.870
So we talked about
how would you modify

00:40:54.870 --> 00:40:57.411
this to make it more realistic
based on what they're actually

00:40:57.411 --> 00:40:58.826
doing, which isn't so hard.

00:40:58.826 --> 00:41:00.200
And they got the
idea, hopefully,

00:41:00.200 --> 00:41:01.814
that this stuff is useful.

00:41:01.814 --> 00:41:03.230
PROFESSOR 1: So
these are details.

00:41:03.230 --> 00:41:05.690
I think part of the point is
we're always getting feedback

00:41:05.690 --> 00:41:07.800
from the students,
because that's

00:41:07.800 --> 00:41:10.120
part of what happens in
a classroom, the kind

00:41:10.120 --> 00:41:12.980
of formative assessment
John mentioned.

00:41:12.980 --> 00:41:15.050
One of the things we've
done, and it differs

00:41:15.050 --> 00:41:17.250
from what TEAL does,
is-- a question

00:41:17.250 --> 00:41:21.010
is, how long do you let
them work on a problem?

00:41:21.010 --> 00:41:23.740
And through their feedback,
through talking to them,

00:41:23.740 --> 00:41:28.070
and through observation, we've
kind of converged on how long

00:41:28.070 --> 00:41:29.127
you go with a problem.

00:41:29.127 --> 00:41:30.960
And I think one of the
differences with TEAL

00:41:30.960 --> 00:41:35.090
is we give them a longer
time to solve problems

00:41:35.090 --> 00:41:36.935
than they do in TEAL,
and they do express

00:41:36.935 --> 00:41:39.683
an appreciation for that.

00:41:39.683 --> 00:41:41.567
How did you learn
to work effectively

00:41:41.567 --> 00:41:42.980
in the TEAL classroom?

00:41:42.980 --> 00:41:44.570
Did you observe others?

00:41:44.570 --> 00:41:45.820
Did you have special training?

00:41:45.820 --> 00:41:47.040
PROFESSOR 1: No.

00:41:47.040 --> 00:41:50.420
We went to Peter Dourmashkin's
class, we talked with Peter.

00:41:53.350 --> 00:41:56.310
The day before our first
class, we went into the room

00:41:56.310 --> 00:41:59.560
and we rehearsed it
down to the minute.

00:41:59.560 --> 00:42:03.100
And then we just sort of
tried it over and over again.

00:42:03.100 --> 00:42:04.890
We were lucky the
first day-- partly

00:42:04.890 --> 00:42:07.500
I think because of the
rehearsal-- was so successful.

00:42:07.500 --> 00:42:09.360
Haynes, you probably
remember how high

00:42:09.360 --> 00:42:11.680
we were flying after that day.

00:42:11.680 --> 00:42:13.057
And so--

00:42:13.057 --> 00:42:14.640
PROFESSOR 2: The
level of preparation,

00:42:14.640 --> 00:42:16.620
quickly deteriorated after.

00:42:16.620 --> 00:42:17.690
PROFESSOR 1: No, no, no.

00:42:17.690 --> 00:42:19.106
We don't do it
down to the minute.

00:42:19.106 --> 00:42:22.090
We're more confident
in 10-minute intervals,

00:42:22.090 --> 00:42:23.675
but we're well prepared.

00:42:23.675 --> 00:42:26.050
PROFESSOR 2: The room is, I
mean there's a little control

00:42:26.050 --> 00:42:27.920
panel you just need to learn
how to make the screens go

00:42:27.920 --> 00:42:29.961
up and down, or make a
video camera go on or off.

00:42:29.961 --> 00:42:31.580
Make the lights go up or down.

00:42:31.580 --> 00:42:34.110
It's not that
that's challenging,

00:42:34.110 --> 00:42:36.700
but it definitely took
a lot of trial and error

00:42:36.700 --> 00:42:40.480
to optimize what's the best
way to use those technologies,

00:42:40.480 --> 00:42:43.831
and that was experimentation,
we got better as we went.

00:42:43.831 --> 00:42:46.080
PROFESSOR 1: I don't think
the technology was as much.

00:42:46.080 --> 00:42:47.955
But things like how long
do you do a problem.

00:42:47.955 --> 00:42:49.957
In the first year, I'd
be off in the corner

00:42:49.957 --> 00:42:52.040
talking with students and
John would come and say,

00:42:52.040 --> 00:42:54.040
you know, it's been a long time.

00:42:54.040 --> 00:42:56.110
It's time to stop.

00:42:56.110 --> 00:42:58.830
And I think maybe it happened
in reverse on occasion.

00:42:58.830 --> 00:43:00.710
PROFESSOR 2: It's a
really common problem

00:43:00.710 --> 00:43:03.200
with active learning is
that you do have groups that

00:43:03.200 --> 00:43:04.420
will go at different speeds.

00:43:04.420 --> 00:43:06.130
One advantage with
having enough staffing

00:43:06.130 --> 00:43:07.921
is that you can try to
target those groups,

00:43:07.921 --> 00:43:09.962
get people to about
the same point.

00:43:09.962 --> 00:43:11.920
There are always going
to be winners and losers

00:43:11.920 --> 00:43:13.900
in any approach you take
to teaching a class.

00:43:13.900 --> 00:43:15.566
We're not claiming
that what we're doing

00:43:15.566 --> 00:43:18.240
is better for everyone.

00:43:18.240 --> 00:43:26.620
But just this feedback you get
as you're doing it, dynamically

00:43:26.620 --> 00:43:29.520
in the hour, it really helps.

00:43:29.520 --> 00:43:31.975
It really helps you, so
know how to reach them.

00:43:31.975 --> 00:43:33.725
AUDIENCE: Did you have
a schedule of we're

00:43:33.725 --> 00:43:35.915
going to do this for 10 minutes,
we're going to do this for 15.

00:43:35.915 --> 00:43:37.300
Did you have [INAUDIBLE]?

00:43:37.300 --> 00:43:39.380
PROFESSOR 1: We have it
broken down by slides.

00:43:39.380 --> 00:43:42.040
And so the slides do it.

00:43:42.040 --> 00:43:44.160
PROFESSOR 2: We have
a slide presentation

00:43:44.160 --> 00:43:46.480
that is prepared ahead, so
it does give you an order.

00:43:46.480 --> 00:43:47.770
PROFESSOR 1: And
at this point, you

00:43:47.770 --> 00:43:49.103
learn when you prepare lectures.

00:43:49.103 --> 00:43:53.380
At this point we know that
16 slides we can get through,

00:43:53.380 --> 00:43:55.280
20 slides we're not
going to get through.

00:43:55.280 --> 00:43:56.220
PROFESSOR 2: We
should also say that--

00:43:56.220 --> 00:43:58.070
PROFESSOR 1: We fall still
into the mistake of thinking,

00:43:58.070 --> 00:43:58.800
oh, we could do it.

00:43:58.800 --> 00:43:59.425
We could do it.

00:43:59.425 --> 00:44:00.256
But you never do.

00:44:00.256 --> 00:44:02.630
PROFESSOR 2: One thing we also
do that students, I think,

00:44:02.630 --> 00:44:05.282
appreciate is that you're still
not going to get every student

00:44:05.282 --> 00:44:07.490
to understand the problem
they just did, necessarily,

00:44:07.490 --> 00:44:08.910
in the confines
of the class time.

00:44:08.910 --> 00:44:10.950
And there might be times
where it doesn't make sense

00:44:10.950 --> 00:44:12.300
to go through every
detail of a problem,

00:44:12.300 --> 00:44:14.425
because it wouldn't be that
efficient, because most

00:44:14.425 --> 00:44:15.610
students basically got it.

00:44:15.610 --> 00:44:18.520
And so the version of the
slides that we post right before

00:44:18.520 --> 00:44:20.764
and use in the class is
different from the updated

00:44:20.764 --> 00:44:22.180
version we post
after class, which

00:44:22.180 --> 00:44:25.917
includes complete careful
solutions to every problem

00:44:25.917 --> 00:44:27.500
that we did in class,
so students know

00:44:27.500 --> 00:44:30.352
that that's there for them.

00:44:30.352 --> 00:44:32.060
They might feel like
they're missing out.

00:44:32.060 --> 00:44:34.630
They want to see an expert
solve the problem the right way.

00:44:34.630 --> 00:44:36.700
Why should they have to come
up with that way completely

00:44:36.700 --> 00:44:37.200
themselves.

00:44:37.200 --> 00:44:40.340
And so we capture some of
that by putting that in there.

00:44:40.340 --> 00:44:42.670
PROFESSOR 1: But you should
say, we usually do solve it

00:44:42.670 --> 00:44:43.630
for them afterwards.

00:44:43.630 --> 00:44:48.295
Most problems we give them the
expert, nice, succinct view.

00:44:48.295 --> 00:44:50.170
And I think that's an
important part of this.

00:44:50.170 --> 00:44:51.586
Students want to
feel that they're

00:44:51.586 --> 00:44:54.150
getting some of your expertise.

00:44:54.150 --> 00:44:55.900
AUDIENCE: You mentioned
students preferred

00:44:55.900 --> 00:44:59.900
the targeted questions
reading to online lectures.

00:44:59.900 --> 00:45:02.436
I was wondering
what students felt

00:45:02.436 --> 00:45:04.410
was so good about
the targeted reading

00:45:04.410 --> 00:45:06.535
or what it was that was so
bad the online lectures.

00:45:06.535 --> 00:45:08.076
PROFESSOR 1: I don't
think they'd say

00:45:08.076 --> 00:45:09.390
the online lectures are bad.

00:45:09.390 --> 00:45:13.270
Online lectures take longer
to watch if you try to MOOC.

00:45:13.270 --> 00:45:15.710
And how many people in
this room have tried.

00:45:15.710 --> 00:45:17.140
The videos are long.

00:45:17.140 --> 00:45:18.800
I mean you always
are looking to try

00:45:18.800 --> 00:45:22.670
to speed it up, find the two
times speed up, et cetera.

00:45:22.670 --> 00:45:23.910
They can take a long time.

00:45:23.910 --> 00:45:26.118
The targeted-- the fact that
the reading is targeted,

00:45:26.118 --> 00:45:27.520
the fact that we
wrote it, and it

00:45:27.520 --> 00:45:31.250
covers what we
expect them to learn,

00:45:31.250 --> 00:45:33.850
is, I think, the
main thing they like.

00:45:33.850 --> 00:45:36.950
There's clear signals in there,
we think this is important.

00:45:36.950 --> 00:45:41.610
Not we said read chapter five
and you have to figure out,

00:45:41.610 --> 00:45:44.575
by listening to us, which parts
of chapter five are important.

00:45:44.575 --> 00:45:45.950
PROFESSOR 2: And
I mean those are

00:45:45.950 --> 00:45:49.270
skills that as you go higher
up in your education at MIT,

00:45:49.270 --> 00:45:51.020
you might be able to
do more for yourself,

00:45:51.020 --> 00:45:52.436
and you should
learn those skills.

00:45:52.436 --> 00:45:54.770
But I think the
level of this course,

00:45:54.770 --> 00:45:56.400
I mean many of these
students, they're

00:45:56.400 --> 00:45:59.022
strong mathematically by
national standards of course,

00:45:59.022 --> 00:46:01.230
but maybe within MIT, if
they're life science majors,

00:46:01.230 --> 00:46:03.355
they're not as confident
as some of the math majors

00:46:03.355 --> 00:46:04.540
in physics majors.

00:46:04.540 --> 00:46:07.410
It helps them to have
a little more guidance

00:46:07.410 --> 00:46:10.937
in what the expectations are.

00:46:10.937 --> 00:46:12.478
What do you do for
small groups where

00:46:12.478 --> 00:46:14.798
there's a group
that's gotten done,

00:46:14.798 --> 00:46:16.364
and everyone else
is still working?

00:46:16.364 --> 00:46:18.372
Do you have extra stuff
for their enrichment?

00:46:18.372 --> 00:46:20.830
PROFESSOR 1: You can walk around
and try to challenge them.

00:46:20.830 --> 00:46:22.762
Most groups it's
rare that there's

00:46:22.762 --> 00:46:24.470
people who have
understood-- particularly

00:46:24.470 --> 00:46:27.040
say the first time we
work through an idea--

00:46:27.040 --> 00:46:28.724
have understood it completely.

00:46:28.724 --> 00:46:29.890
So there's always questions.

00:46:29.890 --> 00:46:31.681
You could say, have
you thought about this,

00:46:31.681 --> 00:46:33.825
or what if you did that?

00:46:33.825 --> 00:46:35.325
AUDIENCE: How do
you form the groups

00:46:35.325 --> 00:46:38.730
and do you change them during
the course of the semester?

00:46:38.730 --> 00:46:39.520
PROFESSOR 1: No.

00:46:39.520 --> 00:46:41.550
We let them form themselves.

00:46:41.550 --> 00:46:44.230
I think there's a debate in
the education community--

00:46:44.230 --> 00:46:47.180
maybe Jenny can speak to this--
about whether it's better

00:46:47.180 --> 00:46:51.360
to form the groups,
for you to form them

00:46:51.360 --> 00:46:57.480
by some sort of heterogeneous
ability, or to let them form.

00:46:57.480 --> 00:47:00.210
We feel that, first
of all, we don't

00:47:00.210 --> 00:47:02.080
know their ability coming in.

00:47:02.080 --> 00:47:04.940
Second, people like autonomy.

00:47:04.940 --> 00:47:08.050
And so allowing them to form
groups makes them comfortable.

00:47:08.050 --> 00:47:11.100
There are students who
are resistant to this idea

00:47:11.100 --> 00:47:12.960
of active learning coming in.

00:47:12.960 --> 00:47:15.360
We win them over
quickly, and part of it

00:47:15.360 --> 00:47:18.066
is that we're generous,
say, in that sense.

00:47:18.066 --> 00:47:19.940
AUDIENCE: What do you
if you see in one group

00:47:19.940 --> 00:47:22.304
there's a dominant
member or there are

00:47:22.304 --> 00:47:23.470
people who are sitting back?

00:47:23.470 --> 00:47:25.000
PROFESSOR 1: You know it
doesn't happen that much.

00:47:25.000 --> 00:47:27.090
Groups of three, they
tend to work together.

00:47:27.090 --> 00:47:30.600
These are largely juniors
and seniors at MIT.

00:47:30.600 --> 00:47:31.820
A few sophomores.

00:47:31.820 --> 00:47:36.810
They have become better
at being students

00:47:36.810 --> 00:47:39.070
and making sure they're
getting what they want.

00:47:39.070 --> 00:47:40.570
There are some
groups where you come

00:47:40.570 --> 00:47:42.110
and you try to draw people in.

00:47:42.110 --> 00:47:44.470
It tends to be the groups
that are overall weaker, where

00:47:44.470 --> 00:47:46.350
you see people hanging back.

00:47:46.350 --> 00:47:48.780
But those are exactly the
groups where you go to first,

00:47:48.780 --> 00:47:52.610
and you spend more
time with them.

00:47:52.610 --> 00:47:55.070
AUDIENCE: You mentioned
the flipped component

00:47:55.070 --> 00:47:56.054
of the course.

00:47:56.054 --> 00:47:58.268
So, can you give a
little bit more idea

00:47:58.268 --> 00:48:01.958
about your expectations
time-wise and material-wise

00:48:01.958 --> 00:48:03.434
and you coordinate it?

00:48:03.434 --> 00:48:05.894
I guess that they have these
readings, maybe 10 pages,

00:48:05.894 --> 00:48:06.878
for every class?

00:48:06.878 --> 00:48:09.197
Or week?

00:48:09.197 --> 00:48:10.280
PROFESSOR 1: Twice a week.

00:48:10.280 --> 00:48:11.480
Tuesday and Thursday.

00:48:11.480 --> 00:48:13.300
So prior to Tuesday
and Thursday,

00:48:13.300 --> 00:48:16.030
they need to do
reading, about 10 pages.

00:48:16.030 --> 00:48:17.814
It's a very fast 10 pages.

00:48:17.814 --> 00:48:18.980
We use a lot of white space.

00:48:18.980 --> 00:48:19.960
AUDIENCE: What are
the problem sets?

00:48:19.960 --> 00:48:21.400
Is there one problem
set per week?

00:48:21.400 --> 00:48:22.900
And also one more
follow-up question

00:48:22.900 --> 00:48:26.820
on that, what's the
turn-around for the assessment

00:48:26.820 --> 00:48:32.700
and feedback for those psets
so that they could hopefully

00:48:32.700 --> 00:48:34.170
learn for the next one?

00:48:34.170 --> 00:48:36.120
PROFESSOR 1: They have
one pset per week,

00:48:36.120 --> 00:48:39.440
except in weeks where we
have an exam and vacations.

00:48:39.440 --> 00:48:43.844
So usually, let's say, one per
week, with some variations.

00:48:43.844 --> 00:48:46.260
PROFESSOR 2: So the psets, the
way we structure it, right?

00:48:46.260 --> 00:48:47.950
They're learning say
some new material

00:48:47.950 --> 00:48:50.120
Tuesday, Thursday,
studio on Friday.

00:48:50.120 --> 00:48:52.150
And the pset we have due Monday.

00:48:52.150 --> 00:48:54.570
And the idea is that by
turning it on Monday,

00:48:54.570 --> 00:48:57.580
they still then,
Monday night, can

00:48:57.580 --> 00:48:59.967
prepare for Tuesday's class.

00:48:59.967 --> 00:49:01.550
We try to get the
graders to turn back

00:49:01.550 --> 00:49:04.370
the pset by the end of
the week, by Friday.

00:49:04.370 --> 00:49:07.150
I'm not convinced that
getting that problem set back

00:49:07.150 --> 00:49:09.760
is actually helping that
many students in the end.

00:49:09.760 --> 00:49:13.064
Like I said, half the students
don't even pick them up.

00:49:13.064 --> 00:49:15.480
PROFESSOR 1: But it is nice
for the students that want it.

00:49:15.480 --> 00:49:17.024
So it's been good.

00:49:17.024 --> 00:49:18.440
The graders get
it back by Friday.

00:49:18.440 --> 00:49:20.648
PROFESSOR 2: Oh yeah, the
graders, they've done that.

00:49:20.648 --> 00:49:24.557
AUDIENCE: How many questions do
you expect them to work through

00:49:24.557 --> 00:49:25.554
in that pset.

00:49:25.554 --> 00:49:26.220
PROFESSOR 1: OK.

00:49:26.220 --> 00:49:29.620
So the piece set is just
the challenging part

00:49:29.620 --> 00:49:33.150
of a problem set we would ask
in a regular lecture class.

00:49:33.150 --> 00:49:34.970
They've done more of
the drill problems

00:49:34.970 --> 00:49:37.740
in class, and somewhat
on the reading questions.

00:49:37.740 --> 00:49:40.400
So five or six
questions, which is

00:49:40.400 --> 00:49:43.790
about what we would have on
what we call the part two

00:49:43.790 --> 00:49:48.260
part of a regular lecture
class, what they would

00:49:48.260 --> 00:49:53.860
see in 18.01 or 18.02 or 18.03.

00:49:53.860 --> 00:49:55.920
AUDIENCE: About the grading
of the problem sets.

00:49:55.920 --> 00:49:58.682
One thing that a lot of
education research shows

00:49:58.682 --> 00:50:02.670
is that if there's a delay
between the act of doing

00:50:02.670 --> 00:50:04.132
the problem and
the feedback, they

00:50:04.132 --> 00:50:06.340
don't pay any attention,
which is what you're saying.

00:50:06.340 --> 00:50:09.328
What if you force them to
actually think about it

00:50:09.328 --> 00:50:12.316
again by grading the
problem set themselves.

00:50:12.316 --> 00:50:14.557
So you could say, everyone
upload their problem

00:50:14.557 --> 00:50:17.349
set, and then you get
somebody else's problem set,

00:50:17.349 --> 00:50:19.390
and you have to then grade
it with your solution.

00:50:19.390 --> 00:50:21.100
PROFESSOR 1: I think
it's a great idea.

00:50:21.100 --> 00:50:23.400
One of my questions about
it would be two-fold.

00:50:23.400 --> 00:50:26.010
The students are not going to
like it, especially when you

00:50:26.010 --> 00:50:26.870
start them doing it.

00:50:26.870 --> 00:50:29.757
And is it the best
use of their time?

00:50:29.757 --> 00:50:31.631
I don't know what the
research shows on that.

00:50:31.631 --> 00:50:32.553
AUDIENCE: We're
writing [INAUDIBLE]

00:50:32.553 --> 00:50:34.397
the solution set is probably
worth their time reading it.

00:50:34.397 --> 00:50:34.897
Otherwise--

00:50:34.897 --> 00:50:35.813
PROFESSOR 1: Oh I see.

00:50:35.813 --> 00:50:38.223
If they have to grade they
have to read our solutions.

00:50:38.223 --> 00:50:39.764
AUDIENCE: And really
understand them,

00:50:39.764 --> 00:50:44.228
and also learn what counts as a
good argument and what doesn't.

00:50:44.228 --> 00:50:45.696
[INAUDIBLE]

00:50:45.696 --> 00:50:47.070
PROFESSOR 1: I
can tell you this.

00:50:47.070 --> 00:50:50.180
[? Analea ?] [? Bronte ?], who
I teach with in the experimental

00:50:50.180 --> 00:50:52.670
study group, who
teaches physics there,

00:50:52.670 --> 00:50:54.750
she did try to do
this at some point,

00:50:54.750 --> 00:50:57.700
and she did learn that students
become good at grading it.

00:50:57.700 --> 00:50:59.330
You don't have to
worry about that.

00:50:59.330 --> 00:51:00.770
Over time with
your feedback, they

00:51:00.770 --> 00:51:03.937
learn to grade pretty close
to what you would grade.

00:51:03.937 --> 00:51:04.520
The questions.

00:51:04.520 --> 00:51:07.710
She would give them a rubric
but she no longer does it.

00:51:07.710 --> 00:51:11.117
There's a lot of
resistance to it.

00:51:11.117 --> 00:51:12.200
AUDIENCE: By the students.

00:51:12.200 --> 00:51:13.300
PROFESSOR 1: By the students.

00:51:13.300 --> 00:51:15.508
PROFESSOR 2: You would hope
that a little bit of that

00:51:15.508 --> 00:51:16.770
occurs before exams.

00:51:16.770 --> 00:51:17.810
But I'm not sure.

00:51:20.804 --> 00:51:23.199
AUDIENCE: Are you
collecting data more

00:51:23.199 --> 00:51:27.040
than you would in
a regular class?

00:51:27.040 --> 00:51:27.600
[INAUDIBLE]

00:51:27.600 --> 00:51:30.830
PROFESSOR 2: Do you mean like
the sort of data coming in

00:51:30.830 --> 00:51:32.302
from the students.

00:51:32.302 --> 00:51:33.241
AUDIENCE: [INAUDIBLE].

00:51:33.241 --> 00:51:34.990
PROFESSOR 1: Here's
the data that we have.

00:51:34.990 --> 00:51:38.790
There's something called the
statistics concept inventory,

00:51:38.790 --> 00:51:42.170
and we had them last
year do a pre-test.

00:51:42.170 --> 00:51:44.700
And we have them do a post-test,
but we got to it too late,

00:51:44.700 --> 00:51:46.587
so we couldn't
assign it to them.

00:51:46.587 --> 00:51:48.170
It was in the last
week of class where

00:51:48.170 --> 00:51:49.920
you can't assign things at MIT.

00:51:49.920 --> 00:51:51.750
So we didn't get a lot
of compliance here.

00:51:51.750 --> 00:51:54.660
This year they did the
pre-test, and the post-test

00:51:54.660 --> 00:51:56.930
will be an assignment
for them to do.

00:51:56.930 --> 00:52:00.330
So we'll be able to measure
how well they improved

00:52:00.330 --> 00:52:01.754
on the concept inventory.

00:52:01.754 --> 00:52:03.420
It's not a perfect
match for what we do.

00:52:03.420 --> 00:52:06.290
It has no Bayesian
component to it.

00:52:06.290 --> 00:52:08.460
But we'll get a measure there.

00:52:08.460 --> 00:52:11.150
The other is what
Glenda has done

00:52:11.150 --> 00:52:13.420
with surveys and interviews.

00:52:13.420 --> 00:52:17.210
And finally there are
analytics on MITx,

00:52:17.210 --> 00:52:22.120
so we can see how long they've
spent at least with the PDFs

00:52:22.120 --> 00:52:26.480
open, how many tries they
took at various questions.

00:52:26.480 --> 00:52:30.597
We haven't tried to analyze
that data yet, but we could.

00:52:30.597 --> 00:52:31.180
MODERATOR: OK.

00:52:31.180 --> 00:52:36.310
I think so we won't have people
starting to get up and leave.

00:52:36.310 --> 00:52:37.090
Thank you guys.

00:52:37.090 --> 00:52:40.170
I think it strikes me as
it's one of those really

00:52:40.170 --> 00:52:41.850
in the trenches talks.

00:52:41.850 --> 00:52:44.800
You're actually doing it
and you can hear the width

00:52:44.800 --> 00:52:46.390
from the questions this time.

00:52:46.390 --> 00:52:49.120
We sort of moved from a sort
of theoretical construct,

00:52:49.120 --> 00:52:50.940
that you could do
it this way, what's

00:52:50.940 --> 00:52:52.278
actually happening around.

00:52:52.278 --> 00:52:54.100
A lot of people are
thinking about it

00:52:54.100 --> 00:52:57.170
and just, thank you
so much for coming.