1
00:00:00,000 --> 00:00:02,430
The following content is
provided under a Creative

2
00:00:02,430 --> 00:00:03,730
Commons license.

3
00:00:03,730 --> 00:00:06,030
Your support will help
MIT OpenCourseWare

4
00:00:06,030 --> 00:00:10,060
continue to offer high quality
educational resources for free.

5
00:00:10,060 --> 00:00:12,690
To make a donation or to
view additional materials

6
00:00:12,690 --> 00:00:16,560
from hundreds of MIT courses,
visit MIT OpenCourseWare

7
00:00:16,560 --> 00:00:17,892
at ocw.mit.edu.

8
00:00:21,030 --> 00:00:24,540
PROFESSOR: Probably the
time has begun to begin.

9
00:00:24,540 --> 00:00:27,240
Welcome.

10
00:00:27,240 --> 00:00:29,160
I've given a guest
lecture in this class

11
00:00:29,160 --> 00:00:30,450
for a few years running now.

12
00:00:30,450 --> 00:00:34,620
So I'm always happy to return.

13
00:00:34,620 --> 00:00:37,560
You're working on a
good subject here.

14
00:00:37,560 --> 00:00:40,050
I'm going to talk today
about experimentation

15
00:00:40,050 --> 00:00:43,380
and robust design of
engineering systems.

16
00:00:43,380 --> 00:00:46,560
I think you've already had
a few lectures on design

17
00:00:46,560 --> 00:00:48,120
of experiments formally.

18
00:00:48,120 --> 00:00:51,470
So this will build on that.

19
00:00:51,470 --> 00:00:55,570
The figure that I show
on this first slide

20
00:00:55,570 --> 00:01:00,310
concerns, on the right,
a method by which

21
00:01:00,310 --> 00:01:03,550
one can make improvements
in robustness of the system

22
00:01:03,550 --> 00:01:06,850
by adaptively experimenting
in the parameter

23
00:01:06,850 --> 00:01:08,440
space of that system.

24
00:01:08,440 --> 00:01:11,230
And the figure on
the left is meant

25
00:01:11,230 --> 00:01:14,350
to indicate why
that approach has

26
00:01:14,350 --> 00:01:19,420
worked rather quite a lot
better than people expected.

27
00:01:19,420 --> 00:01:22,270
And I'll be going into
that in some detail,

28
00:01:22,270 --> 00:01:24,130
and I hope that as I
go through all this,

29
00:01:24,130 --> 00:01:26,428
you'll take time
to ask questions.

30
00:01:26,428 --> 00:01:27,970
Because I think
that's better for you

31
00:01:27,970 --> 00:01:30,790
and more interesting for me too.

32
00:01:30,790 --> 00:01:35,950
So just to give you a
little bit of context,

33
00:01:35,950 --> 00:01:38,470
this topic of adaptive
experimentation

34
00:01:38,470 --> 00:01:41,090
and robust design is really
central to my research.

35
00:01:41,090 --> 00:01:45,340
It's I think the core of it, but
I'm interested in other things

36
00:01:45,340 --> 00:01:45,980
as well.

37
00:01:45,980 --> 00:01:50,440
Such as we do studies
of systems generally

38
00:01:50,440 --> 00:01:54,550
and the regularities that exist
in those engineered systems.

39
00:01:54,550 --> 00:01:59,800
And that gives rise to data
that we can use more generally

40
00:01:59,800 --> 00:02:02,450
in methodology validation.

41
00:02:02,450 --> 00:02:05,080
So we can look at
different ways that people

42
00:02:05,080 --> 00:02:10,030
will do engineering and try
to assess its effectiveness.

43
00:02:10,030 --> 00:02:11,530
And of course,
all this is linked

44
00:02:11,530 --> 00:02:14,830
in with the work on
adaptive experimentation,

45
00:02:14,830 --> 00:02:16,870
because we need a way
to validate our work.

46
00:02:19,870 --> 00:02:21,840
But the main topic
today will be what's

47
00:02:21,840 --> 00:02:23,220
in the middle of that slide.

48
00:02:23,220 --> 00:02:24,640
That's adaptive experimentation.

49
00:02:24,640 --> 00:02:29,910
I think I'll start with
some history and use that

50
00:02:29,910 --> 00:02:31,660
to explain the motivation.

51
00:02:31,660 --> 00:02:33,810
Why is it that we're
addressing this topic?

52
00:02:33,810 --> 00:02:37,095
Why are we addressing it
the way we're addressing it?

53
00:02:37,095 --> 00:02:39,220
And then, I'm going to show
you the research itself

54
00:02:39,220 --> 00:02:43,230
on a depth of experimentation
and robust design.

55
00:02:43,230 --> 00:02:49,140
So I start with this quote
from Sir Edward John Russell.

56
00:02:49,140 --> 00:02:52,060
Now, this fellow was
prominent in his day.

57
00:02:52,060 --> 00:02:52,980
He was knighted.

58
00:02:52,980 --> 00:02:55,810
That's why it says Sir
Edward John Russell there

59
00:02:55,810 --> 00:02:57,270
under his picture.

60
00:02:57,270 --> 00:03:01,650
And as late as
1926, he was still

61
00:03:01,650 --> 00:03:05,730
expressing a view about
how to do experimentation.

62
00:03:05,730 --> 00:03:10,230
He said that you should
seek simplicity in the way

63
00:03:10,230 --> 00:03:13,400
that you conduct
your experiments,

64
00:03:13,400 --> 00:03:16,080
and maybe you should ask just
a few questions at a time,

65
00:03:16,080 --> 00:03:18,500
or maybe even just one
question at a time,

66
00:03:18,500 --> 00:03:21,380
as you're conducting
your experiments.

67
00:03:21,380 --> 00:03:24,950
And this was a prominent view
in that they generally accepted,

68
00:03:24,950 --> 00:03:27,530
and you can trace it
back at least as far

69
00:03:27,530 --> 00:03:30,710
as Francis Bacon
and maybe father.

70
00:03:30,710 --> 00:03:32,750
And what's interesting
about this is just

71
00:03:32,750 --> 00:03:36,200
at the time he's still
expressing his views about how

72
00:03:36,200 --> 00:03:39,260
it is that they conducted
experiments there

73
00:03:39,260 --> 00:03:46,170
at Rothamsted, he hired
another person, R.A. Fisher.

74
00:03:46,170 --> 00:03:49,310
Now, I should say that this
is an interesting confluence.

75
00:03:49,310 --> 00:03:53,525
You have two men, very
prominent, very successful,

76
00:03:53,525 --> 00:03:55,650
coming together in the same
place at the same time.

77
00:03:55,650 --> 00:03:58,790
I think the reason for that was
that this experimental station,

78
00:03:58,790 --> 00:04:02,450
Rothamsted, was
addressing I think

79
00:04:02,450 --> 00:04:04,980
one of the great
problems of their time

80
00:04:04,980 --> 00:04:07,850
which was consistency
and efficiency

81
00:04:07,850 --> 00:04:10,240
of agricultural production.

82
00:04:10,240 --> 00:04:13,840
If you think about the
middle of the 19th century,

83
00:04:13,840 --> 00:04:15,550
the UK and other
parts of the world

84
00:04:15,550 --> 00:04:18,670
experienced terrible famines.

85
00:04:18,670 --> 00:04:23,020
And so trying to do
much better in this area

86
00:04:23,020 --> 00:04:26,680
was a pressing problem, probably
equivalent today to the way

87
00:04:26,680 --> 00:04:32,720
we feel about energy
supply or global warming.

88
00:04:32,720 --> 00:04:35,247
And now in some ways, we
take for granted food supply

89
00:04:35,247 --> 00:04:36,580
at least in the developed world.

90
00:04:39,150 --> 00:04:42,110
So here, we have this
confluence and R.A. Fisher

91
00:04:42,110 --> 00:04:45,590
comes into the lab, and he has
a very different view about how

92
00:04:45,590 --> 00:04:48,640
to do experimentation.

93
00:04:48,640 --> 00:04:53,980
He analyzes this experiment,
in which we see the table

94
00:04:53,980 --> 00:04:54,880
three in the middle.

95
00:04:54,880 --> 00:04:59,290
This is from R.A.
Fisher's 1921 paper,

96
00:04:59,290 --> 00:05:03,190
and what you see is a plot with
different levels of ammonia

97
00:05:03,190 --> 00:05:06,160
being applied as
fertilizer to the fields.

98
00:05:06,160 --> 00:05:11,740
And you see the influence on
the yield per acre of oats

99
00:05:11,740 --> 00:05:15,130
and the decrement in each year.

100
00:05:15,130 --> 00:05:17,320
Because when you
apply the fertilizer

101
00:05:17,320 --> 00:05:20,200
to the same field
year after year,

102
00:05:20,200 --> 00:05:22,690
its effects diminish over time.

103
00:05:22,690 --> 00:05:26,470
And apparently, Fisher
was fairly discouraged

104
00:05:26,470 --> 00:05:27,580
by this experiment.

105
00:05:27,580 --> 00:05:30,970
When you read the paper,
you see the details,

106
00:05:30,970 --> 00:05:33,580
and you see the difficulties
they experienced

107
00:05:33,580 --> 00:05:35,080
in analyzing the data.

108
00:05:35,080 --> 00:05:40,360
They had some pests intruding
on parts of the land,

109
00:05:40,360 --> 00:05:45,700
and this made it complex to draw
inferences from the experiment.

110
00:05:45,700 --> 00:05:51,210
And he was later quoted to
say the following thing which

111
00:05:51,210 --> 00:05:54,150
expressed that
frustration in a way.

112
00:05:54,150 --> 00:05:58,390
You have a bunch of scientists
plan an experiment based

113
00:05:58,390 --> 00:06:00,250
on their knowledge
of what to do.

114
00:06:00,250 --> 00:06:04,300
Later in, you call statisticians
in to analyze the data

115
00:06:04,300 --> 00:06:06,820
and to get the most
precise inferences you can

116
00:06:06,820 --> 00:06:13,120
and precise estimates,
and it's really too late.

117
00:06:13,120 --> 00:06:14,950
The time for
statistical thinking

118
00:06:14,950 --> 00:06:18,050
is before you actually
run the experiment.

119
00:06:18,050 --> 00:06:19,970
That's when you can get
the greatest benefits,

120
00:06:19,970 --> 00:06:24,530
and what he talked about is the
idea that the most finesse you

121
00:06:24,530 --> 00:06:28,520
could apply might improve
your analysis of the data

122
00:06:28,520 --> 00:06:30,932
by a few percent
one way or other.

123
00:06:30,932 --> 00:06:33,140
But if you brought in the
experimenter earlier, maybe

124
00:06:33,140 --> 00:06:39,750
you'd get a factor of 2 or 5
in efficiency and accuracy.

125
00:06:39,750 --> 00:06:42,390
And you can see how
thoughtful Fisher was.

126
00:06:42,390 --> 00:06:46,945
He seems to be struggling
hard to think there,

127
00:06:46,945 --> 00:06:48,820
and I think part of it
is that machine that's

128
00:06:48,820 --> 00:06:51,670
in front of him, which
is I guess what you'd

129
00:06:51,670 --> 00:06:54,170
use for a computer in that day.

130
00:06:54,170 --> 00:06:56,850
I guess it's some
mechanical adding device,

131
00:06:56,850 --> 00:07:00,830
and I think it's nice that we
have so much better machines

132
00:07:00,830 --> 00:07:02,210
now for this purpose.

133
00:07:02,210 --> 00:07:04,100
And in fact, what
we find is that you

134
00:07:04,100 --> 00:07:07,190
can do a lot with statistics
these days by knowing how

135
00:07:07,190 --> 00:07:08,420
to work with computers right.

136
00:07:08,420 --> 00:07:09,837
And in fact, a lot
of the research

137
00:07:09,837 --> 00:07:14,160
results I'll show you
today, we wouldn't

138
00:07:14,160 --> 00:07:16,830
be able to achieve
them with without

139
00:07:16,830 --> 00:07:21,410
modern digital computers.

140
00:07:21,410 --> 00:07:25,440
So Fisher begins to plan
experiments in a different way,

141
00:07:25,440 --> 00:07:28,610
and this is one of the first
ones in the literature,

142
00:07:28,610 --> 00:07:31,880
this figure one
from his 1926 paper

143
00:07:31,880 --> 00:07:34,080
on the arrangement
of field experiments.

144
00:07:34,080 --> 00:07:36,813
And if you've done
some work already

145
00:07:36,813 --> 00:07:38,480
in this course on
design of experiments,

146
00:07:38,480 --> 00:07:41,440
you might recognize this.

147
00:07:41,440 --> 00:07:47,630
You see somewhat
indicated by darker lines

148
00:07:47,630 --> 00:07:52,540
that there are eight different
subplots in this plot.

149
00:07:52,540 --> 00:07:54,890
And if I focus my
attention on just

150
00:07:54,890 --> 00:07:59,790
say the upper left,
what I see is--

151
00:07:59,790 --> 00:08:02,220
and I think the people even
remotely can see the arrow.

152
00:08:02,220 --> 00:08:03,450
Right?

153
00:08:03,450 --> 00:08:09,270
You see the plot has three
different indications

154
00:08:09,270 --> 00:08:10,110
in each plot.

155
00:08:10,110 --> 00:08:12,480
In this case, it's
2, an M, and early.

156
00:08:12,480 --> 00:08:14,830
In the next one, it's
2 and S and late.

157
00:08:14,830 --> 00:08:19,530
So the 2 and the 1
indicate an amount

158
00:08:19,530 --> 00:08:22,500
of a fertilizer
that's been applied.

159
00:08:22,500 --> 00:08:26,650
S and M indicate the
type of the fertilizer,

160
00:08:26,650 --> 00:08:31,240
and early and late
describe the time

161
00:08:31,240 --> 00:08:35,179
in the season, the planting
season, when it's applied.

162
00:08:35,179 --> 00:08:37,640
And we see in this
first block, we

163
00:08:37,640 --> 00:08:44,210
have four of the 2's and four of
the 1's and four of the earlys

164
00:08:44,210 --> 00:08:45,890
and four of the lates and so on.

165
00:08:45,890 --> 00:08:48,320
And we even see that
there is a balance,

166
00:08:48,320 --> 00:08:52,010
in the sense that there are
two of the two M's and two

167
00:08:52,010 --> 00:08:56,090
of the two S's, so pair-wise,
there's a balance and so on.

168
00:08:56,090 --> 00:08:58,130
And you may recognize
that, if I just

169
00:08:58,130 --> 00:09:00,740
focus on 2 and M
and early and late,

170
00:09:00,740 --> 00:09:04,130
and there are 8 of
such blocks in here,

171
00:09:04,130 --> 00:09:07,010
we have every permutation
and combination

172
00:09:07,010 --> 00:09:09,520
of the factors in the level.

173
00:09:09,520 --> 00:09:13,550
So I would say, you could
describe this as a 2

174
00:09:13,550 --> 00:09:16,250
to the 3 full
factorial experiment

175
00:09:16,250 --> 00:09:19,390
within the block, which is 8.

176
00:09:19,390 --> 00:09:27,520
Now, you might also say that
it's a 3 raised to the 1 and 2

177
00:09:27,520 --> 00:09:29,620
raised to the 2,
because there are also

178
00:09:29,620 --> 00:09:33,160
these four plots in which
there are just X's, and

179
00:09:33,160 --> 00:09:35,890
no fertilizers apply.

180
00:09:35,890 --> 00:09:38,110
You could call that
a control, or you

181
00:09:38,110 --> 00:09:46,420
could say that this is
actually a more complex

182
00:09:46,420 --> 00:09:50,560
factorial experiment,
where in one of the levels

183
00:09:50,560 --> 00:09:52,060
obviates the other two.

184
00:09:52,060 --> 00:09:54,370
And once you decide to
apply no fertilizer,

185
00:09:54,370 --> 00:09:57,070
it doesn't matter whether
you apply type 1 or type 2

186
00:09:57,070 --> 00:09:59,540
or apply early or late.

187
00:09:59,540 --> 00:10:06,950
And then, to further complicate
the issue, we have eight plots,

188
00:10:06,950 --> 00:10:10,820
and within those
blocks, those subplots,

189
00:10:10,820 --> 00:10:19,830
we have randomization in space
of the different treatments.

190
00:10:19,830 --> 00:10:23,850
And one of the things that
Fisher talks about here

191
00:10:23,850 --> 00:10:27,320
is how poorly randomization
actually works sometimes.

192
00:10:27,320 --> 00:10:30,900
So if I look down in this
plot, it's on the lower right.

193
00:10:30,900 --> 00:10:35,730
This plot is randomized
in spatial distribution

194
00:10:35,730 --> 00:10:39,660
of the treatments, but you see
that all of the left-hand plots

195
00:10:39,660 --> 00:10:44,520
are early, and all of the
right-hand plots are late.

196
00:10:44,520 --> 00:10:49,790
So sometimes, randomization
gives you such problems,

197
00:10:49,790 --> 00:10:52,100
and in fact, these
days, often people

198
00:10:52,100 --> 00:10:54,950
will not use randomization
and instead use

199
00:10:54,950 --> 00:10:58,370
something like a
Latin hypercube design

200
00:10:58,370 --> 00:11:02,100
rather than randomization.

201
00:11:02,100 --> 00:11:05,400
But this was one of the
early experiments that

202
00:11:05,400 --> 00:11:08,610
had been considered
in a factorial design,

203
00:11:08,610 --> 00:11:12,480
and I think it's interesting
that Fischer decided to call it

204
00:11:12,480 --> 00:11:14,160
a complex experiment.

205
00:11:14,160 --> 00:11:17,830
It's as if he was
trying to emphasize

206
00:11:17,830 --> 00:11:20,050
as strongly as possible
his difference of opinion

207
00:11:20,050 --> 00:11:24,580
with Russell, his boss, who
had said that you should seek

208
00:11:24,580 --> 00:11:26,650
simplicity in your experiments.

209
00:11:26,650 --> 00:11:29,480
And he was saying, no, I'm
going to do the exact opposite.

210
00:11:29,480 --> 00:11:32,040
I'm going to make my
experiment as complex as I can.

211
00:11:35,030 --> 00:11:38,270
And here's the motivation
for doing that,

212
00:11:38,270 --> 00:11:42,290
and this I think is one of
the great results of Fisher,

213
00:11:42,290 --> 00:11:47,150
is that he was able to
demonstrate the specific value

214
00:11:47,150 --> 00:11:54,610
that you could attain by
making the experiment complex.

215
00:11:54,610 --> 00:11:57,340
And what we have here on
the upper right of the slide

216
00:11:57,340 --> 00:12:00,490
is a cuboidal
representation of the 2

217
00:12:00,490 --> 00:12:01,750
to the 3 factorial design.

218
00:12:01,750 --> 00:12:05,530
This may not be the first time
you've seen it in this course,

219
00:12:05,530 --> 00:12:09,370
but you have the three different
levels labeled as A, B, and C,

220
00:12:09,370 --> 00:12:12,030
and the factors
labeled as A, B, and C

221
00:12:12,030 --> 00:12:15,190
and the levels indicate
as pluses and minuses.

222
00:12:15,190 --> 00:12:20,830
And the way that you would
estimate the effect of A

223
00:12:20,830 --> 00:12:25,580
is by looking at the
observations, where

224
00:12:25,580 --> 00:12:29,960
A is at level plus, taking
the average of those four.

225
00:12:29,960 --> 00:12:33,740
And then taking all of
the observations where

226
00:12:33,740 --> 00:12:36,560
A is at minus, taking
the average of those four

227
00:12:36,560 --> 00:12:41,260
and looking at that difference,
and that's the equation here.

228
00:12:41,260 --> 00:12:45,190
The effect if A is defined
as this equation which

229
00:12:45,190 --> 00:12:49,510
just implements what I
just described in words.

230
00:12:49,510 --> 00:12:51,580
And then what Fisher
is able to show

231
00:12:51,580 --> 00:12:55,840
is that the standard deviation
of the effect estimate

232
00:12:55,840 --> 00:13:00,140
has the following form.

233
00:13:00,140 --> 00:13:04,060
It is a function of
the experimental error,

234
00:13:04,060 --> 00:13:12,440
if I model the random variations
in the observation due to,

235
00:13:12,440 --> 00:13:16,030
let's say, soil pH
or the pests that I

236
00:13:16,030 --> 00:13:20,920
spoke of earlier or uneven
distribution of water

237
00:13:20,920 --> 00:13:21,940
on the fields.

238
00:13:21,940 --> 00:13:24,670
If all of those nuisance
factors that I didn't control

239
00:13:24,670 --> 00:13:29,140
influence my yields at
the various locations,

240
00:13:29,140 --> 00:13:33,980
and those contribute
to sigma sub-epsilon,

241
00:13:33,980 --> 00:13:36,470
those will be
reflected in sigma sub

242
00:13:36,470 --> 00:13:39,500
A, my estimate of
the influence of,

243
00:13:39,500 --> 00:13:43,780
say, applying the
fertilizer early or late.

244
00:13:43,780 --> 00:13:48,250
But they are reflected in
sigma sub A substantially

245
00:13:48,250 --> 00:13:51,670
less if I use this
factorial design rather than

246
00:13:51,670 --> 00:13:53,350
a single-factor experiment.

247
00:13:53,350 --> 00:13:57,220
And he could show that for
the case of the 2 to the 3

248
00:13:57,220 --> 00:14:03,390
full factorial, the improvement
in efficiency is a factor of 2.

249
00:14:03,390 --> 00:14:07,050
So it's a large benefit as
compared to the benefits

250
00:14:07,050 --> 00:14:11,430
that you could derive by a
different kind of analysis

251
00:14:11,430 --> 00:14:11,970
of the data.

252
00:14:11,970 --> 00:14:14,336
The planning of the data
made a huge difference.

253
00:14:19,940 --> 00:14:22,280
But the problem
is that, if you're

254
00:14:22,280 --> 00:14:24,140
using a full
factorial experiment,

255
00:14:24,140 --> 00:14:28,070
you see 2 to the
3 or 2 to the k,

256
00:14:28,070 --> 00:14:32,750
the scaling with the number
of factors is terrible.

257
00:14:32,750 --> 00:14:33,740
Right?

258
00:14:33,740 --> 00:14:36,860
And so you find that you can't
run such experiments with more

259
00:14:36,860 --> 00:14:38,250
than, say, seven factors.

260
00:14:38,250 --> 00:14:42,280
That's about the largest that
I've seen published in tables,

261
00:14:42,280 --> 00:14:44,380
so that we can analyze
them, because then you

262
00:14:44,380 --> 00:14:47,500
have 128 data
elements to include

263
00:14:47,500 --> 00:14:49,250
in the appendix of your paper.

264
00:14:49,250 --> 00:14:52,360
So they're rarely run with
more than seven factors.

265
00:14:52,360 --> 00:14:56,410
And yet in an experimentation,
especially in engineering,

266
00:14:56,410 --> 00:14:58,060
we often have many
more than seven

267
00:14:58,060 --> 00:14:59,920
factors we're concerned with.

268
00:14:59,920 --> 00:15:01,730
It's not at all uncommon.

269
00:15:01,730 --> 00:15:03,550
And so almost
immediately, Fisher

270
00:15:03,550 --> 00:15:09,280
begins to consider the
possibility of some means

271
00:15:09,280 --> 00:15:10,930
to reduce the size
of the experiment,

272
00:15:10,930 --> 00:15:14,460
even as the number
of factors goes up.

273
00:15:14,460 --> 00:15:17,150
And this is the way he
describes thinking about that.

274
00:15:17,150 --> 00:15:20,360
You deliberately
sacrifice possibility

275
00:15:20,360 --> 00:15:23,400
of obtaining information
on some points.

276
00:15:23,400 --> 00:15:25,550
So you decide there
are certain things

277
00:15:25,550 --> 00:15:28,520
that you would be able to
estimate in the full factorial,

278
00:15:28,520 --> 00:15:30,500
but you just decide
ahead of time

279
00:15:30,500 --> 00:15:34,430
that they're not worth
investigating, that maybe

280
00:15:34,430 --> 00:15:35,780
they're unlikely a priori.

281
00:15:38,910 --> 00:15:44,250
And so you design the experiment
so that you can't possibly

282
00:15:44,250 --> 00:15:45,500
obtain information about them.

283
00:15:45,500 --> 00:15:47,333
But actually, it's a
little worse than that,

284
00:15:47,333 --> 00:15:48,670
I'm going to show you later.

285
00:15:48,670 --> 00:15:51,780
Not only do you not have
a possibility of obtaining

286
00:15:51,780 --> 00:15:54,660
information about those
points, those specific points,

287
00:15:54,660 --> 00:15:58,140
if they were to be
consequential in the end,

288
00:15:58,140 --> 00:16:02,640
would actually interfere
considerably with your ability

289
00:16:02,640 --> 00:16:03,972
to make inferences.

290
00:16:07,910 --> 00:16:12,920
And so this is the simplest
fractional factorial

291
00:16:12,920 --> 00:16:13,970
I can show you.

292
00:16:13,970 --> 00:16:17,870
Again, we have the cuboidal
representation, and what we do

293
00:16:17,870 --> 00:16:23,310
is just to make observations
at half of the vertices.

294
00:16:23,310 --> 00:16:27,440
So I indicate those by the
black plus's, and these four

295
00:16:27,440 --> 00:16:31,610
are selected in a specific
arrangement to give us the 2

296
00:16:31,610 --> 00:16:34,960
to the 3 minus 1 half
fraction with resolution 3.

297
00:16:34,960 --> 00:16:37,500
I'll describe what resolution
3 means in a second.

298
00:16:37,500 --> 00:16:39,000
One of the things
that's interesting

299
00:16:39,000 --> 00:16:44,240
about the experiment in my mind
is its projective properties.

300
00:16:44,240 --> 00:16:48,190
So we see that, if we
collapse factor A--

301
00:16:51,340 --> 00:16:54,610
that is, if we imagine that
there is a flashlight or such,

302
00:16:54,610 --> 00:16:56,800
a source of light
off to the right,

303
00:16:56,800 --> 00:17:00,010
and we get a projection
of the matrix of the cube,

304
00:17:00,010 --> 00:17:01,780
and now we just get a square.

305
00:17:01,780 --> 00:17:05,470
We'd see that if
factor A is collapsed,

306
00:17:05,470 --> 00:17:12,970
we would have a full factorial
experiment in B and C.

307
00:17:12,970 --> 00:17:15,010
Have you studied these
projective properties

308
00:17:15,010 --> 00:17:16,439
in the other parts
of the course?

309
00:17:16,439 --> 00:17:17,849
OK.

310
00:17:17,849 --> 00:17:22,438
And so it goes in the
other dimensions as well.

311
00:17:22,438 --> 00:17:24,230
And so in my mind,
what's interesting about

312
00:17:24,230 --> 00:17:27,589
this is that in a way the
design of the experiment

313
00:17:27,589 --> 00:17:32,930
allows you to account for a
particular kind of uncertainty.

314
00:17:32,930 --> 00:17:36,380
If you have a long
list of factors,

315
00:17:36,380 --> 00:17:39,410
and you're not sure
which ones are going

316
00:17:39,410 --> 00:17:41,900
to be the important
ones, you know

317
00:17:41,900 --> 00:17:44,780
that some few will turn
out to be important,

318
00:17:44,780 --> 00:17:46,460
but you're not sure which ones.

319
00:17:46,460 --> 00:17:48,020
You can arrange it
so that you will

320
00:17:48,020 --> 00:17:52,190
have done a full factorial
in the few important factors

321
00:17:52,190 --> 00:17:54,620
without knowing exactly
which ones you need to do

322
00:17:54,620 --> 00:17:57,660
a full factorial experiment in.

323
00:17:57,660 --> 00:18:00,700
And that's what the projective
properties allow you to do.

324
00:18:00,700 --> 00:18:05,730
But all this comes at a cost,
and the cost is more easily

325
00:18:05,730 --> 00:18:08,590
described by looking at a
larger fractional factorial

326
00:18:08,590 --> 00:18:09,090
experiment.

327
00:18:09,090 --> 00:18:14,130
This is the 2 to the 7 minus
4 design, and what we see here

328
00:18:14,130 --> 00:18:17,970
is that, if we have 7 factors,
and we want to stuff them

329
00:18:17,970 --> 00:18:21,740
into a small experiment
with just say 8 trials,

330
00:18:21,740 --> 00:18:28,050
then what we can do is put
them in this arrangement.

331
00:18:28,050 --> 00:18:30,540
And the risk is that
if, for example,

332
00:18:30,540 --> 00:18:35,210
there is a two-factor
interaction between F and G,

333
00:18:35,210 --> 00:18:41,300
then that will emerge
in this pattern.

334
00:18:41,300 --> 00:18:45,460
That in the first four
experiments, F and G

335
00:18:45,460 --> 00:18:48,580
are all at the same level,
either plus and plus

336
00:18:48,580 --> 00:18:50,830
or minus and minus.

337
00:18:50,830 --> 00:18:52,720
And in the last
four experiments,

338
00:18:52,720 --> 00:18:57,610
they're all at different
levels, such as minus and plus

339
00:18:57,610 --> 00:18:59,050
or plus and minus.

340
00:18:59,050 --> 00:19:00,670
And so if the two
are interacting

341
00:19:00,670 --> 00:19:03,570
in some important
way, the effect

342
00:19:03,570 --> 00:19:06,600
will emerge in
such a way that it

343
00:19:06,600 --> 00:19:12,900
is coming into being
exactly when A is at minus,

344
00:19:12,900 --> 00:19:17,040
or A is at plus, and
so you will confound

345
00:19:17,040 --> 00:19:19,480
the two in your analysis.

346
00:19:19,480 --> 00:19:22,830
And if you think that
an effect of A itself

347
00:19:22,830 --> 00:19:26,040
is more likely than a two-factor
interaction of F and G,

348
00:19:26,040 --> 00:19:30,900
you're likely to make a mistake,
to assign that effect to A,

349
00:19:30,900 --> 00:19:34,620
whereas, it actually
occurred due to F and G.

350
00:19:34,620 --> 00:19:38,580
That's the risk that you have
to take to make a relatively

351
00:19:38,580 --> 00:19:41,430
small experiment
that's investigating

352
00:19:41,430 --> 00:19:42,600
a lot of potential factors.

353
00:19:45,430 --> 00:19:48,710
So these are the trade-offs we
make in experimental design.

354
00:19:48,710 --> 00:19:51,700
Now, one of the reasons
that this whole topic is

355
00:19:51,700 --> 00:19:53,950
important to engineers
is that we're

356
00:19:53,950 --> 00:19:57,760
concerned with robustness,
robust parameter design.

357
00:19:57,760 --> 00:20:03,130
And what I'm showing
here is the cover of one

358
00:20:03,130 --> 00:20:06,430
of the I think probably
the best modern compilation

359
00:20:06,430 --> 00:20:10,000
of this topic on experimental
design and its application

360
00:20:10,000 --> 00:20:12,610
to engineering
systems, in particular

361
00:20:12,610 --> 00:20:15,400
for the pursuit of robustness.

362
00:20:15,400 --> 00:20:17,650
And Jeff Wu and
Michael [? Hamouda ?]

363
00:20:17,650 --> 00:20:19,960
define robust
parameter design here

364
00:20:19,960 --> 00:20:23,230
as a set of statistical
and engineering methodology

365
00:20:23,230 --> 00:20:27,040
to reduce performance
variation by choosing

366
00:20:27,040 --> 00:20:32,670
settings of control factors to
make the system less sensitive.

367
00:20:32,670 --> 00:20:37,535
So you're trying to find
things that you can control

368
00:20:37,535 --> 00:20:42,110
in an experiment in
the design and to use

369
00:20:42,110 --> 00:20:45,980
those to make performance
less sensitive to things

370
00:20:45,980 --> 00:20:48,930
that you don't control.

371
00:20:48,930 --> 00:20:51,690
Can you think of examples
relevant to your work

372
00:20:51,690 --> 00:20:55,920
of something you
might make robust?

373
00:20:59,840 --> 00:21:02,187
Or even simpler, just
of a noise factor

374
00:21:02,187 --> 00:21:03,770
you might be concerned
with, something

375
00:21:03,770 --> 00:21:05,170
that influences a system.

376
00:21:16,840 --> 00:21:19,660
Well, we'll have
more examples later,

377
00:21:19,660 --> 00:21:21,990
and in fact, I believe
that in the past,

378
00:21:21,990 --> 00:21:26,380
this course has had some
projects associated with it.

379
00:21:26,380 --> 00:21:28,650
And in some cases, people
want to do robust parameter

380
00:21:28,650 --> 00:21:29,580
design and projects.

381
00:21:29,580 --> 00:21:30,120
Yes?

382
00:21:30,120 --> 00:21:32,760
AUDIENCE: [INAUDIBLE]

383
00:21:36,610 --> 00:21:38,060
PROFESSOR: OK.

384
00:21:38,060 --> 00:21:38,560
OK.

385
00:21:38,560 --> 00:21:42,730
So you might be doing injection
molding or a process like that,

386
00:21:42,730 --> 00:21:47,620
and you could control
temperature and humidity

387
00:21:47,620 --> 00:21:51,008
but at some great cost I guess.

388
00:21:51,008 --> 00:21:53,050
So you might choose to
allow those things to vary

389
00:21:53,050 --> 00:21:54,500
to some degree.

390
00:21:54,500 --> 00:21:58,810
And if they influence say
the geometry of your parts,

391
00:21:58,810 --> 00:22:03,250
resulting parts, too
much, you would probably

392
00:22:03,250 --> 00:22:05,140
be producing less
value to your customer.

393
00:22:05,140 --> 00:22:08,290
Probably the function
of your articles

394
00:22:08,290 --> 00:22:10,940
themselves might be
degraded by that,

395
00:22:10,940 --> 00:22:13,690
and so you might like to
choose process parameters,

396
00:22:13,690 --> 00:22:16,480
such as pressures and
compositions of your plastic

397
00:22:16,480 --> 00:22:20,030
and so on, so that that
effect is minimized.

398
00:22:20,030 --> 00:22:22,270
So you decide not to
control the humidity

399
00:22:22,270 --> 00:22:25,220
but just to be
insensitive to it.

400
00:22:25,220 --> 00:22:25,720
There.

401
00:22:25,720 --> 00:22:26,800
Good.

402
00:22:26,800 --> 00:22:29,170
So on the cover
of this textbook,

403
00:22:29,170 --> 00:22:31,900
you see actually
something called

404
00:22:31,900 --> 00:22:37,070
a crossarray it's hard to
pick out, unless you go

405
00:22:37,070 --> 00:22:38,610
and zoom in on
your own computer.

406
00:22:38,610 --> 00:22:42,690
But you see capital
A, B, and C over here,

407
00:22:42,690 --> 00:22:45,050
and those represent
control factors,

408
00:22:45,050 --> 00:22:49,010
such as pressure and
speed of the extruder

409
00:22:49,010 --> 00:22:53,330
and the composition of the
plastic, things that you

410
00:22:53,330 --> 00:22:54,710
might control nominally.

411
00:22:54,710 --> 00:22:56,300
And then there's
little a and b, those

412
00:22:56,300 --> 00:22:59,120
are temperature and
humidity and things

413
00:22:59,120 --> 00:23:02,060
you might decide you're not
going to control so much.

414
00:23:02,060 --> 00:23:05,810
And you see that fractional
factorial designs

415
00:23:05,810 --> 00:23:08,300
or factorial designs
are being used

416
00:23:08,300 --> 00:23:16,220
in each of these arrangements
shown to the left and above,

417
00:23:16,220 --> 00:23:20,530
and then the crossing, the
product of those two designs,

418
00:23:20,530 --> 00:23:22,450
is indicated in the middle.

419
00:23:22,450 --> 00:23:29,490
And here, Wu actually begins
to hint and foreshadow

420
00:23:29,490 --> 00:23:33,570
about the idea of what he
calls a combiner experiment.

421
00:23:33,570 --> 00:23:38,730
So he shows some of these being
white and some being dark,

422
00:23:38,730 --> 00:23:45,760
and he's suggesting that as
a way to address the rather

423
00:23:45,760 --> 00:23:48,220
large size of this experiment.

424
00:23:48,220 --> 00:23:51,540
So it turns out that
the first person

425
00:23:51,540 --> 00:23:57,030
to pioneer this idea of bringing
in fractional factorial designs

426
00:23:57,030 --> 00:23:59,640
into the plant and
using them explicitly

427
00:23:59,640 --> 00:24:02,420
for the purpose of robustness
improvements, person

428
00:24:02,420 --> 00:24:04,740
who seems to most often
get credit for this,

429
00:24:04,740 --> 00:24:07,500
is Taguchi, in Japan.

430
00:24:07,500 --> 00:24:13,080
So he begins to do work, in
Japan, after World War II, when

431
00:24:13,080 --> 00:24:16,060
their manufacturing
base is in disarray,

432
00:24:16,060 --> 00:24:18,277
and their economy is
in not good shape,

433
00:24:18,277 --> 00:24:19,860
and their reputation
for manufacturing

434
00:24:19,860 --> 00:24:21,930
is very poor at that time too.

435
00:24:21,930 --> 00:24:25,600
So if you bought a product, and
it said made in Japan on it,

436
00:24:25,600 --> 00:24:28,360
people would actually
imagine that that meant

437
00:24:28,360 --> 00:24:30,550
it was of rather poor quality.

438
00:24:30,550 --> 00:24:33,980
And of course, our impression
is totally the opposite now,

439
00:24:33,980 --> 00:24:36,190
and it's interesting to
consider how that happened.

440
00:24:36,190 --> 00:24:38,290
It was a lot of
different things.

441
00:24:38,290 --> 00:24:40,180
But maybe one of the
things that brought

442
00:24:40,180 --> 00:24:42,370
that change about
in our perception

443
00:24:42,370 --> 00:24:45,550
of Japanese manufacture was
some of this methodology.

444
00:24:45,550 --> 00:24:49,810
And Taguchi
correctly pointed out

445
00:24:49,810 --> 00:24:51,850
that robustness
could be attained,

446
00:24:51,850 --> 00:24:54,070
and it could be attained
through experimentation.

447
00:24:54,070 --> 00:24:57,100
And he suggested that you might,
for example, run an experiment

448
00:24:57,100 --> 00:25:00,940
like this, an orthogonal array
or factorial design and control

449
00:25:00,940 --> 00:25:03,920
factors crossed
with noise factors.

450
00:25:03,920 --> 00:25:08,500
And in this case, you take your
8-run array of control factors,

451
00:25:08,500 --> 00:25:13,360
cross it with your 4-run
array of noise factors,

452
00:25:13,360 --> 00:25:16,640
and you would have
a 32-run crossarray.

453
00:25:16,640 --> 00:25:21,130
And so you see with
this cross symbol,

454
00:25:21,130 --> 00:25:25,390
the implication is that
the size of the experiment

455
00:25:25,390 --> 00:25:30,040
scales at least like the
product of the number

456
00:25:30,040 --> 00:25:34,720
of control factors plus
1 and the number of noise

457
00:25:34,720 --> 00:25:36,280
factors plus 1.

458
00:25:36,280 --> 00:25:38,260
Which when you begin
to consider lots

459
00:25:38,260 --> 00:25:41,260
of control of noise factors,
it gets awfully burdensome.

460
00:25:41,260 --> 00:25:44,080
So people are looking for ways
to do it more efficiently,

461
00:25:44,080 --> 00:25:50,530
and that leads to the cover of
this textbook, the idea of just

462
00:25:50,530 --> 00:25:54,040
doing half of them, so that
you're actually not running

463
00:25:54,040 --> 00:25:56,500
exactly the same
noise conditions

464
00:25:56,500 --> 00:25:59,150
at every setting of
the control factors.

465
00:25:59,150 --> 00:26:01,340
And what we've done
in our methodology

466
00:26:01,340 --> 00:26:05,030
validation work actually is to
show in a recent publication

467
00:26:05,030 --> 00:26:06,830
in the Journal of
Quality Technology

468
00:26:06,830 --> 00:26:10,400
that this procedure,
although theoretically looks

469
00:26:10,400 --> 00:26:14,840
promising in many specific
cases, it's failing.

470
00:26:21,190 --> 00:26:25,270
And yet, if I look at
industry practice today,

471
00:26:25,270 --> 00:26:28,000
I think the methods
of Taguchi and also

472
00:26:28,000 --> 00:26:31,750
the refinements
proposed more recently

473
00:26:31,750 --> 00:26:35,800
on the basis of
statistical theory

474
00:26:35,800 --> 00:26:38,530
are finding their way
into industry practice

475
00:26:38,530 --> 00:26:41,470
and are really quite
solidly entrenched.

476
00:26:41,470 --> 00:26:44,860
So some of you may have had some
experience with these Six Sigma

477
00:26:44,860 --> 00:26:47,770
programs by now.

478
00:26:47,770 --> 00:26:51,580
Maybe some of you are in General
Electric, maybe some Motorola,

479
00:26:51,580 --> 00:26:55,990
and I guess many other companies
by now have such programs.

480
00:26:55,990 --> 00:27:00,970
And a big part of such programs
is that entering engineers

481
00:27:00,970 --> 00:27:04,240
will take some coursework,
even if they didn't get it

482
00:27:04,240 --> 00:27:06,730
in their undergraduate degrees.

483
00:27:06,730 --> 00:27:09,820
The company feels people
broadly need this,

484
00:27:09,820 --> 00:27:13,870
and so they get some coursework,
and then they do projects.

485
00:27:13,870 --> 00:27:15,353
Right?

486
00:27:15,353 --> 00:27:16,770
And they demonstrate
that they can

487
00:27:16,770 --> 00:27:19,530
do some of this kind of
work for the company,

488
00:27:19,530 --> 00:27:22,530
and at Ford, at
least the last time

489
00:27:22,530 --> 00:27:28,500
I checked, their design
process encoded just this sort

490
00:27:28,500 --> 00:27:31,350
of thing, and they have
their multi-step process

491
00:27:31,350 --> 00:27:34,320
for attaining reliability
and robustness.

492
00:27:34,320 --> 00:27:37,860
And in step four, I dig in
and look at the details,

493
00:27:37,860 --> 00:27:40,080
and the engineers
are asked to select

494
00:27:40,080 --> 00:27:42,420
appropriate orthogonal arrays.

495
00:27:42,420 --> 00:27:45,210
So the idea is that
maybe they have

496
00:27:45,210 --> 00:27:48,090
some choices about what
size of orthogonal array,

497
00:27:48,090 --> 00:27:51,470
number of levels,
the crossed array

498
00:27:51,470 --> 00:27:54,380
versus the combined array,
such as Wu talked about.

499
00:27:54,380 --> 00:27:56,600
But a departure from
orthogonal arrays

500
00:27:56,600 --> 00:28:00,410
itself is not really being
considered anymore in industry,

501
00:28:00,410 --> 00:28:04,220
and so this has become
very much entrenched.

502
00:28:04,220 --> 00:28:06,050
And in fact, in some
of the textbooks,

503
00:28:06,050 --> 00:28:13,150
you see an even more extreme
statement of the point.

504
00:28:13,150 --> 00:28:19,480
In this text, they argue that
the idea that Edward John

505
00:28:19,480 --> 00:28:23,770
Russell expressed on that
earlier slide, that simplicity

506
00:28:23,770 --> 00:28:28,660
should be valued, and that
single-factor experiments

507
00:28:28,660 --> 00:28:33,310
are often advisable, that
has seen its final demise.

508
00:28:33,310 --> 00:28:38,500
So that's the opinion voiced
in this particular textbook.

509
00:28:38,500 --> 00:28:41,560
But we wanted to question
this a little bit.

510
00:28:41,560 --> 00:28:45,190
We wondered if sometimes there
isn't a role for the simpler

511
00:28:45,190 --> 00:28:49,170
experiments still,
and this decision

512
00:28:49,170 --> 00:28:50,940
to spend some time
looking into this

513
00:28:50,940 --> 00:28:53,440
came out of some
observations in industry.

514
00:28:53,440 --> 00:28:57,900
And I thought in some ways
now, it seems a shock to me

515
00:28:57,900 --> 00:29:01,650
that our initial thoughts
along these lines

516
00:29:01,650 --> 00:29:03,810
came out of the farm again.

517
00:29:03,810 --> 00:29:06,120
So that all these ideas
of orthogonal arrays

518
00:29:06,120 --> 00:29:09,000
and so on came from
agriculture in the first place,

519
00:29:09,000 --> 00:29:12,300
and now our wanting to question
them came off of the farm,

520
00:29:12,300 --> 00:29:15,780
because we were working
with a manufacturer

521
00:29:15,780 --> 00:29:18,800
of agricultural equipment.

522
00:29:18,800 --> 00:29:22,040
And it seems to me actually
that farm equipment

523
00:29:22,040 --> 00:29:25,550
is one of the most significant
robustness challenges

524
00:29:25,550 --> 00:29:26,270
in the world.

525
00:29:26,270 --> 00:29:29,600
Because what you have in
a tractor or a windrower

526
00:29:29,600 --> 00:29:33,110
or a combine is
really a factory.

527
00:29:33,110 --> 00:29:36,380
It needs to process
something, sometimes in not

528
00:29:36,380 --> 00:29:37,280
a very simple way.

529
00:29:37,280 --> 00:29:40,340
Sometimes, it needs to take
a piece of cotton off a plant

530
00:29:40,340 --> 00:29:41,960
and then strip away
some part of it.

531
00:29:41,960 --> 00:29:43,310
It's not a simple thing.

532
00:29:43,310 --> 00:29:46,010
And yet here it is, and
it's out in the weather.

533
00:29:46,010 --> 00:29:47,270
It's out in a field.

534
00:29:47,270 --> 00:29:51,090
Whatever conditions prevail,
it has to deal with.

535
00:29:51,090 --> 00:29:53,330
So it's a very difficult
reliability challenge.

536
00:29:53,330 --> 00:29:56,720
And this particular company
had relatively recently

537
00:29:56,720 --> 00:30:00,560
brought in a lot of
robust design methodology,

538
00:30:00,560 --> 00:30:05,240
and they were looking forward to
the benefits that would accrue.

539
00:30:05,240 --> 00:30:07,580
And they got exactly
the opposite,

540
00:30:07,580 --> 00:30:11,480
that this most recent
launch of a new tractor

541
00:30:11,480 --> 00:30:13,250
was one of the worst
they'd ever had,

542
00:30:13,250 --> 00:30:15,230
and it was causing
them real problems.

543
00:30:15,230 --> 00:30:18,170
Their customers were
quite angry with them.

544
00:30:18,170 --> 00:30:20,290
And so we wanted
to look into that,

545
00:30:20,290 --> 00:30:24,010
and we decided to, in
effect, do an audit,

546
00:30:24,010 --> 00:30:27,260
look at their design
process for this tractor.

547
00:30:27,260 --> 00:30:28,990
What had happened?

548
00:30:28,990 --> 00:30:34,900
Were there some mistakes in
choices of design or priority

549
00:30:34,900 --> 00:30:38,080
or management, whatever?

550
00:30:38,080 --> 00:30:39,910
So one of the things
that we asked for

551
00:30:39,910 --> 00:30:42,790
is an accounting of all
the different robust design

552
00:30:42,790 --> 00:30:45,220
experiments they had
done, and they gave us

553
00:30:45,220 --> 00:30:46,690
a list of experiments
they planned.

554
00:30:46,690 --> 00:30:49,540
It was a long list, and
then we said, well, yeah,

555
00:30:49,540 --> 00:30:51,190
but we need more detail in this.

556
00:30:51,190 --> 00:30:55,120
Show us the written reports
from all these experiments.

557
00:30:55,120 --> 00:30:56,320
Show us the data.

558
00:30:56,320 --> 00:30:59,230
Show us the decisions
made basis of the data.

559
00:30:59,230 --> 00:31:03,310
And they gave us a much
smaller list of reports,

560
00:31:03,310 --> 00:31:08,220
and we said, OK, you had
30, and now we have 5.

561
00:31:08,220 --> 00:31:10,080
What about the other 25?

562
00:31:10,080 --> 00:31:13,120
And they said, well,
those were never finished,

563
00:31:13,120 --> 00:31:14,590
and that was interesting to us.

564
00:31:14,590 --> 00:31:18,230
Why is it that the
majority weren't finished?

565
00:31:18,230 --> 00:31:21,130
And we began to ask questions
of the individuals involved,

566
00:31:21,130 --> 00:31:23,200
and they always have
pretty good reasons.

567
00:31:23,200 --> 00:31:26,710
In some cases, they would say
that a piece of test equipment

568
00:31:26,710 --> 00:31:29,500
had broken down at some
point along the way

569
00:31:29,500 --> 00:31:32,000
and was going to take a certain
amount of time to repair,

570
00:31:32,000 --> 00:31:35,450
so we had to move forward
with a partial data set.

571
00:31:35,450 --> 00:31:39,100
In fact, a large fraction of
the experiments aren't finished,

572
00:31:39,100 --> 00:31:42,400
and this prompted us to
wonder, what would you

573
00:31:42,400 --> 00:31:45,760
do if you knew this
from the outset?

574
00:31:45,760 --> 00:31:49,600
Let's say that you
put a 60% probability

575
00:31:49,600 --> 00:31:52,000
on canceling any
individual experiment you

576
00:31:52,000 --> 00:31:58,878
might run somewhere at a
state of partial completion.

577
00:31:58,878 --> 00:32:00,420
Let's say that were
the case, but you

578
00:32:00,420 --> 00:32:02,340
didn't know which
60% we're going

579
00:32:02,340 --> 00:32:06,800
to be canceled, because you
probably don't know that.

580
00:32:06,800 --> 00:32:09,225
What would you do?

581
00:32:09,225 --> 00:32:11,100
Well, we thought, perhaps,
you would do a one

582
00:32:11,100 --> 00:32:14,430
at a time experiment,
because then at least,

583
00:32:14,430 --> 00:32:17,100
no matter what you did,
you would have stopped,

584
00:32:17,100 --> 00:32:20,760
and the data would be
relatively simple to analyze,

585
00:32:20,760 --> 00:32:23,130
such as what Russell
talked about.

586
00:32:23,130 --> 00:32:26,430
And we found that actually
some of the most prominent

587
00:32:26,430 --> 00:32:28,530
statisticians in
the past always felt

588
00:32:28,530 --> 00:32:32,620
that there was a role for one
factor at a time experiments.

589
00:32:32,620 --> 00:32:35,550
So for example, you may have
heard of Milton Friedman

590
00:32:35,550 --> 00:32:37,520
in a different context.

591
00:32:37,520 --> 00:32:39,217
What do you know him for?

592
00:32:39,217 --> 00:32:41,755
AUDIENCE: [INAUDIBLE]

593
00:32:41,755 --> 00:32:42,380
PROFESSOR: Yes.

594
00:32:42,380 --> 00:32:42,880
Yeah.

595
00:32:42,880 --> 00:32:46,730
Nobel Prize in
economics, market view

596
00:32:46,730 --> 00:32:48,320
of economics and
Leonard Savage who

597
00:32:48,320 --> 00:32:53,450
had written an excellent text
on foundations of statistics.

598
00:32:53,450 --> 00:32:56,300
They found that an efficient
design for the present

599
00:32:56,300 --> 00:32:59,630
purposes-- that was
for maximization--

600
00:32:59,630 --> 00:33:02,450
ought to adjust experimental
program at each stage in light

601
00:33:02,450 --> 00:33:04,700
of the prior stages, and
actually, what they described

602
00:33:04,700 --> 00:33:06,788
was a one factor at
a time experiment.

603
00:33:06,788 --> 00:33:08,330
And then Cuthbert
Daniel talked about

604
00:33:08,330 --> 00:33:15,170
a more social or psychological
value of simpler experiments,

605
00:33:15,170 --> 00:33:18,840
just reemphasizing the
points that Edward John

606
00:33:18,840 --> 00:33:20,100
Russell had made.

607
00:33:20,100 --> 00:33:24,960
That somehow you feel that you
can react to data more rapidly

608
00:33:24,960 --> 00:33:27,640
or learn something
from each rung.

609
00:33:27,640 --> 00:33:29,650
And he talked
about some criteria

610
00:33:29,650 --> 00:33:31,660
by which you might
make that choice,

611
00:33:31,660 --> 00:33:34,360
that for example, he
had a demarcation.

612
00:33:34,360 --> 00:33:39,190
Maybe it's OK to do it,
as long as the effects

613
00:33:39,190 --> 00:33:42,500
are at least three or four
times the average random error

614
00:33:42,500 --> 00:33:43,510
per trail.

615
00:33:43,510 --> 00:33:46,120
So he had, based
on his experience,

616
00:33:46,120 --> 00:33:48,550
come up with a demarcation--

617
00:33:48,550 --> 00:33:50,740
effects three or four
times random error.

618
00:33:50,740 --> 00:33:53,650
And we wanted to understand
whether that demarcation was

619
00:33:53,650 --> 00:33:57,100
about right, so
we began to study

620
00:33:57,100 --> 00:34:01,270
an adaptive variant of one
factor of time experimentation.

621
00:34:01,270 --> 00:34:04,540
And in the beginning, we
were interested primarily

622
00:34:04,540 --> 00:34:13,257
in knowing, if you were to do
this adaptive experimentation,

623
00:34:13,257 --> 00:34:16,330
what level of probability
of canceling the experiments

624
00:34:16,330 --> 00:34:19,300
would make this style look good?

625
00:34:19,300 --> 00:34:21,909
And the funny thing about it
is we started with some maybe

626
00:34:21,909 --> 00:34:25,570
50% probability of canceling,
and we looked at the two

627
00:34:25,570 --> 00:34:30,070
experimentation styles, adaptive
and factorial, and on the cases

628
00:34:30,070 --> 00:34:32,330
we were looking at,
adaptive looked better.

629
00:34:32,330 --> 00:34:34,060
And so we dialed
down the probability

630
00:34:34,060 --> 00:34:37,120
of canceling the 10%, and
they still looked better.

631
00:34:37,120 --> 00:34:40,179
And we dialed it down to 0%, and
the adaptive experiment still

632
00:34:40,179 --> 00:34:40,960
looked better.

633
00:34:40,960 --> 00:34:44,840
And we thought, well, now we
know we've made a mistake,

634
00:34:44,840 --> 00:34:47,540
because we saw it
in the textbooks

635
00:34:47,540 --> 00:34:50,239
that this kind of
experimentation

636
00:34:50,239 --> 00:34:51,889
has seen its final demise.

637
00:34:51,889 --> 00:34:55,820
So either we made a mistake in
the way we set up the study,

638
00:34:55,820 --> 00:34:59,950
or we chose a strange
case to start with.

639
00:34:59,950 --> 00:35:02,310
We were perplexed.

640
00:35:02,310 --> 00:35:06,330
So we did eventually a
big empirical evaluation,

641
00:35:06,330 --> 00:35:13,380
and we took 66 different systems
of a broad range of kinds--

642
00:35:13,380 --> 00:35:16,710
chemical systems, mechanical
systems, electrical systems,

643
00:35:16,710 --> 00:35:21,420
and we made our comparison in
this case without canceling.

644
00:35:21,420 --> 00:35:23,708
Now, we were interested
in that issue.

645
00:35:23,708 --> 00:35:26,083
Imagine that you're going to
do this adaptive, one factor

646
00:35:26,083 --> 00:35:27,397
at a time experimentation.

647
00:35:27,397 --> 00:35:29,730
You're going to compare that
with a fractional factorial

648
00:35:29,730 --> 00:35:32,778
experiment and see which
one does better on average.

649
00:35:32,778 --> 00:35:34,320
And this is what we
found is that, as

650
00:35:34,320 --> 00:35:38,550
long as experimental
error was about 25%

651
00:35:38,550 --> 00:35:42,640
of combined factor effect, or if
interactions were large enough,

652
00:35:42,640 --> 00:35:48,610
then you'd prefer the
adaptive experiments.

653
00:35:48,610 --> 00:35:54,970
And so here is that same
result shown in a tabular form.

654
00:35:54,970 --> 00:35:58,240
So we have on the
column headings

655
00:35:58,240 --> 00:36:00,670
the degree of experimental
error expressed

656
00:36:00,670 --> 00:36:10,530
as a fraction of the factor
effects, and we have on the row

657
00:36:10,530 --> 00:36:13,980
headings the strength
of interactions--

658
00:36:13,980 --> 00:36:15,540
mild, moderate,
strong, dominant.

659
00:36:18,100 --> 00:36:21,330
Strong interactions would
indicate that the interactions

660
00:36:21,330 --> 00:36:23,940
are counting for
more than a quarter

661
00:36:23,940 --> 00:36:26,970
of effects in the system,
quite a big fraction,

662
00:36:26,970 --> 00:36:30,450
and moderate is
maybe 10% to 25%.

663
00:36:30,450 --> 00:36:35,860
And the gray boxes indicate
that the adaptive one factor

664
00:36:35,860 --> 00:36:37,980
at a time experiments
were giving

665
00:36:37,980 --> 00:36:40,620
at least as good
or better results

666
00:36:40,620 --> 00:36:43,620
than the fractional factorial,
and so you might prefer them,

667
00:36:43,620 --> 00:36:46,120
especially given
their flexibility.

668
00:36:46,120 --> 00:36:48,240
And so we were surprised
at how much of that

669
00:36:48,240 --> 00:36:51,600
territory turns out to be gray,
and that was the main result

670
00:36:51,600 --> 00:36:54,460
we wanted to get
across in this paper.

671
00:36:54,460 --> 00:36:56,950
And we thought, a strength
of the experimental error

672
00:36:56,950 --> 00:37:00,820
as large as 0.4 looks like
a lot of experimental error

673
00:37:00,820 --> 00:37:02,350
to us, to engineers.

674
00:37:02,350 --> 00:37:05,290
When you see a dependence,
such as this blue line

675
00:37:05,290 --> 00:37:08,260
here, on the figure
and the scatter

676
00:37:08,260 --> 00:37:09,760
being indicated
by the red, that's

677
00:37:09,760 --> 00:37:13,060
about typical of the strength
of experimental error at 0.4

678
00:37:13,060 --> 00:37:16,270
that we expressed,
and to us that

679
00:37:16,270 --> 00:37:19,210
looks like a pretty
noisy experiment.

680
00:37:19,210 --> 00:37:23,510
And so we thought it
was interesting how

681
00:37:23,510 --> 00:37:29,550
much of the time you might
recommend these experiments.

682
00:37:29,550 --> 00:37:32,880
So we wanted to understand
how this could be,

683
00:37:32,880 --> 00:37:36,160
since it seems so
counterintuitive to us.

684
00:37:36,160 --> 00:37:39,600
And so we delved into it
a little more in terms

685
00:37:39,600 --> 00:37:43,920
of the mathematics, and we
tried to develop a theory that

686
00:37:43,920 --> 00:37:46,750
would account for this.

687
00:37:46,750 --> 00:37:50,880
So we started to write
a model of the procedure

688
00:37:50,880 --> 00:37:53,040
of adaptive one factor at
a time experimentation,

689
00:37:53,040 --> 00:37:54,960
and so there's just a
little notation here.

690
00:37:54,960 --> 00:37:58,618
You imagine that you make
an initial observation.

691
00:37:58,618 --> 00:38:00,660
Some of the symbols are
coming up a little wrong.

692
00:38:00,660 --> 00:38:07,620
This is an x with a tilde over
it is an initial starting point

693
00:38:07,620 --> 00:38:09,720
level for factor one.

694
00:38:09,720 --> 00:38:12,600
And x2 with a tilde is
the same, and that k

695
00:38:12,600 --> 00:38:14,190
should be an
ellipsis, three dots.

696
00:38:17,460 --> 00:38:19,950
And then you change
one of the factors.

697
00:38:19,950 --> 00:38:24,610
You toggle it, change it to the
opposite of the starting value,

698
00:38:24,610 --> 00:38:26,500
and that's your
first observation.

699
00:38:26,500 --> 00:38:29,890
And then you make a
decision about the level

700
00:38:29,890 --> 00:38:33,850
of the first factor based on
the sign of the differences

701
00:38:33,850 --> 00:38:35,710
in the observations.

702
00:38:35,710 --> 00:38:44,250
So if the improvement seems to
obtain, we adopt the change,

703
00:38:44,250 --> 00:38:46,440
and then we go through
that for every one

704
00:38:46,440 --> 00:38:51,090
of the different factors,
putting stars on our x's, and

705
00:38:51,090 --> 00:38:51,990
we continue.

706
00:38:51,990 --> 00:38:53,610
And what we're
interested in knowing

707
00:38:53,610 --> 00:39:00,020
is how does the expected
value of improvement

708
00:39:00,020 --> 00:39:03,440
behave as a function of a
number of parameters in a model?

709
00:39:03,440 --> 00:39:07,220
And the parameters we
put in our model are few.

710
00:39:07,220 --> 00:39:11,480
We talked about the size
of main effects in general,

711
00:39:11,480 --> 00:39:14,150
the size of interactions, and
the size of experimental error,

712
00:39:14,150 --> 00:39:16,100
because our empirical
investigation

713
00:39:16,100 --> 00:39:20,000
had shown that those were
interesting to look at.

714
00:39:20,000 --> 00:39:23,900
And here, we show we wanted to
normalize those with respect

715
00:39:23,900 --> 00:39:26,090
to the maximum, the
best that you could do,

716
00:39:26,090 --> 00:39:29,240
so that our results could
be easily interpreted.

717
00:39:29,240 --> 00:39:33,440
And another concept we
introduced in this paper,

718
00:39:33,440 --> 00:39:36,710
we felt we needed to
explain the results,

719
00:39:36,710 --> 00:39:41,470
was the idea of
exploiting an effect.

720
00:39:41,470 --> 00:39:45,830
So we had to introduce
the idea that, when

721
00:39:45,830 --> 00:39:48,920
you do an experiment, if your
system has a number of effects

722
00:39:48,920 --> 00:39:50,840
in it, main effects
or interactions,

723
00:39:50,840 --> 00:39:53,520
they can either be
exploited or not,

724
00:39:53,520 --> 00:39:56,940
and we'll say that they're
exploited, just in case.

725
00:39:56,940 --> 00:40:00,990
For example, if you're trying
to get a higher response out

726
00:40:00,990 --> 00:40:03,720
of the system-- let's say yield
out of a field [? of oats, ?]

727
00:40:03,720 --> 00:40:09,570
or quality of a
manufactured article.

728
00:40:09,570 --> 00:40:14,160
Then, you exploit an
effect, if the coefficient

729
00:40:14,160 --> 00:40:17,550
in the model times the
level of the factor

730
00:40:17,550 --> 00:40:22,310
is such that it's contributing
positively to your outcome.

731
00:40:22,310 --> 00:40:25,880
And you exploit an
interaction in the system

732
00:40:25,880 --> 00:40:27,620
basically under
the same condition.

733
00:40:27,620 --> 00:40:29,420
That if the two
factors involved are

734
00:40:29,420 --> 00:40:32,150
such that you're gaining
benefit from the coefficient

735
00:40:32,150 --> 00:40:35,990
in the model, either
increasing or decreasing

736
00:40:35,990 --> 00:40:37,910
the response depending
on what you prefer,

737
00:40:37,910 --> 00:40:39,530
we call that one exploited.

738
00:40:39,530 --> 00:40:41,390
So we wanted to understand
the probabilities

739
00:40:41,390 --> 00:40:43,580
and conditional
probabilities associated

740
00:40:43,580 --> 00:40:44,960
with this adaptive experiment.

741
00:40:44,960 --> 00:40:49,580
So we begin to do the theorem
proving based on our model,

742
00:40:49,580 --> 00:40:51,590
and we find, first
of all, that if you

743
00:40:51,590 --> 00:40:57,180
change just the first
factor and compute

744
00:40:57,180 --> 00:40:59,060
the expected value
of improvement,

745
00:40:59,060 --> 00:41:03,070
you have two different
kinds of contributions--

746
00:41:03,070 --> 00:41:10,250
1 due to a main effect and n
minus 1 due to interactions.

747
00:41:10,250 --> 00:41:12,940
So you change factor
1, and this figure here

748
00:41:12,940 --> 00:41:16,060
is meant to indicate the
condition with n equals 7.

749
00:41:16,060 --> 00:41:18,070
You've got 7 factors
you're interested in,

750
00:41:18,070 --> 00:41:22,330
and this stack of squares
indicates the contribution

751
00:41:22,330 --> 00:41:23,270
of main effects.

752
00:41:23,270 --> 00:41:25,690
And none of these other factors
are contributing anything,

753
00:41:25,690 --> 00:41:28,960
but now the main effect
of the first factor

754
00:41:28,960 --> 00:41:32,320
is contributing quite a bit
on average, after you've

755
00:41:32,320 --> 00:41:33,850
done the adaptation.

756
00:41:33,850 --> 00:41:37,960
And all of the 6 potential
two-factor interactions

757
00:41:37,960 --> 00:41:41,080
are on average contributing
a little to improvement,

758
00:41:41,080 --> 00:41:44,350
because it may be
that in some cases

759
00:41:44,350 --> 00:41:46,570
there'll be a large
two-factor interaction.

760
00:41:46,570 --> 00:41:48,740
And you'll see it when
you toggle the factors,

761
00:41:48,740 --> 00:41:51,190
and you'll go chase
that, and so on average,

762
00:41:51,190 --> 00:41:53,390
it's contributing something.

763
00:41:53,390 --> 00:41:55,420
And we could write
closed form expressions

764
00:41:55,420 --> 00:41:59,590
for the contribution,
and the contribution

765
00:41:59,590 --> 00:42:01,880
due to main effects
is larger when

766
00:42:01,880 --> 00:42:03,700
the main effects are larger.

767
00:42:03,700 --> 00:42:07,450
It tends to be smaller
when interactions are large

768
00:42:07,450 --> 00:42:10,228
and when error is large.

769
00:42:10,228 --> 00:42:11,770
The contributions
to the interactions

770
00:42:11,770 --> 00:42:15,360
are large when interactions
are large in the model.

771
00:42:15,360 --> 00:42:19,080
And interesting to us, if you
write out the expected value,

772
00:42:19,080 --> 00:42:23,430
normalize by the maximum,
so that the improvements are

773
00:42:23,430 --> 00:42:29,760
on this scale either 0 or
1 or scaled in between.

774
00:42:29,760 --> 00:42:32,820
The contribution you
would get in the case

775
00:42:32,820 --> 00:42:35,790
that experimental error
is low-- so here I

776
00:42:35,790 --> 00:42:40,410
put experimental error
at 10% of a main effect.

777
00:42:40,410 --> 00:42:44,040
The contribution you get after
running one 1 of 7 factors

778
00:42:44,040 --> 00:42:47,180
is actually about 1/7.

779
00:42:47,180 --> 00:42:50,420
So you get 1/7 of the
way toward the maximum

780
00:42:50,420 --> 00:42:52,210
by changing 1/7 of the factors.

781
00:42:52,210 --> 00:42:54,300
That's a good start.

782
00:42:54,300 --> 00:42:56,910
And the influence of error is--

783
00:42:56,910 --> 00:42:57,900
it's there.

784
00:42:57,900 --> 00:43:01,770
As you go to error being
as large as main effects,

785
00:43:01,770 --> 00:43:04,350
it drops a little,
and of course,

786
00:43:04,350 --> 00:43:06,120
if you make the
error really huge,

787
00:43:06,120 --> 00:43:07,780
it drops quite substantially.

788
00:43:07,780 --> 00:43:13,320
But that's a hard test
to pass, but we're just

789
00:43:13,320 --> 00:43:16,180
trying to understand
what the influences are.

790
00:43:16,180 --> 00:43:17,032
OK.

791
00:43:17,032 --> 00:43:18,990
Now, let's talk about
probability of exploiting

792
00:43:18,990 --> 00:43:21,040
the first main effect.

793
00:43:21,040 --> 00:43:27,175
Just when interactions
are small,

794
00:43:27,175 --> 00:43:29,550
the probability of getting
the benefit of the main effect

795
00:43:29,550 --> 00:43:30,450
is large.

796
00:43:30,450 --> 00:43:32,280
Then, as interactions
increase, it's

797
00:43:32,280 --> 00:43:35,760
less likely that you'll get
the benefit of the main effect.

798
00:43:39,530 --> 00:43:41,720
Now, it becomes interesting
at the second step.

799
00:43:41,720 --> 00:43:43,908
Now, you've toggled
two factors, and then

800
00:43:43,908 --> 00:43:45,200
what we think about it is this.

801
00:43:45,200 --> 00:43:48,290
Now, you've got the benefit
of two main effects,

802
00:43:48,290 --> 00:43:53,750
and you get first n minus 1
and then n minus 2 interactions

803
00:43:53,750 --> 00:43:55,940
potentially contributing.

804
00:43:55,940 --> 00:43:58,790
And at this point, I
hash it in, because this

805
00:43:58,790 --> 00:44:02,750
is the first interaction which
you've actually locked in.

806
00:44:02,750 --> 00:44:05,390
In the sense that
any further changes

807
00:44:05,390 --> 00:44:08,090
you make through the
experimentation process

808
00:44:08,090 --> 00:44:11,270
will not reverse whatever
benefits you got.

809
00:44:11,270 --> 00:44:13,430
Because you've already
changed one and changed two,

810
00:44:13,430 --> 00:44:15,600
and you've made your
decision about that.

811
00:44:15,600 --> 00:44:17,480
And now, we see that
there's a contribution

812
00:44:17,480 --> 00:44:21,680
due to the interaction,
and the interaction

813
00:44:21,680 --> 00:44:24,830
is the same functional
form as I showed before.

814
00:44:24,830 --> 00:44:28,700
That the contribution is large
when the interaction is large.

815
00:44:28,700 --> 00:44:35,720
Again, we see that the expected
value has gone from about 1/7

816
00:44:35,720 --> 00:44:36,600
to about 2/7.

817
00:44:36,600 --> 00:44:40,170
So we're continuing to step
forward at a good rate,

818
00:44:40,170 --> 00:44:41,450
and we want to understand why.

819
00:44:41,450 --> 00:44:43,440
Well, OK.

820
00:44:43,440 --> 00:44:46,440
We see the probability of
exploiting the interaction.

821
00:44:46,440 --> 00:44:49,540
We can write it's
closed form expression.

822
00:44:49,540 --> 00:44:53,520
It's strictly better than 50-50.

823
00:44:53,520 --> 00:44:57,900
So it's in any case
better than random,

824
00:44:57,900 --> 00:44:59,910
and it goes up with
interaction strength.

825
00:44:59,910 --> 00:45:01,410
Now, the interesting
thing about it,

826
00:45:01,410 --> 00:45:04,650
it's never very high,
because actually,

827
00:45:04,650 --> 00:45:07,230
any particular
interaction is competing

828
00:45:07,230 --> 00:45:10,820
with n choose 2
other interactions

829
00:45:10,820 --> 00:45:12,590
or n choose 2 minus 1.

830
00:45:12,590 --> 00:45:15,170
And so any particular
one is unlikely to be

831
00:45:15,170 --> 00:45:19,170
exploited by this procedure, and
so maybe we get a 60% chance.

832
00:45:19,170 --> 00:45:21,550
But if you condition that
probability, if you say,

833
00:45:21,550 --> 00:45:23,550
well, let's say that this
particular interaction

834
00:45:23,550 --> 00:45:26,210
that we just locked
in is somehow

835
00:45:26,210 --> 00:45:28,110
the largest one in the system.

836
00:45:28,110 --> 00:45:29,720
Let's say that that happened.

837
00:45:29,720 --> 00:45:32,540
We conditioned the probability,
we get a new expression,

838
00:45:32,540 --> 00:45:36,300
and the probability
could be as high as 80%.

839
00:45:36,300 --> 00:45:38,910
The reason that's
interesting to us

840
00:45:38,910 --> 00:45:42,840
is that we know that the
adaptive one-factor experiments

841
00:45:42,840 --> 00:45:48,030
are just not capable of allowing
you to estimate interactions.

842
00:45:48,030 --> 00:45:52,140
You take a factor, you change
it, you look at the influence,

843
00:45:52,140 --> 00:45:54,840
and what you're seeing
is a combination

844
00:45:54,840 --> 00:45:57,540
of the main effective
the factor and a bunch

845
00:45:57,540 --> 00:46:01,710
of conditional
contributions due to a bunch

846
00:46:01,710 --> 00:46:03,740
of different interactions.

847
00:46:03,740 --> 00:46:05,870
And so you can't sort
out which one is which.

848
00:46:05,870 --> 00:46:08,060
You would need more
experiments to do that.

849
00:46:08,060 --> 00:46:11,330
Nevertheless, if there's
a big effect there,

850
00:46:11,330 --> 00:46:14,030
there's a good chance you'll
get benefit from it, when you

851
00:46:14,030 --> 00:46:18,414
do this adaptive experiment.

852
00:46:18,414 --> 00:46:21,872
AUDIENCE: [INAUDIBLE]

853
00:46:51,253 --> 00:46:51,920
PROFESSOR: Yeah.

854
00:46:51,920 --> 00:46:55,190
It's an interesting question,
and this is something

855
00:46:55,190 --> 00:46:57,920
that we were concerned with.

856
00:46:57,920 --> 00:47:00,440
All the results I'm
showing you today

857
00:47:00,440 --> 00:47:05,180
assume that you've randomized
the ordering of the factors

858
00:47:05,180 --> 00:47:08,980
and that you've randomized
the starting point.

859
00:47:08,980 --> 00:47:12,040
And so far, what our
investigations show

860
00:47:12,040 --> 00:47:13,600
is that the ordering
of the factors

861
00:47:13,600 --> 00:47:18,510
doesn't matter much but
the starting points do.

862
00:47:18,510 --> 00:47:23,280
So I would say, whether or
not you order the factors

863
00:47:23,280 --> 00:47:25,530
from the most important one
to the next to the next,

864
00:47:25,530 --> 00:47:29,010
it's a minor
improvement, not big.

865
00:47:29,010 --> 00:47:35,310
But if you select a very
promising starting point, that

866
00:47:35,310 --> 00:47:39,630
is go with something that
is likely to be very robust,

867
00:47:39,630 --> 00:47:43,100
high performance, that
makes a big difference.

868
00:47:43,100 --> 00:47:46,040
And it wasn't obvious
to us at the start.

869
00:47:46,040 --> 00:47:48,292
I thought probably
what you said was true.

870
00:47:48,292 --> 00:47:50,000
It turned out to be
the other way around.

871
00:47:53,140 --> 00:47:55,890
So the dynamics of this
process of experimentation

872
00:47:55,890 --> 00:47:59,400
are such that it turns out
the probability of exploiting

873
00:47:59,400 --> 00:48:01,890
interactions as
you go gets higher

874
00:48:01,890 --> 00:48:04,810
and higher throughout the
course of the experiment.

875
00:48:04,810 --> 00:48:11,190
So it's true that, if you knew
where the interactions lie,

876
00:48:11,190 --> 00:48:14,010
you'd like to toggle
them for the last time

877
00:48:14,010 --> 00:48:15,840
as late as possible.

878
00:48:15,840 --> 00:48:18,300
But I guess I don't really
believe that people would

879
00:48:18,300 --> 00:48:21,570
be able to identify which
two interactions are going

880
00:48:21,570 --> 00:48:28,150
to be involved, so I ruled that
out and didn't study it much.

881
00:48:28,150 --> 00:48:31,830
But the point is that, even if
you didn't know where they lie,

882
00:48:31,830 --> 00:48:36,930
the benefits continue to
accrue almost linearly.

883
00:48:36,930 --> 00:48:39,000
Even though you're
exploiting fewer and fewer

884
00:48:39,000 --> 00:48:40,620
new interactions
as you go, you see

885
00:48:40,620 --> 00:48:46,920
the white ones left to benefit
are diminishing over time.

886
00:48:46,920 --> 00:48:51,270
But because their likelihood
of coming in your favor

887
00:48:51,270 --> 00:48:54,450
is increasing, you go
up almost linearly,

888
00:48:54,450 --> 00:48:59,390
and you get almost
to the end, 80%.

889
00:48:59,390 --> 00:49:02,630
You can almost to one,
well, pretty close,

890
00:49:02,630 --> 00:49:04,280
And what's interesting
to us about this

891
00:49:04,280 --> 00:49:08,350
is take the case of 7 factors.

892
00:49:08,350 --> 00:49:12,760
The adaptive one factor at
a time, assuming two levels,

893
00:49:12,760 --> 00:49:15,640
that takes 8 experiments.

894
00:49:15,640 --> 00:49:19,480
The full factorial takes 128.

895
00:49:19,480 --> 00:49:24,240
So you've looked at a very small
fraction of the overall space,

896
00:49:24,240 --> 00:49:26,460
and you've gotten 80% of
the way toward the best

897
00:49:26,460 --> 00:49:28,680
possible outcome in the space.

898
00:49:28,680 --> 00:49:32,710
That to us is interesting.

899
00:49:32,710 --> 00:49:37,270
And then if I want to compare
the use of resolution 3 designs

900
00:49:37,270 --> 00:49:41,980
and adaptive designs, we
have expressions for each.

901
00:49:41,980 --> 00:49:44,380
And then to make the
comparison straight,

902
00:49:44,380 --> 00:49:46,360
one against the other,
we do the following.

903
00:49:46,360 --> 00:49:51,540
Just hold them up against
one another, look at--

904
00:49:51,540 --> 00:49:54,370
we see the experimental error
is plotted parametrically.

905
00:49:54,370 --> 00:49:59,070
So let's look at just
the crossing point when

906
00:49:59,070 --> 00:50:02,920
experimental error is about
the size of main effect.

907
00:50:02,920 --> 00:50:07,090
They cross at the point
where interactions are

908
00:50:07,090 --> 00:50:08,770
about a quarter of main effect.

909
00:50:08,770 --> 00:50:10,600
The reason we think
that one is interesting

910
00:50:10,600 --> 00:50:13,930
is that I alluded
to earlier the idea

911
00:50:13,930 --> 00:50:17,290
that we are doing these studies,
big studies of complex systems

912
00:50:17,290 --> 00:50:19,120
and regularities.

913
00:50:19,120 --> 00:50:22,670
And one of the things we
found is that interactions

914
00:50:22,670 --> 00:50:25,790
being about a quarter of the
size of main effects, that's

915
00:50:25,790 --> 00:50:27,530
pretty typical.

916
00:50:27,530 --> 00:50:28,680
That's a typical amount.

917
00:50:28,680 --> 00:50:31,400
So if you go into a factory,
and you get your engineers

918
00:50:31,400 --> 00:50:35,360
to list a number of factors
they're interested in studying.

919
00:50:35,360 --> 00:50:37,660
And then you run
a big experiment

920
00:50:37,660 --> 00:50:39,980
and study some of
the interactions too,

921
00:50:39,980 --> 00:50:42,450
you will generally find they're
about a quarter of the size

922
00:50:42,450 --> 00:50:44,460
of the main effects.

923
00:50:44,460 --> 00:50:47,580
It's just a reasonably
reliable regularity,

924
00:50:47,580 --> 00:50:49,590
and so we think that's typical.

925
00:50:49,590 --> 00:50:51,720
And the way I would
interpret the graph

926
00:50:51,720 --> 00:50:54,090
is that, if your
system is pretty

927
00:50:54,090 --> 00:50:57,750
typical in this regard,
strength of interactions,

928
00:50:57,750 --> 00:51:01,800
you should be doing the adaptive
experiments whenever error

929
00:51:01,800 --> 00:51:05,640
is less than main effects.

930
00:51:05,640 --> 00:51:08,360
So Cuthbert Daniels
criterion for demarcation

931
00:51:08,360 --> 00:51:10,105
was error should
be about a third

932
00:51:10,105 --> 00:51:12,230
or a quarter of main effects,
and we're saying now,

933
00:51:12,230 --> 00:51:16,330
it's actually more like the
size of main effects, not

934
00:51:16,330 --> 00:51:17,590
a third or a quarter.

935
00:51:17,590 --> 00:51:22,360
That's assuming, importantly,
that your main goal is just

936
00:51:22,360 --> 00:51:25,090
getting improvement, knowing
exactly where they came from

937
00:51:25,090 --> 00:51:29,070
or estimating
effects accurately.

938
00:51:29,070 --> 00:51:32,310
Now, our primary reason
for going into all this

939
00:51:32,310 --> 00:51:36,790
in the first place was to deal
with robust design in the end.

940
00:51:36,790 --> 00:51:41,370
And so we attempted lots of
different ways of combining

941
00:51:41,370 --> 00:51:44,460
the noise factors with
the control factors,

942
00:51:44,460 --> 00:51:47,730
and so far, the simplest thing
turned out to be the best.

943
00:51:47,730 --> 00:51:53,030
And that is what you see here
in this cuboidal representation

944
00:51:53,030 --> 00:51:56,600
are controllable
factors A, B, and C

945
00:51:56,600 --> 00:51:59,060
and our noise factors,
little a, b, and c,

946
00:51:59,060 --> 00:52:02,570
are being crossed in effect.

947
00:52:02,570 --> 00:52:04,070
We take the adaptive procedure.

948
00:52:04,070 --> 00:52:08,070
We run a factorial
design in the noise.

949
00:52:08,070 --> 00:52:10,790
Then, change one of the
control factors, just one,

950
00:52:10,790 --> 00:52:14,530
and run again the
exact same design.

951
00:52:14,530 --> 00:52:18,490
And then what we do is we select
our level, in this case of A,

952
00:52:18,490 --> 00:52:23,350
on the basis of our preference
for this set of observations

953
00:52:23,350 --> 00:52:24,490
versus this set.

954
00:52:24,490 --> 00:52:28,000
If we like this set because,
for example, it has lower

955
00:52:28,000 --> 00:52:31,320
variation, then we select it.

956
00:52:31,320 --> 00:52:33,580
We select A as
positive, and then

957
00:52:33,580 --> 00:52:35,890
we just continue to wind
our way through the space,

958
00:52:35,890 --> 00:52:38,530
reversing changes
that we don't like

959
00:52:38,530 --> 00:52:42,620
and adopting ones that we do.

960
00:52:42,620 --> 00:52:44,680
And we study this, this says--

961
00:52:44,680 --> 00:52:46,390
it actually appeared
in ASME journal.

962
00:52:46,390 --> 00:52:48,100
I've got to change that.

963
00:52:52,400 --> 00:52:55,600
So we began to
validate this approach

964
00:52:55,600 --> 00:53:00,200
by applying it to a number
of different systems.

965
00:53:00,200 --> 00:53:03,280
One that might be of interest
to you is sheet metal spinning.

966
00:53:03,280 --> 00:53:05,290
So in this process, you
go into your factory,

967
00:53:05,290 --> 00:53:08,050
you take sheet metal
articles, and you turn them

968
00:53:08,050 --> 00:53:12,610
into manufactured goods,
surfaces of revolution,

969
00:53:12,610 --> 00:53:15,190
cups, and bells and
whatever you need.

970
00:53:15,190 --> 00:53:18,760
By taking the circular
blank, pressing it up

971
00:53:18,760 --> 00:53:24,580
against a mandrel repetitively,
until you get the deformation

972
00:53:24,580 --> 00:53:25,600
that you want.

973
00:53:25,600 --> 00:53:29,860
And you might be interested,
for example, in consistency

974
00:53:29,860 --> 00:53:32,320
of the geometry,
and you might be

975
00:53:32,320 --> 00:53:35,740
interested in how a number
of different parameters

976
00:53:35,740 --> 00:53:37,630
affect that consistency
of the geometry.

977
00:53:37,630 --> 00:53:41,530
Such as what material you
chose, the shape of the mandrel,

978
00:53:41,530 --> 00:53:50,130
the shape of the path which
you use, the number of times

979
00:53:50,130 --> 00:53:53,100
and the force you apply.

980
00:53:53,100 --> 00:53:56,850
And this was studied
by a group in Germany,

981
00:53:56,850 --> 00:54:00,600
and we used their
results to simulate

982
00:54:00,600 --> 00:54:03,810
the process of running
these adaptive experiments

983
00:54:03,810 --> 00:54:06,740
and running an
alternative across the ray

984
00:54:06,740 --> 00:54:08,160
design on these systems.

985
00:54:08,160 --> 00:54:11,820
And we found that the
Taguchi-style crossed

986
00:54:11,820 --> 00:54:14,850
array made indeed improvements
in signal to noise

987
00:54:14,850 --> 00:54:17,790
ratio of the system, but
our adaptive procedures

988
00:54:17,790 --> 00:54:19,740
were doing a little
better, until you

989
00:54:19,740 --> 00:54:23,400
got to some crossing point,
an experimental error up here

990
00:54:23,400 --> 00:54:27,280
of 2 millimeters squared
of this quality measure.

991
00:54:27,280 --> 00:54:30,580
And this is where we found
another result, the one that I

992
00:54:30,580 --> 00:54:32,020
alluded to earlier.

993
00:54:32,020 --> 00:54:37,060
That if you gave the engineers
the benefit of the doubt

994
00:54:37,060 --> 00:54:40,300
and suggested that they didn't
choose their starting points

995
00:54:40,300 --> 00:54:44,120
at random, but instead choose
informed starting points.

996
00:54:44,120 --> 00:54:46,970
And let me define
what I mean by that.

997
00:54:46,970 --> 00:54:50,890
Let's say, you have
a number of factors.

998
00:54:50,890 --> 00:54:54,040
They're all at two levels,
a random starting point

999
00:54:54,040 --> 00:54:55,690
is just flipping coins.

1000
00:54:55,690 --> 00:54:56,830
You choose each one.

1001
00:54:56,830 --> 00:54:58,810
But let's say that
somehow the engineers

1002
00:54:58,810 --> 00:55:01,780
are able to do better than
50% odds of getting the best

1003
00:55:01,780 --> 00:55:06,320
setting for each
factor, but instead have

1004
00:55:06,320 --> 00:55:13,370
75% chance of defining the
superior level of each factor.

1005
00:55:13,370 --> 00:55:17,820
In that case in
the end they would

1006
00:55:17,820 --> 00:55:22,040
get this result,
substantially higher

1007
00:55:22,040 --> 00:55:23,790
than either of the
other two alternatives,

1008
00:55:23,790 --> 00:55:28,940
and they'll never cross, no
matter how much error there is.

1009
00:55:28,940 --> 00:55:29,750
Yes?

1010
00:55:29,750 --> 00:55:31,792
AUDIENCE: Why is it saying
the smaller the better

1011
00:55:31,792 --> 00:55:34,748
on the [INAUDIBLE]

1012
00:55:35,863 --> 00:55:36,530
PROFESSOR: Yeah.

1013
00:55:36,530 --> 00:55:44,330
So smaller the better is
the parameter of interest

1014
00:55:44,330 --> 00:55:47,220
was meant to be
smaller is better.

1015
00:55:47,220 --> 00:55:49,680
But then, when you compute
a signal to noise ratio,

1016
00:55:49,680 --> 00:55:51,040
larger it's better.

1017
00:55:51,040 --> 00:55:52,830
So the underlying
parameter, called

1018
00:55:52,830 --> 00:55:56,380
A20 in this case,
smaller was better.

1019
00:55:56,380 --> 00:55:58,240
But in Taguchi's
methodology, you then

1020
00:55:58,240 --> 00:56:01,970
take that parameter
and its variation

1021
00:56:01,970 --> 00:56:05,030
and transform it to get
a signal to noise ratio,

1022
00:56:05,030 --> 00:56:08,820
and with signal to noise
ratio, larger is always better.

1023
00:56:08,820 --> 00:56:10,719
So I appreciate why
that's confusing.

1024
00:56:15,710 --> 00:56:17,698
This one I included
in the paper,

1025
00:56:17,698 --> 00:56:18,740
and I include it for you.

1026
00:56:18,740 --> 00:56:22,720
Because in some cases,
you'll feel skeptical

1027
00:56:22,720 --> 00:56:24,490
about these results,
because they still

1028
00:56:24,490 --> 00:56:26,500
are pretty counterintuitive.

1029
00:56:26,500 --> 00:56:29,615
And you might need to
demonstrate them to yourself,

1030
00:56:29,615 --> 00:56:31,490
and the paper airplane
allows you to do that.

1031
00:56:31,490 --> 00:56:34,240
So Steve [? Effinger ?]
came up with this template

1032
00:56:34,240 --> 00:56:37,480
for demonstrating
some of the methods

1033
00:56:37,480 --> 00:56:40,000
here in robust design and
design of experiments.

1034
00:56:40,000 --> 00:56:42,040
And the template
allows you to fold

1035
00:56:42,040 --> 00:56:44,960
a variety of different paper
airplanes, in this case,

1036
00:56:44,960 --> 00:56:49,600
it's 3 to the 4th, or 81
different paper airplanes.

1037
00:56:49,600 --> 00:56:52,660
And we ran an
experiment in which

1038
00:56:52,660 --> 00:56:56,020
we had created different noise
conditions for the flights,

1039
00:56:56,020 --> 00:56:58,780
and you could
replicate our results.

1040
00:56:58,780 --> 00:57:01,870
Actually throw these airplanes
and do it adaptively one factor

1041
00:57:01,870 --> 00:57:04,390
at a time, and I think
you'll find the same results

1042
00:57:04,390 --> 00:57:05,350
as we did.

1043
00:57:05,350 --> 00:57:08,920
That if you do the
L9 Taguchi array,

1044
00:57:08,920 --> 00:57:11,920
or else that's a
[? Plackit-Berman ?] design

1045
00:57:11,920 --> 00:57:15,820
by another name, cross with a
2 to the 3 minus 1 noise array,

1046
00:57:15,820 --> 00:57:19,210
you get these kind of
improvements in signal to noise

1047
00:57:19,210 --> 00:57:21,290
ratio.

1048
00:57:21,290 --> 00:57:24,550
But if you run it adaptively
with random starting points,

1049
00:57:24,550 --> 00:57:26,260
you'll get better results.

1050
00:57:26,260 --> 00:57:28,600
And if you use an
informed starting point,

1051
00:57:28,600 --> 00:57:33,220
such as looking at a plane
that has about the right aspect

1052
00:57:33,220 --> 00:57:36,640
ratio, and maybe
you know right ahead

1053
00:57:36,640 --> 00:57:39,490
that folding up the
little winglets out here

1054
00:57:39,490 --> 00:57:45,180
is probably better
for lateral stability.

1055
00:57:45,180 --> 00:57:48,700
If you make such judgments,
you'll get even better results,

1056
00:57:48,700 --> 00:57:51,440
and so that's what the paper
airplane study tells us.

1057
00:57:51,440 --> 00:57:54,900
So we did four such
studies, and here's a point

1058
00:57:54,900 --> 00:57:55,900
that I find interesting.

1059
00:57:55,900 --> 00:57:56,400
OK?

1060
00:57:56,400 --> 00:58:01,530
First of all, if you're to lower
error states, what you find

1061
00:58:01,530 --> 00:58:06,295
is that, on average, the
random approach, starting

1062
00:58:06,295 --> 00:58:07,920
at random starting
points, works better

1063
00:58:07,920 --> 00:58:09,420
than the fractional factorial.

1064
00:58:09,420 --> 00:58:11,340
Informed starting points
are better than that.

1065
00:58:11,340 --> 00:58:15,150
If you raise the error high
enough, things turn around,

1066
00:58:15,150 --> 00:58:18,060
and the factorial design
is a little better

1067
00:58:18,060 --> 00:58:20,940
than the adaptive approach,
unless we use the informed

1068
00:58:20,940 --> 00:58:22,980
starting point.

1069
00:58:22,980 --> 00:58:25,800
But what I think is
further interesting

1070
00:58:25,800 --> 00:58:29,040
is that if you look
at the interquartile--

1071
00:58:29,040 --> 00:58:32,330
if you look at the range
of results across cases,

1072
00:58:32,330 --> 00:58:35,720
the adaptive procedure
even at high error,

1073
00:58:35,720 --> 00:58:44,430
to me that's a preferable
range, 51 to 87 versus 36 to 88.

1074
00:58:44,430 --> 00:58:47,010
I think you actually
take less risk

1075
00:58:47,010 --> 00:58:49,160
from case to case with
the adaptive method.

1076
00:58:51,870 --> 00:58:55,560
The factorial
design is optimally

1077
00:58:55,560 --> 00:59:01,950
suited to minimize the
influence of experimental error

1078
00:59:01,950 --> 00:59:04,280
on your outcomes.

1079
00:59:04,280 --> 00:59:08,090
But some of the uncertainties
you face in experimentation

1080
00:59:08,090 --> 00:59:09,680
aren't that kind.

1081
00:59:09,680 --> 00:59:11,180
They're not experimental error.

1082
00:59:11,180 --> 00:59:14,450
They're actually uncertainties
about where the interactions

1083
00:59:14,450 --> 00:59:18,970
lie, and in fact,
fractional factorial designs

1084
00:59:18,970 --> 00:59:21,130
are among the most
sensitive designs

1085
00:59:21,130 --> 00:59:23,540
to that particular uncertainty.

1086
00:59:23,540 --> 00:59:26,390
Because when you
make this confounding

1087
00:59:26,390 --> 00:59:27,980
between main effects
and interactions,

1088
00:59:27,980 --> 00:59:30,160
it's actually very
harmful to your outcomes.

1089
00:59:36,020 --> 00:59:38,740
So those are the
principle results

1090
00:59:38,740 --> 00:59:41,970
that I meant to show you today.

1091
00:59:41,970 --> 00:59:43,530
There are a couple
other things that

1092
00:59:43,530 --> 00:59:47,130
have emerged more recently
that I can talk about.

1093
00:59:47,130 --> 00:59:49,850
For example,
recently, we've been

1094
00:59:49,850 --> 00:59:54,960
interested in the idea
of producing ensembles.

1095
00:59:54,960 --> 00:59:58,530
Now, here, the
situation is that you

1096
00:59:58,530 --> 01:00:03,450
might find that you have an
adequate budget to run a larger

1097
01:00:03,450 --> 01:00:05,220
fractional factorial design.

1098
01:00:05,220 --> 01:00:08,280
Let's say instead of
a 2 to the 7 minus 4,

1099
01:00:08,280 --> 01:00:12,195
you actually have a budget to
run a much larger experiment, 2

1100
01:00:12,195 --> 01:00:13,920
to the 7 minus 2.

1101
01:00:13,920 --> 01:00:18,180
So the counterargument
from some folks in industry

1102
01:00:18,180 --> 01:00:21,210
was that they rarely ran
resolution 3 designs.

1103
01:00:21,210 --> 01:00:23,760
They thought that that
was too big a risk anyway.

1104
01:00:23,760 --> 01:00:26,880
They were tending to run
larger experiments, such as 2

1105
01:00:26,880 --> 01:00:29,512
to the 7 minus 2
higher resolution

1106
01:00:29,512 --> 01:00:30,720
fractional factorial designs.

1107
01:00:30,720 --> 01:00:32,320
And they said, so
because of that,

1108
01:00:32,320 --> 01:00:34,950
I don't think we should
do the adaptive OFAT,

1109
01:00:34,950 --> 01:00:38,730
and we were interested in
that condition, that issue.

1110
01:00:38,730 --> 01:00:42,240
It turns out that, if you take
the same budget that it takes

1111
01:00:42,240 --> 01:00:45,690
to do the 2 to the
7 minus 2, you would

1112
01:00:45,690 --> 01:00:48,500
be able to run four OFATs.

1113
01:00:48,500 --> 01:00:51,860
Now, if you do
that, if you choose

1114
01:00:51,860 --> 01:00:55,100
four different starting points
and four different orderings

1115
01:00:55,100 --> 01:00:59,290
of the factors and run
four OFATs, and then

1116
01:00:59,290 --> 01:01:04,020
take those results and
make an ensemble of them.

1117
01:01:04,020 --> 01:01:06,420
And for example, a
simple thing you could do

1118
01:01:06,420 --> 01:01:10,250
is just pick the
best of the four.

1119
01:01:10,250 --> 01:01:15,050
Again, the ensemble would
produce a better result

1120
01:01:15,050 --> 01:01:16,615
than the fractional
factorial design.

1121
01:01:16,615 --> 01:01:17,990
And then the
interesting thing we

1122
01:01:17,990 --> 01:01:21,410
find about it is, remember
before, when you increase

1123
01:01:21,410 --> 01:01:26,590
experimental error, you
expect the OFAT to eventually

1124
01:01:26,590 --> 01:01:28,720
degrade in performance
and to cross

1125
01:01:28,720 --> 01:01:30,350
with the fractional factorial.

1126
01:01:30,350 --> 01:01:33,340
So now, the issue is how
much experimental error

1127
01:01:33,340 --> 01:01:36,700
can I endure before the
adaptive experiment is

1128
01:01:36,700 --> 01:01:39,080
no longer recommended?

1129
01:01:39,080 --> 01:01:42,020
With the ensemble method, it's
actually the opposite way.

1130
01:01:42,020 --> 01:01:43,820
As you increase
experimental error,

1131
01:01:43,820 --> 01:01:47,990
eventually, the distance
between the ensemble

1132
01:01:47,990 --> 01:01:51,810
and the fractional
factorial increases.

1133
01:01:51,810 --> 01:01:58,710
So the ensemble method by
itself is providing a robustness

1134
01:01:58,710 --> 01:02:01,600
to experimental error.

1135
01:02:01,600 --> 01:02:07,350
And so we've been
able to address

1136
01:02:07,350 --> 01:02:12,810
another one of the counterpoints
to the adaptive experimentation

1137
01:02:12,810 --> 01:02:14,640
idea.

1138
01:02:14,640 --> 01:02:17,700
That in those cases where
experimental budgets are

1139
01:02:17,700 --> 01:02:23,170
higher, you might still
want to run these.

1140
01:02:23,170 --> 01:02:25,192
The other benefit
of the ensembles

1141
01:02:25,192 --> 01:02:26,900
we've been able to
show in another paper.

1142
01:02:26,900 --> 01:02:28,317
I didn't put some
slides in, but I

1143
01:02:28,317 --> 01:02:30,010
want to tell you a
little bit about it.

1144
01:02:30,010 --> 01:02:31,510
Unless you have a question.

1145
01:02:31,510 --> 01:02:32,768
Maybe I'll take your question.

1146
01:02:32,768 --> 01:02:33,310
AUDIENCE: OK.

1147
01:02:33,310 --> 01:02:39,050
So what are some
practical issues

1148
01:02:39,050 --> 01:02:41,280
that you face influencing
one factor at a time.

1149
01:02:41,280 --> 01:02:42,780
For example, I can
think in my mind,

1150
01:02:42,780 --> 01:02:46,420
say you can't do the
measurement immediately

1151
01:02:46,420 --> 01:02:47,800
after each experiment.

1152
01:02:47,800 --> 01:02:48,400
Right?

1153
01:02:48,400 --> 01:02:48,790
PROFESSOR: That's right.

1154
01:02:48,790 --> 01:02:51,220
AUDIENCE: In that case,
you would probably still

1155
01:02:51,220 --> 01:02:53,620
have to go to some
fractional factorial design.

1156
01:02:53,620 --> 01:02:56,540
Are there other issues
that-- like practically,

1157
01:02:56,540 --> 01:02:58,510
because this is incredible.

1158
01:02:58,510 --> 01:03:01,473
It seems like it could save a
lot of people a lot of time.

1159
01:03:01,473 --> 01:03:02,140
PROFESSOR: Yeah.

1160
01:03:02,140 --> 01:03:03,182
So you make a good point.

1161
01:03:03,182 --> 01:03:06,190
For example, we
know in agriculture,

1162
01:03:06,190 --> 01:03:09,760
you want to run 64 different
treatment conditions,

1163
01:03:09,760 --> 01:03:12,980
and it takes a season
to get your results.

1164
01:03:12,980 --> 01:03:15,130
So you want to run
them in parallel.

1165
01:03:15,130 --> 01:03:16,540
It's important to
run experiments

1166
01:03:16,540 --> 01:03:21,460
in parallel in agriculture,
and in agricultural equipment,

1167
01:03:21,460 --> 01:03:23,950
it's exactly the opposite.

1168
01:03:23,950 --> 01:03:26,100
So what happens is
you're trying to develop

1169
01:03:26,100 --> 01:03:28,350
the next generation of tractor.

1170
01:03:28,350 --> 01:03:31,350
And what they do is they take
the last generation of tractor

1171
01:03:31,350 --> 01:03:35,540
that they built, and they use
that as what they call a mule.

1172
01:03:35,540 --> 01:03:37,920
In the automotive industry,
they use the same term.

1173
01:03:37,920 --> 01:03:39,630
You're making the next Taurus.

1174
01:03:39,630 --> 01:03:42,870
You take the 2007
Taurus, and you

1175
01:03:42,870 --> 01:03:46,260
start using it, putting
additions on it--

1176
01:03:46,260 --> 01:03:49,840
new fuel injection, new
valve timing, and so on.

1177
01:03:49,840 --> 01:03:53,640
You use that as a mule, and
when you're using articles

1178
01:03:53,640 --> 01:03:57,210
like that in that style, and you
have a limited number of them,

1179
01:03:57,210 --> 01:04:01,160
actually, your experiments
are necessarily sequential.

1180
01:04:01,160 --> 01:04:05,260
Whereas, in agriculture,
they're necessarily parallel.

1181
01:04:05,260 --> 01:04:07,620
So I actually agree
with you, when

1182
01:04:07,620 --> 01:04:09,300
you're in an
engineering scenario,

1183
01:04:09,300 --> 01:04:12,240
and your experiments are
necessarily parallel.

1184
01:04:12,240 --> 01:04:14,760
It sometimes happens,
in say lithography.

1185
01:04:14,760 --> 01:04:16,275
You're going to etch it.

1186
01:04:16,275 --> 01:04:18,150
You're going to have to
etch all these things

1187
01:04:18,150 --> 01:04:20,370
all at the same time in a batch.

1188
01:04:20,370 --> 01:04:21,390
Yeah.

1189
01:04:21,390 --> 01:04:23,850
That's a reason to
do it the other way,

1190
01:04:23,850 --> 01:04:27,090
but I think more often than
not, we're doing our experiments

1191
01:04:27,090 --> 01:04:29,910
necessarily
sequentially, or maybe

1192
01:04:29,910 --> 01:04:33,860
it's OK to do it sequentially.

1193
01:04:33,860 --> 01:04:37,550
Another factor, since you
asked, is the possibility

1194
01:04:37,550 --> 01:04:43,450
of time trends, and
indeed, when you

1195
01:04:43,450 --> 01:04:47,800
do adaptive experimentation,
you are necessarily making

1196
01:04:47,800 --> 01:04:49,450
restrictions on randomization.

1197
01:04:49,450 --> 01:04:50,620
Right?

1198
01:04:50,620 --> 01:04:53,800
So let's say that you're
measuring apparatus

1199
01:04:53,800 --> 01:04:55,420
is drifting over time.

1200
01:04:55,420 --> 01:04:57,730
Let's say it's
drifting toward making

1201
01:04:57,730 --> 01:04:59,020
all your results look worse.

1202
01:05:01,540 --> 01:05:04,750
That would give you
somewhat of a bias

1203
01:05:04,750 --> 01:05:08,165
toward your starting points,
as opposed to all the leader

1204
01:05:08,165 --> 01:05:08,665
changes.

1205
01:05:11,620 --> 01:05:15,590
As best we can tell, actually,
the adaptive procedure

1206
01:05:15,590 --> 01:05:19,300
is not so hypersensitive
to those time trends.

1207
01:05:19,300 --> 01:05:21,610
They reflect themselves
to some degree

1208
01:05:21,610 --> 01:05:24,740
as if they were additional
experimental error.

1209
01:05:24,740 --> 01:05:27,790
But if you randomize the
starting points anyway

1210
01:05:27,790 --> 01:05:30,760
and the order of
factors, I don't

1211
01:05:30,760 --> 01:05:33,383
think it's worse than
experimental error.

1212
01:05:33,383 --> 01:05:35,800
Because you're not interested
in factor effects in the end

1213
01:05:35,800 --> 01:05:36,700
anyway.

1214
01:05:36,700 --> 01:05:42,160
This is probably the biggest
concern I think with the method

1215
01:05:42,160 --> 01:05:43,870
as I'm describing it.

1216
01:05:43,870 --> 01:05:47,830
The biggest issue is you're
making an explicit trade-off

1217
01:05:47,830 --> 01:05:52,480
to say, what I'm interested in
doing in this case is getting

1218
01:05:52,480 --> 01:05:54,010
some improvement.

1219
01:05:54,010 --> 01:05:57,970
And I'm willing to make a change
in my experimental procedure

1220
01:05:57,970 --> 01:06:00,400
that will give me somewhat
less knowledge about why

1221
01:06:00,400 --> 01:06:01,900
I got the improvement.

1222
01:06:01,900 --> 01:06:04,720
Now, if you're at
Ford, and you've

1223
01:06:04,720 --> 01:06:07,210
got three months
to product release,

1224
01:06:07,210 --> 01:06:11,980
and you need to meet
your emission standards.

1225
01:06:15,510 --> 01:06:20,580
The goal is to reduce carbon
monoxide in those three months,

1226
01:06:20,580 --> 01:06:22,500
and you think the
learning that you're

1227
01:06:22,500 --> 01:06:25,800
going to gain about
this particular engine

1228
01:06:25,800 --> 01:06:29,130
and its configuration
is not likely to be

1229
01:06:29,130 --> 01:06:32,700
reusable on the next one.

1230
01:06:32,700 --> 01:06:36,600
Then, in that case, I think the
adaptive method makes sense.

1231
01:06:36,600 --> 01:06:40,500
If instead you think the
primary value of your experiment

1232
01:06:40,500 --> 01:06:41,760
is archival.

1233
01:06:41,760 --> 01:06:42,330
Right?

1234
01:06:42,330 --> 01:06:43,710
You're doing the
experiment, you're

1235
01:06:43,710 --> 01:06:45,085
going to take the
results, you're

1236
01:06:45,085 --> 01:06:47,310
going to make them available
to all Ford engineers,

1237
01:06:47,310 --> 01:06:49,320
and you're going to
benefit on the next model

1238
01:06:49,320 --> 01:06:50,460
and the next model.

1239
01:06:50,460 --> 01:06:53,670
Then, yeah, you probably ought
to arrange the experiments,

1240
01:06:53,670 --> 01:06:57,270
so that you get the
highest precision

1241
01:06:57,270 --> 01:07:00,780
of effect estimation, the
highest validity of inference.

1242
01:07:00,780 --> 01:07:04,680
That's factorial
design of experiments.

1243
01:07:04,680 --> 01:07:09,300
But we had an interesting
learning along the way

1244
01:07:09,300 --> 01:07:11,070
and that is to do
our big meta studies,

1245
01:07:11,070 --> 01:07:13,320
where we're trying to decide
how big are interactions?

1246
01:07:13,320 --> 01:07:14,400
When do they occur?

1247
01:07:14,400 --> 01:07:16,770
We went into Ford's databases.

1248
01:07:16,770 --> 01:07:19,980
They had run lots of
experiments in the past,

1249
01:07:19,980 --> 01:07:24,510
and they had them in
some company database.

1250
01:07:24,510 --> 01:07:27,690
And we were drawing these
data sets out to put them

1251
01:07:27,690 --> 01:07:29,610
into one of our studies.

1252
01:07:29,610 --> 01:07:32,850
And very frequently, we'd
go through the table,

1253
01:07:32,850 --> 01:07:35,340
and there was some
ambiguity in our minds.

1254
01:07:35,340 --> 01:07:38,210
Maybe there was a plus in the
experimental matrix, where

1255
01:07:38,210 --> 01:07:39,960
we thought there should
have been a minus,

1256
01:07:39,960 --> 01:07:41,580
or there's some question.

1257
01:07:41,580 --> 01:07:45,450
And in those cases, we would
go track the person down

1258
01:07:45,450 --> 01:07:48,000
and ask them what they meant.

1259
01:07:48,000 --> 01:07:50,685
And in almost every case,
that person would say,

1260
01:07:50,685 --> 01:07:53,400
huh, you're the first
person who ever called me

1261
01:07:53,400 --> 01:07:54,990
about my experiment.

1262
01:07:54,990 --> 01:07:59,370
They'd put it in the database,
and no one was using it.

1263
01:07:59,370 --> 01:08:02,730
We were the first ones
to go back in and use it.

1264
01:08:02,730 --> 01:08:06,870
So we are not so sanguine
about the prospects of people

1265
01:08:06,870 --> 01:08:10,230
reusing experimental
data, at least so far

1266
01:08:10,230 --> 01:08:15,660
as industrial, day-to-day
experimentation is concerned.

1267
01:08:15,660 --> 01:08:19,560
Most often, those experiments
serve their purposes

1268
01:08:19,560 --> 01:08:25,340
at the time, and then they're
gone, for practical purposes.

1269
01:08:25,340 --> 01:08:28,040
AUDIENCE: On that note, I was
working for a startup company,

1270
01:08:28,040 --> 01:08:31,460
and we would use a facility
that we knew other larger

1271
01:08:31,460 --> 01:08:33,340
corporations would use.

1272
01:08:33,340 --> 01:08:35,080
And we had a much
smaller budget,

1273
01:08:35,080 --> 01:08:41,490
and I wouldn't to say that
my mind [INAUDIBLE] we

1274
01:08:41,490 --> 01:08:43,750
didn't go in with a
full factorial design,

1275
01:08:43,750 --> 01:08:46,680
because but we couldn't, but
we had certain objectives,

1276
01:08:46,680 --> 01:08:47,770
certain targets.

1277
01:08:47,770 --> 01:08:50,880
And so we go in and say, all
right, look at the results

1278
01:08:50,880 --> 01:08:54,260
vary and then change
the parameters.

1279
01:08:58,380 --> 01:09:01,229
I think your method has to
be more educated than that,

1280
01:09:01,229 --> 01:09:04,470
but it almost seems like
it's a similar mentality.

1281
01:09:04,470 --> 01:09:05,640
PROFESSOR: It is.

1282
01:09:05,640 --> 01:09:09,569
Yeah, and in fact, I never
formally tried to study this.

1283
01:09:09,569 --> 01:09:15,920
But if you back off a little
bit from our structure

1284
01:09:15,920 --> 01:09:18,470
and apply some
different structure, one

1285
01:09:18,470 --> 01:09:20,899
that suits your
circumstances, it's

1286
01:09:20,899 --> 01:09:24,750
probably still pretty good.

1287
01:09:24,750 --> 01:09:26,880
It might even be better.

1288
01:09:26,880 --> 01:09:32,370
The bigger point is that you're
doing adaptation as you go.

1289
01:09:32,370 --> 01:09:35,300
And I have been asked the
same question, in fact,

1290
01:09:35,300 --> 01:09:37,830
Jeff Wu asked me at the last
time I presented a conference.

1291
01:09:37,830 --> 01:09:42,899
He asked, how much of this
result is due to adaptation,

1292
01:09:42,899 --> 01:09:45,720
and how much is due to the
one factor at a time part?

1293
01:09:45,720 --> 01:09:47,100
Because this is both.

1294
01:09:47,100 --> 01:09:48,000
Right?

1295
01:09:48,000 --> 01:09:50,569
And it's a hard
question to answer.

1296
01:09:50,569 --> 01:09:54,440
My sense is it's
mostly adaptation.

1297
01:09:54,440 --> 01:09:59,570
On the other hand, one factor
at a time in some sense

1298
01:09:59,570 --> 01:10:01,550
optimizes adaptation.

1299
01:10:01,550 --> 01:10:06,980
You can be adaptive all the
time with every new observation

1300
01:10:06,980 --> 01:10:08,550
as much as possible.

1301
01:10:08,550 --> 01:10:12,530
So it's not as if I can
cleanly decomposed the two

1302
01:10:12,530 --> 01:10:14,960
contributions to the
result, but my sense

1303
01:10:14,960 --> 01:10:16,880
is that it's mostly adaptation.

1304
01:10:16,880 --> 01:10:19,790
And if you're doing some
other adaptive procedure,

1305
01:10:19,790 --> 01:10:22,670
especially if it's informed
by some prior knowledge

1306
01:10:22,670 --> 01:10:27,720
or topical area knowledge,
it's probably a good approach.

1307
01:10:33,860 --> 01:10:35,860
So I'll just tell you
about this one last thing.

1308
01:10:35,860 --> 01:10:37,240
OK?

1309
01:10:37,240 --> 01:10:41,930
We did an investigation about
another kind of uncertainty

1310
01:10:41,930 --> 01:10:43,180
in large systems.

1311
01:10:43,180 --> 01:10:47,630
So let's say that you're
developing the next tractor

1312
01:10:47,630 --> 01:10:52,970
or the next car, and
you think about applying

1313
01:10:52,970 --> 01:10:55,100
some robust design
to the system.

1314
01:10:55,100 --> 01:10:59,790
Now, for any reasonable
scale product,

1315
01:10:59,790 --> 01:11:02,100
there are literally
thousands of things

1316
01:11:02,100 --> 01:11:04,740
which you could potentially
apply a robust design

1317
01:11:04,740 --> 01:11:05,970
experiment to.

1318
01:11:05,970 --> 01:11:08,040
You can run one
on fuel injection.

1319
01:11:08,040 --> 01:11:13,830
You can run one on braking and
one on environmental controls

1320
01:11:13,830 --> 01:11:16,090
and one on electrical,
just keep going,

1321
01:11:16,090 --> 01:11:17,590
and you're not going
to do them all.

1322
01:11:17,590 --> 01:11:22,830
In fact, I would say less
than 5% of all opportunities

1323
01:11:22,830 --> 01:11:29,630
to do robustness improvement
work are actually executed.

1324
01:11:29,630 --> 01:11:33,470
And then the question I always
ask myself is I wonder if--

1325
01:11:33,470 --> 01:11:35,210
let's say, it's 5%--

1326
01:11:35,210 --> 01:11:38,043
I wonder if companies
are doing the right 5%.

1327
01:11:38,043 --> 01:11:40,460
I wonder if they know where
their biggest quality problems

1328
01:11:40,460 --> 01:11:41,690
are.

1329
01:11:41,690 --> 01:11:45,800
Now, we know for a fact that,
if you look at retrospectively

1330
01:11:45,800 --> 01:11:47,630
where there have been
big quality problems,

1331
01:11:47,630 --> 01:11:51,250
if I look at Ford Explorer
and the rollover and so on.

1332
01:11:51,250 --> 01:11:52,160
Right?

1333
01:11:52,160 --> 01:11:54,020
I know they didn't
do a robust design

1334
01:11:54,020 --> 01:12:02,540
study on tire inflation and
its delamination and rollover.

1335
01:12:02,540 --> 01:12:04,032
They didn't do it.

1336
01:12:04,032 --> 01:12:05,990
Now, why didn't they know
that that was the one

1337
01:12:05,990 --> 01:12:07,280
to do robust design on?

1338
01:12:07,280 --> 01:12:09,010
Well, they just didn't.

1339
01:12:09,010 --> 01:12:10,660
It was not so likely after all.

1340
01:12:10,660 --> 01:12:14,020
It wasn't happening
to other SUVs.

1341
01:12:14,020 --> 01:12:18,450
So I ask myself,
if you could run

1342
01:12:18,450 --> 01:12:25,910
a relatively expensive
robust design study on 5%,

1343
01:12:25,910 --> 01:12:27,810
let's say that's the case.

1344
01:12:27,810 --> 01:12:29,270
You have the budget for that.

1345
01:12:29,270 --> 01:12:32,220
What if you did something
a tenth as expensive

1346
01:12:32,220 --> 01:12:33,980
and did on 50% of the system?

1347
01:12:37,590 --> 01:12:39,830
So we would call this
idea streamlining.

1348
01:12:39,830 --> 01:12:44,120
We know how to take any robust
design experiment people are

1349
01:12:44,120 --> 01:12:49,310
proposing and to do
something a little bit loose

1350
01:12:49,310 --> 01:12:53,240
but to do it at a tenth of the
cost, and you'll get about 80%

1351
01:12:53,240 --> 01:12:55,910
of the same benefit.

1352
01:12:55,910 --> 01:12:57,540
The trade-off works like that.

1353
01:12:57,540 --> 01:13:01,650
It's a good old Pareto
80-20 sort of trade-off.

1354
01:13:01,650 --> 01:13:06,060
And we ask ourselves
how that trade-off

1355
01:13:06,060 --> 01:13:10,050
in doing relatively
many relatively less

1356
01:13:10,050 --> 01:13:15,310
perfect experiments compares
to doing a few very good ones.

1357
01:13:15,310 --> 01:13:20,080
And we ask ourselves
how this trade-off

1358
01:13:20,080 --> 01:13:23,500
is affected by the
likelihood of identifying

1359
01:13:23,500 --> 01:13:26,020
the one or two biggest quality
problems that you face.

1360
01:13:26,020 --> 01:13:29,790
Because that rollover
thing cost Ford a lot.

1361
01:13:29,790 --> 01:13:33,290
You would pay a lot of
money to avoid that one.

1362
01:13:33,290 --> 01:13:37,720
And so the big
conclusion of our study

1363
01:13:37,720 --> 01:13:40,300
is you would have to have
a pretty high probability

1364
01:13:40,300 --> 01:13:43,277
of knowing exactly where
your problems were in order

1365
01:13:43,277 --> 01:13:44,860
to do it the way
they do it now, which

1366
01:13:44,860 --> 01:13:48,290
is relatively small number
of expensive studies.

1367
01:13:48,290 --> 01:13:50,980
You'd have to know
like 95% certainty

1368
01:13:50,980 --> 01:13:53,200
where your biggest
quality problem is.

1369
01:13:53,200 --> 01:13:56,030
If you are at least
a little uncertain,

1370
01:13:56,030 --> 01:14:00,370
if you think you're only 70%
sure where your problems lie,

1371
01:14:00,370 --> 01:14:05,440
scale them all back by a factor
of 10, and do 10 times as many.

1372
01:14:05,440 --> 01:14:06,940
You'll be much better off.

1373
01:14:06,940 --> 01:14:10,040
That's my view,
and we didn't have

1374
01:14:10,040 --> 01:14:13,800
to make so many assumptions
to come to this conclusion.

1375
01:14:13,800 --> 01:14:16,680
People's uncertainty about
where their biggest issues lie

1376
01:14:16,680 --> 01:14:19,230
are bigger than they
think, and therefore, they

1377
01:14:19,230 --> 01:14:25,540
need to democratize
these kind of processes.

1378
01:14:25,540 --> 01:14:30,160
It's not that you
have to do them very,

1379
01:14:30,160 --> 01:14:32,610
very well in a few cases.

1380
01:14:32,610 --> 01:14:35,220
What you need to do is make
sure every engineer knows

1381
01:14:35,220 --> 01:14:37,740
how to do this kind of
thing, and that they're

1382
01:14:37,740 --> 01:14:43,340
doing it to almost everything.

1383
01:14:43,340 --> 01:14:46,170
At least almost everything
that, for example,

1384
01:14:46,170 --> 01:14:48,130
is not proven in
the field already.

1385
01:14:48,130 --> 01:14:49,880
You don't need to do
it on the alternator,

1386
01:14:49,880 --> 01:14:52,580
because you used that same
alternator on the last three

1387
01:14:52,580 --> 01:14:54,400
models, and it's fine.

1388
01:14:54,400 --> 01:14:56,170
But for anything
that's relatively new,

1389
01:14:56,170 --> 01:14:59,740
you need to be doing some of
this robustness refinement

1390
01:14:59,740 --> 01:15:00,370
work.

1391
01:15:00,370 --> 01:15:03,370
And even if you do
just a few experiments,

1392
01:15:03,370 --> 01:15:07,660
exposing your systems
to harsh conditions,

1393
01:15:07,660 --> 01:15:09,283
making changes,
making improvements

1394
01:15:09,283 --> 01:15:10,700
on the basis of
what that reveals,

1395
01:15:10,700 --> 01:15:14,890
that's the key thing, not so
much the finesse you apply.

1396
01:15:14,890 --> 01:15:16,570
Anyway, that's the
conclusion we came

1397
01:15:16,570 --> 01:15:21,580
to so far due to this research.

1398
01:15:21,580 --> 01:15:27,420
So our main
conclusions are that we

1399
01:15:27,420 --> 01:15:29,760
can show through empirical
work and through theorems

1400
01:15:29,760 --> 01:15:33,120
that this adaptive
procedure gives benefits.

1401
01:15:33,120 --> 01:15:35,970
You get a long way
toward your results

1402
01:15:35,970 --> 01:15:39,270
that are desired, especially
when interactions are not

1403
01:15:39,270 --> 01:15:40,940
so small.

1404
01:15:40,940 --> 01:15:43,490
And that you can cross
these adaptive experiments

1405
01:15:43,490 --> 01:15:45,350
with fractional
factorial designs

1406
01:15:45,350 --> 01:15:49,570
to use them for robustness, and
it seems to work pretty well.

1407
01:15:49,570 --> 01:15:53,680
And now, I think I'm
pretty much out of time.

1408
01:15:53,680 --> 01:15:57,400
Now, if you all have ideas
for your case studies,

1409
01:15:57,400 --> 01:16:00,490
I'm always interested in
head-to-head comparisons

1410
01:16:00,490 --> 01:16:03,130
of experimental
designs being done

1411
01:16:03,130 --> 01:16:05,320
in companies and the
adaptive procedures

1412
01:16:05,320 --> 01:16:07,510
that I'm talking about now.

1413
01:16:07,510 --> 01:16:10,590
So if you're interested in doing
such things for your projects,

1414
01:16:10,590 --> 01:16:14,190
I'm interested in
helping you to do that.

1415
01:16:14,190 --> 01:16:15,980
And you can see my
email there, in case

1416
01:16:15,980 --> 01:16:18,920
you want to take
me up on the offer.

1417
01:16:18,920 --> 01:16:20,630
OK?

1418
01:16:20,630 --> 01:16:21,530
All set?

1419
01:16:21,530 --> 01:16:22,160
All right.

1420
01:16:22,160 --> 01:16:23,100
Good day.

1421
01:16:23,100 --> 01:16:26,530
[APPLAUSE]

1422
01:16:53,562 --> 01:16:57,440
--Problems that we run into
quite often in microfabrication

1423
01:16:57,440 --> 01:17:01,070
is that the metrology of
the things that we've made

1424
01:17:01,070 --> 01:17:04,520
takes up quite a lot
of our time and budget.

1425
01:17:04,520 --> 01:17:07,190
It's quite easy to vary
parameters and make

1426
01:17:07,190 --> 01:17:08,030
many samples.

1427
01:17:08,030 --> 01:17:10,058
But choosing which
ones to measure

1428
01:17:10,058 --> 01:17:11,600
might be the biggest
challenge, and I

1429
01:17:11,600 --> 01:17:16,580
wonder whether you've
looked at situations, where

1430
01:17:16,580 --> 01:17:19,940
the acquisition of the
data lags, the doing

1431
01:17:19,940 --> 01:17:23,030
of the experiments, but
it's sort of concurrent,

1432
01:17:23,030 --> 01:17:26,870
but you can only measure
a certain portion

1433
01:17:26,870 --> 01:17:30,332
of the experiments you do.

1434
01:17:30,332 --> 01:17:34,370
Are there more complicated
situations like that

1435
01:17:34,370 --> 01:17:36,200
where there's a lag?

1436
01:17:36,200 --> 01:17:39,155
But it's not the case that
you do all the experiments,

1437
01:17:39,155 --> 01:17:41,503
and then you do all
the measurements.

1438
01:17:41,503 --> 01:17:43,170
PROFESSOR: It's an
interesting question.

1439
01:17:43,170 --> 01:17:48,240
So you're saying that
you can do an experiment.

1440
01:17:48,240 --> 01:17:49,250
There's a time lag.

1441
01:17:49,250 --> 01:17:52,370
Then you get an outcome.

1442
01:17:52,370 --> 01:17:55,430
And in some cases, it's actually
more complicated than that.

1443
01:17:55,430 --> 01:17:57,890
You get some
indication right away,

1444
01:17:57,890 --> 01:18:01,135
but the more preferred
measurement comes later.

1445
01:18:01,135 --> 01:18:02,510
So for example,
sometimes there's

1446
01:18:02,510 --> 01:18:05,360
a categorical variable that
becomes immediately obvious.

1447
01:18:05,360 --> 01:18:07,610
Either I heard
chatter, or I didn't.

1448
01:18:07,610 --> 01:18:09,500
And then later, I get
a nice measurement

1449
01:18:09,500 --> 01:18:13,100
of surface condition,
something like that.

1450
01:18:13,100 --> 01:18:16,700
Well, any of these
delays tend to weigh

1451
01:18:16,700 --> 01:18:19,790
in favor of the factorial design
as compared to the adaptive.

1452
01:18:19,790 --> 01:18:21,620
You just admit that right away.

1453
01:18:21,620 --> 01:18:23,650
But then I'll say
that sometimes,

1454
01:18:23,650 --> 01:18:27,830
even with categorical
variables, immediately available

1455
01:18:27,830 --> 01:18:31,880
perceptions, you can get a lot
of the benefit of adaptation

1456
01:18:31,880 --> 01:18:33,620
from that.

1457
01:18:33,620 --> 01:18:37,460
PROFESSOR: Yeah, OK,
thank you very much.

1458
01:18:37,460 --> 01:18:38,730
Anybody else?

1459
01:18:38,730 --> 01:18:44,120
So we've arranged to have
an extra half hour today

1460
01:18:44,120 --> 01:18:46,625
where I can answer
questions about problem

1461
01:18:46,625 --> 01:18:49,100
sets and any questions
that you might

1462
01:18:49,100 --> 01:18:51,120
have in the run-up to the quiz.

1463
01:18:51,120 --> 01:18:58,380
So I think what we'll do is
just go straight into that.

1464
01:18:58,380 --> 01:19:02,025
If people want to run off at
this point, then that's fine.

1465
01:19:05,493 --> 01:19:07,618
AUDIENCE: Are you going to
[INAUDIBLE] the solution

1466
01:19:07,618 --> 01:19:09,215
of the problem sets [INAUDIBLE]?

1467
01:19:09,215 --> 01:19:09,840
PROFESSOR: Yes.

1468
01:19:09,840 --> 01:19:10,620
Yes.

1469
01:19:10,620 --> 01:19:16,050
So there's a few
housekeeping things first.

1470
01:19:16,050 --> 01:19:18,210
Yeah, so problem sets, I will--

1471
01:19:18,210 --> 01:19:25,710
I'll put the solutions for
6 and 7 today at some point.

1472
01:19:25,710 --> 01:19:31,380
And I put the 2006 quiz, too,
and solutions on the website.

1473
01:19:31,380 --> 01:19:33,690
I'm still trying to track
down last year's quiz.

1474
01:19:33,690 --> 01:19:37,080
But hopefully that
will be out today.

1475
01:19:37,080 --> 01:19:41,430
And yes, for problem
set 8, I know

1476
01:19:41,430 --> 01:19:46,000
some people have been
sending me questions,

1477
01:19:46,000 --> 01:19:49,560
which I'm happy to continue
to answer by email.

1478
01:19:49,560 --> 01:19:53,010
I'll be around this afternoon
if you want to come and speak

1479
01:19:53,010 --> 01:19:55,170
to me in person.

1480
01:19:55,170 --> 01:20:02,070
And yes, last week, a few
people asked me for extensions.

1481
01:20:02,070 --> 01:20:02,723
That's fine.

1482
01:20:02,723 --> 01:20:04,890
If you need an extension,
you can have an extension.

1483
01:20:04,890 --> 01:20:06,690
I'm not super strict
about these things.

1484
01:20:09,450 --> 01:20:16,860
OK, so really, the floor is
open for you to ask me things.

1485
01:20:16,860 --> 01:20:19,038
And oh, yes grades for
6 and 7 will hopefully

1486
01:20:19,038 --> 01:20:20,580
be done within the
next day, as well.

1487
01:20:23,115 --> 01:20:23,990
AUDIENCE: [INAUDIBLE]

1488
01:20:23,990 --> 01:20:26,490
PROFESSOR: Yes?

1489
01:20:26,490 --> 01:20:29,020
Go ahead.

1490
01:20:29,020 --> 01:20:32,590
AUDIENCE: So we have a question.

1491
01:20:32,590 --> 01:20:37,420
When you compare the Taguchi
method and a surface response,

1492
01:20:37,420 --> 01:20:39,625
this is for problem 4.

1493
01:20:39,625 --> 01:20:42,670
But you want to compare
the number of experiments

1494
01:20:42,670 --> 01:20:45,510
it takes for both methods.

1495
01:20:45,510 --> 01:20:49,020
Are you considering the
quadratic terms [INAUDIBLE]??

1496
01:20:49,020 --> 01:20:50,170
Or are you're not?

1497
01:20:52,930 --> 01:20:56,860
PROFESSOR: In question
4 on problems set 8,

1498
01:20:56,860 --> 01:20:58,220
we're talking about here?

1499
01:20:58,220 --> 01:20:59,032
Yes?

1500
01:20:59,032 --> 01:21:01,240
AUDIENCE: That's right.

1501
01:21:01,240 --> 01:21:05,290
PROFESSOR: I think that
it's good to not include

1502
01:21:05,290 --> 01:21:08,152
the quadratic terms
in that question,

1503
01:21:08,152 --> 01:21:09,610
although you might
want to consider

1504
01:21:09,610 --> 01:21:11,926
what would happen if you did.

1505
01:21:11,926 --> 01:21:14,992
AUDIENCE: OK, so we're only
considering the linear terms

1506
01:21:14,992 --> 01:21:15,700
and interactions?

1507
01:21:15,700 --> 01:21:17,117
PROFESSOR: And
interactions, yeah.

1508
01:21:17,117 --> 01:21:18,758
I think that's right.

1509
01:21:18,758 --> 01:21:19,300
AUDIENCE: OK.

1510
01:21:19,300 --> 01:21:20,770
PROFESSOR: I think that's
the right way forward.

1511
01:21:20,770 --> 01:21:21,100
Yeah?

1512
01:21:21,100 --> 01:21:22,892
AUDIENCE: Only two
[INAUDIBLE] interactions

1513
01:21:22,892 --> 01:21:25,900
or three or four [INAUDIBLE]?

1514
01:21:25,900 --> 01:21:28,480
PROFESSOR: Yeah,
so with this, we're

1515
01:21:28,480 --> 01:21:31,540
not really interested
in quadratic factors.

1516
01:21:31,540 --> 01:21:34,844
Just look at the linear
factors in the interactions.

1517
01:21:37,510 --> 01:21:39,700
OK?

1518
01:21:39,700 --> 01:21:42,241
More questions?

1519
01:21:42,241 --> 01:21:44,230
AUDIENCE: Can I ask
one last question?

1520
01:21:44,230 --> 01:21:46,110
PROFESSOR: Sure.

1521
01:21:46,110 --> 01:21:47,765
AUDIENCE: So
actually, [INAUDIBLE]

1522
01:21:47,765 --> 01:21:50,890
we are kind of confused
about question 2, problem 2.

1523
01:21:50,890 --> 01:21:51,630
PROFESSOR: Yeah, everyone's
confused about that.

1524
01:21:51,630 --> 01:21:51,880
[LAUGHS]

1525
01:21:51,880 --> 01:21:54,047
AUDIENCE: So can you give
us a little [INAUDIBLE] we

1526
01:21:54,047 --> 01:21:55,420
are looking for in question 2?

1527
01:21:55,420 --> 01:21:56,170
PROFESSOR: Yeah.

1528
01:21:56,170 --> 01:22:00,140
So I wrote this question in an
attempt to stretch everyone,

1529
01:22:00,140 --> 01:22:01,340
including myself.

1530
01:22:01,340 --> 01:22:05,620
So it's pretty subtle, I think.

1531
01:22:05,620 --> 01:22:08,290
But I would say the
right way to approach

1532
01:22:08,290 --> 01:22:15,670
this is to recast
that model in terms

1533
01:22:15,670 --> 01:22:20,750
of differences between the
variables and the mean input.

1534
01:22:20,750 --> 01:22:29,140
So if you have a set
of data, and those data

1535
01:22:29,140 --> 01:22:33,160
have a mean value for x1
and a mean value for x2,

1536
01:22:33,160 --> 01:22:39,520
so you might write it with
some new coefficients a--

1537
01:22:47,782 --> 01:22:52,020
x2 minus x-- sorry,
x1 bar, x2 bar.

1538
01:22:59,030 --> 01:23:05,480
And that will be for some values
of the a's that are functions

1539
01:23:05,480 --> 01:23:06,020
of the b's.

1540
01:23:06,020 --> 01:23:09,390
That will be equivalent.

1541
01:23:09,390 --> 01:23:13,700
And then, if it was just
the first two terms,

1542
01:23:13,700 --> 01:23:16,130
that would be
exactly the case that

1543
01:23:16,130 --> 01:23:17,630
was derived in Mayo & Spanos.

1544
01:23:20,930 --> 01:23:24,260
And then this term looks
very much like this term,

1545
01:23:24,260 --> 01:23:26,970
except it's for a
different variable.

1546
01:23:26,970 --> 01:23:32,490
So the standard error
of that coefficient,

1547
01:23:32,490 --> 01:23:35,850
you can get at by the same
way you got at this one.

1548
01:23:35,850 --> 01:23:37,970
And then if you
look at this term,

1549
01:23:37,970 --> 01:23:41,660
well, we have this
coefficient here.

1550
01:23:41,660 --> 01:23:46,040
Well, if you're trying to
evaluate the variance of y hat,

1551
01:23:46,040 --> 01:23:48,740
you've got a product
of two quantities.

1552
01:23:52,010 --> 01:23:57,820
Do you have an expression for
the variance of a11, which

1553
01:23:57,820 --> 01:24:00,880
you're going to get at
in a similar way to how

1554
01:24:00,880 --> 01:24:04,300
you get at the expressions
for these variances.

1555
01:24:04,300 --> 01:24:06,550
And you'll see the expression
for the standard error

1556
01:24:06,550 --> 01:24:09,340
is a function of--

1557
01:24:09,340 --> 01:24:13,480
is going to be a function
of x1 minus x1 bar.

1558
01:24:13,480 --> 01:24:16,630
And you know that
when you're finding

1559
01:24:16,630 --> 01:24:23,320
the invariance of some
constant times something

1560
01:24:23,320 --> 01:24:28,210
that you know the variance
of, you square that constant,

1561
01:24:28,210 --> 01:24:30,700
multiply it by the
variance of the thing

1562
01:24:30,700 --> 01:24:33,910
that the constant
was multiplied by.

1563
01:24:33,910 --> 01:24:36,370
AUDIENCE: Should it x1
star or x1 [INAUDIBLE]

1564
01:24:36,370 --> 01:24:37,780
PROFESSOR: Oh, sorry, yes.

1565
01:24:37,780 --> 01:24:41,520
Yes, these are-- exactly.

1566
01:24:41,520 --> 01:24:43,480
There you go.

1567
01:24:43,480 --> 01:24:48,820
So this is-- yes,
you're trying to find

1568
01:24:48,820 --> 01:24:55,180
an expression for the
confidence interval

1569
01:24:55,180 --> 01:24:59,560
as a function of x1
star and x2 star.

1570
01:24:59,560 --> 01:25:03,610
But at a given at a given
combination of x1 star

1571
01:25:03,610 --> 01:25:09,130
and x2 star, this is the
thing that has variance.

1572
01:25:09,130 --> 01:25:12,590
And this, for a fixed
x1 star, is a constant.

1573
01:25:12,590 --> 01:25:20,380
So I don't want to
give too much away.

1574
01:25:20,380 --> 01:25:22,520
And I think this
is the right way.

1575
01:25:22,520 --> 01:25:25,280
There may be some subtleties
that have escaped me.

1576
01:25:25,280 --> 01:25:29,390
So if you can think
of any, let me know.

1577
01:25:29,390 --> 01:25:33,400
I think because there aren't any
interaction terms in our model,

1578
01:25:33,400 --> 01:25:36,970
I think that means
that covariances

1579
01:25:36,970 --> 01:25:39,400
don't need to concern you.

1580
01:25:39,400 --> 01:25:42,010
But if you think I'm wrong
on that, let me know.

1581
01:25:45,120 --> 01:25:45,660
OK?

1582
01:25:45,660 --> 01:25:51,550
AUDIENCE: So speaking of the
covariance, but x1 and x1

1583
01:25:51,550 --> 01:25:55,270
square, these two
are correlated.

1584
01:25:55,270 --> 01:25:58,195
So the covariance-- you have
to consider that one, right?

1585
01:26:01,920 --> 01:26:04,350
How could you ignore
that? x1 and x2,

1586
01:26:04,350 --> 01:26:05,970
they are not correlated--

1587
01:26:05,970 --> 01:26:09,280
x1 and x1 square.

1588
01:26:09,280 --> 01:26:11,460
PROFESSOR: Yeah.

1589
01:26:11,460 --> 01:26:13,120
Yeah, maybe I'm wrong.

1590
01:26:13,120 --> 01:26:14,930
Well, if you want to--

1591
01:26:14,930 --> 01:26:19,364
if you can work out how to
consider that, then tell me.

1592
01:26:19,364 --> 01:26:24,150
AUDIENCE: OK, but in
terms of the results, can

1593
01:26:24,150 --> 01:26:25,920
we just use an x--

1594
01:26:25,920 --> 01:26:27,630
the matrix as a general--

1595
01:26:27,630 --> 01:26:29,790
we don't need to break
down the matrix, right?

1596
01:26:29,790 --> 01:26:31,758
PROFESSOR: No, I
don't think so, no.

1597
01:26:31,758 --> 01:26:32,300
AUDIENCE: OK.

1598
01:26:38,587 --> 01:26:39,545
PROFESSOR: Anyone else?

1599
01:26:43,520 --> 01:26:45,440
Have we run out of coffee?

1600
01:26:45,440 --> 01:26:46,950
Yes, I expect so.

1601
01:26:50,330 --> 01:26:51,260
Let's see.

1602
01:26:51,260 --> 01:26:55,610
What are the things that
people are most puzzled

1603
01:26:55,610 --> 01:27:02,540
by in the material since quiz 1?

1604
01:27:02,540 --> 01:27:08,780
I mean, if someone
asked you to fit

1605
01:27:08,780 --> 01:27:13,640
a model to a
factorial experiment,

1606
01:27:13,640 --> 01:27:18,200
and you didn't have Minitab
what level of confidence would

1607
01:27:18,200 --> 01:27:20,780
people have in doing that?

1608
01:27:20,780 --> 01:27:23,510
I mean, is it really something
where you rely on the structure

1609
01:27:23,510 --> 01:27:27,670
that the software provides
to know how to calculate some

1610
01:27:27,670 --> 01:27:30,590
of the squares and so forth?

1611
01:27:30,590 --> 01:27:35,120
Because I can foresee
exam questions--

1612
01:27:35,120 --> 01:27:36,620
obviously not going
to have Minitab.

1613
01:27:36,620 --> 01:27:39,920
We're going to be dealing with
very small numbers of data,

1614
01:27:39,920 --> 01:27:42,450
and it's just going to
be a case of knowing what

1615
01:27:42,450 --> 01:27:43,700
to subtract from what, really.

1616
01:27:43,700 --> 01:27:44,930
AUDIENCE: [INAUDIBLE] practice.

1617
01:27:44,930 --> 01:27:46,097
PROFESSOR: Yeah, absolutely.

1618
01:27:46,097 --> 01:27:48,970
AUDIENCE: [INAUDIBLE] with the
[INAUDIBLE],, you could do it.

1619
01:27:48,970 --> 01:27:50,720
But we won't
[INAUDIBLE] practice

1620
01:27:50,720 --> 01:27:54,750
doing it versus using JMP or
Minitab [INAUDIBLE] get used

1621
01:27:54,750 --> 01:27:57,390
to the data being [INAUDIBLE].

1622
01:27:57,390 --> 01:27:59,385
PROFESSOR: Yes.

1623
01:27:59,385 --> 01:28:02,613
AUDIENCE: I think it would
be more time-consuming.

1624
01:28:02,613 --> 01:28:03,530
PROFESSOR: Fair point.

1625
01:28:03,530 --> 01:28:05,240
AUDIENCE: [INAUDIBLE]

1626
01:28:05,240 --> 01:28:06,815
PROFESSOR: Yeah,
I need to I need

1627
01:28:06,815 --> 01:28:08,690
to track it down, because
they didn't post it

1628
01:28:08,690 --> 01:28:09,830
on Stellar last year.

1629
01:28:09,830 --> 01:28:13,190
So I need to get
it up [INAUDIBLE]..

1630
01:28:13,190 --> 01:28:13,970
Yes?

1631
01:28:13,970 --> 01:28:15,490
Who's that?

1632
01:28:15,490 --> 01:28:17,750
AUDIENCE: [INAUDIBLE]
question 4-- can you

1633
01:28:17,750 --> 01:28:22,310
explain more about the noise?

1634
01:28:22,310 --> 01:28:24,480
You mean that you cannot
[INAUDIBLE] during

1635
01:28:24,480 --> 01:28:25,070
an experiment?

1636
01:28:25,070 --> 01:28:29,473
[INAUDIBLE]

1637
01:28:29,473 --> 01:28:30,140
PROFESSOR: Sure.

1638
01:28:30,140 --> 01:28:32,390
Sure, yeah, this
question's a bit cryptic.

1639
01:28:32,390 --> 01:28:34,520
But really, all it's
asking you to do

1640
01:28:34,520 --> 01:28:38,000
is confirm your understanding
of the last question

1641
01:28:38,000 --> 01:28:39,530
on problem set 7.

1642
01:28:39,530 --> 01:28:45,170
You may recall
from that question

1643
01:28:45,170 --> 01:28:50,390
where you made that factor
z, that noise factor,

1644
01:28:50,390 --> 01:28:53,810
you were imagining that
you could control it

1645
01:28:53,810 --> 01:28:55,130
in that situation.

1646
01:28:55,130 --> 01:29:00,980
But then I think you will have
written an expression for y

1647
01:29:00,980 --> 01:29:04,430
in terms of the two
X's and the z factor

1648
01:29:04,430 --> 01:29:11,780
that showed that the sensitivity
of the output to variations

1649
01:29:11,780 --> 01:29:15,860
in z were a function
of x1 and x2.

1650
01:29:15,860 --> 01:29:18,530
So you could choose
a combination

1651
01:29:18,530 --> 01:29:22,670
of the control
parameters at x1 and x2

1652
01:29:22,670 --> 01:29:26,660
that would minimize
the sensitivity to z.

1653
01:29:26,660 --> 01:29:30,290
Now, really parts B
and C of problem 4

1654
01:29:30,290 --> 01:29:34,250
are just sort of checking
that that went in,

1655
01:29:34,250 --> 01:29:41,870
because if you can't control z,
but you know that if you're not

1656
01:29:41,870 --> 01:29:44,870
controlling it, it's probably
going to have a variance,

1657
01:29:44,870 --> 01:29:47,240
the input, the noise
input, is probably

1658
01:29:47,240 --> 01:29:50,040
going to have a variance that
doesn't change with time.

1659
01:29:50,040 --> 01:29:53,690
So you can see how much
of that noise propagates

1660
01:29:53,690 --> 01:29:57,710
through to the outputs
if the noise interacts

1661
01:29:57,710 --> 01:30:00,210
with the control input
variables-- in other words,

1662
01:30:00,210 --> 01:30:03,170
if there are these
product terms of z and x1.

1663
01:30:03,170 --> 01:30:04,130
You can't control z.

1664
01:30:04,130 --> 01:30:05,600
You can control x1.

1665
01:30:05,600 --> 01:30:09,170
Therefore, you can control
how much the noise propagates

1666
01:30:09,170 --> 01:30:10,550
through to the outputs.

1667
01:30:10,550 --> 01:30:15,530
And B and C aren't asking
you to write down very much.

1668
01:30:15,530 --> 01:30:18,740
It's just sort of checking that
the concept is there, really.

1669
01:30:22,330 --> 01:30:25,420
Does that answer your question?

1670
01:30:25,420 --> 01:30:26,170
AUDIENCE: Oh, yes.

1671
01:30:26,170 --> 01:30:26,430
Thank you.

1672
01:30:26,430 --> 01:30:27,180
PROFESSOR: Thanks.

1673
01:30:40,010 --> 01:30:41,450
OK, anyone else?

1674
01:30:41,450 --> 01:30:42,183
Dave?

1675
01:30:42,183 --> 01:30:47,570
AUDIENCE: Do you have an example
of a potential modeled problem

1676
01:30:47,570 --> 01:30:50,308
that might be of a
smaller data set that

1677
01:30:50,308 --> 01:30:52,100
might be a little easier
to go through some

1678
01:30:52,100 --> 01:30:56,513
of the calculations of fitting
a model to the experiment?

1679
01:30:56,513 --> 01:30:58,680
I know a lot of our [INAUDIBLE]
a little bit bigger,

1680
01:30:58,680 --> 01:31:01,710
and it may be useful to be
able to work something through.

1681
01:31:01,710 --> 01:31:03,620
PROFESSOR: That's
a very good idea.

1682
01:31:03,620 --> 01:31:06,200
Yes.

1683
01:31:06,200 --> 01:31:06,740
Yeah.

1684
01:31:06,740 --> 01:31:08,270
I'll try to come
up with something

1685
01:31:08,270 --> 01:31:12,650
that's sort of manageable
on a sheet of paper--

1686
01:31:12,650 --> 01:31:16,060
yes, very, very good thought.

1687
01:31:16,060 --> 01:31:19,940
We may be able to do
with some of the data

1688
01:31:19,940 --> 01:31:21,350
sets we used to look at ANOVA.

1689
01:31:21,350 --> 01:31:24,746
But yeah, fine.

1690
01:31:24,746 --> 01:31:31,090
AUDIENCE: I had some confusion
with problem [INAUDIBLE]

1691
01:31:31,090 --> 01:31:37,781
the lack of [INAUDIBLE]
I was having

1692
01:31:37,781 --> 01:31:41,212
some trouble with [INAUDIBLE]

1693
01:31:41,212 --> 01:31:41,920
PROFESSOR: Right.

1694
01:31:41,920 --> 01:31:42,420
Right.

1695
01:31:42,420 --> 01:31:45,854
AUDIENCE: [INAUDIBLE] if
you could do [INAUDIBLE]

1696
01:31:45,854 --> 01:31:47,970
PROFESSOR: [INAUDIBLE]
points, OK.

1697
01:31:47,970 --> 01:31:48,970
Fine, fine.

1698
01:31:48,970 --> 01:31:50,460
So that was on problem set 7.

1699
01:31:50,460 --> 01:31:53,003
You were at 12-1 or
something, but yeah.

1700
01:31:53,003 --> 01:31:54,420
AUDIENCE: I don't
know [INAUDIBLE]

1701
01:31:54,420 --> 01:31:55,128
PROFESSOR: Right.

1702
01:31:55,128 --> 01:31:56,890
No, that's a good point.

1703
01:31:56,890 --> 01:32:00,130
A lot of people had
difficulties with that.

1704
01:32:00,130 --> 01:32:07,110
So that I will look
at, as well, OK?

1705
01:32:07,110 --> 01:32:12,872
AUDIENCE: [INAUDIBLE] problem
with something [INAUDIBLE]

1706
01:32:12,872 --> 01:32:13,580
PROFESSOR: Sorry?

1707
01:32:13,580 --> 01:32:14,455
Something [INAUDIBLE]

1708
01:32:14,455 --> 01:32:20,990
AUDIENCE: For example,
[INAUDIBLE] minus 1 [INAUDIBLE]

1709
01:32:20,990 --> 01:32:21,800
PROFESSOR: OK.

1710
01:32:21,800 --> 01:32:25,190
AUDIENCE: [INAUDIBLE]
some examples.

1711
01:32:25,190 --> 01:32:33,953
[INAUDIBLE] data versus
[INAUDIBLE] so you don't have

1712
01:32:33,953 --> 01:32:39,800
enough information [INAUDIBLE]
I don't know why, but that's--

1713
01:32:39,800 --> 01:32:44,350
PROFESSOR: Yeah, I think there
is a mistake in Montgomery.

1714
01:32:44,350 --> 01:32:45,350
I need to track it down.

1715
01:32:45,350 --> 01:32:51,740
AUDIENCE: [INAUDIBLE] makes
sense [INAUDIBLE] look at that.

1716
01:32:51,740 --> 01:32:54,870
[INAUDIBLE] you don't
really need to add all data

1717
01:32:54,870 --> 01:32:57,860
and calculate all contrasts.

1718
01:32:57,860 --> 01:33:01,510
So there is some [INAUDIBLE]

1719
01:33:01,510 --> 01:33:03,380
AUDIENCE: [INAUDIBLE]

1720
01:33:03,380 --> 01:33:07,974
AUDIENCE: [INAUDIBLE] example
you have [INAUDIBLE] data.

1721
01:33:07,974 --> 01:33:08,968
AUDIENCE: [INAUDIBLE]

1722
01:33:08,968 --> 01:33:13,450
AUDIENCE: [INAUDIBLE]

1723
01:33:13,450 --> 01:33:15,430
PROFESSOR: OK, sure.

1724
01:33:15,430 --> 01:33:16,450
AUDIENCE: I'm not sure.

1725
01:33:16,450 --> 01:33:21,970
On the Mayo &
Spanos, on the 8.1.3,

1726
01:33:21,970 --> 01:33:28,010
[INAUDIBLE] duration of
[INAUDIBLE] of the parameters.

1727
01:33:28,010 --> 01:33:30,485
It seems to me that
[INAUDIBLE] just jumping

1728
01:33:30,485 --> 01:33:33,550
and without [INAUDIBLE].

1729
01:33:33,550 --> 01:33:36,100
PROFESSOR: Yeah, you're right.

1730
01:33:36,100 --> 01:33:41,380
AUDIENCE: 8.1.3 It's a general
form of the [INAUDIBLE]

1731
01:33:41,380 --> 01:33:48,520
regression [INAUDIBLE]
using the matrix [INAUDIBLE]

1732
01:33:48,520 --> 01:33:51,520
PROFESSOR: Yeah, right.

1733
01:33:51,520 --> 01:33:57,820
Well, sure, I can try to
find a complete derivation.

1734
01:33:57,820 --> 01:34:02,437
I think it's probably not
worth spending a lot of time

1735
01:34:02,437 --> 01:34:03,520
trying to understand that.

1736
01:34:03,520 --> 01:34:07,750
I would focus more
on the stuff that's

1737
01:34:07,750 --> 01:34:09,530
been in the problem sets.

1738
01:34:09,530 --> 01:34:11,830
But sure, I can
try to find that.

1739
01:34:11,830 --> 01:34:16,720
AUDIENCE: Because if you can
get a derivation of that,

1740
01:34:16,720 --> 01:34:20,553
any kind of regression
data just changes form.

1741
01:34:20,553 --> 01:34:21,220
PROFESSOR: Yeah.

1742
01:34:21,220 --> 01:34:24,160
AUDIENCE: So there is no need
to do that kind of thing,

1743
01:34:24,160 --> 01:34:26,500
try to change the form of it.

1744
01:34:26,500 --> 01:34:29,127
[INAUDIBLE] just need to fit it.

1745
01:34:29,127 --> 01:34:30,710
PROFESSOR: Yes, I'm
sure you're right.

1746
01:34:30,710 --> 01:34:32,850
I'm sure you're right.