WEBVTT 00:00:00.000 --> 00:00:02.430 The following content is provided under a Creative 00:00:02.430 --> 00:00:03.730 Commons license. 00:00:03.730 --> 00:00:06.030 Your support will help MIT OpenCourseWare 00:00:06.030 --> 00:00:10.060 continue to offer high quality educational resources for free. 00:00:10.060 --> 00:00:12.690 To make a donation or to view additional materials 00:00:12.690 --> 00:00:16.560 from hundreds of MIT courses, visit MIT OpenCourseWare 00:00:16.560 --> 00:00:17.892 at ocw.mit.edu. 00:00:21.030 --> 00:00:24.540 PROFESSOR: Probably the time has begun to begin. 00:00:24.540 --> 00:00:27.240 Welcome. 00:00:27.240 --> 00:00:29.160 I've given a guest lecture in this class 00:00:29.160 --> 00:00:30.450 for a few years running now. 00:00:30.450 --> 00:00:34.620 So I'm always happy to return. 00:00:34.620 --> 00:00:37.560 You're working on a good subject here. 00:00:37.560 --> 00:00:40.050 I'm going to talk today about experimentation 00:00:40.050 --> 00:00:43.380 and robust design of engineering systems. 00:00:43.380 --> 00:00:46.560 I think you've already had a few lectures on design 00:00:46.560 --> 00:00:48.120 of experiments formally. 00:00:48.120 --> 00:00:51.470 So this will build on that. 00:00:51.470 --> 00:00:55.570 The figure that I show on this first slide 00:00:55.570 --> 00:01:00.310 concerns, on the right, a method by which 00:01:00.310 --> 00:01:03.550 one can make improvements in robustness of the system 00:01:03.550 --> 00:01:06.850 by adaptively experimenting in the parameter 00:01:06.850 --> 00:01:08.440 space of that system. 00:01:08.440 --> 00:01:11.230 And the figure on the left is meant 00:01:11.230 --> 00:01:14.350 to indicate why that approach has 00:01:14.350 --> 00:01:19.420 worked rather quite a lot better than people expected. 00:01:19.420 --> 00:01:22.270 And I'll be going into that in some detail, 00:01:22.270 --> 00:01:24.130 and I hope that as I go through all this, 00:01:24.130 --> 00:01:26.428 you'll take time to ask questions. 00:01:26.428 --> 00:01:27.970 Because I think that's better for you 00:01:27.970 --> 00:01:30.790 and more interesting for me too. 00:01:30.790 --> 00:01:35.950 So just to give you a little bit of context, 00:01:35.950 --> 00:01:38.470 this topic of adaptive experimentation 00:01:38.470 --> 00:01:41.090 and robust design is really central to my research. 00:01:41.090 --> 00:01:45.340 It's I think the core of it, but I'm interested in other things 00:01:45.340 --> 00:01:45.980 as well. 00:01:45.980 --> 00:01:50.440 Such as we do studies of systems generally 00:01:50.440 --> 00:01:54.550 and the regularities that exist in those engineered systems. 00:01:54.550 --> 00:01:59.800 And that gives rise to data that we can use more generally 00:01:59.800 --> 00:02:02.450 in methodology validation. 00:02:02.450 --> 00:02:05.080 So we can look at different ways that people 00:02:05.080 --> 00:02:10.030 will do engineering and try to assess its effectiveness. 00:02:10.030 --> 00:02:11.530 And of course, all this is linked 00:02:11.530 --> 00:02:14.830 in with the work on adaptive experimentation, 00:02:14.830 --> 00:02:16.870 because we need a way to validate our work. 00:02:19.870 --> 00:02:21.840 But the main topic today will be what's 00:02:21.840 --> 00:02:23.220 in the middle of that slide. 00:02:23.220 --> 00:02:24.640 That's adaptive experimentation. 00:02:24.640 --> 00:02:29.910 I think I'll start with some history and use that 00:02:29.910 --> 00:02:31.660 to explain the motivation. 00:02:31.660 --> 00:02:33.810 Why is it that we're addressing this topic? 00:02:33.810 --> 00:02:37.095 Why are we addressing it the way we're addressing it? 00:02:37.095 --> 00:02:39.220 And then, I'm going to show you the research itself 00:02:39.220 --> 00:02:43.230 on a depth of experimentation and robust design. 00:02:43.230 --> 00:02:49.140 So I start with this quote from Sir Edward John Russell. 00:02:49.140 --> 00:02:52.060 Now, this fellow was prominent in his day. 00:02:52.060 --> 00:02:52.980 He was knighted. 00:02:52.980 --> 00:02:55.810 That's why it says Sir Edward John Russell there 00:02:55.810 --> 00:02:57.270 under his picture. 00:02:57.270 --> 00:03:01.650 And as late as 1926, he was still 00:03:01.650 --> 00:03:05.730 expressing a view about how to do experimentation. 00:03:05.730 --> 00:03:10.230 He said that you should seek simplicity in the way 00:03:10.230 --> 00:03:13.400 that you conduct your experiments, 00:03:13.400 --> 00:03:16.080 and maybe you should ask just a few questions at a time, 00:03:16.080 --> 00:03:18.500 or maybe even just one question at a time, 00:03:18.500 --> 00:03:21.380 as you're conducting your experiments. 00:03:21.380 --> 00:03:24.950 And this was a prominent view in that they generally accepted, 00:03:24.950 --> 00:03:27.530 and you can trace it back at least as far 00:03:27.530 --> 00:03:30.710 as Francis Bacon and maybe father. 00:03:30.710 --> 00:03:32.750 And what's interesting about this is just 00:03:32.750 --> 00:03:36.200 at the time he's still expressing his views about how 00:03:36.200 --> 00:03:39.260 it is that they conducted experiments there 00:03:39.260 --> 00:03:46.170 at Rothamsted, he hired another person, R.A. Fisher. 00:03:46.170 --> 00:03:49.310 Now, I should say that this is an interesting confluence. 00:03:49.310 --> 00:03:53.525 You have two men, very prominent, very successful, 00:03:53.525 --> 00:03:55.650 coming together in the same place at the same time. 00:03:55.650 --> 00:03:58.790 I think the reason for that was that this experimental station, 00:03:58.790 --> 00:04:02.450 Rothamsted, was addressing I think 00:04:02.450 --> 00:04:04.980 one of the great problems of their time 00:04:04.980 --> 00:04:07.850 which was consistency and efficiency 00:04:07.850 --> 00:04:10.240 of agricultural production. 00:04:10.240 --> 00:04:13.840 If you think about the middle of the 19th century, 00:04:13.840 --> 00:04:15.550 the UK and other parts of the world 00:04:15.550 --> 00:04:18.670 experienced terrible famines. 00:04:18.670 --> 00:04:23.020 And so trying to do much better in this area 00:04:23.020 --> 00:04:26.680 was a pressing problem, probably equivalent today to the way 00:04:26.680 --> 00:04:32.720 we feel about energy supply or global warming. 00:04:32.720 --> 00:04:35.247 And now in some ways, we take for granted food supply 00:04:35.247 --> 00:04:36.580 at least in the developed world. 00:04:39.150 --> 00:04:42.110 So here, we have this confluence and R.A. Fisher 00:04:42.110 --> 00:04:45.590 comes into the lab, and he has a very different view about how 00:04:45.590 --> 00:04:48.640 to do experimentation. 00:04:48.640 --> 00:04:53.980 He analyzes this experiment, in which we see the table 00:04:53.980 --> 00:04:54.880 three in the middle. 00:04:54.880 --> 00:04:59.290 This is from R.A. Fisher's 1921 paper, 00:04:59.290 --> 00:05:03.190 and what you see is a plot with different levels of ammonia 00:05:03.190 --> 00:05:06.160 being applied as fertilizer to the fields. 00:05:06.160 --> 00:05:11.740 And you see the influence on the yield per acre of oats 00:05:11.740 --> 00:05:15.130 and the decrement in each year. 00:05:15.130 --> 00:05:17.320 Because when you apply the fertilizer 00:05:17.320 --> 00:05:20.200 to the same field year after year, 00:05:20.200 --> 00:05:22.690 its effects diminish over time. 00:05:22.690 --> 00:05:26.470 And apparently, Fisher was fairly discouraged 00:05:26.470 --> 00:05:27.580 by this experiment. 00:05:27.580 --> 00:05:30.970 When you read the paper, you see the details, 00:05:30.970 --> 00:05:33.580 and you see the difficulties they experienced 00:05:33.580 --> 00:05:35.080 in analyzing the data. 00:05:35.080 --> 00:05:40.360 They had some pests intruding on parts of the land, 00:05:40.360 --> 00:05:45.700 and this made it complex to draw inferences from the experiment. 00:05:45.700 --> 00:05:51.210 And he was later quoted to say the following thing which 00:05:51.210 --> 00:05:54.150 expressed that frustration in a way. 00:05:54.150 --> 00:05:58.390 You have a bunch of scientists plan an experiment based 00:05:58.390 --> 00:06:00.250 on their knowledge of what to do. 00:06:00.250 --> 00:06:04.300 Later in, you call statisticians in to analyze the data 00:06:04.300 --> 00:06:06.820 and to get the most precise inferences you can 00:06:06.820 --> 00:06:13.120 and precise estimates, and it's really too late. 00:06:13.120 --> 00:06:14.950 The time for statistical thinking 00:06:14.950 --> 00:06:18.050 is before you actually run the experiment. 00:06:18.050 --> 00:06:19.970 That's when you can get the greatest benefits, 00:06:19.970 --> 00:06:24.530 and what he talked about is the idea that the most finesse you 00:06:24.530 --> 00:06:28.520 could apply might improve your analysis of the data 00:06:28.520 --> 00:06:30.932 by a few percent one way or other. 00:06:30.932 --> 00:06:33.140 But if you brought in the experimenter earlier, maybe 00:06:33.140 --> 00:06:39.750 you'd get a factor of 2 or 5 in efficiency and accuracy. 00:06:39.750 --> 00:06:42.390 And you can see how thoughtful Fisher was. 00:06:42.390 --> 00:06:46.945 He seems to be struggling hard to think there, 00:06:46.945 --> 00:06:48.820 and I think part of it is that machine that's 00:06:48.820 --> 00:06:51.670 in front of him, which is I guess what you'd 00:06:51.670 --> 00:06:54.170 use for a computer in that day. 00:06:54.170 --> 00:06:56.850 I guess it's some mechanical adding device, 00:06:56.850 --> 00:07:00.830 and I think it's nice that we have so much better machines 00:07:00.830 --> 00:07:02.210 now for this purpose. 00:07:02.210 --> 00:07:04.100 And in fact, what we find is that you 00:07:04.100 --> 00:07:07.190 can do a lot with statistics these days by knowing how 00:07:07.190 --> 00:07:08.420 to work with computers right. 00:07:08.420 --> 00:07:09.837 And in fact, a lot of the research 00:07:09.837 --> 00:07:14.160 results I'll show you today, we wouldn't 00:07:14.160 --> 00:07:16.830 be able to achieve them with without 00:07:16.830 --> 00:07:21.410 modern digital computers. 00:07:21.410 --> 00:07:25.440 So Fisher begins to plan experiments in a different way, 00:07:25.440 --> 00:07:28.610 and this is one of the first ones in the literature, 00:07:28.610 --> 00:07:31.880 this figure one from his 1926 paper 00:07:31.880 --> 00:07:34.080 on the arrangement of field experiments. 00:07:34.080 --> 00:07:36.813 And if you've done some work already 00:07:36.813 --> 00:07:38.480 in this course on design of experiments, 00:07:38.480 --> 00:07:41.440 you might recognize this. 00:07:41.440 --> 00:07:47.630 You see somewhat indicated by darker lines 00:07:47.630 --> 00:07:52.540 that there are eight different subplots in this plot. 00:07:52.540 --> 00:07:54.890 And if I focus my attention on just 00:07:54.890 --> 00:07:59.790 say the upper left, what I see is-- 00:07:59.790 --> 00:08:02.220 and I think the people even remotely can see the arrow. 00:08:02.220 --> 00:08:03.450 Right? 00:08:03.450 --> 00:08:09.270 You see the plot has three different indications 00:08:09.270 --> 00:08:10.110 in each plot. 00:08:10.110 --> 00:08:12.480 In this case, it's 2, an M, and early. 00:08:12.480 --> 00:08:14.830 In the next one, it's 2 and S and late. 00:08:14.830 --> 00:08:19.530 So the 2 and the 1 indicate an amount 00:08:19.530 --> 00:08:22.500 of a fertilizer that's been applied. 00:08:22.500 --> 00:08:26.650 S and M indicate the type of the fertilizer, 00:08:26.650 --> 00:08:31.240 and early and late describe the time 00:08:31.240 --> 00:08:35.179 in the season, the planting season, when it's applied. 00:08:35.179 --> 00:08:37.640 And we see in this first block, we 00:08:37.640 --> 00:08:44.210 have four of the 2's and four of the 1's and four of the earlys 00:08:44.210 --> 00:08:45.890 and four of the lates and so on. 00:08:45.890 --> 00:08:48.320 And we even see that there is a balance, 00:08:48.320 --> 00:08:52.010 in the sense that there are two of the two M's and two 00:08:52.010 --> 00:08:56.090 of the two S's, so pair-wise, there's a balance and so on. 00:08:56.090 --> 00:08:58.130 And you may recognize that, if I just 00:08:58.130 --> 00:09:00.740 focus on 2 and M and early and late, 00:09:00.740 --> 00:09:04.130 and there are 8 of such blocks in here, 00:09:04.130 --> 00:09:07.010 we have every permutation and combination 00:09:07.010 --> 00:09:09.520 of the factors in the level. 00:09:09.520 --> 00:09:13.550 So I would say, you could describe this as a 2 00:09:13.550 --> 00:09:16.250 to the 3 full factorial experiment 00:09:16.250 --> 00:09:19.390 within the block, which is 8. 00:09:19.390 --> 00:09:27.520 Now, you might also say that it's a 3 raised to the 1 and 2 00:09:27.520 --> 00:09:29.620 raised to the 2, because there are also 00:09:29.620 --> 00:09:33.160 these four plots in which there are just X's, and 00:09:33.160 --> 00:09:35.890 no fertilizers apply. 00:09:35.890 --> 00:09:38.110 You could call that a control, or you 00:09:38.110 --> 00:09:46.420 could say that this is actually a more complex 00:09:46.420 --> 00:09:50.560 factorial experiment, where in one of the levels 00:09:50.560 --> 00:09:52.060 obviates the other two. 00:09:52.060 --> 00:09:54.370 And once you decide to apply no fertilizer, 00:09:54.370 --> 00:09:57.070 it doesn't matter whether you apply type 1 or type 2 00:09:57.070 --> 00:09:59.540 or apply early or late. 00:09:59.540 --> 00:10:06.950 And then, to further complicate the issue, we have eight plots, 00:10:06.950 --> 00:10:10.820 and within those blocks, those subplots, 00:10:10.820 --> 00:10:19.830 we have randomization in space of the different treatments. 00:10:19.830 --> 00:10:23.850 And one of the things that Fisher talks about here 00:10:23.850 --> 00:10:27.320 is how poorly randomization actually works sometimes. 00:10:27.320 --> 00:10:30.900 So if I look down in this plot, it's on the lower right. 00:10:30.900 --> 00:10:35.730 This plot is randomized in spatial distribution 00:10:35.730 --> 00:10:39.660 of the treatments, but you see that all of the left-hand plots 00:10:39.660 --> 00:10:44.520 are early, and all of the right-hand plots are late. 00:10:44.520 --> 00:10:49.790 So sometimes, randomization gives you such problems, 00:10:49.790 --> 00:10:52.100 and in fact, these days, often people 00:10:52.100 --> 00:10:54.950 will not use randomization and instead use 00:10:54.950 --> 00:10:58.370 something like a Latin hypercube design 00:10:58.370 --> 00:11:02.100 rather than randomization. 00:11:02.100 --> 00:11:05.400 But this was one of the early experiments that 00:11:05.400 --> 00:11:08.610 had been considered in a factorial design, 00:11:08.610 --> 00:11:12.480 and I think it's interesting that Fischer decided to call it 00:11:12.480 --> 00:11:14.160 a complex experiment. 00:11:14.160 --> 00:11:17.830 It's as if he was trying to emphasize 00:11:17.830 --> 00:11:20.050 as strongly as possible his difference of opinion 00:11:20.050 --> 00:11:24.580 with Russell, his boss, who had said that you should seek 00:11:24.580 --> 00:11:26.650 simplicity in your experiments. 00:11:26.650 --> 00:11:29.480 And he was saying, no, I'm going to do the exact opposite. 00:11:29.480 --> 00:11:32.040 I'm going to make my experiment as complex as I can. 00:11:35.030 --> 00:11:38.270 And here's the motivation for doing that, 00:11:38.270 --> 00:11:42.290 and this I think is one of the great results of Fisher, 00:11:42.290 --> 00:11:47.150 is that he was able to demonstrate the specific value 00:11:47.150 --> 00:11:54.610 that you could attain by making the experiment complex. 00:11:54.610 --> 00:11:57.340 And what we have here on the upper right of the slide 00:11:57.340 --> 00:12:00.490 is a cuboidal representation of the 2 00:12:00.490 --> 00:12:01.750 to the 3 factorial design. 00:12:01.750 --> 00:12:05.530 This may not be the first time you've seen it in this course, 00:12:05.530 --> 00:12:09.370 but you have the three different levels labeled as A, B, and C, 00:12:09.370 --> 00:12:12.030 and the factors labeled as A, B, and C 00:12:12.030 --> 00:12:15.190 and the levels indicate as pluses and minuses. 00:12:15.190 --> 00:12:20.830 And the way that you would estimate the effect of A 00:12:20.830 --> 00:12:25.580 is by looking at the observations, where 00:12:25.580 --> 00:12:29.960 A is at level plus, taking the average of those four. 00:12:29.960 --> 00:12:33.740 And then taking all of the observations where 00:12:33.740 --> 00:12:36.560 A is at minus, taking the average of those four 00:12:36.560 --> 00:12:41.260 and looking at that difference, and that's the equation here. 00:12:41.260 --> 00:12:45.190 The effect if A is defined as this equation which 00:12:45.190 --> 00:12:49.510 just implements what I just described in words. 00:12:49.510 --> 00:12:51.580 And then what Fisher is able to show 00:12:51.580 --> 00:12:55.840 is that the standard deviation of the effect estimate 00:12:55.840 --> 00:13:00.140 has the following form. 00:13:00.140 --> 00:13:04.060 It is a function of the experimental error, 00:13:04.060 --> 00:13:12.440 if I model the random variations in the observation due to, 00:13:12.440 --> 00:13:16.030 let's say, soil pH or the pests that I 00:13:16.030 --> 00:13:20.920 spoke of earlier or uneven distribution of water 00:13:20.920 --> 00:13:21.940 on the fields. 00:13:21.940 --> 00:13:24.670 If all of those nuisance factors that I didn't control 00:13:24.670 --> 00:13:29.140 influence my yields at the various locations, 00:13:29.140 --> 00:13:33.980 and those contribute to sigma sub-epsilon, 00:13:33.980 --> 00:13:36.470 those will be reflected in sigma sub 00:13:36.470 --> 00:13:39.500 A, my estimate of the influence of, 00:13:39.500 --> 00:13:43.780 say, applying the fertilizer early or late. 00:13:43.780 --> 00:13:48.250 But they are reflected in sigma sub A substantially 00:13:48.250 --> 00:13:51.670 less if I use this factorial design rather than 00:13:51.670 --> 00:13:53.350 a single-factor experiment. 00:13:53.350 --> 00:13:57.220 And he could show that for the case of the 2 to the 3 00:13:57.220 --> 00:14:03.390 full factorial, the improvement in efficiency is a factor of 2. 00:14:03.390 --> 00:14:07.050 So it's a large benefit as compared to the benefits 00:14:07.050 --> 00:14:11.430 that you could derive by a different kind of analysis 00:14:11.430 --> 00:14:11.970 of the data. 00:14:11.970 --> 00:14:14.336 The planning of the data made a huge difference. 00:14:19.940 --> 00:14:22.280 But the problem is that, if you're 00:14:22.280 --> 00:14:24.140 using a full factorial experiment, 00:14:24.140 --> 00:14:28.070 you see 2 to the 3 or 2 to the k, 00:14:28.070 --> 00:14:32.750 the scaling with the number of factors is terrible. 00:14:32.750 --> 00:14:33.740 Right? 00:14:33.740 --> 00:14:36.860 And so you find that you can't run such experiments with more 00:14:36.860 --> 00:14:38.250 than, say, seven factors. 00:14:38.250 --> 00:14:42.280 That's about the largest that I've seen published in tables, 00:14:42.280 --> 00:14:44.380 so that we can analyze them, because then you 00:14:44.380 --> 00:14:47.500 have 128 data elements to include 00:14:47.500 --> 00:14:49.250 in the appendix of your paper. 00:14:49.250 --> 00:14:52.360 So they're rarely run with more than seven factors. 00:14:52.360 --> 00:14:56.410 And yet in an experimentation, especially in engineering, 00:14:56.410 --> 00:14:58.060 we often have many more than seven 00:14:58.060 --> 00:14:59.920 factors we're concerned with. 00:14:59.920 --> 00:15:01.730 It's not at all uncommon. 00:15:01.730 --> 00:15:03.550 And so almost immediately, Fisher 00:15:03.550 --> 00:15:09.280 begins to consider the possibility of some means 00:15:09.280 --> 00:15:10.930 to reduce the size of the experiment, 00:15:10.930 --> 00:15:14.460 even as the number of factors goes up. 00:15:14.460 --> 00:15:17.150 And this is the way he describes thinking about that. 00:15:17.150 --> 00:15:20.360 You deliberately sacrifice possibility 00:15:20.360 --> 00:15:23.400 of obtaining information on some points. 00:15:23.400 --> 00:15:25.550 So you decide there are certain things 00:15:25.550 --> 00:15:28.520 that you would be able to estimate in the full factorial, 00:15:28.520 --> 00:15:30.500 but you just decide ahead of time 00:15:30.500 --> 00:15:34.430 that they're not worth investigating, that maybe 00:15:34.430 --> 00:15:35.780 they're unlikely a priori. 00:15:38.910 --> 00:15:44.250 And so you design the experiment so that you can't possibly 00:15:44.250 --> 00:15:45.500 obtain information about them. 00:15:45.500 --> 00:15:47.333 But actually, it's a little worse than that, 00:15:47.333 --> 00:15:48.670 I'm going to show you later. 00:15:48.670 --> 00:15:51.780 Not only do you not have a possibility of obtaining 00:15:51.780 --> 00:15:54.660 information about those points, those specific points, 00:15:54.660 --> 00:15:58.140 if they were to be consequential in the end, 00:15:58.140 --> 00:16:02.640 would actually interfere considerably with your ability 00:16:02.640 --> 00:16:03.972 to make inferences. 00:16:07.910 --> 00:16:12.920 And so this is the simplest fractional factorial 00:16:12.920 --> 00:16:13.970 I can show you. 00:16:13.970 --> 00:16:17.870 Again, we have the cuboidal representation, and what we do 00:16:17.870 --> 00:16:23.310 is just to make observations at half of the vertices. 00:16:23.310 --> 00:16:27.440 So I indicate those by the black plus's, and these four 00:16:27.440 --> 00:16:31.610 are selected in a specific arrangement to give us the 2 00:16:31.610 --> 00:16:34.960 to the 3 minus 1 half fraction with resolution 3. 00:16:34.960 --> 00:16:37.500 I'll describe what resolution 3 means in a second. 00:16:37.500 --> 00:16:39.000 One of the things that's interesting 00:16:39.000 --> 00:16:44.240 about the experiment in my mind is its projective properties. 00:16:44.240 --> 00:16:48.190 So we see that, if we collapse factor A-- 00:16:51.340 --> 00:16:54.610 that is, if we imagine that there is a flashlight or such, 00:16:54.610 --> 00:16:56.800 a source of light off to the right, 00:16:56.800 --> 00:17:00.010 and we get a projection of the matrix of the cube, 00:17:00.010 --> 00:17:01.780 and now we just get a square. 00:17:01.780 --> 00:17:05.470 We'd see that if factor A is collapsed, 00:17:05.470 --> 00:17:12.970 we would have a full factorial experiment in B and C. 00:17:12.970 --> 00:17:15.010 Have you studied these projective properties 00:17:15.010 --> 00:17:16.439 in the other parts of the course? 00:17:16.439 --> 00:17:17.849 OK. 00:17:17.849 --> 00:17:22.438 And so it goes in the other dimensions as well. 00:17:22.438 --> 00:17:24.230 And so in my mind, what's interesting about 00:17:24.230 --> 00:17:27.589 this is that in a way the design of the experiment 00:17:27.589 --> 00:17:32.930 allows you to account for a particular kind of uncertainty. 00:17:32.930 --> 00:17:36.380 If you have a long list of factors, 00:17:36.380 --> 00:17:39.410 and you're not sure which ones are going 00:17:39.410 --> 00:17:41.900 to be the important ones, you know 00:17:41.900 --> 00:17:44.780 that some few will turn out to be important, 00:17:44.780 --> 00:17:46.460 but you're not sure which ones. 00:17:46.460 --> 00:17:48.020 You can arrange it so that you will 00:17:48.020 --> 00:17:52.190 have done a full factorial in the few important factors 00:17:52.190 --> 00:17:54.620 without knowing exactly which ones you need to do 00:17:54.620 --> 00:17:57.660 a full factorial experiment in. 00:17:57.660 --> 00:18:00.700 And that's what the projective properties allow you to do. 00:18:00.700 --> 00:18:05.730 But all this comes at a cost, and the cost is more easily 00:18:05.730 --> 00:18:08.590 described by looking at a larger fractional factorial 00:18:08.590 --> 00:18:09.090 experiment. 00:18:09.090 --> 00:18:14.130 This is the 2 to the 7 minus 4 design, and what we see here 00:18:14.130 --> 00:18:17.970 is that, if we have 7 factors, and we want to stuff them 00:18:17.970 --> 00:18:21.740 into a small experiment with just say 8 trials, 00:18:21.740 --> 00:18:28.050 then what we can do is put them in this arrangement. 00:18:28.050 --> 00:18:30.540 And the risk is that if, for example, 00:18:30.540 --> 00:18:35.210 there is a two-factor interaction between F and G, 00:18:35.210 --> 00:18:41.300 then that will emerge in this pattern. 00:18:41.300 --> 00:18:45.460 That in the first four experiments, F and G 00:18:45.460 --> 00:18:48.580 are all at the same level, either plus and plus 00:18:48.580 --> 00:18:50.830 or minus and minus. 00:18:50.830 --> 00:18:52.720 And in the last four experiments, 00:18:52.720 --> 00:18:57.610 they're all at different levels, such as minus and plus 00:18:57.610 --> 00:18:59.050 or plus and minus. 00:18:59.050 --> 00:19:00.670 And so if the two are interacting 00:19:00.670 --> 00:19:03.570 in some important way, the effect 00:19:03.570 --> 00:19:06.600 will emerge in such a way that it 00:19:06.600 --> 00:19:12.900 is coming into being exactly when A is at minus, 00:19:12.900 --> 00:19:17.040 or A is at plus, and so you will confound 00:19:17.040 --> 00:19:19.480 the two in your analysis. 00:19:19.480 --> 00:19:22.830 And if you think that an effect of A itself 00:19:22.830 --> 00:19:26.040 is more likely than a two-factor interaction of F and G, 00:19:26.040 --> 00:19:30.900 you're likely to make a mistake, to assign that effect to A, 00:19:30.900 --> 00:19:34.620 whereas, it actually occurred due to F and G. 00:19:34.620 --> 00:19:38.580 That's the risk that you have to take to make a relatively 00:19:38.580 --> 00:19:41.430 small experiment that's investigating 00:19:41.430 --> 00:19:42.600 a lot of potential factors. 00:19:45.430 --> 00:19:48.710 So these are the trade-offs we make in experimental design. 00:19:48.710 --> 00:19:51.700 Now, one of the reasons that this whole topic is 00:19:51.700 --> 00:19:53.950 important to engineers is that we're 00:19:53.950 --> 00:19:57.760 concerned with robustness, robust parameter design. 00:19:57.760 --> 00:20:03.130 And what I'm showing here is the cover of one 00:20:03.130 --> 00:20:06.430 of the I think probably the best modern compilation 00:20:06.430 --> 00:20:10.000 of this topic on experimental design and its application 00:20:10.000 --> 00:20:12.610 to engineering systems, in particular 00:20:12.610 --> 00:20:15.400 for the pursuit of robustness. 00:20:15.400 --> 00:20:17.650 And Jeff Wu and Michael [? Hamouda ?] 00:20:17.650 --> 00:20:19.960 define robust parameter design here 00:20:19.960 --> 00:20:23.230 as a set of statistical and engineering methodology 00:20:23.230 --> 00:20:27.040 to reduce performance variation by choosing 00:20:27.040 --> 00:20:32.670 settings of control factors to make the system less sensitive. 00:20:32.670 --> 00:20:37.535 So you're trying to find things that you can control 00:20:37.535 --> 00:20:42.110 in an experiment in the design and to use 00:20:42.110 --> 00:20:45.980 those to make performance less sensitive to things 00:20:45.980 --> 00:20:48.930 that you don't control. 00:20:48.930 --> 00:20:51.690 Can you think of examples relevant to your work 00:20:51.690 --> 00:20:55.920 of something you might make robust? 00:20:59.840 --> 00:21:02.187 Or even simpler, just of a noise factor 00:21:02.187 --> 00:21:03.770 you might be concerned with, something 00:21:03.770 --> 00:21:05.170 that influences a system. 00:21:16.840 --> 00:21:19.660 Well, we'll have more examples later, 00:21:19.660 --> 00:21:21.990 and in fact, I believe that in the past, 00:21:21.990 --> 00:21:26.380 this course has had some projects associated with it. 00:21:26.380 --> 00:21:28.650 And in some cases, people want to do robust parameter 00:21:28.650 --> 00:21:29.580 design and projects. 00:21:29.580 --> 00:21:30.120 Yes? 00:21:30.120 --> 00:21:32.760 AUDIENCE: [INAUDIBLE] 00:21:36.610 --> 00:21:38.060 PROFESSOR: OK. 00:21:38.060 --> 00:21:38.560 OK. 00:21:38.560 --> 00:21:42.730 So you might be doing injection molding or a process like that, 00:21:42.730 --> 00:21:47.620 and you could control temperature and humidity 00:21:47.620 --> 00:21:51.008 but at some great cost I guess. 00:21:51.008 --> 00:21:53.050 So you might choose to allow those things to vary 00:21:53.050 --> 00:21:54.500 to some degree. 00:21:54.500 --> 00:21:58.810 And if they influence say the geometry of your parts, 00:21:58.810 --> 00:22:03.250 resulting parts, too much, you would probably 00:22:03.250 --> 00:22:05.140 be producing less value to your customer. 00:22:05.140 --> 00:22:08.290 Probably the function of your articles 00:22:08.290 --> 00:22:10.940 themselves might be degraded by that, 00:22:10.940 --> 00:22:13.690 and so you might like to choose process parameters, 00:22:13.690 --> 00:22:16.480 such as pressures and compositions of your plastic 00:22:16.480 --> 00:22:20.030 and so on, so that that effect is minimized. 00:22:20.030 --> 00:22:22.270 So you decide not to control the humidity 00:22:22.270 --> 00:22:25.220 but just to be insensitive to it. 00:22:25.220 --> 00:22:25.720 There. 00:22:25.720 --> 00:22:26.800 Good. 00:22:26.800 --> 00:22:29.170 So on the cover of this textbook, 00:22:29.170 --> 00:22:31.900 you see actually something called 00:22:31.900 --> 00:22:37.070 a crossarray it's hard to pick out, unless you go 00:22:37.070 --> 00:22:38.610 and zoom in on your own computer. 00:22:38.610 --> 00:22:42.690 But you see capital A, B, and C over here, 00:22:42.690 --> 00:22:45.050 and those represent control factors, 00:22:45.050 --> 00:22:49.010 such as pressure and speed of the extruder 00:22:49.010 --> 00:22:53.330 and the composition of the plastic, things that you 00:22:53.330 --> 00:22:54.710 might control nominally. 00:22:54.710 --> 00:22:56.300 And then there's little a and b, those 00:22:56.300 --> 00:22:59.120 are temperature and humidity and things 00:22:59.120 --> 00:23:02.060 you might decide you're not going to control so much. 00:23:02.060 --> 00:23:05.810 And you see that fractional factorial designs 00:23:05.810 --> 00:23:08.300 or factorial designs are being used 00:23:08.300 --> 00:23:16.220 in each of these arrangements shown to the left and above, 00:23:16.220 --> 00:23:20.530 and then the crossing, the product of those two designs, 00:23:20.530 --> 00:23:22.450 is indicated in the middle. 00:23:22.450 --> 00:23:29.490 And here, Wu actually begins to hint and foreshadow 00:23:29.490 --> 00:23:33.570 about the idea of what he calls a combiner experiment. 00:23:33.570 --> 00:23:38.730 So he shows some of these being white and some being dark, 00:23:38.730 --> 00:23:45.760 and he's suggesting that as a way to address the rather 00:23:45.760 --> 00:23:48.220 large size of this experiment. 00:23:48.220 --> 00:23:51.540 So it turns out that the first person 00:23:51.540 --> 00:23:57.030 to pioneer this idea of bringing in fractional factorial designs 00:23:57.030 --> 00:23:59.640 into the plant and using them explicitly 00:23:59.640 --> 00:24:02.420 for the purpose of robustness improvements, person 00:24:02.420 --> 00:24:04.740 who seems to most often get credit for this, 00:24:04.740 --> 00:24:07.500 is Taguchi, in Japan. 00:24:07.500 --> 00:24:13.080 So he begins to do work, in Japan, after World War II, when 00:24:13.080 --> 00:24:16.060 their manufacturing base is in disarray, 00:24:16.060 --> 00:24:18.277 and their economy is in not good shape, 00:24:18.277 --> 00:24:19.860 and their reputation for manufacturing 00:24:19.860 --> 00:24:21.930 is very poor at that time too. 00:24:21.930 --> 00:24:25.600 So if you bought a product, and it said made in Japan on it, 00:24:25.600 --> 00:24:28.360 people would actually imagine that that meant 00:24:28.360 --> 00:24:30.550 it was of rather poor quality. 00:24:30.550 --> 00:24:33.980 And of course, our impression is totally the opposite now, 00:24:33.980 --> 00:24:36.190 and it's interesting to consider how that happened. 00:24:36.190 --> 00:24:38.290 It was a lot of different things. 00:24:38.290 --> 00:24:40.180 But maybe one of the things that brought 00:24:40.180 --> 00:24:42.370 that change about in our perception 00:24:42.370 --> 00:24:45.550 of Japanese manufacture was some of this methodology. 00:24:45.550 --> 00:24:49.810 And Taguchi correctly pointed out 00:24:49.810 --> 00:24:51.850 that robustness could be attained, 00:24:51.850 --> 00:24:54.070 and it could be attained through experimentation. 00:24:54.070 --> 00:24:57.100 And he suggested that you might, for example, run an experiment 00:24:57.100 --> 00:25:00.940 like this, an orthogonal array or factorial design and control 00:25:00.940 --> 00:25:03.920 factors crossed with noise factors. 00:25:03.920 --> 00:25:08.500 And in this case, you take your 8-run array of control factors, 00:25:08.500 --> 00:25:13.360 cross it with your 4-run array of noise factors, 00:25:13.360 --> 00:25:16.640 and you would have a 32-run crossarray. 00:25:16.640 --> 00:25:21.130 And so you see with this cross symbol, 00:25:21.130 --> 00:25:25.390 the implication is that the size of the experiment 00:25:25.390 --> 00:25:30.040 scales at least like the product of the number 00:25:30.040 --> 00:25:34.720 of control factors plus 1 and the number of noise 00:25:34.720 --> 00:25:36.280 factors plus 1. 00:25:36.280 --> 00:25:38.260 Which when you begin to consider lots 00:25:38.260 --> 00:25:41.260 of control of noise factors, it gets awfully burdensome. 00:25:41.260 --> 00:25:44.080 So people are looking for ways to do it more efficiently, 00:25:44.080 --> 00:25:50.530 and that leads to the cover of this textbook, the idea of just 00:25:50.530 --> 00:25:54.040 doing half of them, so that you're actually not running 00:25:54.040 --> 00:25:56.500 exactly the same noise conditions 00:25:56.500 --> 00:25:59.150 at every setting of the control factors. 00:25:59.150 --> 00:26:01.340 And what we've done in our methodology 00:26:01.340 --> 00:26:05.030 validation work actually is to show in a recent publication 00:26:05.030 --> 00:26:06.830 in the Journal of Quality Technology 00:26:06.830 --> 00:26:10.400 that this procedure, although theoretically looks 00:26:10.400 --> 00:26:14.840 promising in many specific cases, it's failing. 00:26:21.190 --> 00:26:25.270 And yet, if I look at industry practice today, 00:26:25.270 --> 00:26:28.000 I think the methods of Taguchi and also 00:26:28.000 --> 00:26:31.750 the refinements proposed more recently 00:26:31.750 --> 00:26:35.800 on the basis of statistical theory 00:26:35.800 --> 00:26:38.530 are finding their way into industry practice 00:26:38.530 --> 00:26:41.470 and are really quite solidly entrenched. 00:26:41.470 --> 00:26:44.860 So some of you may have had some experience with these Six Sigma 00:26:44.860 --> 00:26:47.770 programs by now. 00:26:47.770 --> 00:26:51.580 Maybe some of you are in General Electric, maybe some Motorola, 00:26:51.580 --> 00:26:55.990 and I guess many other companies by now have such programs. 00:26:55.990 --> 00:27:00.970 And a big part of such programs is that entering engineers 00:27:00.970 --> 00:27:04.240 will take some coursework, even if they didn't get it 00:27:04.240 --> 00:27:06.730 in their undergraduate degrees. 00:27:06.730 --> 00:27:09.820 The company feels people broadly need this, 00:27:09.820 --> 00:27:13.870 and so they get some coursework, and then they do projects. 00:27:13.870 --> 00:27:15.353 Right? 00:27:15.353 --> 00:27:16.770 And they demonstrate that they can 00:27:16.770 --> 00:27:19.530 do some of this kind of work for the company, 00:27:19.530 --> 00:27:22.530 and at Ford, at least the last time 00:27:22.530 --> 00:27:28.500 I checked, their design process encoded just this sort 00:27:28.500 --> 00:27:31.350 of thing, and they have their multi-step process 00:27:31.350 --> 00:27:34.320 for attaining reliability and robustness. 00:27:34.320 --> 00:27:37.860 And in step four, I dig in and look at the details, 00:27:37.860 --> 00:27:40.080 and the engineers are asked to select 00:27:40.080 --> 00:27:42.420 appropriate orthogonal arrays. 00:27:42.420 --> 00:27:45.210 So the idea is that maybe they have 00:27:45.210 --> 00:27:48.090 some choices about what size of orthogonal array, 00:27:48.090 --> 00:27:51.470 number of levels, the crossed array 00:27:51.470 --> 00:27:54.380 versus the combined array, such as Wu talked about. 00:27:54.380 --> 00:27:56.600 But a departure from orthogonal arrays 00:27:56.600 --> 00:28:00.410 itself is not really being considered anymore in industry, 00:28:00.410 --> 00:28:04.220 and so this has become very much entrenched. 00:28:04.220 --> 00:28:06.050 And in fact, in some of the textbooks, 00:28:06.050 --> 00:28:13.150 you see an even more extreme statement of the point. 00:28:13.150 --> 00:28:19.480 In this text, they argue that the idea that Edward John 00:28:19.480 --> 00:28:23.770 Russell expressed on that earlier slide, that simplicity 00:28:23.770 --> 00:28:28.660 should be valued, and that single-factor experiments 00:28:28.660 --> 00:28:33.310 are often advisable, that has seen its final demise. 00:28:33.310 --> 00:28:38.500 So that's the opinion voiced in this particular textbook. 00:28:38.500 --> 00:28:41.560 But we wanted to question this a little bit. 00:28:41.560 --> 00:28:45.190 We wondered if sometimes there isn't a role for the simpler 00:28:45.190 --> 00:28:49.170 experiments still, and this decision 00:28:49.170 --> 00:28:50.940 to spend some time looking into this 00:28:50.940 --> 00:28:53.440 came out of some observations in industry. 00:28:53.440 --> 00:28:57.900 And I thought in some ways now, it seems a shock to me 00:28:57.900 --> 00:29:01.650 that our initial thoughts along these lines 00:29:01.650 --> 00:29:03.810 came out of the farm again. 00:29:03.810 --> 00:29:06.120 So that all these ideas of orthogonal arrays 00:29:06.120 --> 00:29:09.000 and so on came from agriculture in the first place, 00:29:09.000 --> 00:29:12.300 and now our wanting to question them came off of the farm, 00:29:12.300 --> 00:29:15.780 because we were working with a manufacturer 00:29:15.780 --> 00:29:18.800 of agricultural equipment. 00:29:18.800 --> 00:29:22.040 And it seems to me actually that farm equipment 00:29:22.040 --> 00:29:25.550 is one of the most significant robustness challenges 00:29:25.550 --> 00:29:26.270 in the world. 00:29:26.270 --> 00:29:29.600 Because what you have in a tractor or a windrower 00:29:29.600 --> 00:29:33.110 or a combine is really a factory. 00:29:33.110 --> 00:29:36.380 It needs to process something, sometimes in not 00:29:36.380 --> 00:29:37.280 a very simple way. 00:29:37.280 --> 00:29:40.340 Sometimes, it needs to take a piece of cotton off a plant 00:29:40.340 --> 00:29:41.960 and then strip away some part of it. 00:29:41.960 --> 00:29:43.310 It's not a simple thing. 00:29:43.310 --> 00:29:46.010 And yet here it is, and it's out in the weather. 00:29:46.010 --> 00:29:47.270 It's out in a field. 00:29:47.270 --> 00:29:51.090 Whatever conditions prevail, it has to deal with. 00:29:51.090 --> 00:29:53.330 So it's a very difficult reliability challenge. 00:29:53.330 --> 00:29:56.720 And this particular company had relatively recently 00:29:56.720 --> 00:30:00.560 brought in a lot of robust design methodology, 00:30:00.560 --> 00:30:05.240 and they were looking forward to the benefits that would accrue. 00:30:05.240 --> 00:30:07.580 And they got exactly the opposite, 00:30:07.580 --> 00:30:11.480 that this most recent launch of a new tractor 00:30:11.480 --> 00:30:13.250 was one of the worst they'd ever had, 00:30:13.250 --> 00:30:15.230 and it was causing them real problems. 00:30:15.230 --> 00:30:18.170 Their customers were quite angry with them. 00:30:18.170 --> 00:30:20.290 And so we wanted to look into that, 00:30:20.290 --> 00:30:24.010 and we decided to, in effect, do an audit, 00:30:24.010 --> 00:30:27.260 look at their design process for this tractor. 00:30:27.260 --> 00:30:28.990 What had happened? 00:30:28.990 --> 00:30:34.900 Were there some mistakes in choices of design or priority 00:30:34.900 --> 00:30:38.080 or management, whatever? 00:30:38.080 --> 00:30:39.910 So one of the things that we asked for 00:30:39.910 --> 00:30:42.790 is an accounting of all the different robust design 00:30:42.790 --> 00:30:45.220 experiments they had done, and they gave us 00:30:45.220 --> 00:30:46.690 a list of experiments they planned. 00:30:46.690 --> 00:30:49.540 It was a long list, and then we said, well, yeah, 00:30:49.540 --> 00:30:51.190 but we need more detail in this. 00:30:51.190 --> 00:30:55.120 Show us the written reports from all these experiments. 00:30:55.120 --> 00:30:56.320 Show us the data. 00:30:56.320 --> 00:30:59.230 Show us the decisions made basis of the data. 00:30:59.230 --> 00:31:03.310 And they gave us a much smaller list of reports, 00:31:03.310 --> 00:31:08.220 and we said, OK, you had 30, and now we have 5. 00:31:08.220 --> 00:31:10.080 What about the other 25? 00:31:10.080 --> 00:31:13.120 And they said, well, those were never finished, 00:31:13.120 --> 00:31:14.590 and that was interesting to us. 00:31:14.590 --> 00:31:18.230 Why is it that the majority weren't finished? 00:31:18.230 --> 00:31:21.130 And we began to ask questions of the individuals involved, 00:31:21.130 --> 00:31:23.200 and they always have pretty good reasons. 00:31:23.200 --> 00:31:26.710 In some cases, they would say that a piece of test equipment 00:31:26.710 --> 00:31:29.500 had broken down at some point along the way 00:31:29.500 --> 00:31:32.000 and was going to take a certain amount of time to repair, 00:31:32.000 --> 00:31:35.450 so we had to move forward with a partial data set. 00:31:35.450 --> 00:31:39.100 In fact, a large fraction of the experiments aren't finished, 00:31:39.100 --> 00:31:42.400 and this prompted us to wonder, what would you 00:31:42.400 --> 00:31:45.760 do if you knew this from the outset? 00:31:45.760 --> 00:31:49.600 Let's say that you put a 60% probability 00:31:49.600 --> 00:31:52.000 on canceling any individual experiment you 00:31:52.000 --> 00:31:58.878 might run somewhere at a state of partial completion. 00:31:58.878 --> 00:32:00.420 Let's say that were the case, but you 00:32:00.420 --> 00:32:02.340 didn't know which 60% we're going 00:32:02.340 --> 00:32:06.800 to be canceled, because you probably don't know that. 00:32:06.800 --> 00:32:09.225 What would you do? 00:32:09.225 --> 00:32:11.100 Well, we thought, perhaps, you would do a one 00:32:11.100 --> 00:32:14.430 at a time experiment, because then at least, 00:32:14.430 --> 00:32:17.100 no matter what you did, you would have stopped, 00:32:17.100 --> 00:32:20.760 and the data would be relatively simple to analyze, 00:32:20.760 --> 00:32:23.130 such as what Russell talked about. 00:32:23.130 --> 00:32:26.430 And we found that actually some of the most prominent 00:32:26.430 --> 00:32:28.530 statisticians in the past always felt 00:32:28.530 --> 00:32:32.620 that there was a role for one factor at a time experiments. 00:32:32.620 --> 00:32:35.550 So for example, you may have heard of Milton Friedman 00:32:35.550 --> 00:32:37.520 in a different context. 00:32:37.520 --> 00:32:39.217 What do you know him for? 00:32:39.217 --> 00:32:41.755 AUDIENCE: [INAUDIBLE] 00:32:41.755 --> 00:32:42.380 PROFESSOR: Yes. 00:32:42.380 --> 00:32:42.880 Yeah. 00:32:42.880 --> 00:32:46.730 Nobel Prize in economics, market view 00:32:46.730 --> 00:32:48.320 of economics and Leonard Savage who 00:32:48.320 --> 00:32:53.450 had written an excellent text on foundations of statistics. 00:32:53.450 --> 00:32:56.300 They found that an efficient design for the present 00:32:56.300 --> 00:32:59.630 purposes-- that was for maximization-- 00:32:59.630 --> 00:33:02.450 ought to adjust experimental program at each stage in light 00:33:02.450 --> 00:33:04.700 of the prior stages, and actually, what they described 00:33:04.700 --> 00:33:06.788 was a one factor at a time experiment. 00:33:06.788 --> 00:33:08.330 And then Cuthbert Daniel talked about 00:33:08.330 --> 00:33:15.170 a more social or psychological value of simpler experiments, 00:33:15.170 --> 00:33:18.840 just reemphasizing the points that Edward John 00:33:18.840 --> 00:33:20.100 Russell had made. 00:33:20.100 --> 00:33:24.960 That somehow you feel that you can react to data more rapidly 00:33:24.960 --> 00:33:27.640 or learn something from each rung. 00:33:27.640 --> 00:33:29.650 And he talked about some criteria 00:33:29.650 --> 00:33:31.660 by which you might make that choice, 00:33:31.660 --> 00:33:34.360 that for example, he had a demarcation. 00:33:34.360 --> 00:33:39.190 Maybe it's OK to do it, as long as the effects 00:33:39.190 --> 00:33:42.500 are at least three or four times the average random error 00:33:42.500 --> 00:33:43.510 per trail. 00:33:43.510 --> 00:33:46.120 So he had, based on his experience, 00:33:46.120 --> 00:33:48.550 come up with a demarcation-- 00:33:48.550 --> 00:33:50.740 effects three or four times random error. 00:33:50.740 --> 00:33:53.650 And we wanted to understand whether that demarcation was 00:33:53.650 --> 00:33:57.100 about right, so we began to study 00:33:57.100 --> 00:34:01.270 an adaptive variant of one factor of time experimentation. 00:34:01.270 --> 00:34:04.540 And in the beginning, we were interested primarily 00:34:04.540 --> 00:34:13.257 in knowing, if you were to do this adaptive experimentation, 00:34:13.257 --> 00:34:16.330 what level of probability of canceling the experiments 00:34:16.330 --> 00:34:19.300 would make this style look good? 00:34:19.300 --> 00:34:21.909 And the funny thing about it is we started with some maybe 00:34:21.909 --> 00:34:25.570 50% probability of canceling, and we looked at the two 00:34:25.570 --> 00:34:30.070 experimentation styles, adaptive and factorial, and on the cases 00:34:30.070 --> 00:34:32.330 we were looking at, adaptive looked better. 00:34:32.330 --> 00:34:34.060 And so we dialed down the probability 00:34:34.060 --> 00:34:37.120 of canceling the 10%, and they still looked better. 00:34:37.120 --> 00:34:40.179 And we dialed it down to 0%, and the adaptive experiment still 00:34:40.179 --> 00:34:40.960 looked better. 00:34:40.960 --> 00:34:44.840 And we thought, well, now we know we've made a mistake, 00:34:44.840 --> 00:34:47.540 because we saw it in the textbooks 00:34:47.540 --> 00:34:50.239 that this kind of experimentation 00:34:50.239 --> 00:34:51.889 has seen its final demise. 00:34:51.889 --> 00:34:55.820 So either we made a mistake in the way we set up the study, 00:34:55.820 --> 00:34:59.950 or we chose a strange case to start with. 00:34:59.950 --> 00:35:02.310 We were perplexed. 00:35:02.310 --> 00:35:06.330 So we did eventually a big empirical evaluation, 00:35:06.330 --> 00:35:13.380 and we took 66 different systems of a broad range of kinds-- 00:35:13.380 --> 00:35:16.710 chemical systems, mechanical systems, electrical systems, 00:35:16.710 --> 00:35:21.420 and we made our comparison in this case without canceling. 00:35:21.420 --> 00:35:23.708 Now, we were interested in that issue. 00:35:23.708 --> 00:35:26.083 Imagine that you're going to do this adaptive, one factor 00:35:26.083 --> 00:35:27.397 at a time experimentation. 00:35:27.397 --> 00:35:29.730 You're going to compare that with a fractional factorial 00:35:29.730 --> 00:35:32.778 experiment and see which one does better on average. 00:35:32.778 --> 00:35:34.320 And this is what we found is that, as 00:35:34.320 --> 00:35:38.550 long as experimental error was about 25% 00:35:38.550 --> 00:35:42.640 of combined factor effect, or if interactions were large enough, 00:35:42.640 --> 00:35:48.610 then you'd prefer the adaptive experiments. 00:35:48.610 --> 00:35:54.970 And so here is that same result shown in a tabular form. 00:35:54.970 --> 00:35:58.240 So we have on the column headings 00:35:58.240 --> 00:36:00.670 the degree of experimental error expressed 00:36:00.670 --> 00:36:10.530 as a fraction of the factor effects, and we have on the row 00:36:10.530 --> 00:36:13.980 headings the strength of interactions-- 00:36:13.980 --> 00:36:15.540 mild, moderate, strong, dominant. 00:36:18.100 --> 00:36:21.330 Strong interactions would indicate that the interactions 00:36:21.330 --> 00:36:23.940 are counting for more than a quarter 00:36:23.940 --> 00:36:26.970 of effects in the system, quite a big fraction, 00:36:26.970 --> 00:36:30.450 and moderate is maybe 10% to 25%. 00:36:30.450 --> 00:36:35.860 And the gray boxes indicate that the adaptive one factor 00:36:35.860 --> 00:36:37.980 at a time experiments were giving 00:36:37.980 --> 00:36:40.620 at least as good or better results 00:36:40.620 --> 00:36:43.620 than the fractional factorial, and so you might prefer them, 00:36:43.620 --> 00:36:46.120 especially given their flexibility. 00:36:46.120 --> 00:36:48.240 And so we were surprised at how much of that 00:36:48.240 --> 00:36:51.600 territory turns out to be gray, and that was the main result 00:36:51.600 --> 00:36:54.460 we wanted to get across in this paper. 00:36:54.460 --> 00:36:56.950 And we thought, a strength of the experimental error 00:36:56.950 --> 00:37:00.820 as large as 0.4 looks like a lot of experimental error 00:37:00.820 --> 00:37:02.350 to us, to engineers. 00:37:02.350 --> 00:37:05.290 When you see a dependence, such as this blue line 00:37:05.290 --> 00:37:08.260 here, on the figure and the scatter 00:37:08.260 --> 00:37:09.760 being indicated by the red, that's 00:37:09.760 --> 00:37:13.060 about typical of the strength of experimental error at 0.4 00:37:13.060 --> 00:37:16.270 that we expressed, and to us that 00:37:16.270 --> 00:37:19.210 looks like a pretty noisy experiment. 00:37:19.210 --> 00:37:23.510 And so we thought it was interesting how 00:37:23.510 --> 00:37:29.550 much of the time you might recommend these experiments. 00:37:29.550 --> 00:37:32.880 So we wanted to understand how this could be, 00:37:32.880 --> 00:37:36.160 since it seems so counterintuitive to us. 00:37:36.160 --> 00:37:39.600 And so we delved into it a little more in terms 00:37:39.600 --> 00:37:43.920 of the mathematics, and we tried to develop a theory that 00:37:43.920 --> 00:37:46.750 would account for this. 00:37:46.750 --> 00:37:50.880 So we started to write a model of the procedure 00:37:50.880 --> 00:37:53.040 of adaptive one factor at a time experimentation, 00:37:53.040 --> 00:37:54.960 and so there's just a little notation here. 00:37:54.960 --> 00:37:58.618 You imagine that you make an initial observation. 00:37:58.618 --> 00:38:00.660 Some of the symbols are coming up a little wrong. 00:38:00.660 --> 00:38:07.620 This is an x with a tilde over it is an initial starting point 00:38:07.620 --> 00:38:09.720 level for factor one. 00:38:09.720 --> 00:38:12.600 And x2 with a tilde is the same, and that k 00:38:12.600 --> 00:38:14.190 should be an ellipsis, three dots. 00:38:17.460 --> 00:38:19.950 And then you change one of the factors. 00:38:19.950 --> 00:38:24.610 You toggle it, change it to the opposite of the starting value, 00:38:24.610 --> 00:38:26.500 and that's your first observation. 00:38:26.500 --> 00:38:29.890 And then you make a decision about the level 00:38:29.890 --> 00:38:33.850 of the first factor based on the sign of the differences 00:38:33.850 --> 00:38:35.710 in the observations. 00:38:35.710 --> 00:38:44.250 So if the improvement seems to obtain, we adopt the change, 00:38:44.250 --> 00:38:46.440 and then we go through that for every one 00:38:46.440 --> 00:38:51.090 of the different factors, putting stars on our x's, and 00:38:51.090 --> 00:38:51.990 we continue. 00:38:51.990 --> 00:38:53.610 And what we're interested in knowing 00:38:53.610 --> 00:39:00.020 is how does the expected value of improvement 00:39:00.020 --> 00:39:03.440 behave as a function of a number of parameters in a model? 00:39:03.440 --> 00:39:07.220 And the parameters we put in our model are few. 00:39:07.220 --> 00:39:11.480 We talked about the size of main effects in general, 00:39:11.480 --> 00:39:14.150 the size of interactions, and the size of experimental error, 00:39:14.150 --> 00:39:16.100 because our empirical investigation 00:39:16.100 --> 00:39:20.000 had shown that those were interesting to look at. 00:39:20.000 --> 00:39:23.900 And here, we show we wanted to normalize those with respect 00:39:23.900 --> 00:39:26.090 to the maximum, the best that you could do, 00:39:26.090 --> 00:39:29.240 so that our results could be easily interpreted. 00:39:29.240 --> 00:39:33.440 And another concept we introduced in this paper, 00:39:33.440 --> 00:39:36.710 we felt we needed to explain the results, 00:39:36.710 --> 00:39:41.470 was the idea of exploiting an effect. 00:39:41.470 --> 00:39:45.830 So we had to introduce the idea that, when 00:39:45.830 --> 00:39:48.920 you do an experiment, if your system has a number of effects 00:39:48.920 --> 00:39:50.840 in it, main effects or interactions, 00:39:50.840 --> 00:39:53.520 they can either be exploited or not, 00:39:53.520 --> 00:39:56.940 and we'll say that they're exploited, just in case. 00:39:56.940 --> 00:40:00.990 For example, if you're trying to get a higher response out 00:40:00.990 --> 00:40:03.720 of the system-- let's say yield out of a field [? of oats, ?] 00:40:03.720 --> 00:40:09.570 or quality of a manufactured article. 00:40:09.570 --> 00:40:14.160 Then, you exploit an effect, if the coefficient 00:40:14.160 --> 00:40:17.550 in the model times the level of the factor 00:40:17.550 --> 00:40:22.310 is such that it's contributing positively to your outcome. 00:40:22.310 --> 00:40:25.880 And you exploit an interaction in the system 00:40:25.880 --> 00:40:27.620 basically under the same condition. 00:40:27.620 --> 00:40:29.420 That if the two factors involved are 00:40:29.420 --> 00:40:32.150 such that you're gaining benefit from the coefficient 00:40:32.150 --> 00:40:35.990 in the model, either increasing or decreasing 00:40:35.990 --> 00:40:37.910 the response depending on what you prefer, 00:40:37.910 --> 00:40:39.530 we call that one exploited. 00:40:39.530 --> 00:40:41.390 So we wanted to understand the probabilities 00:40:41.390 --> 00:40:43.580 and conditional probabilities associated 00:40:43.580 --> 00:40:44.960 with this adaptive experiment. 00:40:44.960 --> 00:40:49.580 So we begin to do the theorem proving based on our model, 00:40:49.580 --> 00:40:51.590 and we find, first of all, that if you 00:40:51.590 --> 00:40:57.180 change just the first factor and compute 00:40:57.180 --> 00:40:59.060 the expected value of improvement, 00:40:59.060 --> 00:41:03.070 you have two different kinds of contributions-- 00:41:03.070 --> 00:41:10.250 1 due to a main effect and n minus 1 due to interactions. 00:41:10.250 --> 00:41:12.940 So you change factor 1, and this figure here 00:41:12.940 --> 00:41:16.060 is meant to indicate the condition with n equals 7. 00:41:16.060 --> 00:41:18.070 You've got 7 factors you're interested in, 00:41:18.070 --> 00:41:22.330 and this stack of squares indicates the contribution 00:41:22.330 --> 00:41:23.270 of main effects. 00:41:23.270 --> 00:41:25.690 And none of these other factors are contributing anything, 00:41:25.690 --> 00:41:28.960 but now the main effect of the first factor 00:41:28.960 --> 00:41:32.320 is contributing quite a bit on average, after you've 00:41:32.320 --> 00:41:33.850 done the adaptation. 00:41:33.850 --> 00:41:37.960 And all of the 6 potential two-factor interactions 00:41:37.960 --> 00:41:41.080 are on average contributing a little to improvement, 00:41:41.080 --> 00:41:44.350 because it may be that in some cases 00:41:44.350 --> 00:41:46.570 there'll be a large two-factor interaction. 00:41:46.570 --> 00:41:48.740 And you'll see it when you toggle the factors, 00:41:48.740 --> 00:41:51.190 and you'll go chase that, and so on average, 00:41:51.190 --> 00:41:53.390 it's contributing something. 00:41:53.390 --> 00:41:55.420 And we could write closed form expressions 00:41:55.420 --> 00:41:59.590 for the contribution, and the contribution 00:41:59.590 --> 00:42:01.880 due to main effects is larger when 00:42:01.880 --> 00:42:03.700 the main effects are larger. 00:42:03.700 --> 00:42:07.450 It tends to be smaller when interactions are large 00:42:07.450 --> 00:42:10.228 and when error is large. 00:42:10.228 --> 00:42:11.770 The contributions to the interactions 00:42:11.770 --> 00:42:15.360 are large when interactions are large in the model. 00:42:15.360 --> 00:42:19.080 And interesting to us, if you write out the expected value, 00:42:19.080 --> 00:42:23.430 normalize by the maximum, so that the improvements are 00:42:23.430 --> 00:42:29.760 on this scale either 0 or 1 or scaled in between. 00:42:29.760 --> 00:42:32.820 The contribution you would get in the case 00:42:32.820 --> 00:42:35.790 that experimental error is low-- so here I 00:42:35.790 --> 00:42:40.410 put experimental error at 10% of a main effect. 00:42:40.410 --> 00:42:44.040 The contribution you get after running one 1 of 7 factors 00:42:44.040 --> 00:42:47.180 is actually about 1/7. 00:42:47.180 --> 00:42:50.420 So you get 1/7 of the way toward the maximum 00:42:50.420 --> 00:42:52.210 by changing 1/7 of the factors. 00:42:52.210 --> 00:42:54.300 That's a good start. 00:42:54.300 --> 00:42:56.910 And the influence of error is-- 00:42:56.910 --> 00:42:57.900 it's there. 00:42:57.900 --> 00:43:01.770 As you go to error being as large as main effects, 00:43:01.770 --> 00:43:04.350 it drops a little, and of course, 00:43:04.350 --> 00:43:06.120 if you make the error really huge, 00:43:06.120 --> 00:43:07.780 it drops quite substantially. 00:43:07.780 --> 00:43:13.320 But that's a hard test to pass, but we're just 00:43:13.320 --> 00:43:16.180 trying to understand what the influences are. 00:43:16.180 --> 00:43:17.032 OK. 00:43:17.032 --> 00:43:18.990 Now, let's talk about probability of exploiting 00:43:18.990 --> 00:43:21.040 the first main effect. 00:43:21.040 --> 00:43:27.175 Just when interactions are small, 00:43:27.175 --> 00:43:29.550 the probability of getting the benefit of the main effect 00:43:29.550 --> 00:43:30.450 is large. 00:43:30.450 --> 00:43:32.280 Then, as interactions increase, it's 00:43:32.280 --> 00:43:35.760 less likely that you'll get the benefit of the main effect. 00:43:39.530 --> 00:43:41.720 Now, it becomes interesting at the second step. 00:43:41.720 --> 00:43:43.908 Now, you've toggled two factors, and then 00:43:43.908 --> 00:43:45.200 what we think about it is this. 00:43:45.200 --> 00:43:48.290 Now, you've got the benefit of two main effects, 00:43:48.290 --> 00:43:53.750 and you get first n minus 1 and then n minus 2 interactions 00:43:53.750 --> 00:43:55.940 potentially contributing. 00:43:55.940 --> 00:43:58.790 And at this point, I hash it in, because this 00:43:58.790 --> 00:44:02.750 is the first interaction which you've actually locked in. 00:44:02.750 --> 00:44:05.390 In the sense that any further changes 00:44:05.390 --> 00:44:08.090 you make through the experimentation process 00:44:08.090 --> 00:44:11.270 will not reverse whatever benefits you got. 00:44:11.270 --> 00:44:13.430 Because you've already changed one and changed two, 00:44:13.430 --> 00:44:15.600 and you've made your decision about that. 00:44:15.600 --> 00:44:17.480 And now, we see that there's a contribution 00:44:17.480 --> 00:44:21.680 due to the interaction, and the interaction 00:44:21.680 --> 00:44:24.830 is the same functional form as I showed before. 00:44:24.830 --> 00:44:28.700 That the contribution is large when the interaction is large. 00:44:28.700 --> 00:44:35.720 Again, we see that the expected value has gone from about 1/7 00:44:35.720 --> 00:44:36.600 to about 2/7. 00:44:36.600 --> 00:44:40.170 So we're continuing to step forward at a good rate, 00:44:40.170 --> 00:44:41.450 and we want to understand why. 00:44:41.450 --> 00:44:43.440 Well, OK. 00:44:43.440 --> 00:44:46.440 We see the probability of exploiting the interaction. 00:44:46.440 --> 00:44:49.540 We can write it's closed form expression. 00:44:49.540 --> 00:44:53.520 It's strictly better than 50-50. 00:44:53.520 --> 00:44:57.900 So it's in any case better than random, 00:44:57.900 --> 00:44:59.910 and it goes up with interaction strength. 00:44:59.910 --> 00:45:01.410 Now, the interesting thing about it, 00:45:01.410 --> 00:45:04.650 it's never very high, because actually, 00:45:04.650 --> 00:45:07.230 any particular interaction is competing 00:45:07.230 --> 00:45:10.820 with n choose 2 other interactions 00:45:10.820 --> 00:45:12.590 or n choose 2 minus 1. 00:45:12.590 --> 00:45:15.170 And so any particular one is unlikely to be 00:45:15.170 --> 00:45:19.170 exploited by this procedure, and so maybe we get a 60% chance. 00:45:19.170 --> 00:45:21.550 But if you condition that probability, if you say, 00:45:21.550 --> 00:45:23.550 well, let's say that this particular interaction 00:45:23.550 --> 00:45:26.210 that we just locked in is somehow 00:45:26.210 --> 00:45:28.110 the largest one in the system. 00:45:28.110 --> 00:45:29.720 Let's say that that happened. 00:45:29.720 --> 00:45:32.540 We conditioned the probability, we get a new expression, 00:45:32.540 --> 00:45:36.300 and the probability could be as high as 80%. 00:45:36.300 --> 00:45:38.910 The reason that's interesting to us 00:45:38.910 --> 00:45:42.840 is that we know that the adaptive one-factor experiments 00:45:42.840 --> 00:45:48.030 are just not capable of allowing you to estimate interactions. 00:45:48.030 --> 00:45:52.140 You take a factor, you change it, you look at the influence, 00:45:52.140 --> 00:45:54.840 and what you're seeing is a combination 00:45:54.840 --> 00:45:57.540 of the main effective the factor and a bunch 00:45:57.540 --> 00:46:01.710 of conditional contributions due to a bunch 00:46:01.710 --> 00:46:03.740 of different interactions. 00:46:03.740 --> 00:46:05.870 And so you can't sort out which one is which. 00:46:05.870 --> 00:46:08.060 You would need more experiments to do that. 00:46:08.060 --> 00:46:11.330 Nevertheless, if there's a big effect there, 00:46:11.330 --> 00:46:14.030 there's a good chance you'll get benefit from it, when you 00:46:14.030 --> 00:46:18.414 do this adaptive experiment. 00:46:18.414 --> 00:46:21.872 AUDIENCE: [INAUDIBLE] 00:46:51.253 --> 00:46:51.920 PROFESSOR: Yeah. 00:46:51.920 --> 00:46:55.190 It's an interesting question, and this is something 00:46:55.190 --> 00:46:57.920 that we were concerned with. 00:46:57.920 --> 00:47:00.440 All the results I'm showing you today 00:47:00.440 --> 00:47:05.180 assume that you've randomized the ordering of the factors 00:47:05.180 --> 00:47:08.980 and that you've randomized the starting point. 00:47:08.980 --> 00:47:12.040 And so far, what our investigations show 00:47:12.040 --> 00:47:13.600 is that the ordering of the factors 00:47:13.600 --> 00:47:18.510 doesn't matter much but the starting points do. 00:47:18.510 --> 00:47:23.280 So I would say, whether or not you order the factors 00:47:23.280 --> 00:47:25.530 from the most important one to the next to the next, 00:47:25.530 --> 00:47:29.010 it's a minor improvement, not big. 00:47:29.010 --> 00:47:35.310 But if you select a very promising starting point, that 00:47:35.310 --> 00:47:39.630 is go with something that is likely to be very robust, 00:47:39.630 --> 00:47:43.100 high performance, that makes a big difference. 00:47:43.100 --> 00:47:46.040 And it wasn't obvious to us at the start. 00:47:46.040 --> 00:47:48.292 I thought probably what you said was true. 00:47:48.292 --> 00:47:50.000 It turned out to be the other way around. 00:47:53.140 --> 00:47:55.890 So the dynamics of this process of experimentation 00:47:55.890 --> 00:47:59.400 are such that it turns out the probability of exploiting 00:47:59.400 --> 00:48:01.890 interactions as you go gets higher 00:48:01.890 --> 00:48:04.810 and higher throughout the course of the experiment. 00:48:04.810 --> 00:48:11.190 So it's true that, if you knew where the interactions lie, 00:48:11.190 --> 00:48:14.010 you'd like to toggle them for the last time 00:48:14.010 --> 00:48:15.840 as late as possible. 00:48:15.840 --> 00:48:18.300 But I guess I don't really believe that people would 00:48:18.300 --> 00:48:21.570 be able to identify which two interactions are going 00:48:21.570 --> 00:48:28.150 to be involved, so I ruled that out and didn't study it much. 00:48:28.150 --> 00:48:31.830 But the point is that, even if you didn't know where they lie, 00:48:31.830 --> 00:48:36.930 the benefits continue to accrue almost linearly. 00:48:36.930 --> 00:48:39.000 Even though you're exploiting fewer and fewer 00:48:39.000 --> 00:48:40.620 new interactions as you go, you see 00:48:40.620 --> 00:48:46.920 the white ones left to benefit are diminishing over time. 00:48:46.920 --> 00:48:51.270 But because their likelihood of coming in your favor 00:48:51.270 --> 00:48:54.450 is increasing, you go up almost linearly, 00:48:54.450 --> 00:48:59.390 and you get almost to the end, 80%. 00:48:59.390 --> 00:49:02.630 You can almost to one, well, pretty close, 00:49:02.630 --> 00:49:04.280 And what's interesting to us about this 00:49:04.280 --> 00:49:08.350 is take the case of 7 factors. 00:49:08.350 --> 00:49:12.760 The adaptive one factor at a time, assuming two levels, 00:49:12.760 --> 00:49:15.640 that takes 8 experiments. 00:49:15.640 --> 00:49:19.480 The full factorial takes 128. 00:49:19.480 --> 00:49:24.240 So you've looked at a very small fraction of the overall space, 00:49:24.240 --> 00:49:26.460 and you've gotten 80% of the way toward the best 00:49:26.460 --> 00:49:28.680 possible outcome in the space. 00:49:28.680 --> 00:49:32.710 That to us is interesting. 00:49:32.710 --> 00:49:37.270 And then if I want to compare the use of resolution 3 designs 00:49:37.270 --> 00:49:41.980 and adaptive designs, we have expressions for each. 00:49:41.980 --> 00:49:44.380 And then to make the comparison straight, 00:49:44.380 --> 00:49:46.360 one against the other, we do the following. 00:49:46.360 --> 00:49:51.540 Just hold them up against one another, look at-- 00:49:51.540 --> 00:49:54.370 we see the experimental error is plotted parametrically. 00:49:54.370 --> 00:49:59.070 So let's look at just the crossing point when 00:49:59.070 --> 00:50:02.920 experimental error is about the size of main effect. 00:50:02.920 --> 00:50:07.090 They cross at the point where interactions are 00:50:07.090 --> 00:50:08.770 about a quarter of main effect. 00:50:08.770 --> 00:50:10.600 The reason we think that one is interesting 00:50:10.600 --> 00:50:13.930 is that I alluded to earlier the idea 00:50:13.930 --> 00:50:17.290 that we are doing these studies, big studies of complex systems 00:50:17.290 --> 00:50:19.120 and regularities. 00:50:19.120 --> 00:50:22.670 And one of the things we found is that interactions 00:50:22.670 --> 00:50:25.790 being about a quarter of the size of main effects, that's 00:50:25.790 --> 00:50:27.530 pretty typical. 00:50:27.530 --> 00:50:28.680 That's a typical amount. 00:50:28.680 --> 00:50:31.400 So if you go into a factory, and you get your engineers 00:50:31.400 --> 00:50:35.360 to list a number of factors they're interested in studying. 00:50:35.360 --> 00:50:37.660 And then you run a big experiment 00:50:37.660 --> 00:50:39.980 and study some of the interactions too, 00:50:39.980 --> 00:50:42.450 you will generally find they're about a quarter of the size 00:50:42.450 --> 00:50:44.460 of the main effects. 00:50:44.460 --> 00:50:47.580 It's just a reasonably reliable regularity, 00:50:47.580 --> 00:50:49.590 and so we think that's typical. 00:50:49.590 --> 00:50:51.720 And the way I would interpret the graph 00:50:51.720 --> 00:50:54.090 is that, if your system is pretty 00:50:54.090 --> 00:50:57.750 typical in this regard, strength of interactions, 00:50:57.750 --> 00:51:01.800 you should be doing the adaptive experiments whenever error 00:51:01.800 --> 00:51:05.640 is less than main effects. 00:51:05.640 --> 00:51:08.360 So Cuthbert Daniels criterion for demarcation 00:51:08.360 --> 00:51:10.105 was error should be about a third 00:51:10.105 --> 00:51:12.230 or a quarter of main effects, and we're saying now, 00:51:12.230 --> 00:51:16.330 it's actually more like the size of main effects, not 00:51:16.330 --> 00:51:17.590 a third or a quarter. 00:51:17.590 --> 00:51:22.360 That's assuming, importantly, that your main goal is just 00:51:22.360 --> 00:51:25.090 getting improvement, knowing exactly where they came from 00:51:25.090 --> 00:51:29.070 or estimating effects accurately. 00:51:29.070 --> 00:51:32.310 Now, our primary reason for going into all this 00:51:32.310 --> 00:51:36.790 in the first place was to deal with robust design in the end. 00:51:36.790 --> 00:51:41.370 And so we attempted lots of different ways of combining 00:51:41.370 --> 00:51:44.460 the noise factors with the control factors, 00:51:44.460 --> 00:51:47.730 and so far, the simplest thing turned out to be the best. 00:51:47.730 --> 00:51:53.030 And that is what you see here in this cuboidal representation 00:51:53.030 --> 00:51:56.600 are controllable factors A, B, and C 00:51:56.600 --> 00:51:59.060 and our noise factors, little a, b, and c, 00:51:59.060 --> 00:52:02.570 are being crossed in effect. 00:52:02.570 --> 00:52:04.070 We take the adaptive procedure. 00:52:04.070 --> 00:52:08.070 We run a factorial design in the noise. 00:52:08.070 --> 00:52:10.790 Then, change one of the control factors, just one, 00:52:10.790 --> 00:52:14.530 and run again the exact same design. 00:52:14.530 --> 00:52:18.490 And then what we do is we select our level, in this case of A, 00:52:18.490 --> 00:52:23.350 on the basis of our preference for this set of observations 00:52:23.350 --> 00:52:24.490 versus this set. 00:52:24.490 --> 00:52:28.000 If we like this set because, for example, it has lower 00:52:28.000 --> 00:52:31.320 variation, then we select it. 00:52:31.320 --> 00:52:33.580 We select A as positive, and then 00:52:33.580 --> 00:52:35.890 we just continue to wind our way through the space, 00:52:35.890 --> 00:52:38.530 reversing changes that we don't like 00:52:38.530 --> 00:52:42.620 and adopting ones that we do. 00:52:42.620 --> 00:52:44.680 And we study this, this says-- 00:52:44.680 --> 00:52:46.390 it actually appeared in ASME journal. 00:52:46.390 --> 00:52:48.100 I've got to change that. 00:52:52.400 --> 00:52:55.600 So we began to validate this approach 00:52:55.600 --> 00:53:00.200 by applying it to a number of different systems. 00:53:00.200 --> 00:53:03.280 One that might be of interest to you is sheet metal spinning. 00:53:03.280 --> 00:53:05.290 So in this process, you go into your factory, 00:53:05.290 --> 00:53:08.050 you take sheet metal articles, and you turn them 00:53:08.050 --> 00:53:12.610 into manufactured goods, surfaces of revolution, 00:53:12.610 --> 00:53:15.190 cups, and bells and whatever you need. 00:53:15.190 --> 00:53:18.760 By taking the circular blank, pressing it up 00:53:18.760 --> 00:53:24.580 against a mandrel repetitively, until you get the deformation 00:53:24.580 --> 00:53:25.600 that you want. 00:53:25.600 --> 00:53:29.860 And you might be interested, for example, in consistency 00:53:29.860 --> 00:53:32.320 of the geometry, and you might be 00:53:32.320 --> 00:53:35.740 interested in how a number of different parameters 00:53:35.740 --> 00:53:37.630 affect that consistency of the geometry. 00:53:37.630 --> 00:53:41.530 Such as what material you chose, the shape of the mandrel, 00:53:41.530 --> 00:53:50.130 the shape of the path which you use, the number of times 00:53:50.130 --> 00:53:53.100 and the force you apply. 00:53:53.100 --> 00:53:56.850 And this was studied by a group in Germany, 00:53:56.850 --> 00:54:00.600 and we used their results to simulate 00:54:00.600 --> 00:54:03.810 the process of running these adaptive experiments 00:54:03.810 --> 00:54:06.740 and running an alternative across the ray 00:54:06.740 --> 00:54:08.160 design on these systems. 00:54:08.160 --> 00:54:11.820 And we found that the Taguchi-style crossed 00:54:11.820 --> 00:54:14.850 array made indeed improvements in signal to noise 00:54:14.850 --> 00:54:17.790 ratio of the system, but our adaptive procedures 00:54:17.790 --> 00:54:19.740 were doing a little better, until you 00:54:19.740 --> 00:54:23.400 got to some crossing point, an experimental error up here 00:54:23.400 --> 00:54:27.280 of 2 millimeters squared of this quality measure. 00:54:27.280 --> 00:54:30.580 And this is where we found another result, the one that I 00:54:30.580 --> 00:54:32.020 alluded to earlier. 00:54:32.020 --> 00:54:37.060 That if you gave the engineers the benefit of the doubt 00:54:37.060 --> 00:54:40.300 and suggested that they didn't choose their starting points 00:54:40.300 --> 00:54:44.120 at random, but instead choose informed starting points. 00:54:44.120 --> 00:54:46.970 And let me define what I mean by that. 00:54:46.970 --> 00:54:50.890 Let's say, you have a number of factors. 00:54:50.890 --> 00:54:54.040 They're all at two levels, a random starting point 00:54:54.040 --> 00:54:55.690 is just flipping coins. 00:54:55.690 --> 00:54:56.830 You choose each one. 00:54:56.830 --> 00:54:58.810 But let's say that somehow the engineers 00:54:58.810 --> 00:55:01.780 are able to do better than 50% odds of getting the best 00:55:01.780 --> 00:55:06.320 setting for each factor, but instead have 00:55:06.320 --> 00:55:13.370 75% chance of defining the superior level of each factor. 00:55:13.370 --> 00:55:17.820 In that case in the end they would 00:55:17.820 --> 00:55:22.040 get this result, substantially higher 00:55:22.040 --> 00:55:23.790 than either of the other two alternatives, 00:55:23.790 --> 00:55:28.940 and they'll never cross, no matter how much error there is. 00:55:28.940 --> 00:55:29.750 Yes? 00:55:29.750 --> 00:55:31.792 AUDIENCE: Why is it saying the smaller the better 00:55:31.792 --> 00:55:34.748 on the [INAUDIBLE] 00:55:35.863 --> 00:55:36.530 PROFESSOR: Yeah. 00:55:36.530 --> 00:55:44.330 So smaller the better is the parameter of interest 00:55:44.330 --> 00:55:47.220 was meant to be smaller is better. 00:55:47.220 --> 00:55:49.680 But then, when you compute a signal to noise ratio, 00:55:49.680 --> 00:55:51.040 larger it's better. 00:55:51.040 --> 00:55:52.830 So the underlying parameter, called 00:55:52.830 --> 00:55:56.380 A20 in this case, smaller was better. 00:55:56.380 --> 00:55:58.240 But in Taguchi's methodology, you then 00:55:58.240 --> 00:56:01.970 take that parameter and its variation 00:56:01.970 --> 00:56:05.030 and transform it to get a signal to noise ratio, 00:56:05.030 --> 00:56:08.820 and with signal to noise ratio, larger is always better. 00:56:08.820 --> 00:56:10.719 So I appreciate why that's confusing. 00:56:15.710 --> 00:56:17.698 This one I included in the paper, 00:56:17.698 --> 00:56:18.740 and I include it for you. 00:56:18.740 --> 00:56:22.720 Because in some cases, you'll feel skeptical 00:56:22.720 --> 00:56:24.490 about these results, because they still 00:56:24.490 --> 00:56:26.500 are pretty counterintuitive. 00:56:26.500 --> 00:56:29.615 And you might need to demonstrate them to yourself, 00:56:29.615 --> 00:56:31.490 and the paper airplane allows you to do that. 00:56:31.490 --> 00:56:34.240 So Steve [? Effinger ?] came up with this template 00:56:34.240 --> 00:56:37.480 for demonstrating some of the methods 00:56:37.480 --> 00:56:40.000 here in robust design and design of experiments. 00:56:40.000 --> 00:56:42.040 And the template allows you to fold 00:56:42.040 --> 00:56:44.960 a variety of different paper airplanes, in this case, 00:56:44.960 --> 00:56:49.600 it's 3 to the 4th, or 81 different paper airplanes. 00:56:49.600 --> 00:56:52.660 And we ran an experiment in which 00:56:52.660 --> 00:56:56.020 we had created different noise conditions for the flights, 00:56:56.020 --> 00:56:58.780 and you could replicate our results. 00:56:58.780 --> 00:57:01.870 Actually throw these airplanes and do it adaptively one factor 00:57:01.870 --> 00:57:04.390 at a time, and I think you'll find the same results 00:57:04.390 --> 00:57:05.350 as we did. 00:57:05.350 --> 00:57:08.920 That if you do the L9 Taguchi array, 00:57:08.920 --> 00:57:11.920 or else that's a [? Plackit-Berman ?] design 00:57:11.920 --> 00:57:15.820 by another name, cross with a 2 to the 3 minus 1 noise array, 00:57:15.820 --> 00:57:19.210 you get these kind of improvements in signal to noise 00:57:19.210 --> 00:57:21.290 ratio. 00:57:21.290 --> 00:57:24.550 But if you run it adaptively with random starting points, 00:57:24.550 --> 00:57:26.260 you'll get better results. 00:57:26.260 --> 00:57:28.600 And if you use an informed starting point, 00:57:28.600 --> 00:57:33.220 such as looking at a plane that has about the right aspect 00:57:33.220 --> 00:57:36.640 ratio, and maybe you know right ahead 00:57:36.640 --> 00:57:39.490 that folding up the little winglets out here 00:57:39.490 --> 00:57:45.180 is probably better for lateral stability. 00:57:45.180 --> 00:57:48.700 If you make such judgments, you'll get even better results, 00:57:48.700 --> 00:57:51.440 and so that's what the paper airplane study tells us. 00:57:51.440 --> 00:57:54.900 So we did four such studies, and here's a point 00:57:54.900 --> 00:57:55.900 that I find interesting. 00:57:55.900 --> 00:57:56.400 OK? 00:57:56.400 --> 00:58:01.530 First of all, if you're to lower error states, what you find 00:58:01.530 --> 00:58:06.295 is that, on average, the random approach, starting 00:58:06.295 --> 00:58:07.920 at random starting points, works better 00:58:07.920 --> 00:58:09.420 than the fractional factorial. 00:58:09.420 --> 00:58:11.340 Informed starting points are better than that. 00:58:11.340 --> 00:58:15.150 If you raise the error high enough, things turn around, 00:58:15.150 --> 00:58:18.060 and the factorial design is a little better 00:58:18.060 --> 00:58:20.940 than the adaptive approach, unless we use the informed 00:58:20.940 --> 00:58:22.980 starting point. 00:58:22.980 --> 00:58:25.800 But what I think is further interesting 00:58:25.800 --> 00:58:29.040 is that if you look at the interquartile-- 00:58:29.040 --> 00:58:32.330 if you look at the range of results across cases, 00:58:32.330 --> 00:58:35.720 the adaptive procedure even at high error, 00:58:35.720 --> 00:58:44.430 to me that's a preferable range, 51 to 87 versus 36 to 88. 00:58:44.430 --> 00:58:47.010 I think you actually take less risk 00:58:47.010 --> 00:58:49.160 from case to case with the adaptive method. 00:58:51.870 --> 00:58:55.560 The factorial design is optimally 00:58:55.560 --> 00:59:01.950 suited to minimize the influence of experimental error 00:59:01.950 --> 00:59:04.280 on your outcomes. 00:59:04.280 --> 00:59:08.090 But some of the uncertainties you face in experimentation 00:59:08.090 --> 00:59:09.680 aren't that kind. 00:59:09.680 --> 00:59:11.180 They're not experimental error. 00:59:11.180 --> 00:59:14.450 They're actually uncertainties about where the interactions 00:59:14.450 --> 00:59:18.970 lie, and in fact, fractional factorial designs 00:59:18.970 --> 00:59:21.130 are among the most sensitive designs 00:59:21.130 --> 00:59:23.540 to that particular uncertainty. 00:59:23.540 --> 00:59:26.390 Because when you make this confounding 00:59:26.390 --> 00:59:27.980 between main effects and interactions, 00:59:27.980 --> 00:59:30.160 it's actually very harmful to your outcomes. 00:59:36.020 --> 00:59:38.740 So those are the principle results 00:59:38.740 --> 00:59:41.970 that I meant to show you today. 00:59:41.970 --> 00:59:43.530 There are a couple other things that 00:59:43.530 --> 00:59:47.130 have emerged more recently that I can talk about. 00:59:47.130 --> 00:59:49.850 For example, recently, we've been 00:59:49.850 --> 00:59:54.960 interested in the idea of producing ensembles. 00:59:54.960 --> 00:59:58.530 Now, here, the situation is that you 00:59:58.530 --> 01:00:03.450 might find that you have an adequate budget to run a larger 01:00:03.450 --> 01:00:05.220 fractional factorial design. 01:00:05.220 --> 01:00:08.280 Let's say instead of a 2 to the 7 minus 4, 01:00:08.280 --> 01:00:12.195 you actually have a budget to run a much larger experiment, 2 01:00:12.195 --> 01:00:13.920 to the 7 minus 2. 01:00:13.920 --> 01:00:18.180 So the counterargument from some folks in industry 01:00:18.180 --> 01:00:21.210 was that they rarely ran resolution 3 designs. 01:00:21.210 --> 01:00:23.760 They thought that that was too big a risk anyway. 01:00:23.760 --> 01:00:26.880 They were tending to run larger experiments, such as 2 01:00:26.880 --> 01:00:29.512 to the 7 minus 2 higher resolution 01:00:29.512 --> 01:00:30.720 fractional factorial designs. 01:00:30.720 --> 01:00:32.320 And they said, so because of that, 01:00:32.320 --> 01:00:34.950 I don't think we should do the adaptive OFAT, 01:00:34.950 --> 01:00:38.730 and we were interested in that condition, that issue. 01:00:38.730 --> 01:00:42.240 It turns out that, if you take the same budget that it takes 01:00:42.240 --> 01:00:45.690 to do the 2 to the 7 minus 2, you would 01:00:45.690 --> 01:00:48.500 be able to run four OFATs. 01:00:48.500 --> 01:00:51.860 Now, if you do that, if you choose 01:00:51.860 --> 01:00:55.100 four different starting points and four different orderings 01:00:55.100 --> 01:00:59.290 of the factors and run four OFATs, and then 01:00:59.290 --> 01:01:04.020 take those results and make an ensemble of them. 01:01:04.020 --> 01:01:06.420 And for example, a simple thing you could do 01:01:06.420 --> 01:01:10.250 is just pick the best of the four. 01:01:10.250 --> 01:01:15.050 Again, the ensemble would produce a better result 01:01:15.050 --> 01:01:16.615 than the fractional factorial design. 01:01:16.615 --> 01:01:17.990 And then the interesting thing we 01:01:17.990 --> 01:01:21.410 find about it is, remember before, when you increase 01:01:21.410 --> 01:01:26.590 experimental error, you expect the OFAT to eventually 01:01:26.590 --> 01:01:28.720 degrade in performance and to cross 01:01:28.720 --> 01:01:30.350 with the fractional factorial. 01:01:30.350 --> 01:01:33.340 So now, the issue is how much experimental error 01:01:33.340 --> 01:01:36.700 can I endure before the adaptive experiment is 01:01:36.700 --> 01:01:39.080 no longer recommended? 01:01:39.080 --> 01:01:42.020 With the ensemble method, it's actually the opposite way. 01:01:42.020 --> 01:01:43.820 As you increase experimental error, 01:01:43.820 --> 01:01:47.990 eventually, the distance between the ensemble 01:01:47.990 --> 01:01:51.810 and the fractional factorial increases. 01:01:51.810 --> 01:01:58.710 So the ensemble method by itself is providing a robustness 01:01:58.710 --> 01:02:01.600 to experimental error. 01:02:01.600 --> 01:02:07.350 And so we've been able to address 01:02:07.350 --> 01:02:12.810 another one of the counterpoints to the adaptive experimentation 01:02:12.810 --> 01:02:14.640 idea. 01:02:14.640 --> 01:02:17.700 That in those cases where experimental budgets are 01:02:17.700 --> 01:02:23.170 higher, you might still want to run these. 01:02:23.170 --> 01:02:25.192 The other benefit of the ensembles 01:02:25.192 --> 01:02:26.900 we've been able to show in another paper. 01:02:26.900 --> 01:02:28.317 I didn't put some slides in, but I 01:02:28.317 --> 01:02:30.010 want to tell you a little bit about it. 01:02:30.010 --> 01:02:31.510 Unless you have a question. 01:02:31.510 --> 01:02:32.768 Maybe I'll take your question. 01:02:32.768 --> 01:02:33.310 AUDIENCE: OK. 01:02:33.310 --> 01:02:39.050 So what are some practical issues 01:02:39.050 --> 01:02:41.280 that you face influencing one factor at a time. 01:02:41.280 --> 01:02:42.780 For example, I can think in my mind, 01:02:42.780 --> 01:02:46.420 say you can't do the measurement immediately 01:02:46.420 --> 01:02:47.800 after each experiment. 01:02:47.800 --> 01:02:48.400 Right? 01:02:48.400 --> 01:02:48.790 PROFESSOR: That's right. 01:02:48.790 --> 01:02:51.220 AUDIENCE: In that case, you would probably still 01:02:51.220 --> 01:02:53.620 have to go to some fractional factorial design. 01:02:53.620 --> 01:02:56.540 Are there other issues that-- like practically, 01:02:56.540 --> 01:02:58.510 because this is incredible. 01:02:58.510 --> 01:03:01.473 It seems like it could save a lot of people a lot of time. 01:03:01.473 --> 01:03:02.140 PROFESSOR: Yeah. 01:03:02.140 --> 01:03:03.182 So you make a good point. 01:03:03.182 --> 01:03:06.190 For example, we know in agriculture, 01:03:06.190 --> 01:03:09.760 you want to run 64 different treatment conditions, 01:03:09.760 --> 01:03:12.980 and it takes a season to get your results. 01:03:12.980 --> 01:03:15.130 So you want to run them in parallel. 01:03:15.130 --> 01:03:16.540 It's important to run experiments 01:03:16.540 --> 01:03:21.460 in parallel in agriculture, and in agricultural equipment, 01:03:21.460 --> 01:03:23.950 it's exactly the opposite. 01:03:23.950 --> 01:03:26.100 So what happens is you're trying to develop 01:03:26.100 --> 01:03:28.350 the next generation of tractor. 01:03:28.350 --> 01:03:31.350 And what they do is they take the last generation of tractor 01:03:31.350 --> 01:03:35.540 that they built, and they use that as what they call a mule. 01:03:35.540 --> 01:03:37.920 In the automotive industry, they use the same term. 01:03:37.920 --> 01:03:39.630 You're making the next Taurus. 01:03:39.630 --> 01:03:42.870 You take the 2007 Taurus, and you 01:03:42.870 --> 01:03:46.260 start using it, putting additions on it-- 01:03:46.260 --> 01:03:49.840 new fuel injection, new valve timing, and so on. 01:03:49.840 --> 01:03:53.640 You use that as a mule, and when you're using articles 01:03:53.640 --> 01:03:57.210 like that in that style, and you have a limited number of them, 01:03:57.210 --> 01:04:01.160 actually, your experiments are necessarily sequential. 01:04:01.160 --> 01:04:05.260 Whereas, in agriculture, they're necessarily parallel. 01:04:05.260 --> 01:04:07.620 So I actually agree with you, when 01:04:07.620 --> 01:04:09.300 you're in an engineering scenario, 01:04:09.300 --> 01:04:12.240 and your experiments are necessarily parallel. 01:04:12.240 --> 01:04:14.760 It sometimes happens, in say lithography. 01:04:14.760 --> 01:04:16.275 You're going to etch it. 01:04:16.275 --> 01:04:18.150 You're going to have to etch all these things 01:04:18.150 --> 01:04:20.370 all at the same time in a batch. 01:04:20.370 --> 01:04:21.390 Yeah. 01:04:21.390 --> 01:04:23.850 That's a reason to do it the other way, 01:04:23.850 --> 01:04:27.090 but I think more often than not, we're doing our experiments 01:04:27.090 --> 01:04:29.910 necessarily sequentially, or maybe 01:04:29.910 --> 01:04:33.860 it's OK to do it sequentially. 01:04:33.860 --> 01:04:37.550 Another factor, since you asked, is the possibility 01:04:37.550 --> 01:04:43.450 of time trends, and indeed, when you 01:04:43.450 --> 01:04:47.800 do adaptive experimentation, you are necessarily making 01:04:47.800 --> 01:04:49.450 restrictions on randomization. 01:04:49.450 --> 01:04:50.620 Right? 01:04:50.620 --> 01:04:53.800 So let's say that you're measuring apparatus 01:04:53.800 --> 01:04:55.420 is drifting over time. 01:04:55.420 --> 01:04:57.730 Let's say it's drifting toward making 01:04:57.730 --> 01:04:59.020 all your results look worse. 01:05:01.540 --> 01:05:04.750 That would give you somewhat of a bias 01:05:04.750 --> 01:05:08.165 toward your starting points, as opposed to all the leader 01:05:08.165 --> 01:05:08.665 changes. 01:05:11.620 --> 01:05:15.590 As best we can tell, actually, the adaptive procedure 01:05:15.590 --> 01:05:19.300 is not so hypersensitive to those time trends. 01:05:19.300 --> 01:05:21.610 They reflect themselves to some degree 01:05:21.610 --> 01:05:24.740 as if they were additional experimental error. 01:05:24.740 --> 01:05:27.790 But if you randomize the starting points anyway 01:05:27.790 --> 01:05:30.760 and the order of factors, I don't 01:05:30.760 --> 01:05:33.383 think it's worse than experimental error. 01:05:33.383 --> 01:05:35.800 Because you're not interested in factor effects in the end 01:05:35.800 --> 01:05:36.700 anyway. 01:05:36.700 --> 01:05:42.160 This is probably the biggest concern I think with the method 01:05:42.160 --> 01:05:43.870 as I'm describing it. 01:05:43.870 --> 01:05:47.830 The biggest issue is you're making an explicit trade-off 01:05:47.830 --> 01:05:52.480 to say, what I'm interested in doing in this case is getting 01:05:52.480 --> 01:05:54.010 some improvement. 01:05:54.010 --> 01:05:57.970 And I'm willing to make a change in my experimental procedure 01:05:57.970 --> 01:06:00.400 that will give me somewhat less knowledge about why 01:06:00.400 --> 01:06:01.900 I got the improvement. 01:06:01.900 --> 01:06:04.720 Now, if you're at Ford, and you've 01:06:04.720 --> 01:06:07.210 got three months to product release, 01:06:07.210 --> 01:06:11.980 and you need to meet your emission standards. 01:06:15.510 --> 01:06:20.580 The goal is to reduce carbon monoxide in those three months, 01:06:20.580 --> 01:06:22.500 and you think the learning that you're 01:06:22.500 --> 01:06:25.800 going to gain about this particular engine 01:06:25.800 --> 01:06:29.130 and its configuration is not likely to be 01:06:29.130 --> 01:06:32.700 reusable on the next one. 01:06:32.700 --> 01:06:36.600 Then, in that case, I think the adaptive method makes sense. 01:06:36.600 --> 01:06:40.500 If instead you think the primary value of your experiment 01:06:40.500 --> 01:06:41.760 is archival. 01:06:41.760 --> 01:06:42.330 Right? 01:06:42.330 --> 01:06:43.710 You're doing the experiment, you're 01:06:43.710 --> 01:06:45.085 going to take the results, you're 01:06:45.085 --> 01:06:47.310 going to make them available to all Ford engineers, 01:06:47.310 --> 01:06:49.320 and you're going to benefit on the next model 01:06:49.320 --> 01:06:50.460 and the next model. 01:06:50.460 --> 01:06:53.670 Then, yeah, you probably ought to arrange the experiments, 01:06:53.670 --> 01:06:57.270 so that you get the highest precision 01:06:57.270 --> 01:07:00.780 of effect estimation, the highest validity of inference. 01:07:00.780 --> 01:07:04.680 That's factorial design of experiments. 01:07:04.680 --> 01:07:09.300 But we had an interesting learning along the way 01:07:09.300 --> 01:07:11.070 and that is to do our big meta studies, 01:07:11.070 --> 01:07:13.320 where we're trying to decide how big are interactions? 01:07:13.320 --> 01:07:14.400 When do they occur? 01:07:14.400 --> 01:07:16.770 We went into Ford's databases. 01:07:16.770 --> 01:07:19.980 They had run lots of experiments in the past, 01:07:19.980 --> 01:07:24.510 and they had them in some company database. 01:07:24.510 --> 01:07:27.690 And we were drawing these data sets out to put them 01:07:27.690 --> 01:07:29.610 into one of our studies. 01:07:29.610 --> 01:07:32.850 And very frequently, we'd go through the table, 01:07:32.850 --> 01:07:35.340 and there was some ambiguity in our minds. 01:07:35.340 --> 01:07:38.210 Maybe there was a plus in the experimental matrix, where 01:07:38.210 --> 01:07:39.960 we thought there should have been a minus, 01:07:39.960 --> 01:07:41.580 or there's some question. 01:07:41.580 --> 01:07:45.450 And in those cases, we would go track the person down 01:07:45.450 --> 01:07:48.000 and ask them what they meant. 01:07:48.000 --> 01:07:50.685 And in almost every case, that person would say, 01:07:50.685 --> 01:07:53.400 huh, you're the first person who ever called me 01:07:53.400 --> 01:07:54.990 about my experiment. 01:07:54.990 --> 01:07:59.370 They'd put it in the database, and no one was using it. 01:07:59.370 --> 01:08:02.730 We were the first ones to go back in and use it. 01:08:02.730 --> 01:08:06.870 So we are not so sanguine about the prospects of people 01:08:06.870 --> 01:08:10.230 reusing experimental data, at least so far 01:08:10.230 --> 01:08:15.660 as industrial, day-to-day experimentation is concerned. 01:08:15.660 --> 01:08:19.560 Most often, those experiments serve their purposes 01:08:19.560 --> 01:08:25.340 at the time, and then they're gone, for practical purposes. 01:08:25.340 --> 01:08:28.040 AUDIENCE: On that note, I was working for a startup company, 01:08:28.040 --> 01:08:31.460 and we would use a facility that we knew other larger 01:08:31.460 --> 01:08:33.340 corporations would use. 01:08:33.340 --> 01:08:35.080 And we had a much smaller budget, 01:08:35.080 --> 01:08:41.490 and I wouldn't to say that my mind [INAUDIBLE] we 01:08:41.490 --> 01:08:43.750 didn't go in with a full factorial design, 01:08:43.750 --> 01:08:46.680 because but we couldn't, but we had certain objectives, 01:08:46.680 --> 01:08:47.770 certain targets. 01:08:47.770 --> 01:08:50.880 And so we go in and say, all right, look at the results 01:08:50.880 --> 01:08:54.260 vary and then change the parameters. 01:08:58.380 --> 01:09:01.229 I think your method has to be more educated than that, 01:09:01.229 --> 01:09:04.470 but it almost seems like it's a similar mentality. 01:09:04.470 --> 01:09:05.640 PROFESSOR: It is. 01:09:05.640 --> 01:09:09.569 Yeah, and in fact, I never formally tried to study this. 01:09:09.569 --> 01:09:15.920 But if you back off a little bit from our structure 01:09:15.920 --> 01:09:18.470 and apply some different structure, one 01:09:18.470 --> 01:09:20.899 that suits your circumstances, it's 01:09:20.899 --> 01:09:24.750 probably still pretty good. 01:09:24.750 --> 01:09:26.880 It might even be better. 01:09:26.880 --> 01:09:32.370 The bigger point is that you're doing adaptation as you go. 01:09:32.370 --> 01:09:35.300 And I have been asked the same question, in fact, 01:09:35.300 --> 01:09:37.830 Jeff Wu asked me at the last time I presented a conference. 01:09:37.830 --> 01:09:42.899 He asked, how much of this result is due to adaptation, 01:09:42.899 --> 01:09:45.720 and how much is due to the one factor at a time part? 01:09:45.720 --> 01:09:47.100 Because this is both. 01:09:47.100 --> 01:09:48.000 Right? 01:09:48.000 --> 01:09:50.569 And it's a hard question to answer. 01:09:50.569 --> 01:09:54.440 My sense is it's mostly adaptation. 01:09:54.440 --> 01:09:59.570 On the other hand, one factor at a time in some sense 01:09:59.570 --> 01:10:01.550 optimizes adaptation. 01:10:01.550 --> 01:10:06.980 You can be adaptive all the time with every new observation 01:10:06.980 --> 01:10:08.550 as much as possible. 01:10:08.550 --> 01:10:12.530 So it's not as if I can cleanly decomposed the two 01:10:12.530 --> 01:10:14.960 contributions to the result, but my sense 01:10:14.960 --> 01:10:16.880 is that it's mostly adaptation. 01:10:16.880 --> 01:10:19.790 And if you're doing some other adaptive procedure, 01:10:19.790 --> 01:10:22.670 especially if it's informed by some prior knowledge 01:10:22.670 --> 01:10:27.720 or topical area knowledge, it's probably a good approach. 01:10:33.860 --> 01:10:35.860 So I'll just tell you about this one last thing. 01:10:35.860 --> 01:10:37.240 OK? 01:10:37.240 --> 01:10:41.930 We did an investigation about another kind of uncertainty 01:10:41.930 --> 01:10:43.180 in large systems. 01:10:43.180 --> 01:10:47.630 So let's say that you're developing the next tractor 01:10:47.630 --> 01:10:52.970 or the next car, and you think about applying 01:10:52.970 --> 01:10:55.100 some robust design to the system. 01:10:55.100 --> 01:10:59.790 Now, for any reasonable scale product, 01:10:59.790 --> 01:11:02.100 there are literally thousands of things 01:11:02.100 --> 01:11:04.740 which you could potentially apply a robust design 01:11:04.740 --> 01:11:05.970 experiment to. 01:11:05.970 --> 01:11:08.040 You can run one on fuel injection. 01:11:08.040 --> 01:11:13.830 You can run one on braking and one on environmental controls 01:11:13.830 --> 01:11:16.090 and one on electrical, just keep going, 01:11:16.090 --> 01:11:17.590 and you're not going to do them all. 01:11:17.590 --> 01:11:22.830 In fact, I would say less than 5% of all opportunities 01:11:22.830 --> 01:11:29.630 to do robustness improvement work are actually executed. 01:11:29.630 --> 01:11:33.470 And then the question I always ask myself is I wonder if-- 01:11:33.470 --> 01:11:35.210 let's say, it's 5%-- 01:11:35.210 --> 01:11:38.043 I wonder if companies are doing the right 5%. 01:11:38.043 --> 01:11:40.460 I wonder if they know where their biggest quality problems 01:11:40.460 --> 01:11:41.690 are. 01:11:41.690 --> 01:11:45.800 Now, we know for a fact that, if you look at retrospectively 01:11:45.800 --> 01:11:47.630 where there have been big quality problems, 01:11:47.630 --> 01:11:51.250 if I look at Ford Explorer and the rollover and so on. 01:11:51.250 --> 01:11:52.160 Right? 01:11:52.160 --> 01:11:54.020 I know they didn't do a robust design 01:11:54.020 --> 01:12:02.540 study on tire inflation and its delamination and rollover. 01:12:02.540 --> 01:12:04.032 They didn't do it. 01:12:04.032 --> 01:12:05.990 Now, why didn't they know that that was the one 01:12:05.990 --> 01:12:07.280 to do robust design on? 01:12:07.280 --> 01:12:09.010 Well, they just didn't. 01:12:09.010 --> 01:12:10.660 It was not so likely after all. 01:12:10.660 --> 01:12:14.020 It wasn't happening to other SUVs. 01:12:14.020 --> 01:12:18.450 So I ask myself, if you could run 01:12:18.450 --> 01:12:25.910 a relatively expensive robust design study on 5%, 01:12:25.910 --> 01:12:27.810 let's say that's the case. 01:12:27.810 --> 01:12:29.270 You have the budget for that. 01:12:29.270 --> 01:12:32.220 What if you did something a tenth as expensive 01:12:32.220 --> 01:12:33.980 and did on 50% of the system? 01:12:37.590 --> 01:12:39.830 So we would call this idea streamlining. 01:12:39.830 --> 01:12:44.120 We know how to take any robust design experiment people are 01:12:44.120 --> 01:12:49.310 proposing and to do something a little bit loose 01:12:49.310 --> 01:12:53.240 but to do it at a tenth of the cost, and you'll get about 80% 01:12:53.240 --> 01:12:55.910 of the same benefit. 01:12:55.910 --> 01:12:57.540 The trade-off works like that. 01:12:57.540 --> 01:13:01.650 It's a good old Pareto 80-20 sort of trade-off. 01:13:01.650 --> 01:13:06.060 And we ask ourselves how that trade-off 01:13:06.060 --> 01:13:10.050 in doing relatively many relatively less 01:13:10.050 --> 01:13:15.310 perfect experiments compares to doing a few very good ones. 01:13:15.310 --> 01:13:20.080 And we ask ourselves how this trade-off 01:13:20.080 --> 01:13:23.500 is affected by the likelihood of identifying 01:13:23.500 --> 01:13:26.020 the one or two biggest quality problems that you face. 01:13:26.020 --> 01:13:29.790 Because that rollover thing cost Ford a lot. 01:13:29.790 --> 01:13:33.290 You would pay a lot of money to avoid that one. 01:13:33.290 --> 01:13:37.720 And so the big conclusion of our study 01:13:37.720 --> 01:13:40.300 is you would have to have a pretty high probability 01:13:40.300 --> 01:13:43.277 of knowing exactly where your problems were in order 01:13:43.277 --> 01:13:44.860 to do it the way they do it now, which 01:13:44.860 --> 01:13:48.290 is relatively small number of expensive studies. 01:13:48.290 --> 01:13:50.980 You'd have to know like 95% certainty 01:13:50.980 --> 01:13:53.200 where your biggest quality problem is. 01:13:53.200 --> 01:13:56.030 If you are at least a little uncertain, 01:13:56.030 --> 01:14:00.370 if you think you're only 70% sure where your problems lie, 01:14:00.370 --> 01:14:05.440 scale them all back by a factor of 10, and do 10 times as many. 01:14:05.440 --> 01:14:06.940 You'll be much better off. 01:14:06.940 --> 01:14:10.040 That's my view, and we didn't have 01:14:10.040 --> 01:14:13.800 to make so many assumptions to come to this conclusion. 01:14:13.800 --> 01:14:16.680 People's uncertainty about where their biggest issues lie 01:14:16.680 --> 01:14:19.230 are bigger than they think, and therefore, they 01:14:19.230 --> 01:14:25.540 need to democratize these kind of processes. 01:14:25.540 --> 01:14:30.160 It's not that you have to do them very, 01:14:30.160 --> 01:14:32.610 very well in a few cases. 01:14:32.610 --> 01:14:35.220 What you need to do is make sure every engineer knows 01:14:35.220 --> 01:14:37.740 how to do this kind of thing, and that they're 01:14:37.740 --> 01:14:43.340 doing it to almost everything. 01:14:43.340 --> 01:14:46.170 At least almost everything that, for example, 01:14:46.170 --> 01:14:48.130 is not proven in the field already. 01:14:48.130 --> 01:14:49.880 You don't need to do it on the alternator, 01:14:49.880 --> 01:14:52.580 because you used that same alternator on the last three 01:14:52.580 --> 01:14:54.400 models, and it's fine. 01:14:54.400 --> 01:14:56.170 But for anything that's relatively new, 01:14:56.170 --> 01:14:59.740 you need to be doing some of this robustness refinement 01:14:59.740 --> 01:15:00.370 work. 01:15:00.370 --> 01:15:03.370 And even if you do just a few experiments, 01:15:03.370 --> 01:15:07.660 exposing your systems to harsh conditions, 01:15:07.660 --> 01:15:09.283 making changes, making improvements 01:15:09.283 --> 01:15:10.700 on the basis of what that reveals, 01:15:10.700 --> 01:15:14.890 that's the key thing, not so much the finesse you apply. 01:15:14.890 --> 01:15:16.570 Anyway, that's the conclusion we came 01:15:16.570 --> 01:15:21.580 to so far due to this research. 01:15:21.580 --> 01:15:27.420 So our main conclusions are that we 01:15:27.420 --> 01:15:29.760 can show through empirical work and through theorems 01:15:29.760 --> 01:15:33.120 that this adaptive procedure gives benefits. 01:15:33.120 --> 01:15:35.970 You get a long way toward your results 01:15:35.970 --> 01:15:39.270 that are desired, especially when interactions are not 01:15:39.270 --> 01:15:40.940 so small. 01:15:40.940 --> 01:15:43.490 And that you can cross these adaptive experiments 01:15:43.490 --> 01:15:45.350 with fractional factorial designs 01:15:45.350 --> 01:15:49.570 to use them for robustness, and it seems to work pretty well. 01:15:49.570 --> 01:15:53.680 And now, I think I'm pretty much out of time. 01:15:53.680 --> 01:15:57.400 Now, if you all have ideas for your case studies, 01:15:57.400 --> 01:16:00.490 I'm always interested in head-to-head comparisons 01:16:00.490 --> 01:16:03.130 of experimental designs being done 01:16:03.130 --> 01:16:05.320 in companies and the adaptive procedures 01:16:05.320 --> 01:16:07.510 that I'm talking about now. 01:16:07.510 --> 01:16:10.590 So if you're interested in doing such things for your projects, 01:16:10.590 --> 01:16:14.190 I'm interested in helping you to do that. 01:16:14.190 --> 01:16:15.980 And you can see my email there, in case 01:16:15.980 --> 01:16:18.920 you want to take me up on the offer. 01:16:18.920 --> 01:16:20.630 OK? 01:16:20.630 --> 01:16:21.530 All set? 01:16:21.530 --> 01:16:22.160 All right. 01:16:22.160 --> 01:16:23.100 Good day. 01:16:23.100 --> 01:16:26.530 [APPLAUSE] 01:16:53.562 --> 01:16:57.440 --Problems that we run into quite often in microfabrication 01:16:57.440 --> 01:17:01.070 is that the metrology of the things that we've made 01:17:01.070 --> 01:17:04.520 takes up quite a lot of our time and budget. 01:17:04.520 --> 01:17:07.190 It's quite easy to vary parameters and make 01:17:07.190 --> 01:17:08.030 many samples. 01:17:08.030 --> 01:17:10.058 But choosing which ones to measure 01:17:10.058 --> 01:17:11.600 might be the biggest challenge, and I 01:17:11.600 --> 01:17:16.580 wonder whether you've looked at situations, where 01:17:16.580 --> 01:17:19.940 the acquisition of the data lags, the doing 01:17:19.940 --> 01:17:23.030 of the experiments, but it's sort of concurrent, 01:17:23.030 --> 01:17:26.870 but you can only measure a certain portion 01:17:26.870 --> 01:17:30.332 of the experiments you do. 01:17:30.332 --> 01:17:34.370 Are there more complicated situations like that 01:17:34.370 --> 01:17:36.200 where there's a lag? 01:17:36.200 --> 01:17:39.155 But it's not the case that you do all the experiments, 01:17:39.155 --> 01:17:41.503 and then you do all the measurements. 01:17:41.503 --> 01:17:43.170 PROFESSOR: It's an interesting question. 01:17:43.170 --> 01:17:48.240 So you're saying that you can do an experiment. 01:17:48.240 --> 01:17:49.250 There's a time lag. 01:17:49.250 --> 01:17:52.370 Then you get an outcome. 01:17:52.370 --> 01:17:55.430 And in some cases, it's actually more complicated than that. 01:17:55.430 --> 01:17:57.890 You get some indication right away, 01:17:57.890 --> 01:18:01.135 but the more preferred measurement comes later. 01:18:01.135 --> 01:18:02.510 So for example, sometimes there's 01:18:02.510 --> 01:18:05.360 a categorical variable that becomes immediately obvious. 01:18:05.360 --> 01:18:07.610 Either I heard chatter, or I didn't. 01:18:07.610 --> 01:18:09.500 And then later, I get a nice measurement 01:18:09.500 --> 01:18:13.100 of surface condition, something like that. 01:18:13.100 --> 01:18:16.700 Well, any of these delays tend to weigh 01:18:16.700 --> 01:18:19.790 in favor of the factorial design as compared to the adaptive. 01:18:19.790 --> 01:18:21.620 You just admit that right away. 01:18:21.620 --> 01:18:23.650 But then I'll say that sometimes, 01:18:23.650 --> 01:18:27.830 even with categorical variables, immediately available 01:18:27.830 --> 01:18:31.880 perceptions, you can get a lot of the benefit of adaptation 01:18:31.880 --> 01:18:33.620 from that. 01:18:33.620 --> 01:18:37.460 PROFESSOR: Yeah, OK, thank you very much. 01:18:37.460 --> 01:18:38.730 Anybody else? 01:18:38.730 --> 01:18:44.120 So we've arranged to have an extra half hour today 01:18:44.120 --> 01:18:46.625 where I can answer questions about problem 01:18:46.625 --> 01:18:49.100 sets and any questions that you might 01:18:49.100 --> 01:18:51.120 have in the run-up to the quiz. 01:18:51.120 --> 01:18:58.380 So I think what we'll do is just go straight into that. 01:18:58.380 --> 01:19:02.025 If people want to run off at this point, then that's fine. 01:19:05.493 --> 01:19:07.618 AUDIENCE: Are you going to [INAUDIBLE] the solution 01:19:07.618 --> 01:19:09.215 of the problem sets [INAUDIBLE]? 01:19:09.215 --> 01:19:09.840 PROFESSOR: Yes. 01:19:09.840 --> 01:19:10.620 Yes. 01:19:10.620 --> 01:19:16.050 So there's a few housekeeping things first. 01:19:16.050 --> 01:19:18.210 Yeah, so problem sets, I will-- 01:19:18.210 --> 01:19:25.710 I'll put the solutions for 6 and 7 today at some point. 01:19:25.710 --> 01:19:31.380 And I put the 2006 quiz, too, and solutions on the website. 01:19:31.380 --> 01:19:33.690 I'm still trying to track down last year's quiz. 01:19:33.690 --> 01:19:37.080 But hopefully that will be out today. 01:19:37.080 --> 01:19:41.430 And yes, for problem set 8, I know 01:19:41.430 --> 01:19:46.000 some people have been sending me questions, 01:19:46.000 --> 01:19:49.560 which I'm happy to continue to answer by email. 01:19:49.560 --> 01:19:53.010 I'll be around this afternoon if you want to come and speak 01:19:53.010 --> 01:19:55.170 to me in person. 01:19:55.170 --> 01:20:02.070 And yes, last week, a few people asked me for extensions. 01:20:02.070 --> 01:20:02.723 That's fine. 01:20:02.723 --> 01:20:04.890 If you need an extension, you can have an extension. 01:20:04.890 --> 01:20:06.690 I'm not super strict about these things. 01:20:09.450 --> 01:20:16.860 OK, so really, the floor is open for you to ask me things. 01:20:16.860 --> 01:20:19.038 And oh, yes grades for 6 and 7 will hopefully 01:20:19.038 --> 01:20:20.580 be done within the next day, as well. 01:20:23.115 --> 01:20:23.990 AUDIENCE: [INAUDIBLE] 01:20:23.990 --> 01:20:26.490 PROFESSOR: Yes? 01:20:26.490 --> 01:20:29.020 Go ahead. 01:20:29.020 --> 01:20:32.590 AUDIENCE: So we have a question. 01:20:32.590 --> 01:20:37.420 When you compare the Taguchi method and a surface response, 01:20:37.420 --> 01:20:39.625 this is for problem 4. 01:20:39.625 --> 01:20:42.670 But you want to compare the number of experiments 01:20:42.670 --> 01:20:45.510 it takes for both methods. 01:20:45.510 --> 01:20:49.020 Are you considering the quadratic terms [INAUDIBLE]?? 01:20:49.020 --> 01:20:50.170 Or are you're not? 01:20:52.930 --> 01:20:56.860 PROFESSOR: In question 4 on problems set 8, 01:20:56.860 --> 01:20:58.220 we're talking about here? 01:20:58.220 --> 01:20:59.032 Yes? 01:20:59.032 --> 01:21:01.240 AUDIENCE: That's right. 01:21:01.240 --> 01:21:05.290 PROFESSOR: I think that it's good to not include 01:21:05.290 --> 01:21:08.152 the quadratic terms in that question, 01:21:08.152 --> 01:21:09.610 although you might want to consider 01:21:09.610 --> 01:21:11.926 what would happen if you did. 01:21:11.926 --> 01:21:14.992 AUDIENCE: OK, so we're only considering the linear terms 01:21:14.992 --> 01:21:15.700 and interactions? 01:21:15.700 --> 01:21:17.117 PROFESSOR: And interactions, yeah. 01:21:17.117 --> 01:21:18.758 I think that's right. 01:21:18.758 --> 01:21:19.300 AUDIENCE: OK. 01:21:19.300 --> 01:21:20.770 PROFESSOR: I think that's the right way forward. 01:21:20.770 --> 01:21:21.100 Yeah? 01:21:21.100 --> 01:21:22.892 AUDIENCE: Only two [INAUDIBLE] interactions 01:21:22.892 --> 01:21:25.900 or three or four [INAUDIBLE]? 01:21:25.900 --> 01:21:28.480 PROFESSOR: Yeah, so with this, we're 01:21:28.480 --> 01:21:31.540 not really interested in quadratic factors. 01:21:31.540 --> 01:21:34.844 Just look at the linear factors in the interactions. 01:21:37.510 --> 01:21:39.700 OK? 01:21:39.700 --> 01:21:42.241 More questions? 01:21:42.241 --> 01:21:44.230 AUDIENCE: Can I ask one last question? 01:21:44.230 --> 01:21:46.110 PROFESSOR: Sure. 01:21:46.110 --> 01:21:47.765 AUDIENCE: So actually, [INAUDIBLE] 01:21:47.765 --> 01:21:50.890 we are kind of confused about question 2, problem 2. 01:21:50.890 --> 01:21:51.630 PROFESSOR: Yeah, everyone's confused about that. 01:21:51.630 --> 01:21:51.880 [LAUGHS] 01:21:51.880 --> 01:21:54.047 AUDIENCE: So can you give us a little [INAUDIBLE] we 01:21:54.047 --> 01:21:55.420 are looking for in question 2? 01:21:55.420 --> 01:21:56.170 PROFESSOR: Yeah. 01:21:56.170 --> 01:22:00.140 So I wrote this question in an attempt to stretch everyone, 01:22:00.140 --> 01:22:01.340 including myself. 01:22:01.340 --> 01:22:05.620 So it's pretty subtle, I think. 01:22:05.620 --> 01:22:08.290 But I would say the right way to approach 01:22:08.290 --> 01:22:15.670 this is to recast that model in terms 01:22:15.670 --> 01:22:20.750 of differences between the variables and the mean input. 01:22:20.750 --> 01:22:29.140 So if you have a set of data, and those data 01:22:29.140 --> 01:22:33.160 have a mean value for x1 and a mean value for x2, 01:22:33.160 --> 01:22:39.520 so you might write it with some new coefficients a-- 01:22:47.782 --> 01:22:52.020 x2 minus x-- sorry, x1 bar, x2 bar. 01:22:59.030 --> 01:23:05.480 And that will be for some values of the a's that are functions 01:23:05.480 --> 01:23:06.020 of the b's. 01:23:06.020 --> 01:23:09.390 That will be equivalent. 01:23:09.390 --> 01:23:13.700 And then, if it was just the first two terms, 01:23:13.700 --> 01:23:16.130 that would be exactly the case that 01:23:16.130 --> 01:23:17.630 was derived in Mayo & Spanos. 01:23:20.930 --> 01:23:24.260 And then this term looks very much like this term, 01:23:24.260 --> 01:23:26.970 except it's for a different variable. 01:23:26.970 --> 01:23:32.490 So the standard error of that coefficient, 01:23:32.490 --> 01:23:35.850 you can get at by the same way you got at this one. 01:23:35.850 --> 01:23:37.970 And then if you look at this term, 01:23:37.970 --> 01:23:41.660 well, we have this coefficient here. 01:23:41.660 --> 01:23:46.040 Well, if you're trying to evaluate the variance of y hat, 01:23:46.040 --> 01:23:48.740 you've got a product of two quantities. 01:23:52.010 --> 01:23:57.820 Do you have an expression for the variance of a11, which 01:23:57.820 --> 01:24:00.880 you're going to get at in a similar way to how 01:24:00.880 --> 01:24:04.300 you get at the expressions for these variances. 01:24:04.300 --> 01:24:06.550 And you'll see the expression for the standard error 01:24:06.550 --> 01:24:09.340 is a function of-- 01:24:09.340 --> 01:24:13.480 is going to be a function of x1 minus x1 bar. 01:24:13.480 --> 01:24:16.630 And you know that when you're finding 01:24:16.630 --> 01:24:23.320 the invariance of some constant times something 01:24:23.320 --> 01:24:28.210 that you know the variance of, you square that constant, 01:24:28.210 --> 01:24:30.700 multiply it by the variance of the thing 01:24:30.700 --> 01:24:33.910 that the constant was multiplied by. 01:24:33.910 --> 01:24:36.370 AUDIENCE: Should it x1 star or x1 [INAUDIBLE] 01:24:36.370 --> 01:24:37.780 PROFESSOR: Oh, sorry, yes. 01:24:37.780 --> 01:24:41.520 Yes, these are-- exactly. 01:24:41.520 --> 01:24:43.480 There you go. 01:24:43.480 --> 01:24:48.820 So this is-- yes, you're trying to find 01:24:48.820 --> 01:24:55.180 an expression for the confidence interval 01:24:55.180 --> 01:24:59.560 as a function of x1 star and x2 star. 01:24:59.560 --> 01:25:03.610 But at a given at a given combination of x1 star 01:25:03.610 --> 01:25:09.130 and x2 star, this is the thing that has variance. 01:25:09.130 --> 01:25:12.590 And this, for a fixed x1 star, is a constant. 01:25:12.590 --> 01:25:20.380 So I don't want to give too much away. 01:25:20.380 --> 01:25:22.520 And I think this is the right way. 01:25:22.520 --> 01:25:25.280 There may be some subtleties that have escaped me. 01:25:25.280 --> 01:25:29.390 So if you can think of any, let me know. 01:25:29.390 --> 01:25:33.400 I think because there aren't any interaction terms in our model, 01:25:33.400 --> 01:25:36.970 I think that means that covariances 01:25:36.970 --> 01:25:39.400 don't need to concern you. 01:25:39.400 --> 01:25:42.010 But if you think I'm wrong on that, let me know. 01:25:45.120 --> 01:25:45.660 OK? 01:25:45.660 --> 01:25:51.550 AUDIENCE: So speaking of the covariance, but x1 and x1 01:25:51.550 --> 01:25:55.270 square, these two are correlated. 01:25:55.270 --> 01:25:58.195 So the covariance-- you have to consider that one, right? 01:26:01.920 --> 01:26:04.350 How could you ignore that? x1 and x2, 01:26:04.350 --> 01:26:05.970 they are not correlated-- 01:26:05.970 --> 01:26:09.280 x1 and x1 square. 01:26:09.280 --> 01:26:11.460 PROFESSOR: Yeah. 01:26:11.460 --> 01:26:13.120 Yeah, maybe I'm wrong. 01:26:13.120 --> 01:26:14.930 Well, if you want to-- 01:26:14.930 --> 01:26:19.364 if you can work out how to consider that, then tell me. 01:26:19.364 --> 01:26:24.150 AUDIENCE: OK, but in terms of the results, can 01:26:24.150 --> 01:26:25.920 we just use an x-- 01:26:25.920 --> 01:26:27.630 the matrix as a general-- 01:26:27.630 --> 01:26:29.790 we don't need to break down the matrix, right? 01:26:29.790 --> 01:26:31.758 PROFESSOR: No, I don't think so, no. 01:26:31.758 --> 01:26:32.300 AUDIENCE: OK. 01:26:38.587 --> 01:26:39.545 PROFESSOR: Anyone else? 01:26:43.520 --> 01:26:45.440 Have we run out of coffee? 01:26:45.440 --> 01:26:46.950 Yes, I expect so. 01:26:50.330 --> 01:26:51.260 Let's see. 01:26:51.260 --> 01:26:55.610 What are the things that people are most puzzled 01:26:55.610 --> 01:27:02.540 by in the material since quiz 1? 01:27:02.540 --> 01:27:08.780 I mean, if someone asked you to fit 01:27:08.780 --> 01:27:13.640 a model to a factorial experiment, 01:27:13.640 --> 01:27:18.200 and you didn't have Minitab what level of confidence would 01:27:18.200 --> 01:27:20.780 people have in doing that? 01:27:20.780 --> 01:27:23.510 I mean, is it really something where you rely on the structure 01:27:23.510 --> 01:27:27.670 that the software provides to know how to calculate some 01:27:27.670 --> 01:27:30.590 of the squares and so forth? 01:27:30.590 --> 01:27:35.120 Because I can foresee exam questions-- 01:27:35.120 --> 01:27:36.620 obviously not going to have Minitab. 01:27:36.620 --> 01:27:39.920 We're going to be dealing with very small numbers of data, 01:27:39.920 --> 01:27:42.450 and it's just going to be a case of knowing what 01:27:42.450 --> 01:27:43.700 to subtract from what, really. 01:27:43.700 --> 01:27:44.930 AUDIENCE: [INAUDIBLE] practice. 01:27:44.930 --> 01:27:46.097 PROFESSOR: Yeah, absolutely. 01:27:46.097 --> 01:27:48.970 AUDIENCE: [INAUDIBLE] with the [INAUDIBLE],, you could do it. 01:27:48.970 --> 01:27:50.720 But we won't [INAUDIBLE] practice 01:27:50.720 --> 01:27:54.750 doing it versus using JMP or Minitab [INAUDIBLE] get used 01:27:54.750 --> 01:27:57.390 to the data being [INAUDIBLE]. 01:27:57.390 --> 01:27:59.385 PROFESSOR: Yes. 01:27:59.385 --> 01:28:02.613 AUDIENCE: I think it would be more time-consuming. 01:28:02.613 --> 01:28:03.530 PROFESSOR: Fair point. 01:28:03.530 --> 01:28:05.240 AUDIENCE: [INAUDIBLE] 01:28:05.240 --> 01:28:06.815 PROFESSOR: Yeah, I need to I need 01:28:06.815 --> 01:28:08.690 to track it down, because they didn't post it 01:28:08.690 --> 01:28:09.830 on Stellar last year. 01:28:09.830 --> 01:28:13.190 So I need to get it up [INAUDIBLE].. 01:28:13.190 --> 01:28:13.970 Yes? 01:28:13.970 --> 01:28:15.490 Who's that? 01:28:15.490 --> 01:28:17.750 AUDIENCE: [INAUDIBLE] question 4-- can you 01:28:17.750 --> 01:28:22.310 explain more about the noise? 01:28:22.310 --> 01:28:24.480 You mean that you cannot [INAUDIBLE] during 01:28:24.480 --> 01:28:25.070 an experiment? 01:28:25.070 --> 01:28:29.473 [INAUDIBLE] 01:28:29.473 --> 01:28:30.140 PROFESSOR: Sure. 01:28:30.140 --> 01:28:32.390 Sure, yeah, this question's a bit cryptic. 01:28:32.390 --> 01:28:34.520 But really, all it's asking you to do 01:28:34.520 --> 01:28:38.000 is confirm your understanding of the last question 01:28:38.000 --> 01:28:39.530 on problem set 7. 01:28:39.530 --> 01:28:45.170 You may recall from that question 01:28:45.170 --> 01:28:50.390 where you made that factor z, that noise factor, 01:28:50.390 --> 01:28:53.810 you were imagining that you could control it 01:28:53.810 --> 01:28:55.130 in that situation. 01:28:55.130 --> 01:29:00.980 But then I think you will have written an expression for y 01:29:00.980 --> 01:29:04.430 in terms of the two X's and the z factor 01:29:04.430 --> 01:29:11.780 that showed that the sensitivity of the output to variations 01:29:11.780 --> 01:29:15.860 in z were a function of x1 and x2. 01:29:15.860 --> 01:29:18.530 So you could choose a combination 01:29:18.530 --> 01:29:22.670 of the control parameters at x1 and x2 01:29:22.670 --> 01:29:26.660 that would minimize the sensitivity to z. 01:29:26.660 --> 01:29:30.290 Now, really parts B and C of problem 4 01:29:30.290 --> 01:29:34.250 are just sort of checking that that went in, 01:29:34.250 --> 01:29:41.870 because if you can't control z, but you know that if you're not 01:29:41.870 --> 01:29:44.870 controlling it, it's probably going to have a variance, 01:29:44.870 --> 01:29:47.240 the input, the noise input, is probably 01:29:47.240 --> 01:29:50.040 going to have a variance that doesn't change with time. 01:29:50.040 --> 01:29:53.690 So you can see how much of that noise propagates 01:29:53.690 --> 01:29:57.710 through to the outputs if the noise interacts 01:29:57.710 --> 01:30:00.210 with the control input variables-- in other words, 01:30:00.210 --> 01:30:03.170 if there are these product terms of z and x1. 01:30:03.170 --> 01:30:04.130 You can't control z. 01:30:04.130 --> 01:30:05.600 You can control x1. 01:30:05.600 --> 01:30:09.170 Therefore, you can control how much the noise propagates 01:30:09.170 --> 01:30:10.550 through to the outputs. 01:30:10.550 --> 01:30:15.530 And B and C aren't asking you to write down very much. 01:30:15.530 --> 01:30:18.740 It's just sort of checking that the concept is there, really. 01:30:22.330 --> 01:30:25.420 Does that answer your question? 01:30:25.420 --> 01:30:26.170 AUDIENCE: Oh, yes. 01:30:26.170 --> 01:30:26.430 Thank you. 01:30:26.430 --> 01:30:27.180 PROFESSOR: Thanks. 01:30:40.010 --> 01:30:41.450 OK, anyone else? 01:30:41.450 --> 01:30:42.183 Dave? 01:30:42.183 --> 01:30:47.570 AUDIENCE: Do you have an example of a potential modeled problem 01:30:47.570 --> 01:30:50.308 that might be of a smaller data set that 01:30:50.308 --> 01:30:52.100 might be a little easier to go through some 01:30:52.100 --> 01:30:56.513 of the calculations of fitting a model to the experiment? 01:30:56.513 --> 01:30:58.680 I know a lot of our [INAUDIBLE] a little bit bigger, 01:30:58.680 --> 01:31:01.710 and it may be useful to be able to work something through. 01:31:01.710 --> 01:31:03.620 PROFESSOR: That's a very good idea. 01:31:03.620 --> 01:31:06.200 Yes. 01:31:06.200 --> 01:31:06.740 Yeah. 01:31:06.740 --> 01:31:08.270 I'll try to come up with something 01:31:08.270 --> 01:31:12.650 that's sort of manageable on a sheet of paper-- 01:31:12.650 --> 01:31:16.060 yes, very, very good thought. 01:31:16.060 --> 01:31:19.940 We may be able to do with some of the data 01:31:19.940 --> 01:31:21.350 sets we used to look at ANOVA. 01:31:21.350 --> 01:31:24.746 But yeah, fine. 01:31:24.746 --> 01:31:31.090 AUDIENCE: I had some confusion with problem [INAUDIBLE] 01:31:31.090 --> 01:31:37.781 the lack of [INAUDIBLE] I was having 01:31:37.781 --> 01:31:41.212 some trouble with [INAUDIBLE] 01:31:41.212 --> 01:31:41.920 PROFESSOR: Right. 01:31:41.920 --> 01:31:42.420 Right. 01:31:42.420 --> 01:31:45.854 AUDIENCE: [INAUDIBLE] if you could do [INAUDIBLE] 01:31:45.854 --> 01:31:47.970 PROFESSOR: [INAUDIBLE] points, OK. 01:31:47.970 --> 01:31:48.970 Fine, fine. 01:31:48.970 --> 01:31:50.460 So that was on problem set 7. 01:31:50.460 --> 01:31:53.003 You were at 12-1 or something, but yeah. 01:31:53.003 --> 01:31:54.420 AUDIENCE: I don't know [INAUDIBLE] 01:31:54.420 --> 01:31:55.128 PROFESSOR: Right. 01:31:55.128 --> 01:31:56.890 No, that's a good point. 01:31:56.890 --> 01:32:00.130 A lot of people had difficulties with that. 01:32:00.130 --> 01:32:07.110 So that I will look at, as well, OK? 01:32:07.110 --> 01:32:12.872 AUDIENCE: [INAUDIBLE] problem with something [INAUDIBLE] 01:32:12.872 --> 01:32:13.580 PROFESSOR: Sorry? 01:32:13.580 --> 01:32:14.455 Something [INAUDIBLE] 01:32:14.455 --> 01:32:20.990 AUDIENCE: For example, [INAUDIBLE] minus 1 [INAUDIBLE] 01:32:20.990 --> 01:32:21.800 PROFESSOR: OK. 01:32:21.800 --> 01:32:25.190 AUDIENCE: [INAUDIBLE] some examples. 01:32:25.190 --> 01:32:33.953 [INAUDIBLE] data versus [INAUDIBLE] so you don't have 01:32:33.953 --> 01:32:39.800 enough information [INAUDIBLE] I don't know why, but that's-- 01:32:39.800 --> 01:32:44.350 PROFESSOR: Yeah, I think there is a mistake in Montgomery. 01:32:44.350 --> 01:32:45.350 I need to track it down. 01:32:45.350 --> 01:32:51.740 AUDIENCE: [INAUDIBLE] makes sense [INAUDIBLE] look at that. 01:32:51.740 --> 01:32:54.870 [INAUDIBLE] you don't really need to add all data 01:32:54.870 --> 01:32:57.860 and calculate all contrasts. 01:32:57.860 --> 01:33:01.510 So there is some [INAUDIBLE] 01:33:01.510 --> 01:33:03.380 AUDIENCE: [INAUDIBLE] 01:33:03.380 --> 01:33:07.974 AUDIENCE: [INAUDIBLE] example you have [INAUDIBLE] data. 01:33:07.974 --> 01:33:08.968 AUDIENCE: [INAUDIBLE] 01:33:08.968 --> 01:33:13.450 AUDIENCE: [INAUDIBLE] 01:33:13.450 --> 01:33:15.430 PROFESSOR: OK, sure. 01:33:15.430 --> 01:33:16.450 AUDIENCE: I'm not sure. 01:33:16.450 --> 01:33:21.970 On the Mayo & Spanos, on the 8.1.3, 01:33:21.970 --> 01:33:28.010 [INAUDIBLE] duration of [INAUDIBLE] of the parameters. 01:33:28.010 --> 01:33:30.485 It seems to me that [INAUDIBLE] just jumping 01:33:30.485 --> 01:33:33.550 and without [INAUDIBLE]. 01:33:33.550 --> 01:33:36.100 PROFESSOR: Yeah, you're right. 01:33:36.100 --> 01:33:41.380 AUDIENCE: 8.1.3 It's a general form of the [INAUDIBLE] 01:33:41.380 --> 01:33:48.520 regression [INAUDIBLE] using the matrix [INAUDIBLE] 01:33:48.520 --> 01:33:51.520 PROFESSOR: Yeah, right. 01:33:51.520 --> 01:33:57.820 Well, sure, I can try to find a complete derivation. 01:33:57.820 --> 01:34:02.437 I think it's probably not worth spending a lot of time 01:34:02.437 --> 01:34:03.520 trying to understand that. 01:34:03.520 --> 01:34:07.750 I would focus more on the stuff that's 01:34:07.750 --> 01:34:09.530 been in the problem sets. 01:34:09.530 --> 01:34:11.830 But sure, I can try to find that. 01:34:11.830 --> 01:34:16.720 AUDIENCE: Because if you can get a derivation of that, 01:34:16.720 --> 01:34:20.553 any kind of regression data just changes form. 01:34:20.553 --> 01:34:21.220 PROFESSOR: Yeah. 01:34:21.220 --> 01:34:24.160 AUDIENCE: So there is no need to do that kind of thing, 01:34:24.160 --> 01:34:26.500 try to change the form of it. 01:34:26.500 --> 01:34:29.127 [INAUDIBLE] just need to fit it. 01:34:29.127 --> 01:34:30.710 PROFESSOR: Yes, I'm sure you're right. 01:34:30.710 --> 01:34:32.850 I'm sure you're right.