1 00:00:00,000 --> 00:00:02,460 The following content is provided under a Creative 2 00:00:02,460 --> 00:00:03,730 Commons license. 3 00:00:03,730 --> 00:00:06,060 Your support will help MIT OpenCourseWare 4 00:00:06,060 --> 00:00:10,090 continue to offer high quality educational resources for free. 5 00:00:10,090 --> 00:00:12,690 To make a donation or to view additional materials 6 00:00:12,690 --> 00:00:16,560 from hundreds of MIT courses, visit MIT OpenCourseWare 7 00:00:16,560 --> 00:00:17,904 at ocw.mit.edu. 8 00:00:20,970 --> 00:00:23,220 DUANE BONING: OK, so what I'm going to try to do today 9 00:00:23,220 --> 00:00:28,290 is kind of a fun case study on spatial modeling. 10 00:00:28,290 --> 00:00:31,080 And then we will have time afterwards, 11 00:00:31,080 --> 00:00:32,580 especially with the Singapore folks, 12 00:00:32,580 --> 00:00:34,800 to try to at least have brief meetings 13 00:00:34,800 --> 00:00:36,450 on a couple of the projects. 14 00:00:36,450 --> 00:00:38,190 I've met with at least one group. 15 00:00:38,190 --> 00:00:42,180 I think one group you guys met with Dave Hardt. 16 00:00:42,180 --> 00:00:43,950 At least talked, email. 17 00:00:43,950 --> 00:00:46,161 Did you meet also? 18 00:00:46,161 --> 00:00:47,772 AUDIENCE: [INAUDIBLE] 19 00:00:47,772 --> 00:00:48,480 DUANE BONING: OK. 20 00:00:48,480 --> 00:00:52,094 Because it looked like you guys are in good shape to, right? 21 00:00:52,094 --> 00:00:53,710 AUDIENCE: I don't know. 22 00:00:53,710 --> 00:00:55,730 DUANE BONING: He said do more. 23 00:00:55,730 --> 00:00:58,682 AUDIENCE: [INAUDIBLE] 24 00:00:58,682 --> 00:00:59,390 DUANE BONING: OK. 25 00:00:59,390 --> 00:01:01,940 So we can have some follow up on that as well. 26 00:01:01,940 --> 00:01:04,450 But what I'm going to do is go through this case study. 27 00:01:07,030 --> 00:01:08,947 My preferences actually this is-- 28 00:01:08,947 --> 00:01:10,780 I would have liked to have been able to talk 29 00:01:10,780 --> 00:01:13,197 about this a little bit earlier, because this is some very 30 00:01:13,197 --> 00:01:15,730 neat stuff on spatial modeling that in the past 31 00:01:15,730 --> 00:01:17,728 has sometimes been a part of some projects. 32 00:01:17,728 --> 00:01:19,270 So maybe you can look at this and see 33 00:01:19,270 --> 00:01:22,960 if there's something extra on spatial modeling you can do. 34 00:01:22,960 --> 00:01:25,780 Essentially, this is going to be based on three papers that 35 00:01:25,780 --> 00:01:27,580 are also on the website. 36 00:01:27,580 --> 00:01:31,000 There's a paper by Davis, one by Mozumder, 37 00:01:31,000 --> 00:01:34,070 and one by Guo and Sachs. 38 00:01:34,070 --> 00:01:37,070 So you can grab those papers and learn more. 39 00:01:37,070 --> 00:01:40,310 But my basic agenda is as shown here. 40 00:01:40,310 --> 00:01:43,600 First I want to sort of talk a little bit about applying 41 00:01:43,600 --> 00:01:46,180 the same response surface modeling and regression 42 00:01:46,180 --> 00:01:48,400 methodology that we've talked about 43 00:01:48,400 --> 00:01:50,630 for things like process conditions 44 00:01:50,630 --> 00:01:53,250 also to spatial parameters. 45 00:01:53,250 --> 00:01:56,050 So if you were spatially sampling, 46 00:01:56,050 --> 00:02:00,130 could you build a model as a function of spatial coordinates 47 00:02:00,130 --> 00:02:03,580 and get a sense of spatial uniformity across a wafer, 48 00:02:03,580 --> 00:02:06,340 across a part, across some spatial dimension? 49 00:02:06,340 --> 00:02:08,350 And of course, the answer is yes. 50 00:02:08,350 --> 00:02:11,470 What's interesting is then when you go and calculate things 51 00:02:11,470 --> 00:02:14,380 like uniformity or uniformity metrics, 52 00:02:14,380 --> 00:02:16,610 there's some special issues that come up. 53 00:02:16,610 --> 00:02:19,870 And in particular, your sampling plan 54 00:02:19,870 --> 00:02:24,820 can affect dramatically what number you get for uniformity. 55 00:02:24,820 --> 00:02:30,100 And so we'll talk about ways to not fool yourself on that. 56 00:02:30,100 --> 00:02:34,660 The second part will be then to go back and say, OK, 57 00:02:34,660 --> 00:02:38,670 if I can do a response surface model on spatial coordinates 58 00:02:38,670 --> 00:02:42,270 and the process is changing, how do I put those two things 59 00:02:42,270 --> 00:02:44,910 together to model spatial variation 60 00:02:44,910 --> 00:02:46,830 as a function of the process? 61 00:02:46,830 --> 00:02:49,350 And there's two different ideas, sort 62 00:02:49,350 --> 00:02:52,860 of opposite cuts at the same problem, one based 63 00:02:52,860 --> 00:02:56,760 on the Mozumder paper and the other by Guo and Sachs 64 00:02:56,760 --> 00:03:00,960 that I think are a very interesting case 65 00:03:00,960 --> 00:03:06,300 study in that they show an evolution in the thinking 66 00:03:06,300 --> 00:03:09,750 about how to efficiently and effectively do that. 67 00:03:09,750 --> 00:03:13,080 So let me talk first about just purely spatial modeling. 68 00:03:13,080 --> 00:03:14,760 And of course, we know that there 69 00:03:14,760 --> 00:03:19,740 can be a spatial trend, systematic trend 70 00:03:19,740 --> 00:03:22,290 in many, many processes. 71 00:03:22,290 --> 00:03:25,230 Most of the methods we've been talking about so far kind 72 00:03:25,230 --> 00:03:27,330 of we're looking at random variation, ways 73 00:03:27,330 --> 00:03:33,120 of assessing as if we were sampling 74 00:03:33,120 --> 00:03:36,120 from a Gaussian or some other process. 75 00:03:36,120 --> 00:03:39,330 But underlying that, you may have that noise, measurement 76 00:03:39,330 --> 00:03:41,670 noise, or whatever, but underlying that, 77 00:03:41,670 --> 00:03:44,640 there may be a systematic spatial trend. 78 00:03:44,640 --> 00:03:50,760 And these may result from inherent equipment or process 79 00:03:50,760 --> 00:03:54,970 asymmetries, be highly repeatable. 80 00:03:54,970 --> 00:03:57,300 You would like to be able to capture and express them. 81 00:03:57,300 --> 00:03:59,040 So I've sort of pictured here an example 82 00:03:59,040 --> 00:04:04,650 where you might have a concave or convex shape. 83 00:04:04,650 --> 00:04:07,710 If I were measuring some parameter across the wafer, 84 00:04:07,710 --> 00:04:11,190 like a film thickness, other geometric parameters, 85 00:04:11,190 --> 00:04:15,030 or some material property like film resistivity. 86 00:04:15,030 --> 00:04:17,279 And very often maybe gas flows are 87 00:04:17,279 --> 00:04:20,730 a little bit different in the center of the wafer and so on. 88 00:04:20,730 --> 00:04:23,760 You get these sort of bowl shaped patterns. 89 00:04:23,760 --> 00:04:25,890 But there may be other kinds of spatial patterns, 90 00:04:25,890 --> 00:04:29,490 and you'd like to be able to capture that. 91 00:04:29,490 --> 00:04:31,420 So our goal is how do we model that? 92 00:04:31,420 --> 00:04:33,330 And then how do we boil that down 93 00:04:33,330 --> 00:04:36,690 into an overall metric that says how good this is and then drive 94 00:04:36,690 --> 00:04:38,550 the process to try to improve that? 95 00:04:38,550 --> 00:04:42,430 Decrease non-uniformity, improve uniformity. 96 00:04:42,430 --> 00:04:44,530 So what I'm going to do is show an example. 97 00:04:44,530 --> 00:04:48,780 And what I'm going to do here is use synthetic data. 98 00:04:48,780 --> 00:04:52,410 By the way, what I mean by synthetic data is 99 00:04:52,410 --> 00:04:58,210 I'm going to generate data that I know it's properties of. 100 00:04:58,210 --> 00:04:59,970 I am going to generate a data with 101 00:04:59,970 --> 00:05:05,220 a known statistical and systematic, a known random 102 00:05:05,220 --> 00:05:10,260 and a known systematic component to it. 103 00:05:10,260 --> 00:05:13,680 This is a useful technique that you 104 00:05:13,680 --> 00:05:18,460 might consider somewhere in one of your team projects as well. 105 00:05:18,460 --> 00:05:20,760 You may be dealing with real data. 106 00:05:20,760 --> 00:05:23,280 And very often you don't know what ground truth is. 107 00:05:23,280 --> 00:05:25,530 And that's the whole interesting thing about a project 108 00:05:25,530 --> 00:05:27,450 is you don't really know the truth. 109 00:05:27,450 --> 00:05:29,940 What we're trying to do is infer things about it 110 00:05:29,940 --> 00:05:31,740 from experimental data. 111 00:05:31,740 --> 00:05:34,810 If you're also exploring some methodology, 112 00:05:34,810 --> 00:05:38,010 you want to see does this new methodology that I've come up 113 00:05:38,010 --> 00:05:40,980 with that I think solves some problem 114 00:05:40,980 --> 00:05:44,370 or applies to some aspect of this problem? 115 00:05:44,370 --> 00:05:47,610 You can also test that with synthetic data 116 00:05:47,610 --> 00:05:49,110 that you generate yourself. 117 00:05:49,110 --> 00:05:54,600 And it's a nice complement in some cases to the other project 118 00:05:54,600 --> 00:05:56,670 data that you're doing. 119 00:05:56,670 --> 00:06:01,020 So the data that I'm generating has a systematic component 120 00:06:01,020 --> 00:06:06,710 that is a circular or elliptic kind of wafer map. 121 00:06:06,710 --> 00:06:10,840 Plus it's got some superimposed random noise on it. 122 00:06:10,840 --> 00:06:14,310 And what I'm going to do is sample it 123 00:06:14,310 --> 00:06:15,820 in two different ways. 124 00:06:15,820 --> 00:06:21,000 So I'm going to generate two sampled data sets off 125 00:06:21,000 --> 00:06:24,930 of a more basic data set that I create. 126 00:06:24,930 --> 00:06:26,670 And the sampling patterns are going 127 00:06:26,670 --> 00:06:31,290 to be two that are commonly used in semiconductor manufacturing. 128 00:06:31,290 --> 00:06:33,850 One is a circular wafer map. 129 00:06:33,850 --> 00:06:34,350 Let's see. 130 00:06:34,350 --> 00:06:35,370 Do I have a picture of it? 131 00:06:35,370 --> 00:06:35,870 Yeah. 132 00:06:35,870 --> 00:06:40,110 So here's a radial or circular sampling plan. 133 00:06:40,110 --> 00:06:43,830 The basic idea that I take a point at the center, 134 00:06:43,830 --> 00:06:47,340 then I take points around some radius, 135 00:06:47,340 --> 00:06:50,250 I take additional points around another radius, 136 00:06:50,250 --> 00:06:53,350 and I take additional points at a third radius 137 00:06:53,350 --> 00:06:57,510 so that I've got these rings of concentric circles 138 00:06:57,510 --> 00:06:59,470 as a sampling plan. 139 00:06:59,470 --> 00:07:02,220 An alternative one is to simply use 140 00:07:02,220 --> 00:07:07,410 a rectangular or square sampling plan. 141 00:07:07,410 --> 00:07:09,750 So those are going to be two different sampling plans. 142 00:07:09,750 --> 00:07:12,840 What you would hope is you get the same uniformity 143 00:07:12,840 --> 00:07:14,640 metric off of the two, that it shouldn't 144 00:07:14,640 --> 00:07:16,200 be sensitive to sampling. 145 00:07:16,200 --> 00:07:21,480 I'm going to show you, in fact, that it can be quite sensitive. 146 00:07:21,480 --> 00:07:23,940 OK, so what's my synthetic data? 147 00:07:23,940 --> 00:07:24,880 It's shown here. 148 00:07:24,880 --> 00:07:27,030 This is in fact the elliptic component, 149 00:07:27,030 --> 00:07:31,120 the systematic component, that I create some output. 150 00:07:31,120 --> 00:07:33,550 Call it resistivity here just to be concrete. 151 00:07:36,160 --> 00:07:41,210 Plus random Gaussian 0 mean noise. 152 00:07:41,210 --> 00:07:42,970 So I've got, say, some measurement noise, 153 00:07:42,970 --> 00:07:46,120 process noise superimposed on this. 154 00:07:46,120 --> 00:07:48,740 What I'm going to try to do is two things. 155 00:07:48,740 --> 00:07:52,930 One is build a response surface model. 156 00:07:52,930 --> 00:07:56,770 Maybe, hopefully, recover something close 157 00:07:56,770 --> 00:08:00,760 to the underlying ground truth that I've imposed here. 158 00:08:00,760 --> 00:08:04,600 And we can now assess how good a response surface model 159 00:08:04,600 --> 00:08:07,510 fit is, because I know what the truth is. 160 00:08:07,510 --> 00:08:09,940 Second, I'm also going to calculate 161 00:08:09,940 --> 00:08:11,860 a non-uniformity metric. 162 00:08:11,860 --> 00:08:14,440 And the basic metric is going to be something 163 00:08:14,440 --> 00:08:18,880 like a noise to signal ratio. 164 00:08:18,880 --> 00:08:22,180 That is to say, the normalized standard deviation 165 00:08:22,180 --> 00:08:24,080 across my measurement points. 166 00:08:24,080 --> 00:08:25,480 So I calculate the mean. 167 00:08:25,480 --> 00:08:28,450 I calculate the standard deviation, normalize or divide. 168 00:08:28,450 --> 00:08:30,490 So I'm getting something, if I multiply 169 00:08:30,490 --> 00:08:34,659 by 100, something like a percentage non-uniformity 170 00:08:34,659 --> 00:08:35,710 across this wafer. 171 00:08:38,260 --> 00:08:41,230 Here's the data, just so you have it. 172 00:08:41,230 --> 00:08:49,720 What I get in this sampling plan is 25 total data points, 173 00:08:49,720 --> 00:08:53,110 one at the center and then eight at each of 25, 50, 174 00:08:53,110 --> 00:08:55,930 and 75 millimeters away from the center. 175 00:08:55,930 --> 00:09:00,820 This is assuming sort of a 200 millimeter wafer, if you will. 176 00:09:00,820 --> 00:09:09,010 And here's the actual data based on my underlying synthetic data 177 00:09:09,010 --> 00:09:09,950 model. 178 00:09:09,950 --> 00:09:12,520 Here's the square sampling plan. 179 00:09:12,520 --> 00:09:15,160 In this case, you'll notice that a few 180 00:09:15,160 --> 00:09:17,710 of the points, these corner points, 181 00:09:17,710 --> 00:09:21,190 actually fall off the edge of a round wafer. 182 00:09:21,190 --> 00:09:23,170 So they are discarded. 183 00:09:23,170 --> 00:09:25,390 And those are the ones in bold here. 184 00:09:25,390 --> 00:09:28,030 Those ones are removed, because they're 185 00:09:28,030 --> 00:09:30,670 outside of the wafer boundary. 186 00:09:30,670 --> 00:09:33,760 That should be a plus minus. 187 00:09:33,760 --> 00:09:38,530 So this actually has, in fact, slightly fewer data points. 188 00:09:38,530 --> 00:09:43,060 It has only 21 data points, whereas the previous sampling 189 00:09:43,060 --> 00:09:44,610 plan actually has more data. 190 00:09:44,610 --> 00:09:46,390 It's 25. 191 00:09:46,390 --> 00:09:47,170 Yeah. 192 00:09:47,170 --> 00:09:48,420 3 times 8 plus 1. 193 00:09:48,420 --> 00:09:51,310 25 data points. 194 00:09:51,310 --> 00:09:52,750 OK. 195 00:09:52,750 --> 00:09:54,610 One thing I want to-- 196 00:09:54,610 --> 00:09:59,170 I'll be showing some jump plots, some surface plots, contours, 197 00:09:59,170 --> 00:10:00,100 and so on. 198 00:10:00,100 --> 00:10:03,820 And the contouring does show a little bit of difference 199 00:10:03,820 --> 00:10:08,140 because it interpolates just to plot things. 200 00:10:08,140 --> 00:10:10,870 But in both of these cases, they pretty much 201 00:10:10,870 --> 00:10:13,780 look similar except for some adjustment 202 00:10:13,780 --> 00:10:16,300 because of the location of the data points. 203 00:10:16,300 --> 00:10:20,530 And if anything, the radial 25 points 204 00:10:20,530 --> 00:10:23,740 looks like it actually kind of interpolates a little bit more 205 00:10:23,740 --> 00:10:28,240 to the roundish or elliptic underlying pattern 206 00:10:28,240 --> 00:10:29,950 that I said I was imposing. 207 00:10:29,950 --> 00:10:31,690 This one because of the square grid 208 00:10:31,690 --> 00:10:33,820 looks like it may have some artifacts. 209 00:10:33,820 --> 00:10:36,160 But that gives you a feel for what's going on. 210 00:10:36,160 --> 00:10:41,320 What we've got is a not quite center high. 211 00:10:41,320 --> 00:10:43,960 It's not a pure bowl right in the center. 212 00:10:43,960 --> 00:10:46,300 It's a little bit off center, and it's not 213 00:10:46,300 --> 00:10:48,190 perfectly circular. 214 00:10:48,190 --> 00:10:49,690 It's kind of elliptic. 215 00:10:49,690 --> 00:10:53,410 So that gives you a feel for what the underlying pattern is. 216 00:10:53,410 --> 00:10:56,260 So I say I want to fit a model. 217 00:10:56,260 --> 00:11:01,150 I'm going to fit a generic second order model in x and y. 218 00:11:01,150 --> 00:11:04,990 So it's got linear coefficients in x and y, an interaction 219 00:11:04,990 --> 00:11:09,330 term, and second order terms, x squared and y squared. 220 00:11:09,330 --> 00:11:12,490 So six total coefficients. 221 00:11:12,490 --> 00:11:15,760 What I'm doing first here is the radial pattern 222 00:11:15,760 --> 00:11:18,460 where I had 25 data points. 223 00:11:18,460 --> 00:11:21,220 And you can sort of see what happens here. 224 00:11:21,220 --> 00:11:23,680 What we've got is an r squared that 225 00:11:23,680 --> 00:11:26,680 looks like it's about 0.65. 226 00:11:26,680 --> 00:11:31,250 Doesn't sound great, but I know there's noise in the process. 227 00:11:31,250 --> 00:11:34,330 So I got to look at it in a little bit 228 00:11:34,330 --> 00:11:38,800 more detail at the ANOVA and see is my model significant. 229 00:11:38,800 --> 00:11:40,900 The r squared is make me a little bit worried. 230 00:11:40,900 --> 00:11:42,400 It's not great. 231 00:11:42,400 --> 00:11:46,180 But I look, and overall the model compared to the error 232 00:11:46,180 --> 00:11:51,100 terms is significant. 233 00:11:53,620 --> 00:11:58,420 So I've got an f that is much, much larger 234 00:11:58,420 --> 00:12:00,220 than I would expect by chance alone. 235 00:12:00,220 --> 00:12:04,660 It's 99.94% significant. 236 00:12:04,660 --> 00:12:05,320 OK, great. 237 00:12:05,320 --> 00:12:09,520 So I've got a model that is capturing something going on. 238 00:12:09,520 --> 00:12:12,160 Then I can go in and look at the parameter estimates, 239 00:12:12,160 --> 00:12:15,190 look at each of the individual terms, look at the t ratio, 240 00:12:15,190 --> 00:12:17,980 and look at the probability of those being significant. 241 00:12:17,980 --> 00:12:20,530 And they all look pretty significant 242 00:12:20,530 --> 00:12:26,785 except this one for sure, a y term. 243 00:12:26,785 --> 00:12:29,540 It says no, 0.5 chance that would 244 00:12:29,540 --> 00:12:32,050 have occurred by chance alone. 245 00:12:32,050 --> 00:12:40,100 And I've only got maybe 92 %or 91% chance or level 246 00:12:40,100 --> 00:12:44,040 of confidence in this xy term. 247 00:12:44,040 --> 00:12:48,020 But if I was looking at, say, 95% confidence 248 00:12:48,020 --> 00:12:52,190 interval cutoffs, I might reject that term. 249 00:12:52,190 --> 00:12:55,160 And we'll see later, that's correctly rejecting 250 00:12:55,160 --> 00:12:57,320 that term if you look back. 251 00:12:57,320 --> 00:12:59,570 So knowing ground truth, actually this 252 00:12:59,570 --> 00:13:01,400 raises already a worry. 253 00:13:01,400 --> 00:13:05,480 How come on correctly rejecting the xy term 254 00:13:05,480 --> 00:13:08,450 but incorrectly rejecting another term? 255 00:13:08,450 --> 00:13:09,890 Interesting. 256 00:13:09,890 --> 00:13:10,390 OK. 257 00:13:14,030 --> 00:13:16,470 Let's look at the other case. 258 00:13:16,470 --> 00:13:18,710 I'm using a rectangular pattern fitting 259 00:13:18,710 --> 00:13:21,870 the same model in this case. 260 00:13:21,870 --> 00:13:23,040 What happens now? 261 00:13:23,040 --> 00:13:25,350 Well, now we only have 21 data points. 262 00:13:25,350 --> 00:13:28,250 Notice that the r squared appears to be better. 263 00:13:28,250 --> 00:13:30,560 That's interesting. 264 00:13:30,560 --> 00:13:33,210 Again, the model is highly significant. 265 00:13:33,210 --> 00:13:36,050 And now if I look at the individual terms, 266 00:13:36,050 --> 00:13:39,450 the x is significant, the y is significant, 267 00:13:39,450 --> 00:13:43,640 the xy is not significant. 268 00:13:43,640 --> 00:13:45,620 That's rejected. 269 00:13:45,620 --> 00:13:49,220 The x squared and the y squared look-- 270 00:13:49,220 --> 00:13:52,940 I guess the y squared is kind of right marginal right at about 271 00:13:52,940 --> 00:13:54,810 95%. 272 00:13:54,810 --> 00:13:59,000 So it seems like it's coming up with different terms 273 00:13:59,000 --> 00:14:00,350 in the model. 274 00:14:00,350 --> 00:14:03,500 And also we'll see in a second when I compare them, 275 00:14:03,500 --> 00:14:06,680 it's coming up with some different estimates. 276 00:14:06,680 --> 00:14:09,230 So maybe this isn't a surprise. 277 00:14:09,230 --> 00:14:14,280 The sampling plan is affecting the model that I'm building. 278 00:14:14,280 --> 00:14:16,220 Doesn't sound too bad so far. 279 00:14:16,220 --> 00:14:18,407 But what I also want to do-- 280 00:14:18,407 --> 00:14:19,490 let me compare them first. 281 00:14:19,490 --> 00:14:21,230 And then what I want to do is also 282 00:14:21,230 --> 00:14:24,560 calculate a uniformity metric across all of this data 283 00:14:24,560 --> 00:14:26,280 and see what happens. 284 00:14:26,280 --> 00:14:27,500 So here's our truth. 285 00:14:27,500 --> 00:14:29,930 I've expanded out that original model 286 00:14:29,930 --> 00:14:32,690 of the systematic component contaminated 287 00:14:32,690 --> 00:14:38,780 by noise out to the x terms, the xy, and the x squared terms. 288 00:14:38,780 --> 00:14:42,050 And here is the radial sampling plan and the square sampling 289 00:14:42,050 --> 00:14:43,310 plan. 290 00:14:43,310 --> 00:14:45,020 And the coefficients, some of them 291 00:14:45,020 --> 00:14:48,110 are not radically different. 292 00:14:48,110 --> 00:14:50,660 They're kind of in the ballpark. 293 00:14:50,660 --> 00:14:56,360 But in terms of deciding which terms are kept 294 00:14:56,360 --> 00:15:00,110 and which are rejected, we know that this one, in fact, 295 00:15:00,110 --> 00:15:04,910 really is 0, whereas this one tries to keep it. 296 00:15:04,910 --> 00:15:08,300 This one is very much smaller, and we actually get rid of it. 297 00:15:08,300 --> 00:15:12,350 This one's small and this one's small but real. 298 00:15:12,350 --> 00:15:17,810 Both of the models fit parameters for it 299 00:15:17,810 --> 00:15:20,635 but make different cutoff points on what's significant 300 00:15:20,635 --> 00:15:23,420 and what's not. 301 00:15:23,420 --> 00:15:26,210 Now, what's really interesting, I think, 302 00:15:26,210 --> 00:15:30,585 is I now take that same data and I go back-- ah, great. 303 00:15:30,585 --> 00:15:31,460 Come on in, Charlene. 304 00:15:33,750 --> 00:15:34,250 What? 305 00:15:34,250 --> 00:15:35,792 AUDIENCE: She's setting up right now. 306 00:15:35,792 --> 00:15:39,740 DUANE BONING: Oh, Charlene, you can set up in here. 307 00:15:39,740 --> 00:15:42,260 Thank you. 308 00:15:42,260 --> 00:15:44,930 Is if I take those two sampling plans 309 00:15:44,930 --> 00:15:51,020 and I look back at the mean and the standard deviation 310 00:15:51,020 --> 00:15:53,390 just across my sampling points and calculate 311 00:15:53,390 --> 00:15:55,400 that non-uniformity metric. 312 00:15:55,400 --> 00:15:58,760 In one case, it's about a factor of two 313 00:15:58,760 --> 00:16:00,020 different than the other. 314 00:16:06,160 --> 00:16:07,940 The question is why? 315 00:16:07,940 --> 00:16:11,600 And you can ponder that question as you get coffee, 316 00:16:11,600 --> 00:16:13,606 because that's what I'm going to do. 317 00:16:13,606 --> 00:16:15,620 So now I think everybody's had a chance 318 00:16:15,620 --> 00:16:18,710 to ruminate on this question of non-uniformities 319 00:16:18,710 --> 00:16:21,830 or at least read the next two sub-bullets at the bottom 320 00:16:21,830 --> 00:16:24,200 of the page, and then everybody has a good idea 321 00:16:24,200 --> 00:16:26,490 of what might be going on. 322 00:16:26,490 --> 00:16:30,320 One is, of course, we've got systematic curvature. 323 00:16:30,320 --> 00:16:32,690 I'm not random sampling, so I may, in fact, 324 00:16:32,690 --> 00:16:36,650 hit different points on that curvature. 325 00:16:36,650 --> 00:16:39,590 So that's one hypothesis is just because 326 00:16:39,590 --> 00:16:44,060 of the particular places where my points happened to fall, 327 00:16:44,060 --> 00:16:45,170 that's one idea. 328 00:16:45,170 --> 00:16:48,410 What we'll find out, and I'll show that by more densely 329 00:16:48,410 --> 00:16:51,710 sampling, that's actually not what's going on necessarily 330 00:16:51,710 --> 00:16:54,800 in this problem. 331 00:16:54,800 --> 00:16:59,900 Actually, it's the structure of the sampling plan. 332 00:16:59,900 --> 00:17:05,030 And underlying that is how much data or what spatial region 333 00:17:05,030 --> 00:17:08,510 is each data point representing of the underlying wafer 334 00:17:08,510 --> 00:17:10,040 surface. 335 00:17:10,040 --> 00:17:11,970 So let me try to explain that. 336 00:17:11,970 --> 00:17:15,950 So what I've done here is gone back and tried to say maybe 337 00:17:15,950 --> 00:17:20,599 I just wasn't sampling enough with just 25 or 21 data points. 338 00:17:20,599 --> 00:17:25,980 What's my best guess at the true non-uniformity? 339 00:17:25,980 --> 00:17:29,090 Which one of these two should I believe? 340 00:17:29,090 --> 00:17:33,140 Which one is closer, the 1.9%, the 3.2%? 341 00:17:33,140 --> 00:17:34,430 What's a better sampling plan? 342 00:17:37,490 --> 00:17:40,810 Actually, before I go into that, what do you think? 343 00:17:40,810 --> 00:17:43,640 I need a vote here. 344 00:17:43,640 --> 00:17:46,270 How many people think that the radial sampling 345 00:17:46,270 --> 00:17:49,870 plan is probably closer to the truth in terms 346 00:17:49,870 --> 00:17:53,200 of both the model, but more importantly, 347 00:17:53,200 --> 00:17:55,450 the non-uniformity number that comes out? 348 00:17:55,450 --> 00:17:58,210 And how many people think it's a rectangular sampling plan? 349 00:17:58,210 --> 00:18:02,290 So all those think the radial sampling plan is going 350 00:18:02,290 --> 00:18:05,350 to be better, raise your hand. 351 00:18:05,350 --> 00:18:10,370 I got I got two here and a half. three. 352 00:18:10,370 --> 00:18:14,740 We got two, three, three in Singapore, four in Singapore. 353 00:18:14,740 --> 00:18:16,210 Oh, we're talking it up. 354 00:18:16,210 --> 00:18:17,560 We're talking it up. 355 00:18:17,560 --> 00:18:21,730 OK, how about the rectangular sampling plan? 356 00:18:21,730 --> 00:18:22,870 About evenly divided. 357 00:18:22,870 --> 00:18:26,590 I got three here, one in Singapore. 358 00:18:26,590 --> 00:18:31,930 And OK, that must leave four asleep people who didn't vote, 359 00:18:31,930 --> 00:18:33,580 something like that. 360 00:18:33,580 --> 00:18:34,150 OK. 361 00:18:34,150 --> 00:18:36,700 What I'm going to try to do is get a close estimate 362 00:18:36,700 --> 00:18:38,950 to what maybe better truth is. 363 00:18:38,950 --> 00:18:40,640 And the way I'm going to do that is 364 00:18:40,640 --> 00:18:47,650 try to take a very dense spatial sample so that hopefully it's 365 00:18:47,650 --> 00:18:49,690 not just sort of the random location of points. 366 00:18:49,690 --> 00:18:52,570 I'm going to kind of get really close to almost 367 00:18:52,570 --> 00:18:54,670 a full representation of the surface. 368 00:18:54,670 --> 00:18:58,090 And this is almost a 900 point-- 369 00:18:58,090 --> 00:19:00,140 slightly under 900 point sample. 370 00:19:00,140 --> 00:19:00,640 What is it? 371 00:19:00,640 --> 00:19:02,710 665, I guess. 372 00:19:02,710 --> 00:19:05,530 A 29 by 29 spatial sample and then the 373 00:19:05,530 --> 00:19:10,000 points that fall within the radius of the wafer. 374 00:19:10,000 --> 00:19:13,660 And as you do that, you actually get a, quote, 375 00:19:13,660 --> 00:19:18,430 a "true non-uniformity" that's about 3% 376 00:19:18,430 --> 00:19:20,620 closer to the other number. 377 00:19:20,620 --> 00:19:25,240 And if one does a response surface model fit, 378 00:19:25,240 --> 00:19:28,730 there's a couple of important points to come up with this. 379 00:19:28,730 --> 00:19:31,430 I go in and I fit the model. 380 00:19:31,430 --> 00:19:35,080 In this case, the coefficients that come out of the model 381 00:19:35,080 --> 00:19:39,490 are actually quite close to the true systematic piece. 382 00:19:39,490 --> 00:19:43,630 But notice that the r squared is still only about 0.77, 383 00:19:43,630 --> 00:19:45,520 something like that. 384 00:19:45,520 --> 00:19:47,980 Why is the r squared so bad? 385 00:19:47,980 --> 00:19:49,940 I got tons of data. 386 00:19:49,940 --> 00:19:51,190 Why is the r squared so bad? 387 00:19:56,110 --> 00:19:58,510 And the answer is kind of highlighted right here. 388 00:19:58,510 --> 00:20:01,060 I've still got lots of random noise in the process. 389 00:20:01,060 --> 00:20:09,820 I was adding magnitude variance 0.49 or 0.7 standard deviation 390 00:20:09,820 --> 00:20:14,050 values to all of my measurements. 391 00:20:14,050 --> 00:20:16,780 I cannot model the random noise. 392 00:20:16,780 --> 00:20:18,820 That's my residual that's leftover. 393 00:20:18,820 --> 00:20:21,640 I could include it as an estimate of the random noise 394 00:20:21,640 --> 00:20:25,000 and know what that is, but I can't pull that 395 00:20:25,000 --> 00:20:26,290 into the systematic model. 396 00:20:26,290 --> 00:20:31,210 So there's always a ground if there's noise and randomness 397 00:20:31,210 --> 00:20:33,610 that you can never capture. 398 00:20:33,610 --> 00:20:35,830 OK, so this seems to be suggesting 399 00:20:35,830 --> 00:20:39,790 that maybe that square sample with the smaller 400 00:20:39,790 --> 00:20:41,680 number of points was a little bit better. 401 00:20:41,680 --> 00:20:43,240 What's going on? 402 00:20:43,240 --> 00:20:46,270 This finally gets me to the Davis paper. 403 00:20:46,270 --> 00:20:48,400 And what they're going to propose 404 00:20:48,400 --> 00:20:51,850 is a method for looking at systematic trends. 405 00:20:51,850 --> 00:20:53,770 They're first going to make the point 406 00:20:53,770 --> 00:20:58,210 that the signal to noise ratio, this standard deviation over mu 407 00:20:58,210 --> 00:21:01,150 or mu over standard deviation, if I turn it 408 00:21:01,150 --> 00:21:05,290 into a signal to noise, is sensitive to both the location 409 00:21:05,290 --> 00:21:07,000 and the number of measurements. 410 00:21:07,000 --> 00:21:11,770 And they're going to propose a difference statistic rather 411 00:21:11,770 --> 00:21:14,350 than just the sample standard deviation 412 00:21:14,350 --> 00:21:18,730 to try to get at what a better, truer measure 413 00:21:18,730 --> 00:21:21,610 of overall non-uniformity will be. 414 00:21:21,610 --> 00:21:23,620 Turns out this integration statistic 415 00:21:23,620 --> 00:21:27,430 is based on fitting of splines to your data, 416 00:21:27,430 --> 00:21:30,470 and not a lot of people go in for that. 417 00:21:30,470 --> 00:21:34,750 What I'm going to show is that a relatively simple approximation 418 00:21:34,750 --> 00:21:37,030 achieves a lot of the goals of this integration 419 00:21:37,030 --> 00:21:41,680 statistic, which is worrying about uniform sampling 420 00:21:41,680 --> 00:21:44,950 spatially, or alternatively, by weighting 421 00:21:44,950 --> 00:21:47,200 the importance of each of your measurement points 422 00:21:47,200 --> 00:21:49,330 by the amount of area that it's representing. 423 00:21:49,330 --> 00:21:51,730 So that's the core of the problem. 424 00:21:51,730 --> 00:21:57,460 So what Davis does is looks at, first off, a sort of a smaller 425 00:21:57,460 --> 00:22:01,240 radial kind of picture compared to the one I showed. 426 00:22:01,240 --> 00:22:03,980 It's only got 13 data points. 427 00:22:03,980 --> 00:22:07,090 Here's the 13 data points with just linear interpolation 428 00:22:07,090 --> 00:22:07,940 between them. 429 00:22:07,940 --> 00:22:11,470 So that's the surface example that he's using. 430 00:22:11,470 --> 00:22:14,810 And if you just purely linearly interpolate between them, 431 00:22:14,810 --> 00:22:20,110 you can see that's a pretty coarse approximation 432 00:22:20,110 --> 00:22:21,310 to that surface. 433 00:22:21,310 --> 00:22:25,360 One can easily be worried that you might get bias or variance 434 00:22:25,360 --> 00:22:30,640 errors in trying to calculate with just those data points. 435 00:22:30,640 --> 00:22:34,270 What they're proposing is using thin plate splines, these TPS 436 00:22:34,270 --> 00:22:39,250 methods, which are conceptually, basically fitting localized 437 00:22:39,250 --> 00:22:45,340 polynomials such that you have minimum curvature over the data 438 00:22:45,340 --> 00:22:48,100 points or knots in the data. 439 00:22:48,100 --> 00:22:48,850 All right. 440 00:22:48,850 --> 00:22:51,190 I'm sure there are statistical packages and others that 441 00:22:51,190 --> 00:22:53,560 can help fit those thin plate splines. 442 00:22:53,560 --> 00:22:55,570 I found when I've tried this it's 443 00:22:55,570 --> 00:22:59,620 very tricky, because what localized means can be a little 444 00:22:59,620 --> 00:23:00,560 bit tricky. 445 00:23:00,560 --> 00:23:03,970 You've got other kind of human intervention 446 00:23:03,970 --> 00:23:06,910 going on in trying to fit these things. 447 00:23:06,910 --> 00:23:08,800 So it can be a little bit tricky. 448 00:23:08,800 --> 00:23:10,750 But essentially, what they're after is 449 00:23:10,750 --> 00:23:15,530 trying to get a model for the whole surface. 450 00:23:15,530 --> 00:23:17,680 And then once they have the whole surface, 451 00:23:17,680 --> 00:23:22,720 then calculate an overall metric as 452 00:23:22,720 --> 00:23:26,200 if I had an infinite number of measurement points representing 453 00:23:26,200 --> 00:23:27,400 the whole surface. 454 00:23:27,400 --> 00:23:32,395 So I want an integrated measure, an integrated statistic i, 455 00:23:32,395 --> 00:23:35,350 that captures the total deviation 456 00:23:35,350 --> 00:23:39,820 from a target or total deviation from some nominal value. 457 00:23:39,820 --> 00:23:43,120 And so what they're doing here is calculating 458 00:23:43,120 --> 00:23:47,530 a normalized integral of deviation 459 00:23:47,530 --> 00:23:52,450 of this interpolated surface, gr is their thin plate spline, 460 00:23:52,450 --> 00:23:55,390 from some target value, and then just integrating 461 00:23:55,390 --> 00:23:56,980 that over the surface. 462 00:23:56,980 --> 00:24:00,040 Total deviation from some target. 463 00:24:00,040 --> 00:24:02,530 Presumably the target would be just one value. 464 00:24:02,530 --> 00:24:06,130 I guess you could extend to whatever target 465 00:24:06,130 --> 00:24:08,410 spatial distribution you wanted. 466 00:24:08,410 --> 00:24:11,140 And then they're normalizing it by the total volume 467 00:24:11,140 --> 00:24:14,230 of the surface. 468 00:24:14,230 --> 00:24:17,800 So it's sort of like a standard deviation over mu 469 00:24:17,800 --> 00:24:25,090 except what it is is total deviation over total volume. 470 00:24:25,090 --> 00:24:28,220 Now, this is just total deviation. 471 00:24:28,220 --> 00:24:30,490 And if I was just summing up deviations 472 00:24:30,490 --> 00:24:36,940 as a measure of variation just in some randomly sampled data 473 00:24:36,940 --> 00:24:39,160 set not having to do with space, you 474 00:24:39,160 --> 00:24:41,330 might be worried about that. 475 00:24:41,330 --> 00:24:44,230 Why would you be worried about just summing up 476 00:24:44,230 --> 00:24:46,540 total deviations? 477 00:24:46,540 --> 00:24:50,560 Positive deviations, negative deviations, they'd cancel out. 478 00:24:50,560 --> 00:24:53,470 So if I was plus and minus huge amounts but about 479 00:24:53,470 --> 00:24:54,980 the same amount plus and minus, I 480 00:24:54,980 --> 00:24:58,340 might fool myself and think they might cancel out and come to 0. 481 00:24:58,340 --> 00:24:59,990 And I'd say, oh, everything's great. 482 00:24:59,990 --> 00:25:02,030 And they're really completely different. 483 00:25:02,030 --> 00:25:04,130 And that's why we often use things 484 00:25:04,130 --> 00:25:07,940 like squares or absolute value to try 485 00:25:07,940 --> 00:25:14,030 to account for or penalize for both plus and minus deviations. 486 00:25:14,030 --> 00:25:16,370 And so in the Davis paper, one can 487 00:25:16,370 --> 00:25:21,260 go in and apply some other generalized loss transformation 488 00:25:21,260 --> 00:25:23,610 to that deviation. 489 00:25:23,610 --> 00:25:27,620 So you can do a sum of squared deviations 490 00:25:27,620 --> 00:25:31,370 that looks a lot more like a standard deviation. 491 00:25:31,370 --> 00:25:33,650 But again, the idea here is they want to integrate 492 00:25:33,650 --> 00:25:36,020 that over the whole surface. 493 00:25:36,020 --> 00:25:39,140 Represent all of the surface, not just the few data 494 00:25:39,140 --> 00:25:41,870 points that you had, but do that using 495 00:25:41,870 --> 00:25:44,810 the interpolated function. 496 00:25:44,810 --> 00:25:47,090 Now, there's an approximation to this integral 497 00:25:47,090 --> 00:25:52,730 that they mention, which is if I have my lost transformation, 498 00:25:52,730 --> 00:25:55,050 I have my actual data points. 499 00:25:55,050 --> 00:25:58,580 So these are my measured data points. 500 00:25:58,580 --> 00:26:01,640 And I have the deviation of that measured point 501 00:26:01,640 --> 00:26:06,020 and then my h might be the square or the absolute value. 502 00:26:06,020 --> 00:26:09,710 If I simply sum those for my data points, 503 00:26:09,710 --> 00:26:12,650 I get something close, again, to standard deviation, 504 00:26:12,650 --> 00:26:14,240 a sum of squares. 505 00:26:14,240 --> 00:26:17,432 But there's one big difference. 506 00:26:17,432 --> 00:26:18,530 And that's right here. 507 00:26:18,530 --> 00:26:20,480 The c sub j. 508 00:26:20,480 --> 00:26:24,770 And essentially the c sub j is a weight 509 00:26:24,770 --> 00:26:30,080 that corresponds to if I was doing the spatial integration, 510 00:26:30,080 --> 00:26:33,650 I would be integrating over these thin little patches 511 00:26:33,650 --> 00:26:36,478 around each of my measurement points. 512 00:26:36,478 --> 00:26:38,270 It wouldn't be just that measurement point. 513 00:26:38,270 --> 00:26:40,910 It would be representing the surface near that measurement 514 00:26:40,910 --> 00:26:41,720 point. 515 00:26:41,720 --> 00:26:44,330 And the c sub j is a weight that says, 516 00:26:44,330 --> 00:26:48,410 how much area on my surface does that measurement point 517 00:26:48,410 --> 00:26:50,360 represent? 518 00:26:50,360 --> 00:26:53,300 How much vote does that one point have? 519 00:26:53,300 --> 00:26:56,840 And that can be very different in these two sampling plans. 520 00:26:56,840 --> 00:27:02,500 So what's neat is that if you weight appropriately for area 521 00:27:02,500 --> 00:27:05,360 that the point represents, you actually 522 00:27:05,360 --> 00:27:09,830 get fairly close to something like the integration coming 523 00:27:09,830 --> 00:27:12,200 from a thin plate spline. 524 00:27:12,200 --> 00:27:12,980 OK. 525 00:27:12,980 --> 00:27:17,210 So there's a couple of plots in Davis's paper 526 00:27:17,210 --> 00:27:19,970 where they're taking that typical radial sampling 527 00:27:19,970 --> 00:27:20,610 pattern. 528 00:27:20,610 --> 00:27:28,850 Here's a perfectly circularly symmetric pattern, 15% 529 00:27:28,850 --> 00:27:33,380 systematic variation, 2% random superimposed on it. 530 00:27:33,380 --> 00:27:37,640 And then what they're doing is looking at an SNR. 531 00:27:37,640 --> 00:27:41,270 This is basically just sigma over mu of just the data 532 00:27:41,270 --> 00:27:42,320 points. 533 00:27:42,320 --> 00:27:44,870 And they're showing what estimates 534 00:27:44,870 --> 00:27:49,700 over 300 different runs of this model 535 00:27:49,700 --> 00:27:51,530 with different amounts of noise each time, 536 00:27:51,530 --> 00:27:58,460 just different instantiations of the random 2% noise, what 537 00:27:58,460 --> 00:28:02,060 do they calculate for sigma over mu? 538 00:28:02,060 --> 00:28:05,570 So each time they do 300 runs, out here 539 00:28:05,570 --> 00:28:11,570 they do it taking 73 different measurements 540 00:28:11,570 --> 00:28:16,970 around the wafer with a typical radial spatial pattern. 541 00:28:16,970 --> 00:28:19,860 This is the problematic one we saw earlier. 542 00:28:19,860 --> 00:28:22,370 And notice there is a spread, as you would expect, 543 00:28:22,370 --> 00:28:26,100 because the noise is in there. 544 00:28:26,100 --> 00:28:28,310 But look what happens as you go to smaller 545 00:28:28,310 --> 00:28:30,110 numbers of measurements. 546 00:28:30,110 --> 00:28:32,750 First off, the spread or the variance 547 00:28:32,750 --> 00:28:37,530 in your estimate of sigma over mu increases. 548 00:28:37,530 --> 00:28:39,170 That makes sense, right? 549 00:28:39,170 --> 00:28:40,130 I have less data. 550 00:28:40,130 --> 00:28:41,990 I'm going to have more variance in how I'm 551 00:28:41,990 --> 00:28:43,910 estimating standard deviation. 552 00:28:43,910 --> 00:28:46,820 We already know that estimating standard deviation in general 553 00:28:46,820 --> 00:28:48,770 requires a lot of data. 554 00:28:48,770 --> 00:28:50,210 It's really a tough thing. 555 00:28:50,210 --> 00:28:52,310 So it's getting worse. 556 00:28:52,310 --> 00:28:55,730 But even worse than that, especially for, say, 557 00:28:55,730 --> 00:28:58,190 five measurements or even 13 measurements, 558 00:28:58,190 --> 00:29:03,890 a little bit there, is the average value coming 559 00:29:03,890 --> 00:29:12,680 from repeated evaluations is not equal to the true underlying 560 00:29:12,680 --> 00:29:13,470 average? 561 00:29:13,470 --> 00:29:16,490 In other words, there's bias. 562 00:29:16,490 --> 00:29:21,170 The estimator of sigma over mu is biased. 563 00:29:21,170 --> 00:29:23,840 It's not a very good estimator. 564 00:29:23,840 --> 00:29:27,050 And as you have smaller numbers of parameters, 565 00:29:27,050 --> 00:29:29,690 that unequal weighting of the surface 566 00:29:29,690 --> 00:29:32,480 is biasing or fooling you. 567 00:29:32,480 --> 00:29:35,810 Even on average you're going to on average be wrong. 568 00:29:38,550 --> 00:29:40,220 What they're showing is if they actually 569 00:29:40,220 --> 00:29:44,540 do their whole thin plate spline I stat statistic, 570 00:29:44,540 --> 00:29:48,620 they argue both the variance is smaller, but more importantly 571 00:29:48,620 --> 00:29:49,790 is it's unbiased. 572 00:29:49,790 --> 00:29:54,350 Even when you get down to small numbers of measurements, 573 00:29:54,350 --> 00:29:58,430 you're at least on average right. 574 00:29:58,430 --> 00:30:01,700 And that's just these two plots that are simply boiled down 575 00:30:01,700 --> 00:30:04,490 into this comparison that shows as a function 576 00:30:04,490 --> 00:30:10,550 of the number of measurements how good the statistic is. 577 00:30:10,550 --> 00:30:15,530 So the spread of the statistic over the mean of the statistic. 578 00:30:15,530 --> 00:30:18,680 And showing that the variance of the statistic 579 00:30:18,680 --> 00:30:23,030 itself is getting smaller with the integration statistic. 580 00:30:23,030 --> 00:30:25,310 So you're basically doing a better job 581 00:30:25,310 --> 00:30:31,500 of estimating with their proposed integration statistic. 582 00:30:31,500 --> 00:30:33,380 Now, it's not perfect. 583 00:30:33,380 --> 00:30:36,590 And in fact, conceptually, if I only 584 00:30:36,590 --> 00:30:38,510 have a small number of sampling points, 585 00:30:38,510 --> 00:30:41,750 I can still be prone to where those sampling points happen 586 00:30:41,750 --> 00:30:44,550 to lie if I have a complex surface. 587 00:30:44,550 --> 00:30:49,040 And so they're also showing an asymmetric underlying 588 00:30:49,040 --> 00:30:53,960 non-uniformity, this crazy shape down here in the lower left. 589 00:30:53,960 --> 00:30:56,990 And imagine I'm only sampling with five data points. 590 00:30:56,990 --> 00:31:02,480 Well, if those five data points land in one orientation 591 00:31:02,480 --> 00:31:08,450 or if they land then slightly offset in angle, 592 00:31:08,450 --> 00:31:10,640 I am spatially sampling different parts 593 00:31:10,640 --> 00:31:11,820 of that surface. 594 00:31:11,820 --> 00:31:13,460 So what happens then? 595 00:31:13,460 --> 00:31:17,180 And what they basically show is, yes, their I stat 596 00:31:17,180 --> 00:31:19,460 does have some sensitivity. 597 00:31:19,460 --> 00:31:21,590 But because you're trying to interpolate 598 00:31:21,590 --> 00:31:26,630 the rest of the surface, you do a better job. 599 00:31:26,630 --> 00:31:29,540 You're not completely dependent just on that local measurement 600 00:31:29,540 --> 00:31:30,350 point. 601 00:31:30,350 --> 00:31:33,680 Both approaches are still highly sensitive to angle, 602 00:31:33,680 --> 00:31:39,110 but the pure sigma over mu is much more sensitive. 603 00:31:39,110 --> 00:31:45,290 You get about 20% smaller variation with their I stat. 604 00:31:45,290 --> 00:31:46,370 OK. 605 00:31:46,370 --> 00:31:50,570 So what are the key lessons out of this part 606 00:31:50,570 --> 00:31:53,840 of the case so far, this paper? 607 00:31:53,840 --> 00:31:56,450 I think the key lessons are watch out 608 00:31:56,450 --> 00:31:58,220 for your sampling plan. 609 00:31:58,220 --> 00:32:03,050 And especially if any of you go into the semiconductor industry 610 00:32:03,050 --> 00:32:06,440 or I guess any industry dealing with round substrates, 611 00:32:06,440 --> 00:32:10,430 be careful, because you will run into this circular sampling 612 00:32:10,430 --> 00:32:12,290 plan again and again. 613 00:32:12,290 --> 00:32:16,340 And if you're calculating some metric that is equally 614 00:32:16,340 --> 00:32:19,370 weighting each of these points where each of these points is, 615 00:32:19,370 --> 00:32:22,460 in fact, points off on the edge of the wafer 616 00:32:22,460 --> 00:32:25,550 are representing a bigger area. 617 00:32:25,550 --> 00:32:29,600 But they're equally weighted with smaller sampling points 618 00:32:29,600 --> 00:32:31,130 near the center of the wafer. 619 00:32:31,130 --> 00:32:37,340 And this is a problem that I've seen come up again and again 620 00:32:37,340 --> 00:32:41,420 and again that people, I think, are not 621 00:32:41,420 --> 00:32:45,930 as aware of as they ought to be in the industry. 622 00:32:45,930 --> 00:32:47,616 It's easy to fix. 623 00:32:47,616 --> 00:32:48,540 It's easy to fix. 624 00:32:48,540 --> 00:32:52,010 One fix that I recommend, that I think is the cleanest, 625 00:32:52,010 --> 00:32:55,820 is simply do a more uniform sampling. 626 00:32:55,820 --> 00:32:59,090 That rectangular sampling has a great benefit. 627 00:32:59,090 --> 00:33:02,210 How much area does each point represent? 628 00:33:02,210 --> 00:33:04,100 An equal area. 629 00:33:04,100 --> 00:33:11,120 It's not dependent on an r theta kind of calculation. 630 00:33:11,120 --> 00:33:15,890 Second is if you do have some non-uniform spatially sampling 631 00:33:15,890 --> 00:33:19,760 plan, you can kind of fix it by doing a weighted metric 632 00:33:19,760 --> 00:33:21,560 or a weighted regression. 633 00:33:21,560 --> 00:33:22,348 Yeah? 634 00:33:22,348 --> 00:33:24,409 AUDIENCE: Normally, you have a different kind 635 00:33:24,409 --> 00:33:29,080 of sampling plan like that for, say, the [INAUDIBLE] terms. 636 00:33:29,080 --> 00:33:31,130 But it would be four ranges of the [INAUDIBLE] 637 00:33:31,130 --> 00:33:33,920 and four ranges of the wafer. 638 00:33:33,920 --> 00:33:36,980 And then we would applicate hundreds of times 639 00:33:36,980 --> 00:33:40,920 over the wafer. 640 00:33:40,920 --> 00:33:46,100 So in that case, does that factor then, because you're-- 641 00:33:46,100 --> 00:33:48,560 DUANE BONING: So Nalish's point is often, say, 642 00:33:48,560 --> 00:33:50,930 if you're sampling within each chip, 643 00:33:50,930 --> 00:33:54,020 so here's our big wafer with lots and lots of these chips, 644 00:33:54,020 --> 00:33:56,930 you might sample spatially the same ring 645 00:33:56,930 --> 00:33:59,570 oscillator in different corners of the chip 646 00:33:59,570 --> 00:34:02,210 and then-- since there are multiple chips. 647 00:34:02,210 --> 00:34:04,940 I think, again, you've got to be a little bit alert to what 648 00:34:04,940 --> 00:34:07,070 it is you're doing with that data 649 00:34:07,070 --> 00:34:08,690 and how you're calculating it. 650 00:34:08,690 --> 00:34:12,560 But that chip by chip kind of sampling plan 651 00:34:12,560 --> 00:34:16,219 has the advantage that chips are generally rectangular. 652 00:34:16,219 --> 00:34:19,489 And so maybe by accident or by free, 653 00:34:19,489 --> 00:34:26,360 you're usually getting a more equally representative sampling 654 00:34:26,360 --> 00:34:30,770 plan that lets you fit or calculate metrics 655 00:34:30,770 --> 00:34:32,822 a little bit more fairly. 656 00:34:32,822 --> 00:34:36,119 AUDIENCE: [INAUDIBLE] you had mentioned the sensitivity 657 00:34:36,119 --> 00:34:38,480 to the amount of area cover. 658 00:34:38,480 --> 00:34:42,290 But a lot of times you have more of a sensitivity to factor, 659 00:34:42,290 --> 00:34:49,228 depending on if SRAM is nearby or a populated logic is nearby. 660 00:34:49,228 --> 00:34:50,020 DUANE BONING: Yeah. 661 00:34:50,020 --> 00:34:56,150 So this is kind of good for wafer level modeling. 662 00:34:56,150 --> 00:34:59,450 Nalish is also pointing out in many cases, 663 00:34:59,450 --> 00:35:03,600 maybe this is an SRAM and this is logic over here, 664 00:35:03,600 --> 00:35:06,020 and you're sensitive to other, what 665 00:35:06,020 --> 00:35:10,370 I would call, finer grained systematic sources 666 00:35:10,370 --> 00:35:13,550 of variation, like layout pattern density, 667 00:35:13,550 --> 00:35:17,120 proximity to other structures, other perturbing effects. 668 00:35:17,120 --> 00:35:20,660 And those are extremely interesting and fascinating 669 00:35:20,660 --> 00:35:21,380 to look at. 670 00:35:21,380 --> 00:35:24,440 And I think the basic strategy is 671 00:35:24,440 --> 00:35:26,360 if you know what those are as factors 672 00:35:26,360 --> 00:35:30,020 and can represent them as factors, 673 00:35:30,020 --> 00:35:33,980 you can actually do some of the kind of ANOVA 674 00:35:33,980 --> 00:35:37,250 and model fitting with those as factors in addition 675 00:35:37,250 --> 00:35:42,830 to or separating from spatial xy dependencies. 676 00:35:42,830 --> 00:35:45,860 So it's usually better if you know what those factors are not 677 00:35:45,860 --> 00:35:49,010 just trying to let x and y be the stand in for them 678 00:35:49,010 --> 00:35:52,070 but actually explicitly include those things 679 00:35:52,070 --> 00:35:54,140 so you know what causes what. 680 00:35:54,140 --> 00:35:57,980 And that's a very interesting area. 681 00:35:57,980 --> 00:36:02,180 Any other questions on this spatial sampling? 682 00:36:02,180 --> 00:36:05,390 I think it's kind of cool but relatively intuitive 683 00:36:05,390 --> 00:36:08,910 once you think about it. 684 00:36:08,910 --> 00:36:12,050 So the next thing I want to do is say, OK, now I 685 00:36:12,050 --> 00:36:17,480 know how to be careful in building a spatial response 686 00:36:17,480 --> 00:36:19,020 surface model. 687 00:36:19,020 --> 00:36:21,500 But what if I'm worried about how that surface 688 00:36:21,500 --> 00:36:24,500 changes as a function of process conditions. 689 00:36:24,500 --> 00:36:27,230 What I'd like to do is have both process and spatial 690 00:36:27,230 --> 00:36:28,890 dependencies in it. 691 00:36:28,890 --> 00:36:31,760 I'm going to show first off an approach that 692 00:36:31,760 --> 00:36:36,320 basically builds a two layered response surface model. 693 00:36:36,320 --> 00:36:39,050 This is the Mozumder and Loewenstein paper. 694 00:36:39,050 --> 00:36:41,180 And then I'll come back to it and flip it 695 00:36:41,180 --> 00:36:46,740 around that says in the first case, 696 00:36:46,740 --> 00:36:51,080 the first approach is basically to build the model of space 697 00:36:51,080 --> 00:36:52,820 and then take each of the coefficients 698 00:36:52,820 --> 00:36:58,010 in your spatial model, your a3 times the xy term, 699 00:36:58,010 --> 00:37:01,430 and now model that a3 term as a function of your process 700 00:37:01,430 --> 00:37:03,830 conditions. 701 00:37:03,830 --> 00:37:05,390 Got it? 702 00:37:05,390 --> 00:37:08,040 The second approach does the opposite. 703 00:37:08,040 --> 00:37:11,780 It says, I'm going to build a model of this particular site 704 00:37:11,780 --> 00:37:15,450 location as a function of process conditions. 705 00:37:15,450 --> 00:37:19,520 And then I can fit a spatial model once I have that. 706 00:37:19,520 --> 00:37:22,370 So it's two completely separate or orthogonal ways 707 00:37:22,370 --> 00:37:24,360 of looking at the problem. 708 00:37:24,360 --> 00:37:28,370 So the first one here is "Model for Semiconductor Process 709 00:37:28,370 --> 00:37:31,820 Optimization Using Functional Representations of Spatial 710 00:37:31,820 --> 00:37:33,800 Variations in Selectivity." 711 00:37:33,800 --> 00:37:38,120 And they make some of the same kinds of observations 712 00:37:38,120 --> 00:37:39,960 that we've talked about. 713 00:37:39,960 --> 00:37:43,880 But what they're doing is basically 714 00:37:43,880 --> 00:37:47,480 trying to say, OK, I have to be careful in how I calculate 715 00:37:47,480 --> 00:37:53,300 my non-uniformity metric, but I want to calculate that and then 716 00:37:53,300 --> 00:37:55,910 understand how that changes as the function 717 00:37:55,910 --> 00:37:58,760 of different process conditions. 718 00:37:58,760 --> 00:38:00,290 And in particular, they're looking 719 00:38:00,290 --> 00:38:02,960 at a silicon nitride wet etch looking-- 720 00:38:02,960 --> 00:38:06,050 or silicon nitride plasma etch, excuse me. 721 00:38:06,050 --> 00:38:11,510 And looking at things like the etch traits of the silicon 722 00:38:11,510 --> 00:38:14,120 nitride, etch rates of the silicon dioxide 723 00:38:14,120 --> 00:38:19,370 as a function of things like gas flows process conditions. 724 00:38:19,370 --> 00:38:21,920 Now, they're actually worried about two key parameters 725 00:38:21,920 --> 00:38:23,400 or three key parameters. 726 00:38:23,400 --> 00:38:25,910 One is the etch rate. 727 00:38:25,910 --> 00:38:27,560 Faster rates are better. 728 00:38:27,560 --> 00:38:29,250 Uniformity is better. 729 00:38:29,250 --> 00:38:31,460 But they also worry about the relative etch 730 00:38:31,460 --> 00:38:36,330 rate between nitride and oxide and they want high selectivity. 731 00:38:36,330 --> 00:38:38,570 They want to be able to etch through the oxide 732 00:38:38,570 --> 00:38:44,210 but stop on the underlying nitride or vice versa, I guess. 733 00:38:44,210 --> 00:38:46,970 This is a nitride etch. 734 00:38:46,970 --> 00:38:51,830 OK, so what they do here, they take a typical spatial map, 735 00:38:51,830 --> 00:38:53,990 19 measurements. 736 00:38:53,990 --> 00:38:56,840 The paper says two concentric hexagons. 737 00:38:56,840 --> 00:39:00,590 But if I look at it, they look like two concentric octagons, 738 00:39:00,590 --> 00:39:01,778 because it's eight points. 739 00:39:01,778 --> 00:39:03,320 So I don't know where that came from. 740 00:39:03,320 --> 00:39:10,850 But it's basically these octagons plus the center point. 741 00:39:10,850 --> 00:39:13,490 There may have been replicates to get to the 19 measurements. 742 00:39:13,490 --> 00:39:15,110 Not entirely clear. 743 00:39:15,110 --> 00:39:16,640 But then their process conditions 744 00:39:16,640 --> 00:39:20,960 are temperature, microwave power, pressure in the chamber, 745 00:39:20,960 --> 00:39:24,350 as well as nitrogen and hydrogen flows. 746 00:39:24,350 --> 00:39:26,990 So it's sort of a five factor DOE 747 00:39:26,990 --> 00:39:29,360 that they're going to be exploring. 748 00:39:29,360 --> 00:39:32,120 And what they note is that the removal rates 749 00:39:32,120 --> 00:39:36,360 do have a spatial dependence as a function of these conditions. 750 00:39:36,360 --> 00:39:39,560 Things like gas flow rate means more of the gas 751 00:39:39,560 --> 00:39:44,660 is coming in contact with the center or the edge of the wafer 752 00:39:44,660 --> 00:39:46,880 perhaps at different temperatures, different process 753 00:39:46,880 --> 00:39:47,720 conditions. 754 00:39:47,720 --> 00:39:52,580 So they do have an etch rate that varies spatially 755 00:39:52,580 --> 00:39:55,730 across the wafer, and they would like to model that. 756 00:39:55,730 --> 00:40:00,140 Now, what they're going to do is say if I know my rate 757 00:40:00,140 --> 00:40:02,780 spatially across the wafer, I am going 758 00:40:02,780 --> 00:40:07,250 to then calculate non-uniformity as a derived parameter. 759 00:40:07,250 --> 00:40:10,460 I'm not just going to sum up my data points. 760 00:40:10,460 --> 00:40:13,760 I am actually going to try to use kind of the model 761 00:40:13,760 --> 00:40:16,580 and do a ratio of standard deviation to mean. 762 00:40:16,580 --> 00:40:20,450 But it is derived from the model. 763 00:40:20,450 --> 00:40:23,810 And so their basic idea is to do this two level 764 00:40:23,810 --> 00:40:27,890 model or two layered model of the etch rates. 765 00:40:27,890 --> 00:40:32,450 So what they're going to do is say the etch rate is 766 00:40:32,450 --> 00:40:37,110 a function of both spatial terms, 767 00:40:37,110 --> 00:40:41,000 sort of the same thing we saw before, xy, xy, x squared, 768 00:40:41,000 --> 00:40:42,950 y squared. 769 00:40:42,950 --> 00:40:46,650 But also of processed terms. 770 00:40:46,650 --> 00:40:49,490 So what they would like to be able to do is plug in 771 00:40:49,490 --> 00:40:52,190 and say, OK, if I know my hydrogen flow rate 772 00:40:52,190 --> 00:40:56,540 and I want to look at some particular xy location, 773 00:40:56,540 --> 00:40:59,510 I would like to know at this process condition 774 00:40:59,510 --> 00:41:04,340 and at this spatial location what is my etch rate. 775 00:41:04,340 --> 00:41:06,170 So what they have to do is basically 776 00:41:06,170 --> 00:41:09,710 fit this two layered model that has 777 00:41:09,710 --> 00:41:17,030 both dependencies on the spatial terms and on the process 778 00:41:17,030 --> 00:41:19,190 terms in it. 779 00:41:19,190 --> 00:41:21,350 And another way of looking at this is essentially 780 00:41:21,350 --> 00:41:24,320 what they're doing is, as I mentioned earlier, they 781 00:41:24,320 --> 00:41:26,480 are building a spatial model where 782 00:41:26,480 --> 00:41:30,110 each coefficient, sort of an a1, is also then 783 00:41:30,110 --> 00:41:32,710 a function of the process conditions. 784 00:41:35,540 --> 00:41:38,300 So here's an example of a regression surface 785 00:41:38,300 --> 00:41:39,710 that comes out. 786 00:41:39,710 --> 00:41:41,600 This is the spatial regression. 787 00:41:41,600 --> 00:41:45,200 Again, as a function of xy and so on for the removal 788 00:41:45,200 --> 00:41:48,110 rate or etch rate of silicon nitride. 789 00:41:48,110 --> 00:41:51,920 And then these coefficients themselves, they go in 790 00:41:51,920 --> 00:41:54,350 and they build a model of those coefficients 791 00:41:54,350 --> 00:41:58,987 as a function of the process. 792 00:41:58,987 --> 00:42:01,320 And when you do that, you can get some pretty good fits. 793 00:42:01,320 --> 00:42:01,970 Look at these. 794 00:42:01,970 --> 00:42:09,110 These are square root of in the range 0.93, 0.90 is the worst, 795 00:42:09,110 --> 00:42:16,320 0.99 for each of the significance for each of those 796 00:42:16,320 --> 00:42:16,820 terms. 797 00:42:16,820 --> 00:42:21,980 In other words, this is the spatial model 798 00:42:21,980 --> 00:42:26,720 for that c sub 2n whatever. 799 00:42:26,720 --> 00:42:29,590 C sub 2x. 800 00:42:29,590 --> 00:42:32,810 I guess c sub 2y yn. 801 00:42:32,810 --> 00:42:34,640 But the goodness of fit, then, of doing 802 00:42:34,640 --> 00:42:37,610 that spatial coordinate model is a function of the process. 803 00:42:37,610 --> 00:42:40,250 It's a pretty good fit. 804 00:42:40,250 --> 00:42:43,340 And then what they do is say, OK, I'm 805 00:42:43,340 --> 00:42:47,150 going to do polynomial regression, as we saw before. 806 00:42:47,150 --> 00:42:50,630 They actually find kind of a nasty thing. 807 00:42:50,630 --> 00:42:53,690 To get good fits, they had to go to cubic fits. 808 00:42:57,340 --> 00:42:58,592 That may be. 809 00:42:58,592 --> 00:43:00,550 Just a couple of other points of interest here. 810 00:43:00,550 --> 00:43:03,250 They use the Latin hypercube sampling rather than 811 00:43:03,250 --> 00:43:06,880 a central composite sampling. 812 00:43:06,880 --> 00:43:08,740 This is a really cute sampling plan. 813 00:43:08,740 --> 00:43:13,000 Did I describe this Latin hypercube sampling plan before? 814 00:43:13,000 --> 00:43:13,750 I think I did. 815 00:43:13,750 --> 00:43:16,120 But basically this was saying, OK, 816 00:43:16,120 --> 00:43:19,690 if I have a space, some x1 parameter 817 00:43:19,690 --> 00:43:21,820 and some x2 parameter, and I only 818 00:43:21,820 --> 00:43:25,720 have five data points I'm going to allow myself 819 00:43:25,720 --> 00:43:29,380 to sample, what you do is you basically divide both 820 00:43:29,380 --> 00:43:35,140 your coordinates or dimensions up into five spaces, five 821 00:43:35,140 --> 00:43:35,755 equal spaces. 822 00:43:39,200 --> 00:43:43,430 And then you take a sample point and make sure 823 00:43:43,430 --> 00:43:45,290 that every one of your five sample points 824 00:43:45,290 --> 00:43:50,420 uniquely follows in one row and one column. 825 00:43:50,420 --> 00:43:53,600 So you might get something like this. 826 00:43:53,600 --> 00:43:55,610 What am I missing? 827 00:43:55,610 --> 00:43:57,680 Maybe that. 828 00:43:57,680 --> 00:44:00,200 And what's cool there is if you project down 829 00:44:00,200 --> 00:44:03,770 onto either x1 coordinate, now I've 830 00:44:03,770 --> 00:44:09,080 got sort of five data points spread throughout that space. 831 00:44:09,080 --> 00:44:11,060 Or if I project down onto the x2, 832 00:44:11,060 --> 00:44:12,980 I similarly have five coordinates 833 00:44:12,980 --> 00:44:14,992 spread throughout that space. 834 00:44:14,992 --> 00:44:17,450 You have to be a little bit careful, because that algorithm 835 00:44:17,450 --> 00:44:19,760 could have accidentally given you 836 00:44:19,760 --> 00:44:21,660 a spatial correlation in your sampling. 837 00:44:21,660 --> 00:44:25,400 So for example, those points follow 838 00:44:25,400 --> 00:44:27,320 my rule of one row, one column. 839 00:44:27,320 --> 00:44:29,688 But that has an unintentional correlation in it. 840 00:44:29,688 --> 00:44:31,730 So there's a part of the Latin hypercube sampling 841 00:44:31,730 --> 00:44:36,792 algorithm that swaps rows and columns to avoid that. 842 00:44:36,792 --> 00:44:38,000 So that's what they do there. 843 00:44:38,000 --> 00:44:39,542 Actually, it's good that you're aware 844 00:44:39,542 --> 00:44:41,990 of Latin hypercube sampling, because especially 845 00:44:41,990 --> 00:44:46,160 for constrained cases where you know how many data points 846 00:44:46,160 --> 00:44:52,400 you can sample, it's actually quite an interesting approach. 847 00:44:52,400 --> 00:44:52,940 OK. 848 00:44:52,940 --> 00:44:56,540 Then what they go in and do is build out 849 00:44:56,540 --> 00:44:59,990 of these fundamental rate functions that they fit, 850 00:44:59,990 --> 00:45:03,710 they built derived functions. 851 00:45:03,710 --> 00:45:06,770 These are derived. 852 00:45:06,770 --> 00:45:09,140 It says if I know the removal rate or etch 853 00:45:09,140 --> 00:45:12,680 rate at different locations, now I can fold that together 854 00:45:12,680 --> 00:45:16,760 to calculate an aggregate uniformity 855 00:45:16,760 --> 00:45:19,760 kind of using some of these ideas 856 00:45:19,760 --> 00:45:21,230 we were talking about before. 857 00:45:21,230 --> 00:45:25,730 I'm not simply calculating from my raw data uniformity number 858 00:45:25,730 --> 00:45:29,720 and then building a separate model, a separate response 859 00:45:29,720 --> 00:45:33,920 surface model, a separate polynomial model of uniformity. 860 00:45:33,920 --> 00:45:37,040 Instead I'm getting close to the physics as I can 861 00:45:37,040 --> 00:45:40,580 and build this great model and then recognize 862 00:45:40,580 --> 00:45:45,560 that uniformity is simply a calculated function 863 00:45:45,560 --> 00:45:50,000 across that underlying rate function. 864 00:45:50,000 --> 00:45:57,140 And they happen to be using something like a rate 865 00:45:57,140 --> 00:45:59,420 sigma divided by rate mean. 866 00:45:59,420 --> 00:46:03,140 So they're integrating that over their space. 867 00:46:03,140 --> 00:46:05,450 By the way, this little n simply indicates 868 00:46:05,450 --> 00:46:08,630 that they're calculating separately or building models 869 00:46:08,630 --> 00:46:14,240 separately for the oxide rate and the nitride etch rate. 870 00:46:14,240 --> 00:46:16,160 They both vary spatially. 871 00:46:16,160 --> 00:46:18,410 So they built these two models. 872 00:46:18,410 --> 00:46:20,510 They do the same thing for this selectivity. 873 00:46:20,510 --> 00:46:23,570 Again, this is the ratio of etch rate 874 00:46:23,570 --> 00:46:26,660 of nitride to silicon dioxide. 875 00:46:26,660 --> 00:46:30,260 And that also is a derived model. 876 00:46:33,270 --> 00:46:36,270 And notice that there's something really important 877 00:46:36,270 --> 00:46:42,540 about these derived models and why they help. 878 00:46:42,540 --> 00:46:45,240 May be less obvious, although we've already 879 00:46:45,240 --> 00:46:48,360 talked about it here with the uniformity metric. 880 00:46:48,360 --> 00:46:52,650 Maybe they'll focus first here on the selectivity. 881 00:46:52,650 --> 00:46:58,150 Is selectivity a linear function? 882 00:46:58,150 --> 00:46:58,650 No. 883 00:46:58,650 --> 00:46:59,670 It's kind of nasty. 884 00:46:59,670 --> 00:47:03,870 It's a ratio of two other rates. 885 00:47:03,870 --> 00:47:06,790 Why would that necessarily be linear? 886 00:47:06,790 --> 00:47:09,780 This is kind of a complicated functional form 887 00:47:09,780 --> 00:47:13,590 that if you were trying to directly model selectivity, 888 00:47:13,590 --> 00:47:16,920 it might be very difficult to capture 889 00:47:16,920 --> 00:47:20,880 by chance that complicated functional form. 890 00:47:20,880 --> 00:47:24,090 And even worse, think about the computations 891 00:47:24,090 --> 00:47:27,580 that go in the calculation of standard deviation. 892 00:47:27,580 --> 00:47:30,060 Remember standard deviation? 893 00:47:30,060 --> 00:47:34,940 It's the square root of the sum of some x sub 894 00:47:34,940 --> 00:47:44,940 i minus the mean squared divided by 1 over n minus 1. 895 00:47:44,940 --> 00:47:46,320 So you got squaring. 896 00:47:46,320 --> 00:47:48,240 You got a square root in there. 897 00:47:48,240 --> 00:47:50,670 That's a very nonlinear operation. 898 00:47:50,670 --> 00:47:56,080 Why would a linear model be very good at capturing that? 899 00:47:56,080 --> 00:47:59,260 In fact, if you tried to do it directly, 900 00:47:59,260 --> 00:48:04,770 you might get a non-uniformity that 901 00:48:04,770 --> 00:48:07,450 needed not only cubic terms, but who knows, 902 00:48:07,450 --> 00:48:10,230 fourth order terms, fifth order terms, which would be quite 903 00:48:10,230 --> 00:48:12,240 complicated to actually fit it. 904 00:48:12,240 --> 00:48:15,510 And what they're showing in the paper is an example where 905 00:48:15,510 --> 00:48:20,550 a relatively simple rate function, 906 00:48:20,550 --> 00:48:22,890 they're doing it here also as a function of, say, 907 00:48:22,890 --> 00:48:28,890 theta and r, not just, say, xy, but that transformation 908 00:48:28,890 --> 00:48:30,370 is pretty simple. 909 00:48:30,370 --> 00:48:33,480 If you then feed that into the uniformity calculation, 910 00:48:33,480 --> 00:48:36,330 the expansions of squares and square roots, 911 00:48:36,330 --> 00:48:39,400 you get kind of almost, if you will, 912 00:48:39,400 --> 00:48:45,860 for free a functional form that is 913 00:48:45,860 --> 00:48:48,800 quite complicated in capturing the true functional 914 00:48:48,800 --> 00:48:51,620 dependencies of uniformity. 915 00:48:51,620 --> 00:48:54,020 That's the important idea here is you 916 00:48:54,020 --> 00:48:58,062 have a simple, direct, underlying model of rate. 917 00:48:58,062 --> 00:49:00,020 And then if you've got a complicated functional 918 00:49:00,020 --> 00:49:02,750 dependence, like non-uniformity metric on that, 919 00:49:02,750 --> 00:49:07,370 you calculate that from the simpler underlying model. 920 00:49:07,370 --> 00:49:08,480 I like that. 921 00:49:08,480 --> 00:49:10,070 I think that that's nice. 922 00:49:10,070 --> 00:49:12,470 That's a key idea in this paper. 923 00:49:12,470 --> 00:49:15,740 Then they go on and do, once they've got these models, 924 00:49:15,740 --> 00:49:19,400 they do multiple objective optimization 925 00:49:19,400 --> 00:49:23,690 trying to minimize or maximize selectivity, minimize 926 00:49:23,690 --> 00:49:27,890 their non-uniformity value, maximize the rate. 927 00:49:27,890 --> 00:49:29,660 They've got some constraints. 928 00:49:29,660 --> 00:49:34,280 They go in and kind of do a multiple objective. 929 00:49:34,280 --> 00:49:39,620 Come up with some optimum values and show how much improvement 930 00:49:39,620 --> 00:49:41,720 they get. 931 00:49:41,720 --> 00:49:45,170 But the key idea, I think, was that contribution 932 00:49:45,170 --> 00:49:48,620 of this notion that you want to start with something 933 00:49:48,620 --> 00:49:51,150 that's simpler model. 934 00:49:51,150 --> 00:49:54,890 Then if you've got a complicated optimization objective, 935 00:49:54,890 --> 00:49:57,740 use the simple model, build the complicated objective 936 00:49:57,740 --> 00:50:03,550 or complicated derived function, and then use that. 937 00:50:03,550 --> 00:50:05,433 So any questions on that? 938 00:50:05,433 --> 00:50:06,600 Does that seem pretty clear? 939 00:50:09,320 --> 00:50:13,590 What I'm going to show next is the last piece of this puzzle, 940 00:50:13,590 --> 00:50:14,780 which is-- 941 00:50:14,780 --> 00:50:17,104 yes, before I do that? 942 00:50:17,104 --> 00:50:18,080 AUDIENCE: [INAUDIBLE] 943 00:50:18,080 --> 00:50:18,980 DUANE BONING: Yes. 944 00:50:18,980 --> 00:50:23,846 AUDIENCE: It's different to [INAUDIBLE] when you 945 00:50:23,846 --> 00:50:26,490 compare rectangle to circle. 946 00:50:26,490 --> 00:50:30,646 So the circle has the minimal area to perimeter ratio, 947 00:50:30,646 --> 00:50:33,470 while the rectangle has the maximum. 948 00:50:33,470 --> 00:50:37,220 So it's most likely when we have a rectangle we most likely have 949 00:50:37,220 --> 00:50:40,988 to have a uniform sampling. 950 00:50:40,988 --> 00:50:42,530 Is it correct to look at it that way? 951 00:50:47,283 --> 00:50:48,950 DUANE BONING: I don't see a relationship 952 00:50:48,950 --> 00:50:51,680 between the perimeter to area. 953 00:50:51,680 --> 00:50:56,630 AUDIENCE: For a given perimeter. 954 00:50:56,630 --> 00:50:58,700 DUANE BONING: And the reason I'm not sure 955 00:50:58,700 --> 00:51:03,590 that works is if I have-- here's my wafer. 956 00:51:03,590 --> 00:51:06,440 In the rectangular case, it's easy to sort of tessellate 957 00:51:06,440 --> 00:51:12,710 the space and cover the space with non-overlapping regions 958 00:51:12,710 --> 00:51:15,410 that each data point represents. 959 00:51:15,410 --> 00:51:20,030 In order to do that with the circular sampling plan, 960 00:51:20,030 --> 00:51:21,845 each one of these is not a circle. 961 00:51:24,800 --> 00:51:28,100 The way you have to sort of do it is each one of these 962 00:51:28,100 --> 00:51:35,285 is this arc to cover the space with non-overlapping. 963 00:51:35,285 --> 00:51:35,910 Now, who knows? 964 00:51:35,910 --> 00:51:38,790 If you allow yourself sort of overlapping coverage, 965 00:51:38,790 --> 00:51:42,540 maybe what you're talking about makes sense. 966 00:51:42,540 --> 00:51:46,800 But I don't think it's really a perimeter that each area-- 967 00:51:46,800 --> 00:51:49,200 I think it really is area, just purely 968 00:51:49,200 --> 00:51:52,170 area that it's representing. 969 00:51:52,170 --> 00:51:55,710 But I haven't thought about the perimeter issue. 970 00:51:55,710 --> 00:51:58,470 It's kind of interesting. 971 00:51:58,470 --> 00:52:00,630 And by the way, this is the kind of calculation 972 00:52:00,630 --> 00:52:02,940 you would do back in spatial modeling 973 00:52:02,940 --> 00:52:06,900 to figure out as a function of your xy 974 00:52:06,900 --> 00:52:08,610 coordinate for your measurement point 975 00:52:08,610 --> 00:52:11,280 how much area that ought to represent. 976 00:52:11,280 --> 00:52:13,410 You're basically dividing it up and you 977 00:52:13,410 --> 00:52:15,960 have to kind of do those calculations. 978 00:52:19,910 --> 00:52:20,780 OK. 979 00:52:20,780 --> 00:52:24,380 The last thing I want to show is this paper by Guo and Sachs. 980 00:52:24,380 --> 00:52:26,720 And again, I remind you Elliot Sachs 981 00:52:26,720 --> 00:52:29,150 is a professor here in mechanical engineering 982 00:52:29,150 --> 00:52:33,200 who two lives ago did a lot of work 983 00:52:33,200 --> 00:52:34,880 in semiconductor process control. 984 00:52:38,390 --> 00:52:41,810 His life after that was 3D printing. 985 00:52:41,810 --> 00:52:46,290 And his current life is now solar energy. 986 00:52:46,290 --> 00:52:49,130 So he's made some wonderful contributions 987 00:52:49,130 --> 00:52:50,090 and some big moves. 988 00:52:50,090 --> 00:52:53,840 And I actually like this paper quite a lot. 989 00:52:53,840 --> 00:52:56,630 The basic idea in this, it's looking at this issue, again, 990 00:52:56,630 --> 00:53:00,260 of spatial uniformity, how one does modeling, and then uses 991 00:53:00,260 --> 00:53:04,190 that for optimization and control of uniformity. 992 00:53:04,190 --> 00:53:08,120 And basically, here he builds on, I think, 993 00:53:08,120 --> 00:53:12,980 he builds on this notion that we learned from the Mozumder 994 00:53:12,980 --> 00:53:17,840 and Loewenstein paper, which is build simple underlying models 995 00:53:17,840 --> 00:53:20,720 and then combine them and use them to solve the larger 996 00:53:20,720 --> 00:53:21,800 problems. 997 00:53:21,800 --> 00:53:25,190 But keep it as simple as you possibly can. 998 00:53:25,190 --> 00:53:29,480 And he suggests, together with Andy Guo, 999 00:53:29,480 --> 00:53:32,480 he suggests that we should flip around 1000 00:53:32,480 --> 00:53:35,330 what Loewenstein and Mozumder did 1001 00:53:35,330 --> 00:53:41,450 and actually build models for each spatial location first. 1002 00:53:41,450 --> 00:53:43,497 Don't fit the spatial model first 1003 00:53:43,497 --> 00:53:45,080 and then take that coefficient and try 1004 00:53:45,080 --> 00:53:47,240 to build a model of that coefficient 1005 00:53:47,240 --> 00:53:49,340 as a function of process. 1006 00:53:49,340 --> 00:53:52,520 Instead just look at that one spatial location 1007 00:53:52,520 --> 00:53:56,570 and build a model of that spatial location's response 1008 00:53:56,570 --> 00:53:59,420 as a function of the process conditions. 1009 00:53:59,420 --> 00:54:04,190 And now I can combine that to other spatial derived things, 1010 00:54:04,190 --> 00:54:08,240 like fitting a spatial model or not necessarily even fitting 1011 00:54:08,240 --> 00:54:11,030 a spatial model, but now I can basically 1012 00:54:11,030 --> 00:54:15,170 use spatial information, not lose it. 1013 00:54:15,170 --> 00:54:17,540 I still know the left side or the right side 1014 00:54:17,540 --> 00:54:20,510 is higher or lower. 1015 00:54:20,510 --> 00:54:23,730 What happens when you calculate standard deviation over mu? 1016 00:54:23,730 --> 00:54:26,450 You lose information of left side or right side 1017 00:54:26,450 --> 00:54:27,920 was higher or lower. 1018 00:54:27,920 --> 00:54:30,800 You just boil it down to they were different. 1019 00:54:30,800 --> 00:54:32,690 He says keep the sites. 1020 00:54:32,690 --> 00:54:33,920 Keep the sites. 1021 00:54:33,920 --> 00:54:40,430 Use that information to drive towards an improvement. 1022 00:54:40,430 --> 00:54:42,720 Keep the simple model, build those, 1023 00:54:42,720 --> 00:54:44,340 and then you can combine them. 1024 00:54:44,340 --> 00:54:46,440 So let's look and see how that works. 1025 00:54:46,440 --> 00:54:47,590 Here's the basic idea. 1026 00:54:50,120 --> 00:54:54,050 Let's say I was measuring three different locations 1027 00:54:54,050 --> 00:54:58,100 on the wafer, y1, y2, y3. 1028 00:54:58,100 --> 00:55:00,140 And I have two equipment settings. 1029 00:55:00,140 --> 00:55:02,990 So two different process parameters. 1030 00:55:02,990 --> 00:55:06,800 What you're going to do here is build a response surface 1031 00:55:06,800 --> 00:55:09,260 model and a DOE. 1032 00:55:09,260 --> 00:55:12,080 And there's two different approaches one can take. 1033 00:55:12,080 --> 00:55:16,280 One can take what I'll called the single response 1034 00:55:16,280 --> 00:55:18,320 surface, the classical approach that 1035 00:55:18,320 --> 00:55:21,170 says I'm looking at different combinations of my input 1036 00:55:21,170 --> 00:55:22,220 parameters. 1037 00:55:22,220 --> 00:55:24,200 I measure my three points. 1038 00:55:24,200 --> 00:55:27,830 I've got three data points I can then calculate for that process 1039 00:55:27,830 --> 00:55:29,840 condition, that run number one. 1040 00:55:29,840 --> 00:55:32,480 I can calculate the standard deviation across those three 1041 00:55:32,480 --> 00:55:35,630 data points, divide by the mean of those three data points. 1042 00:55:35,630 --> 00:55:38,630 That gives me what the uniformity spatially 1043 00:55:38,630 --> 00:55:40,910 was for that run. 1044 00:55:40,910 --> 00:55:43,400 I can then repeat that for all of my different process 1045 00:55:43,400 --> 00:55:44,910 conditions. 1046 00:55:44,910 --> 00:55:48,530 And then I can build a single response surface 1047 00:55:48,530 --> 00:55:52,820 at the end, SRS, Single Response Surface, 1048 00:55:52,820 --> 00:55:58,730 that is the response surface of the response of non-uniformity 1049 00:55:58,730 --> 00:56:02,760 as a function of my process conditions. 1050 00:56:02,760 --> 00:56:04,820 That's a classic approach, and you will still 1051 00:56:04,820 --> 00:56:06,620 run into that in many papers. 1052 00:56:06,620 --> 00:56:09,230 We didn't see it in the Loewenstein paper. 1053 00:56:09,230 --> 00:56:11,610 They used a slightly different approach. 1054 00:56:11,610 --> 00:56:13,580 But this is a classic approach. 1055 00:56:13,580 --> 00:56:18,140 And what he says is no, don't do that. 1056 00:56:18,140 --> 00:56:20,300 Do not do that. 1057 00:56:20,300 --> 00:56:25,910 Instead use the same design of experiments, same data, 1058 00:56:25,910 --> 00:56:29,240 but do something that I think is a little bit smarter, which 1059 00:56:29,240 --> 00:56:32,090 says take your different site. 1060 00:56:32,090 --> 00:56:33,890 This is a site one. 1061 00:56:33,890 --> 00:56:35,630 I'm always measuring the left side. 1062 00:56:35,630 --> 00:56:36,830 Maybe this is the center. 1063 00:56:36,830 --> 00:56:38,720 This is the right side of the wafer. 1064 00:56:38,720 --> 00:56:44,150 And build a response surface for that site response, 1065 00:56:44,150 --> 00:56:46,700 maybe it's the thickness or the etch rate or whatever, 1066 00:56:46,700 --> 00:56:51,110 as a function of the process conditions. 1067 00:56:51,110 --> 00:56:53,750 Do the same for all of the sites that you've got. 1068 00:56:53,750 --> 00:56:59,150 Build separate multiple response surfaces. 1069 00:56:59,150 --> 00:57:02,120 And now you know how each site responds for whatever process 1070 00:57:02,120 --> 00:57:03,710 conditions you want. 1071 00:57:03,710 --> 00:57:06,020 And now you can combine them if you want a uniformity 1072 00:57:06,020 --> 00:57:07,850 metric to see which is better. 1073 00:57:07,850 --> 00:57:09,350 But you can also do smarter things 1074 00:57:09,350 --> 00:57:11,090 like try to balance them. 1075 00:57:11,090 --> 00:57:13,610 Get the left side back up to match the right side. 1076 00:57:13,610 --> 00:57:15,440 Do other sorts of things. 1077 00:57:15,440 --> 00:57:18,950 OK, let's see how that works. 1078 00:57:18,950 --> 00:57:21,740 His basic point here is very similar to the Mozumder 1079 00:57:21,740 --> 00:57:26,840 and Loewenstein, which is the single response 1080 00:57:26,840 --> 00:57:31,580 surface is a very tough job to directly model 1081 00:57:31,580 --> 00:57:35,390 in one response surface, this highly nonlinear sigma over mu. 1082 00:57:35,390 --> 00:57:38,780 And in fact, if I try to build a second order model, 1083 00:57:38,780 --> 00:57:41,630 you often need a second order model 1084 00:57:41,630 --> 00:57:45,180 as a function of the process conditions, 1085 00:57:45,180 --> 00:57:48,740 which means lots of sampling in your multiple levels 1086 00:57:48,740 --> 00:57:53,090 in your process conditions, at least three, maybe more. 1087 00:57:53,090 --> 00:57:55,460 And you get a very complicated model. 1088 00:57:55,460 --> 00:57:59,930 What he also shows is that very often 1089 00:57:59,930 --> 00:58:02,510 with the multiple response surface model, 1090 00:58:02,510 --> 00:58:08,750 you can have much simpler lower order models for each site. 1091 00:58:08,750 --> 00:58:11,030 And in fact, what he will do is show 1092 00:58:11,030 --> 00:58:15,890 if I just build a linear model for each site as a function 1093 00:58:15,890 --> 00:58:18,320 of my process conditions, meaning I 1094 00:58:18,320 --> 00:58:22,160 actually need fewer levels in my DOE. 1095 00:58:22,160 --> 00:58:28,010 I can actually get simple linear models for my individual sites. 1096 00:58:28,010 --> 00:58:30,530 And then when I functionally calculate 1097 00:58:30,530 --> 00:58:34,550 the mu, the mean of those three, that's a simple formula, 1098 00:58:34,550 --> 00:58:38,690 and standard deviation over mu is a simple formula. 1099 00:58:38,690 --> 00:58:43,970 But embedded in it is the right nonlinear functional form 1100 00:58:43,970 --> 00:58:45,860 for sigma over mu. 1101 00:58:45,860 --> 00:58:49,700 And so you get this non-linearity out 1102 00:58:49,700 --> 00:58:52,970 of very simple linear models. 1103 00:58:52,970 --> 00:58:55,040 And that's the key idea. 1104 00:58:55,040 --> 00:58:58,070 Similar to Mozumder and Loewenstein. 1105 00:58:58,070 --> 00:58:59,990 Build the simple model and then use it 1106 00:58:59,990 --> 00:59:02,870 and combine it in whatever complicated functional form 1107 00:59:02,870 --> 00:59:04,670 you have. 1108 00:59:04,670 --> 00:59:06,890 So he argues that you can actually 1109 00:59:06,890 --> 00:59:09,020 get a smaller number of data. 1110 00:59:09,020 --> 00:59:12,440 Not just use the same data in a smarter way, but actually 1111 00:59:12,440 --> 00:59:17,370 sample less in your process conditions in many cases. 1112 00:59:17,370 --> 00:59:19,700 So you get a savings in the DOE. 1113 00:59:19,700 --> 00:59:22,250 Another very nice advantage that's 1114 00:59:22,250 --> 00:59:26,180 articulated in the paper, I'm not going to go into here, 1115 00:59:26,180 --> 00:59:30,770 is that each of those models of the process 1116 00:59:30,770 --> 00:59:34,550 is a nice, simple linear model. 1117 00:59:34,550 --> 00:59:37,910 And you can use simple linear models with the cycle 1118 00:59:37,910 --> 00:59:42,620 to cycle run by run control that Dave Hardt talked 1119 00:59:42,620 --> 00:59:44,540 about a couple of lectures ago. 1120 00:59:44,540 --> 00:59:46,880 You can basically take that as a model, 1121 00:59:46,880 --> 00:59:50,630 adapt that model rapidly to changing process drifts 1122 00:59:50,630 --> 00:59:55,070 or conditions in your equipment, use that updated model in sort 1123 00:59:55,070 --> 00:59:59,090 of a PI or PID kind of fashion, and use 1124 00:59:59,090 --> 01:00:02,180 that to improve the selection or pick 1125 01:00:02,180 --> 01:00:04,220 the selection of the right process condition 1126 01:00:04,220 --> 01:00:06,230 for the next wafer run. 1127 01:00:06,230 --> 01:00:09,410 It's a lot easier to use this kind of a model for cycle 1128 01:00:09,410 --> 01:00:11,010 to cycle control. 1129 01:00:11,010 --> 01:00:14,090 He also has some other very nice points. 1130 01:00:14,090 --> 01:00:17,120 He does a bit of analysis in the paper saying which one 1131 01:00:17,120 --> 01:00:20,610 is less susceptible to noise. 1132 01:00:20,610 --> 01:00:23,840 It turns out that the site models kind of average 1133 01:00:23,840 --> 01:00:26,510 out noise a little bit better than putting everything 1134 01:00:26,510 --> 01:00:28,250 into signal to noise. 1135 01:00:28,250 --> 01:00:33,170 And then the last point is a really important one, which 1136 01:00:33,170 --> 01:00:35,690 is if you get this really complicated 1137 01:00:35,690 --> 01:00:39,860 functional cubic model that tells you 1138 01:00:39,860 --> 01:00:42,920 how non-uniformity changes as a function of your process 1139 01:00:42,920 --> 01:00:47,930 condition, do you as the process engineer have any idea really 1140 01:00:47,930 --> 01:00:51,260 intuitively what will happen if you change a knob 1141 01:00:51,260 --> 01:00:54,260 or what's happening spatially across your wafer? 1142 01:00:54,260 --> 01:00:57,620 His argument is you've lost a lot of that information. 1143 01:00:57,620 --> 01:00:59,930 If you actually have your site models, 1144 01:00:59,930 --> 01:01:02,120 you can build that full spatial map, 1145 01:01:02,120 --> 01:01:05,300 see how each site is changing as you change your process 1146 01:01:05,300 --> 01:01:08,830 condition, and actually see-- 1147 01:01:08,830 --> 01:01:10,940 build some process knowledge, see 1148 01:01:10,940 --> 01:01:14,280 what's going on much closer to the process. 1149 01:01:14,280 --> 01:01:16,860 So here's a couple of examples. 1150 01:01:16,860 --> 01:01:19,130 This relates to, again, the point 1151 01:01:19,130 --> 01:01:22,490 that the complexity of the uniformity metric 1152 01:01:22,490 --> 01:01:25,620 has a lot in it and you actually lose information. 1153 01:01:25,620 --> 01:01:28,160 So here's the point where if I measure the thickness 1154 01:01:28,160 --> 01:01:35,180 at the left site, thickness at the right site, so red 1155 01:01:35,180 --> 01:01:41,000 is my left and blue here are my right measurement points. 1156 01:01:41,000 --> 01:01:45,020 Let's say I have simple linear responses 1157 01:01:45,020 --> 01:01:49,870 for how those two sides change as a function of the input 1158 01:01:49,870 --> 01:01:51,340 parameter. 1159 01:01:51,340 --> 01:01:53,800 They really do change linearly. 1160 01:01:53,800 --> 01:01:56,450 Notice what happens, in one case, 1161 01:01:56,450 --> 01:02:00,320 the left side is much thicker than the right 1162 01:02:00,320 --> 01:02:01,840 and in the other process condition, 1163 01:02:01,840 --> 01:02:03,940 the right side is much thicker than the left. 1164 01:02:03,940 --> 01:02:07,150 If you actually just directly do a single response surface 1165 01:02:07,150 --> 01:02:12,550 kind of idea or do our calculation of sigma over mu, 1166 01:02:12,550 --> 01:02:17,230 you can easily get a little bit fooled here. 1167 01:02:17,230 --> 01:02:20,440 They're both equal in terms of a uniformity value, 1168 01:02:20,440 --> 01:02:23,620 but you've lost track just looking at these values 1169 01:02:23,620 --> 01:02:24,620 by themselves. 1170 01:02:24,620 --> 01:02:28,780 You don't know what's going on on the wafer. 1171 01:02:28,780 --> 01:02:30,520 So looking back here, you can see 1172 01:02:30,520 --> 01:02:31,930 what's going on in the wafer. 1173 01:02:31,930 --> 01:02:34,960 The second point is if one wants to actually go 1174 01:02:34,960 --> 01:02:38,680 in and do a medium input and interpolate 1175 01:02:38,680 --> 01:02:41,320 if I have these site models, I could actually 1176 01:02:41,320 --> 01:02:45,400 project and guess and say, I think 1177 01:02:45,400 --> 01:02:47,380 that if I were to run at the middle, 1178 01:02:47,380 --> 01:02:48,970 I would have very good uniformity. 1179 01:02:48,970 --> 01:02:51,350 The thickness would be the same in both cases. 1180 01:02:51,350 --> 01:02:55,120 And if I calculate a value based on these underlying models, 1181 01:02:55,120 --> 01:02:57,310 I can actually project what happens 1182 01:02:57,310 --> 01:02:59,590 at the intermediate values. 1183 01:02:59,590 --> 01:03:03,760 This would suggest for a control problem or an optimization 1184 01:03:03,760 --> 01:03:06,400 try the center point. 1185 01:03:06,400 --> 01:03:09,010 Whereas if all I had were these two data points, 1186 01:03:09,010 --> 01:03:11,290 I would say they're the same. 1187 01:03:11,290 --> 01:03:13,870 I can't really improve things much. 1188 01:03:13,870 --> 01:03:15,050 Yeah? 1189 01:03:15,050 --> 01:03:17,655 AUDIENCE: Left or right with respect to [INAUDIBLE] 1190 01:03:17,655 --> 01:03:22,343 if you have a wafer and you have a knot and then [INAUDIBLE] 1191 01:03:22,343 --> 01:03:26,470 wafer is in a certain way with respect to the notch, 1192 01:03:26,470 --> 01:03:28,870 but you still have a reticle. 1193 01:03:28,870 --> 01:03:31,390 And that can be with any direction with respect 1194 01:03:31,390 --> 01:03:32,020 to the notch. 1195 01:03:32,020 --> 01:03:34,240 So any information that you get, you 1196 01:03:34,240 --> 01:03:39,353 would never be sure for the friendships how-- 1197 01:03:39,353 --> 01:03:41,770 DUANE BONING: Yeah, I'm not projecting down to chip level. 1198 01:03:41,770 --> 01:03:43,790 I'm thinking here wafer level. 1199 01:03:43,790 --> 01:03:45,790 And I think you always know-- you can always 1200 01:03:45,790 --> 01:03:50,170 refer to something with the wafer notch or the wafer flat. 1201 01:03:50,170 --> 01:03:52,000 So I think that's-- 1202 01:03:52,000 --> 01:03:55,150 right and left, these are conceptual here, 1203 01:03:55,150 --> 01:03:57,580 but I think they can be very well defined. 1204 01:03:57,580 --> 01:04:01,270 If you are also worried about repeated chip scale 1205 01:04:01,270 --> 01:04:03,070 things coming from lithography fields, 1206 01:04:03,070 --> 01:04:08,390 that's an extra layer of spatial concerns. 1207 01:04:08,390 --> 01:04:08,890 OK. 1208 01:04:08,890 --> 01:04:11,620 And then this is essentially the implications for control 1209 01:04:11,620 --> 01:04:15,280 are pretty much what I was just describing. 1210 01:04:15,280 --> 01:04:20,800 If you just use sort of the high and the low, 1211 01:04:20,800 --> 01:04:23,290 and this is your intended input, you 1212 01:04:23,290 --> 01:04:25,990 can now predict a little bit better 1213 01:04:25,990 --> 01:04:30,260 what your actual output would be with these simplified cases. 1214 01:04:30,260 --> 01:04:33,520 So in the paper, they do some examples 1215 01:04:33,520 --> 01:04:36,790 actually using some experimental data that they generated. 1216 01:04:36,790 --> 01:04:39,370 This was for the low pressure chemical vapor 1217 01:04:39,370 --> 01:04:41,500 deposition of polysilicon. 1218 01:04:41,500 --> 01:04:46,060 And it has spatial uniformity in it not just on the wafer. 1219 01:04:46,060 --> 01:04:49,990 Actually what they're doing here is spatial uniformity 1220 01:04:49,990 --> 01:04:52,000 across the tube. 1221 01:04:52,000 --> 01:04:54,610 So as a function of wafer position, 1222 01:04:54,610 --> 01:04:59,260 these are big multi-wafer tubes that might actually process 24 1223 01:04:59,260 --> 01:05:03,040 or 48 or 96 wafers all at once. 1224 01:05:03,040 --> 01:05:04,630 And there's a gas flow. 1225 01:05:04,630 --> 01:05:07,750 There's an injector for the gas, and the gas flows. 1226 01:05:07,750 --> 01:05:11,330 We've got a center injector, source injector. 1227 01:05:11,330 --> 01:05:13,720 The gas flows are somewhat non-uniform. 1228 01:05:13,720 --> 01:05:17,500 And you may, in fact, have systematic spatial dependencies 1229 01:05:17,500 --> 01:05:22,540 as a function of temperatures, gas flows, and so on. 1230 01:05:22,540 --> 01:05:24,670 And what they basically went in and did 1231 01:05:24,670 --> 01:05:28,120 is compared single response surface models 1232 01:05:28,120 --> 01:05:31,210 to multiple response surface models. 1233 01:05:31,210 --> 01:05:34,960 Pictured here is basically showing 1234 01:05:34,960 --> 01:05:37,540 this complicated dependence of signal 1235 01:05:37,540 --> 01:05:42,610 to noise ratio, the single response surface model, 1236 01:05:42,610 --> 01:05:45,940 but the simple dependence of deposition rates 1237 01:05:45,940 --> 01:05:48,230 on these two parameters. 1238 01:05:48,230 --> 01:05:53,890 So down here, this is basically saying my site here 1239 01:05:53,890 --> 01:05:55,240 is wafer number 26. 1240 01:05:55,240 --> 01:05:58,450 I want to know what the average thickness on wafer 26 1241 01:05:58,450 --> 01:06:01,210 is as a function of these two flow rates. 1242 01:06:01,210 --> 01:06:05,530 And it's a nice, simple dependence on the flow rates. 1243 01:06:05,530 --> 01:06:08,500 If I look at a different site, a little bit 1244 01:06:08,500 --> 01:06:11,080 further down the tube at wafer number 124, 1245 01:06:11,080 --> 01:06:14,240 I also get a simple model. 1246 01:06:14,240 --> 01:06:19,150 But if I now combine these into a sigma over mu, 1247 01:06:19,150 --> 01:06:23,470 it has a very complicated shape. 1248 01:06:23,470 --> 01:06:26,620 Built out of these simple dependencies, simple responses 1249 01:06:26,620 --> 01:06:29,830 spatially, they combine to a very complicated 1250 01:06:29,830 --> 01:06:34,000 non-uniformity signature. 1251 01:06:34,000 --> 01:06:37,150 Then what they did is say, OK, I'm 1252 01:06:37,150 --> 01:06:42,550 going to compare SRS to MRS. And I'm, furthermore, 1253 01:06:42,550 --> 01:06:45,340 going to tie one arm behind my back 1254 01:06:45,340 --> 01:06:49,480 when I do MRS. The arm I'm going to tie behind my back 1255 01:06:49,480 --> 01:06:54,810 is I'm going to let SRS do three level DOEs. 1256 01:06:54,810 --> 01:06:57,090 Two parameters, flow rate one and flow rate two. 1257 01:06:57,090 --> 01:07:00,990 But I'm doing a full factorial three level. 1258 01:07:00,990 --> 01:07:03,750 I do nine different process experiments. 1259 01:07:03,750 --> 01:07:07,620 And I fit the model for non-uniformity for that. 1260 01:07:07,620 --> 01:07:10,020 For MRS, I'm just going to let myself 1261 01:07:10,020 --> 01:07:12,060 pick the high and the low. 1262 01:07:12,060 --> 01:07:16,770 I'm just going to do less than half the number of experiments 1263 01:07:16,770 --> 01:07:19,350 and build my site models and then 1264 01:07:19,350 --> 01:07:24,000 see what kind of predicted signal to noise ratio 1265 01:07:24,000 --> 01:07:25,380 I will get. 1266 01:07:25,380 --> 01:07:27,180 And I'm going to look now at also 1267 01:07:27,180 --> 01:07:31,320 how noise factors into this and see, do I do better with SRS 1268 01:07:31,320 --> 01:07:33,450 or do I do better with MRS? 1269 01:07:33,450 --> 01:07:36,210 And this is a picture that shows, hey, in these two cases, 1270 01:07:36,210 --> 01:07:40,170 they're kind of suggesting different non-uniformity 1271 01:07:40,170 --> 01:07:41,400 signatures. 1272 01:07:41,400 --> 01:07:44,040 But what happens now if I repeat that 1273 01:07:44,040 --> 01:07:48,510 with different amounts of injected noise? 1274 01:07:48,510 --> 01:07:51,420 And it's basically showing that the MRS 1275 01:07:51,420 --> 01:07:54,660 in four repeats of this kind of experiment 1276 01:07:54,660 --> 01:07:56,820 with different amounts of injected noise 1277 01:07:56,820 --> 01:08:00,480 pretty much gives the same surface each time. 1278 01:08:00,480 --> 01:08:06,180 Look what happens over here in the SRS case. 1279 01:08:06,180 --> 01:08:07,920 Three out of the four times, they're 1280 01:08:07,920 --> 01:08:10,290 kind of the same bowl shaped. 1281 01:08:10,290 --> 01:08:14,760 But one case, you actually get this weird hyperbolic surface. 1282 01:08:14,760 --> 01:08:17,310 More susceptible, this is just kind of an example 1283 01:08:17,310 --> 01:08:20,700 not proving anything, but just showing an example 1284 01:08:20,700 --> 01:08:23,580 that actually the models you get with the SRS 1285 01:08:23,580 --> 01:08:26,819 can actually vary and be much more sensitive to noise 1286 01:08:26,819 --> 01:08:29,760 even though you got more measurements, more 1287 01:08:29,760 --> 01:08:34,200 spatial sample or more levels of your DOE. 1288 01:08:34,200 --> 01:08:36,240 And then they went in and did sort 1289 01:08:36,240 --> 01:08:40,740 of a more thorough example, building lots of models, 1290 01:08:40,740 --> 01:08:45,029 and then using that model to drive or select 1291 01:08:45,029 --> 01:08:46,740 what an optimum point is. 1292 01:08:46,740 --> 01:08:50,910 So maybe you were trying to maximize the removal rate, 1293 01:08:50,910 --> 01:08:55,229 find the optimum point in each of those cases 1294 01:08:55,229 --> 01:08:59,010 and see what the spread is with the SRS and MRS 1295 01:08:59,010 --> 01:09:01,170 in the predicted optimal points. 1296 01:09:01,170 --> 01:09:03,210 And you can see over here on the right 1297 01:09:03,210 --> 01:09:05,100 the basic conclusion that they come 1298 01:09:05,100 --> 01:09:09,899 to is based on the single response surface 1299 01:09:09,899 --> 01:09:13,170 model and the multiple response surface model. 1300 01:09:13,170 --> 01:09:16,590 They're spread in the two in terms of the optimum. 1301 01:09:16,590 --> 01:09:20,700 But the single response surface model is basically biased. 1302 01:09:20,700 --> 01:09:23,760 It drives you to an optimum point 1303 01:09:23,760 --> 01:09:27,510 that is actually not close to the measured optimum point 1304 01:09:27,510 --> 01:09:29,450 on average. 1305 01:09:29,450 --> 01:09:33,450 They basically say single response surface modeling 1306 01:09:33,450 --> 01:09:34,800 is dangerous. 1307 01:09:34,800 --> 01:09:36,720 Not only is it inefficient, but it 1308 01:09:36,720 --> 01:09:39,819 can drive you to make errors. 1309 01:09:39,819 --> 01:09:42,970 So that's a very interesting observation. 1310 01:09:42,970 --> 01:09:45,227 So I think I'm going to wrap it up here so we have 1311 01:09:45,227 --> 01:09:47,560 a little bit of time to talk about some of the projects, 1312 01:09:47,560 --> 01:09:50,130 especially with Singapore people while we've got the video. 1313 01:09:50,130 --> 01:09:53,340 But I think these are really neat papers, really neat ideas. 1314 01:09:53,340 --> 01:09:58,200 Basically first idea is spatial sampling, how you sample, 1315 01:09:58,200 --> 01:10:00,120 and what area each of those sampling points 1316 01:10:00,120 --> 01:10:02,340 represents matters. 1317 01:10:02,340 --> 01:10:04,830 So you can be smart about that, using either weighting 1318 01:10:04,830 --> 01:10:06,360 or uniform sampling. 1319 01:10:06,360 --> 01:10:09,300 And then there's some neat ideas about combined process 1320 01:10:09,300 --> 01:10:10,860 and spatial modeling. 1321 01:10:10,860 --> 01:10:13,620 I basically really like the multiple response surface 1322 01:10:13,620 --> 01:10:14,850 modeling approach. 1323 01:10:14,850 --> 01:10:17,130 I think the Mozumder and Loewenstein 1324 01:10:17,130 --> 01:10:19,020 was a nice stepping stone on the way 1325 01:10:19,020 --> 01:10:21,390 to recognize that you'd like to build simple models 1326 01:10:21,390 --> 01:10:22,740 and then derive on them. 1327 01:10:22,740 --> 01:10:26,190 I like the Guo and Sachs because it carries it one step further. 1328 01:10:26,190 --> 01:10:28,680 It says build simple models of your response 1329 01:10:28,680 --> 01:10:30,720 at each of your spatial locations 1330 01:10:30,720 --> 01:10:32,820 and then combine those or use those 1331 01:10:32,820 --> 01:10:37,200 as you see fit to either boil down and come up 1332 01:10:37,200 --> 01:10:40,560 with an overall uniformity or make other kinds of control 1333 01:10:40,560 --> 01:10:42,730 or optimization decisions. 1334 01:10:42,730 --> 01:10:45,540 So any questions on that before we switch over 1335 01:10:45,540 --> 01:10:47,010 to talking about some projects? 1336 01:10:50,880 --> 01:10:51,940 Think about it. 1337 01:10:51,940 --> 01:10:53,440 Maybe some of your data is actually 1338 01:10:53,440 --> 01:10:55,470 a spatially sampled for one of your projects. 1339 01:10:55,470 --> 01:10:56,880 It'd be cool. 1340 01:10:56,880 --> 01:10:58,710 Cool to add some spatial modeling in there. 1341 01:11:02,320 --> 01:11:06,040 How finely refined can you probe that monkey brain? 1342 01:11:08,740 --> 01:11:10,690 OK, that's it. 1343 01:11:10,690 --> 01:11:15,490 I know some of the folks here in Cambridge, 1344 01:11:15,490 --> 01:11:17,230 I've met with one group. 1345 01:11:17,230 --> 01:11:20,830 I think Dave has talked to Matt and you guys are thinking. 1346 01:11:20,830 --> 01:11:25,390 I talked also-- will want to try to set up a meeting. 1347 01:11:25,390 --> 01:11:28,000 I think I sent email pretty much to everybody 1348 01:11:28,000 --> 01:11:29,770 saying I think the projects look good. 1349 01:11:29,770 --> 01:11:31,810 Here's some ideas you might think about. 1350 01:11:31,810 --> 01:11:34,960 The purpose of the meeting is not to get approval. 1351 01:11:34,960 --> 01:11:36,610 Consider this now. 1352 01:11:36,610 --> 01:11:37,660 You're approved. 1353 01:11:37,660 --> 01:11:38,680 Go, run. 1354 01:11:38,680 --> 01:11:40,390 You've only got a week. 1355 01:11:40,390 --> 01:11:41,620 Less than a week. 1356 01:11:41,620 --> 01:11:42,700 Go full boar. 1357 01:11:42,700 --> 01:11:46,930 The purpose of the meeting is to answer questions, help out 1358 01:11:46,930 --> 01:11:49,870 if you've got additional inquiries. 1359 01:11:49,870 --> 01:11:54,400 Feel free to contact me, Dave Hardt, Hayden, at any time. 1360 01:11:54,400 --> 01:11:58,040 But I would like, if we haven't had a chance to talk yet, 1361 01:11:58,040 --> 01:12:00,960 just try to set something up. 1362 01:12:00,960 --> 01:12:07,570 AUDIENCE: [INAUDIBLE] 1363 01:12:07,570 --> 01:12:12,400 DUANE BONING: Well, one and a half hours less than one week 1364 01:12:12,400 --> 01:12:14,080 from today will be the-- 1365 01:12:14,080 --> 01:12:18,250 so what I will also be doing is making assignments 1366 01:12:18,250 --> 01:12:21,760 for what groups will present on Tuesday for the next week 1367 01:12:21,760 --> 01:12:25,750 and what groups will present on Thursday of next week. 1368 01:12:25,750 --> 01:12:29,170 So it'll be kind of random luck of the draw. 1369 01:12:29,170 --> 01:12:33,760 But I'll let you know by the end of this week 1370 01:12:33,760 --> 01:12:35,140 when your time will be. 1371 01:12:35,140 --> 01:12:38,560 And then everybody's reports are due on Friday 1372 01:12:38,560 --> 01:12:41,300 of next week, the end of class. 1373 01:12:41,300 --> 01:12:43,030 OK? 1374 01:12:43,030 --> 01:12:45,190 I didn't mean to panic you. 1375 01:12:45,190 --> 01:12:49,150 You've got almost at least a full week for the presentation. 1376 01:12:49,150 --> 01:12:53,920 And then in fact, one idea is if you get some quick questions 1377 01:12:53,920 --> 01:12:57,910 or feedback from the class, from us, during the presentation, 1378 01:12:57,910 --> 01:13:00,640 it gives you a quick chance to maybe add a little bit 1379 01:13:00,640 --> 01:13:02,680 or do a quick additional analysis 1380 01:13:02,680 --> 01:13:05,120 before the report is due. 1381 01:13:05,120 --> 01:13:07,630 So I actually do recommend that you 1382 01:13:07,630 --> 01:13:10,930 take that opportunity to fold in some feedback 1383 01:13:10,930 --> 01:13:13,030 from the presentation. 1384 01:13:13,030 --> 01:13:16,990 Do not think that you're locked in completely 1385 01:13:16,990 --> 01:13:17,890 at the presentation. 1386 01:13:17,890 --> 01:13:20,110 That's most of the way there, but it 1387 01:13:20,110 --> 01:13:25,960 does give a chance for a little bit of additional work 1388 01:13:25,960 --> 01:13:27,940 in the last couple of days. 1389 01:13:27,940 --> 01:13:29,310 OK? 1390 01:13:29,310 --> 01:13:30,870 Thanks. 1391 01:13:30,870 --> 01:13:31,980 So let's see. 1392 01:13:31,980 --> 01:13:32,970 Singapore folks. 1393 01:13:36,570 --> 01:13:39,350 First I think I have-- 1394 01:13:39,350 --> 01:13:42,570 let me get out of this. 1395 01:13:42,570 --> 01:13:44,243 Somebody sent me slides. 1396 01:13:44,243 --> 01:13:45,660 Let me talk with that group first. 1397 01:13:50,620 --> 01:13:51,190 Yeah. 1398 01:13:51,190 --> 01:13:53,560 So David, Stephen, Jenny. 1399 01:13:53,560 --> 01:13:56,020 These were the slides you guys sent. 1400 01:13:59,100 --> 01:14:01,170 So somebody want to-- 1401 01:14:01,170 --> 01:14:02,670 somebody talk me through these. 1402 01:14:02,670 --> 01:14:05,685 I can push buttons if you want or you can come up. 1403 01:14:05,685 --> 01:14:07,680 It's probably easier to do this than try 1404 01:14:07,680 --> 01:14:11,520 to switch control to Singapore. 1405 01:14:11,520 --> 01:14:16,200 AUDIENCE: OK, so our project try to apply the [INAUDIBLE] method 1406 01:14:16,200 --> 01:14:18,520 to supply chain case study. 1407 01:14:18,520 --> 01:14:22,770 So we can click to next slide. 1408 01:14:22,770 --> 01:14:24,300 Next slide. 1409 01:14:24,300 --> 01:14:26,200 I think we can skip this. 1410 01:14:26,200 --> 01:14:28,110 Also this will be basically the model. 1411 01:14:28,110 --> 01:14:32,700 We can consider the supplier and Company X 1412 01:14:32,700 --> 01:14:35,260 and can consider the customers. 1413 01:14:35,260 --> 01:14:38,310 So following other parameters that we will consider, 1414 01:14:38,310 --> 01:14:43,170 [INAUDIBLE] service level that your supplier promised to you 1415 01:14:43,170 --> 01:14:48,810 will be the buffer size in the company between the machines. 1416 01:14:48,810 --> 01:14:51,000 And I have TDF and TDR. 1417 01:14:51,000 --> 01:14:53,640 And p, the production rate of the flow line. 1418 01:14:53,640 --> 01:14:58,110 LT, the transportation lead time from company to a customer. 1419 01:14:58,110 --> 01:15:00,570 LC is the lead time that the company 1420 01:15:00,570 --> 01:15:02,100 promised to the customer. 1421 01:15:02,100 --> 01:15:04,517 And mu and sigma for the demand. 1422 01:15:04,517 --> 01:15:05,850 DUANE BONING: So quick question. 1423 01:15:05,850 --> 01:15:08,640 Are all of these sort of input parameters 1424 01:15:08,640 --> 01:15:11,550 or are some of these output parameters? 1425 01:15:16,780 --> 01:15:19,800 AUDIENCE: These are the input parameters I think. 1426 01:15:19,800 --> 01:15:23,482 All of these parameters are the DOE parameters. 1427 01:15:23,482 --> 01:15:24,190 DUANE BONING: OK. 1428 01:15:24,190 --> 01:15:31,080 So some will be sort of set that essentially define the setup. 1429 01:15:31,080 --> 01:15:35,400 And some will be DOE. 1430 01:15:35,400 --> 01:15:37,025 That's what you're saying? 1431 01:15:37,025 --> 01:15:37,650 AUDIENCE: Yeah. 1432 01:15:37,650 --> 01:15:40,290 DUANE BONING: OK, OK. 1433 01:15:40,290 --> 01:15:43,350 AUDIENCE: And next slide. 1434 01:15:43,350 --> 01:15:49,590 It's about the problem that is for the company. 1435 01:15:49,590 --> 01:15:54,790 Sometimes the service level will out of the range. 1436 01:15:54,790 --> 01:15:58,100 And also using the data for the service level 1437 01:15:58,100 --> 01:16:02,220 is unlike other parameters. 1438 01:16:02,220 --> 01:16:03,390 The data is quite less. 1439 01:16:03,390 --> 01:16:07,900 So we need to detect the shift as soon as possible. 1440 01:16:07,900 --> 01:16:11,340 And after that, you may want to improve some parameters 1441 01:16:11,340 --> 01:16:15,600 to improve the service level. 1442 01:16:15,600 --> 01:16:18,780 So the objective-- yeah. 1443 01:16:18,780 --> 01:16:22,350 DUANE BONING: So you've got sort of two branches here in part. 1444 01:16:22,350 --> 01:16:25,890 One is essentially SPC, right? 1445 01:16:25,890 --> 01:16:27,360 So that's what you're saying? 1446 01:16:27,360 --> 01:16:30,532 And the other is optimization. 1447 01:16:30,532 --> 01:16:31,740 AUDIENCE: Yeah, that's right. 1448 01:16:31,740 --> 01:16:32,448 DUANE BONING: OK. 1449 01:16:32,448 --> 01:16:33,960 Got it. 1450 01:16:33,960 --> 01:16:34,590 AUDIENCE: OK. 1451 01:16:34,590 --> 01:16:37,920 And you click again. 1452 01:16:37,920 --> 01:16:40,700 This objective. 1453 01:16:40,700 --> 01:16:45,230 To find the first step is apply the SPC. 1454 01:16:45,230 --> 01:16:48,050 And you find out what are the main effects 1455 01:16:48,050 --> 01:16:51,560 in the second stage that you apply the RSM to do 1456 01:16:51,560 --> 01:16:56,570 optimization to give advice for the company that 1457 01:16:56,570 --> 01:16:59,240 how can you distribute the budget 1458 01:16:59,240 --> 01:17:03,840 to adjust some of the parameters to get the best 1459 01:17:03,840 --> 01:17:06,410 result for improving the service level. 1460 01:17:06,410 --> 01:17:07,250 DUANE BONING: OK. 1461 01:17:07,250 --> 01:17:10,580 So one thought here, just going back. 1462 01:17:10,580 --> 01:17:11,630 Let's see. 1463 01:17:11,630 --> 01:17:15,560 Back to this slide. 1464 01:17:15,560 --> 01:17:19,860 Some of these parameters might be discrete, 1465 01:17:19,860 --> 01:17:23,120 which is interesting, I think. 1466 01:17:23,120 --> 01:17:26,090 So it looks like buffer size is probably discrete. 1467 01:17:26,090 --> 01:17:27,690 Is that the plan? 1468 01:17:27,690 --> 01:17:30,770 AUDIENCE: Yeah, yeah. 1469 01:17:30,770 --> 01:17:32,930 DUANE BONING: Is that the only discrete parameter? 1470 01:17:36,357 --> 01:17:38,190 AUDIENCE: Yeah, I think that's the only one. 1471 01:17:38,190 --> 01:17:40,523 DUANE BONING: Because that might be interesting in terms 1472 01:17:40,523 --> 01:17:41,160 of the DOE. 1473 01:17:41,160 --> 01:17:43,530 Does the discreetness matter? 1474 01:17:43,530 --> 01:17:47,880 You have only choice A, B, C, and it might be hard. 1475 01:17:47,880 --> 01:17:51,420 Here it might still be monotonic, 1476 01:17:51,420 --> 01:17:54,180 meaning there really is a progression. 1477 01:17:54,180 --> 01:17:55,770 So you could have discrete levels, 1478 01:17:55,770 --> 01:17:58,200 and they might have three levels that are 1479 01:17:58,200 --> 01:18:02,022 ordered in some meaningful way. 1480 01:18:02,022 --> 01:18:03,480 So that'll be an interesting-- that 1481 01:18:03,480 --> 01:18:08,190 might be an interesting twist both on the DOE, but more 1482 01:18:08,190 --> 01:18:12,030 interestingly, for response surface 1483 01:18:12,030 --> 01:18:14,850 modeling and optimization. 1484 01:18:14,850 --> 01:18:17,250 Because what do you do if your optimization 1485 01:18:17,250 --> 01:18:21,078 says pick a buffer size of 1.2? 1486 01:18:21,078 --> 01:18:22,510 AUDIENCE: Oh, I see. 1487 01:18:22,510 --> 01:18:24,240 DUANE BONING: So I think that's an interesting twist you'll 1488 01:18:24,240 --> 01:18:24,990 have to deal with. 1489 01:18:28,020 --> 01:18:30,150 So you might actually have to sort of come up 1490 01:18:30,150 --> 01:18:35,640 with some, on that point, some strategies for rounding 1491 01:18:35,640 --> 01:18:40,170 off and maybe assessing both the high and low round off points 1492 01:18:40,170 --> 01:18:43,730 to see which one's actually better. 1493 01:18:43,730 --> 01:18:46,370 That might be an example strategy. 1494 01:18:46,370 --> 01:18:47,410 OK, keep going. 1495 01:18:47,410 --> 01:18:48,410 AUDIENCE: OK, thank you. 1496 01:18:48,410 --> 01:18:49,040 OK. 1497 01:18:49,040 --> 01:18:52,940 That's the three stages that we summarize. 1498 01:18:52,940 --> 01:18:57,680 First stage is the advanced SPC. The second stage the DOE. 1499 01:18:57,680 --> 01:19:02,010 And the last stage is response of this model. 1500 01:19:02,010 --> 01:19:04,220 DUANE BONING: When you say stage, 1501 01:19:04,220 --> 01:19:06,530 does it have to be in that order? 1502 01:19:10,110 --> 01:19:11,850 AUDIENCE: I think yeah. 1503 01:19:11,850 --> 01:19:16,920 First because you try to plot the control 1504 01:19:16,920 --> 01:19:18,660 chart for the service level. 1505 01:19:18,660 --> 01:19:21,090 Then you find the problem. 1506 01:19:21,090 --> 01:19:23,640 Then you start to focus on the effects. 1507 01:19:23,640 --> 01:19:25,470 That's stage two. 1508 01:19:25,470 --> 01:19:27,610 And after that you do the optimization. 1509 01:19:30,550 --> 01:19:34,300 That's what we think should be order. 1510 01:19:34,300 --> 01:19:36,880 DUANE BONING: Because one thought maybe to think about 1511 01:19:36,880 --> 01:19:41,680 is whether switching that order might make sense. 1512 01:19:41,680 --> 01:19:44,710 I mean, in one scenario, I think you're right. 1513 01:19:44,710 --> 01:19:47,440 And in one actual sort of real life scenario, 1514 01:19:47,440 --> 01:19:51,010 you may have SPC charts already up. 1515 01:19:51,010 --> 01:19:52,990 And then you go out of control. 1516 01:19:52,990 --> 01:19:56,710 You see you're out of spec somewhere. 1517 01:19:56,710 --> 01:19:59,080 And then if I understand your approach here, 1518 01:19:59,080 --> 01:20:02,440 you're saying to figure out what the problem was, 1519 01:20:02,440 --> 01:20:05,365 you want to do a DOE and solve the problem. 1520 01:20:09,070 --> 01:20:10,870 That might be an interesting scenario, 1521 01:20:10,870 --> 01:20:18,280 but often the other way around is do a DOE up front 1522 01:20:18,280 --> 01:20:21,310 so you actually know what you should be monitoring. 1523 01:20:21,310 --> 01:20:25,510 You actually understand larger the basic relationships, 1524 01:20:25,510 --> 01:20:30,820 what's significant, what you should be monitoring, 1525 01:20:30,820 --> 01:20:34,330 where there's a likely actual effect. 1526 01:20:34,330 --> 01:20:43,330 And then help use the DOE to set up the SPC. 1527 01:20:43,330 --> 01:20:45,970 Or maybe the sequence-- maybe what I would do, 1528 01:20:45,970 --> 01:20:48,010 maybe it goes the other way around. 1529 01:20:48,010 --> 01:20:56,290 Stage four, revise SPC. Maybe that 1530 01:20:56,290 --> 01:20:58,180 would be a good compromise. 1531 01:20:58,180 --> 01:21:00,260 Because I think once you've built a DOE, 1532 01:21:00,260 --> 01:21:05,170 now you've got a lot more information about what the-- 1533 01:21:05,170 --> 01:21:09,130 you've got a more solid model, a more functional model. 1534 01:21:09,130 --> 01:21:15,175 And you can use that to perhaps set better limits on SPC 1535 01:21:15,175 --> 01:21:18,290 and decide what parameters are most important. 1536 01:21:18,290 --> 01:21:18,940 So I like that. 1537 01:21:18,940 --> 01:21:21,910 AUDIENCE: That's right, we can revise the control chart 1538 01:21:21,910 --> 01:21:22,460 parameters. 1539 01:21:22,460 --> 01:21:23,830 Yeah, that's right. 1540 01:21:23,830 --> 01:21:27,070 And I think we'll use CellSim. 1541 01:21:27,070 --> 01:21:29,980 It's an Excel based simulation software 1542 01:21:29,980 --> 01:21:32,140 to build a model, supply chain model, 1543 01:21:32,140 --> 01:21:35,510 and used Minitab to follow data analysis. 1544 01:21:35,510 --> 01:21:36,220 OK? 1545 01:21:36,220 --> 01:21:37,730 DUANE BONING: OK. 1546 01:21:37,730 --> 01:21:38,230 Great. 1547 01:21:38,230 --> 01:21:39,640 So this is all going to be-- 1548 01:21:42,150 --> 01:21:44,410 do you actually have some company data 1549 01:21:44,410 --> 01:21:48,130 or are you going to pretty much mostly use the CellSim model? 1550 01:21:53,675 --> 01:21:54,550 AUDIENCE: Next slide. 1551 01:21:54,550 --> 01:21:56,110 You can change to next slide. 1552 01:21:56,110 --> 01:21:58,090 DUANE BONING: Ah, good, yeah. 1553 01:21:58,090 --> 01:21:59,200 AUDIENCE: Yeah. 1554 01:21:59,200 --> 01:22:02,500 So for the service level, I think 1555 01:22:02,500 --> 01:22:05,050 we will get from the simulation. 1556 01:22:05,050 --> 01:22:07,600 But the input from the company. 1557 01:22:07,600 --> 01:22:11,212 Because we can't get the service level data from the company. 1558 01:22:11,212 --> 01:22:11,920 DUANE BONING: OK. 1559 01:22:15,010 --> 01:22:17,080 So if I understand what you're saying 1560 01:22:17,080 --> 01:22:20,890 is you might actually try to model kind of a real line 1561 01:22:20,890 --> 01:22:24,160 with some of the parameters that tell you how that line is 1562 01:22:24,160 --> 01:22:29,290 structured and then use that to do simulations and generate 1563 01:22:29,290 --> 01:22:30,932 service level? 1564 01:22:30,932 --> 01:22:32,140 AUDIENCE: Yeah, that's right. 1565 01:22:32,140 --> 01:22:32,740 DUANE BONING: I see. 1566 01:22:32,740 --> 01:22:34,720 AUDIENCE: So the input from company, yeah. 1567 01:22:34,720 --> 01:22:35,428 DUANE BONING: OK. 1568 01:22:35,428 --> 01:22:38,290 But you have no output data from the company? 1569 01:22:38,290 --> 01:22:40,532 It's all going to be simulation? 1570 01:22:40,532 --> 01:22:41,740 AUDIENCE: Yeah, that's right. 1571 01:22:41,740 --> 01:22:43,990 DUANE BONING: OK, OK. 1572 01:22:43,990 --> 01:22:45,520 That certainly gives the advantage 1573 01:22:45,520 --> 01:22:50,260 I talked about today of synthetic data. 1574 01:22:50,260 --> 01:22:54,730 I mean, it's nice that you've got a realistic scenario based 1575 01:22:54,730 --> 01:23:00,250 on a company scenario for setting up your simulation. 1576 01:23:00,250 --> 01:23:04,480 But you have lots of latitude in your synthetic data 1577 01:23:04,480 --> 01:23:06,910 to actually-- 1578 01:23:06,910 --> 01:23:12,860 I would encourage you to consider exploring 1579 01:23:12,860 --> 01:23:15,860 intentional perturbations. 1580 01:23:15,860 --> 01:23:20,120 You can introduce different faults into your line 1581 01:23:20,120 --> 01:23:25,250 and see what happens both in the SPC and in the model. 1582 01:23:25,250 --> 01:23:27,320 You've got a lot of flexibility because you're 1583 01:23:27,320 --> 01:23:28,800 doing the simulation. 1584 01:23:28,800 --> 01:23:31,850 And you can play with and say what 1585 01:23:31,850 --> 01:23:35,360 happens if this piece of equipment goes down 1586 01:23:35,360 --> 01:23:38,660 or the mean time to failure triples. 1587 01:23:38,660 --> 01:23:41,810 And I didn't know that. 1588 01:23:41,810 --> 01:23:43,190 How does things improve? 1589 01:23:43,190 --> 01:23:47,750 Or mean time to failure becomes 1/10 what it was. 1590 01:23:47,750 --> 01:23:49,950 I'm failing much more often. 1591 01:23:49,950 --> 01:23:52,280 So I think you can construct very nicely 1592 01:23:52,280 --> 01:23:58,280 a lot of different scenarios based on your knowledge 1593 01:23:58,280 --> 01:24:03,350 from sort of factory modeling that might be relevant ones 1594 01:24:03,350 --> 01:24:06,990 to study with a DOE approach. 1595 01:24:06,990 --> 01:24:08,720 So I think that's rich. 1596 01:24:08,720 --> 01:24:10,500 That gives you a lot of opportunity. 1597 01:24:10,500 --> 01:24:11,520 So that's good. 1598 01:24:14,178 --> 01:24:14,720 AUDIENCE: OK. 1599 01:24:14,720 --> 01:24:16,538 I think that's the end of the slides. 1600 01:24:16,538 --> 01:24:17,330 DUANE BONING: Yeah. 1601 01:24:17,330 --> 01:24:18,770 Hayden, do you have-- 1602 01:24:18,770 --> 01:24:20,740 you want to-- here, why don't you? 1603 01:24:23,235 --> 01:24:24,860 GUEST SPEAKER: We're just thinking more 1604 01:24:24,860 --> 01:24:27,110 about additional things you might explore. 1605 01:24:27,110 --> 01:24:30,320 So let me just make sure I understand 1606 01:24:30,320 --> 01:24:34,200 how the Excel model works. 1607 01:24:34,200 --> 01:24:36,800 I mean, it sounds like you actually 1608 01:24:36,800 --> 01:24:41,270 have a function that determines y as a function of all 1609 01:24:41,270 --> 01:24:42,470 these input variables. 1610 01:24:42,470 --> 01:24:44,180 And you know what that function is. 1611 01:24:44,180 --> 01:24:46,975 DUANE BONING: Well, it's very complicated. 1612 01:24:46,975 --> 01:24:48,600 GUEST SPEAKER: But you know what it is. 1613 01:24:48,600 --> 01:24:53,540 So if you're doing DOE, you're trying to build a model, 1614 01:24:53,540 --> 01:24:55,650 but you already have the model. 1615 01:24:55,650 --> 01:24:59,990 So I'm not sure the timeline. 1616 01:24:59,990 --> 01:25:03,290 Basically, you're going to be trying to get to the model-- 1617 01:25:03,290 --> 01:25:06,830 AUDIENCE: Actually you don't have to the exact model. 1618 01:25:06,830 --> 01:25:10,670 The service level relates to all of the parameters. 1619 01:25:10,670 --> 01:25:14,960 But you cannot derive a model. 1620 01:25:14,960 --> 01:25:18,590 GUEST SPEAKER: You can't derive a-- so how is this exact model? 1621 01:25:18,590 --> 01:25:21,050 Is it a lookup table or something? 1622 01:25:21,050 --> 01:25:24,230 AUDIENCE: We will simulate the supply chain 1623 01:25:24,230 --> 01:25:29,540 and we run the simulation to get a production lead 1624 01:25:29,540 --> 01:25:31,940 time, a production cycle time. 1625 01:25:31,940 --> 01:25:37,730 That is influenced by the buffer size, by the MTTF, MTTR. 1626 01:25:37,730 --> 01:25:41,660 And we will check for the production lead time 1627 01:25:41,660 --> 01:25:43,580 plus transportation lead time. 1628 01:25:43,580 --> 01:25:49,730 If this is less than the lead time 1629 01:25:49,730 --> 01:25:52,370 that you promised your customer, then that 1630 01:25:52,370 --> 01:25:55,280 means you satisfy the customer. 1631 01:25:55,280 --> 01:25:59,350 And if the production lead time plus the transportation 1632 01:25:59,350 --> 01:26:01,430 lead time larger than LC, then it 1633 01:26:01,430 --> 01:26:05,690 will account for one [INAUDIBLE].. 1634 01:26:05,690 --> 01:26:07,790 That's what we generate the service level. 1635 01:26:07,790 --> 01:26:09,200 GUEST SPEAKER: OK, I see. 1636 01:26:09,200 --> 01:26:11,913 So you want some approximate way of predicting 1637 01:26:11,913 --> 01:26:12,830 what it's going to be. 1638 01:26:12,830 --> 01:26:13,760 OK, fair enough. 1639 01:26:13,760 --> 01:26:15,260 DUANE BONING: I think Hayden's point 1640 01:26:15,260 --> 01:26:18,710 is if you have, in general, very often you 1641 01:26:18,710 --> 01:26:21,530 have very complicated models that 1642 01:26:21,530 --> 01:26:24,540 may take a long time to actually run any one point. 1643 01:26:24,540 --> 01:26:26,450 And so I think that's lurking in your mind. 1644 01:26:26,450 --> 01:26:28,180 What you would like to do is build 1645 01:26:28,180 --> 01:26:30,950 simplified approximate derived models 1646 01:26:30,950 --> 01:26:35,240 off of a complicated long simulation time 1647 01:26:35,240 --> 01:26:38,390 sophisticated CellSim model. 1648 01:26:38,390 --> 01:26:38,980 Is that right? 1649 01:26:44,000 --> 01:26:44,750 GUEST SPEAKER: OK. 1650 01:26:44,750 --> 01:26:45,080 Great. 1651 01:26:45,080 --> 01:26:47,330 DUANE BONING: I mean, even though CellSim may actually 1652 01:26:47,330 --> 01:26:48,200 be fairly fast. 1653 01:26:48,200 --> 01:26:51,770 In what I've seen in many of these factory simulations, 1654 01:26:51,770 --> 01:26:56,350 running the factory simulation itself may take hours or days. 1655 01:26:56,350 --> 01:26:58,968 And that's where you might actually be building 1656 01:26:58,968 --> 01:27:00,260 a simplified approximate model. 1657 01:27:03,910 --> 01:27:07,760 AUDIENCE: I have one question here. 1658 01:27:07,760 --> 01:27:09,830 I'm not very sure about the demand data. 1659 01:27:09,830 --> 01:27:12,710 But seems like the CellSim can only 1660 01:27:12,710 --> 01:27:16,620 have a normal distribution, an exponential distribution input. 1661 01:27:16,620 --> 01:27:19,220 So if the normal distribution is not 1662 01:27:19,220 --> 01:27:22,700 very appropriate for the demand, what can we do? 1663 01:27:28,103 --> 01:27:29,270 DUANE BONING: Good question. 1664 01:27:29,270 --> 01:27:34,520 I think if it can at least do two different distributions, 1665 01:27:34,520 --> 01:27:37,310 one interesting thing would be to see how sensitive 1666 01:27:37,310 --> 01:27:40,880 the simulation is to whether it's-- you said a normal 1667 01:27:40,880 --> 01:27:42,590 distribution and the other was a Poisson? 1668 01:27:42,590 --> 01:27:44,270 Or what was the other one? 1669 01:27:44,270 --> 01:27:45,845 AUDIENCE: It's a exponential. 1670 01:27:45,845 --> 01:27:48,380 DUANE BONING: Exponential. 1671 01:27:48,380 --> 01:27:51,560 I think usually an exponential is 1672 01:27:51,560 --> 01:27:55,400 what's often assumed for these kinds of arrival processes. 1673 01:27:55,400 --> 01:27:59,870 But it would be interesting, actually, to see how much-- 1674 01:27:59,870 --> 01:28:02,060 that might end up being a nice question for you 1675 01:28:02,060 --> 01:28:07,070 to ask you and try with your data and your simulation. 1676 01:28:07,070 --> 01:28:11,930 How do things differ as a function of not just 1677 01:28:11,930 --> 01:28:13,730 the parameters of the distribution 1678 01:28:13,730 --> 01:28:17,540 but the type of distribution? 1679 01:28:17,540 --> 01:28:19,320 It might not matter at all. 1680 01:28:19,320 --> 01:28:21,500 That might be crucial. 1681 01:28:21,500 --> 01:28:23,690 That might be the most important thing. 1682 01:28:23,690 --> 01:28:24,260 I don't know. 1683 01:28:29,240 --> 01:28:29,820 Good. 1684 01:28:29,820 --> 01:28:33,300 Any other questions on this one? 1685 01:28:33,300 --> 01:28:34,440 AUDIENCE: No, we're fine. 1686 01:28:34,440 --> 01:28:35,273 DUANE BONING: Great. 1687 01:28:35,273 --> 01:28:35,790 Go for it. 1688 01:28:35,790 --> 01:28:38,010 It sounds interesting. 1689 01:28:38,010 --> 01:28:41,220 I like how this integrates your other classes 1690 01:28:41,220 --> 01:28:44,070 and your other learning too. 1691 01:28:44,070 --> 01:28:47,010 AUDIENCE: That's the inspiration from another class. 1692 01:28:47,010 --> 01:28:48,500 DUANE BONING: Exactly. 1693 01:28:48,500 --> 01:28:49,000 OK. 1694 01:28:57,060 --> 01:28:57,560 All right. 1695 01:28:57,560 --> 01:28:58,400 So let's see. 1696 01:28:58,400 --> 01:29:01,250 I had talked with-- 1697 01:29:01,250 --> 01:29:06,560 is there-- who wants to go next? 1698 01:29:06,560 --> 01:29:09,770 Other people have not sent slides, but let's see. 1699 01:29:09,770 --> 01:29:11,570 Who else have-- 1700 01:29:11,570 --> 01:29:13,490 I sent email to I think everybody 1701 01:29:13,490 --> 01:29:16,460 kind of late last night or early morning for you guys. 1702 01:29:21,850 --> 01:29:23,055 They're all mixing together. 1703 01:29:23,055 --> 01:29:25,180 Who wants to talk about one of their projects next? 1704 01:29:25,180 --> 01:29:27,790 Because we still have about 20 minutes on the video time. 1705 01:29:33,860 --> 01:29:36,200 Just talk me through your written thing and some 1706 01:29:36,200 --> 01:29:37,280 of the comments I sent. 1707 01:29:46,100 --> 01:29:49,280 How about the-- let's see. 1708 01:29:49,280 --> 01:29:56,750 I'm looking at the one for the iron sole plate. 1709 01:29:56,750 --> 01:30:00,110 Who's kind of a lead person on that? 1710 01:30:00,110 --> 01:30:02,510 By the way, I liked this one, because I 1711 01:30:02,510 --> 01:30:05,720 got to go and tour the Phillips factory when we were there 1712 01:30:05,720 --> 01:30:07,440 and see some of these things being made. 1713 01:30:07,440 --> 01:30:09,030 So I'm like, oh, this is cool. 1714 01:30:09,030 --> 01:30:09,650 This is great. 1715 01:30:12,716 --> 01:30:16,405 AUDIENCE: Well, so actually one we have sent to you. 1716 01:30:16,405 --> 01:30:17,780 The other part we have in mind is 1717 01:30:17,780 --> 01:30:21,350 that the company is going to run some intermediate points. 1718 01:30:21,350 --> 01:30:25,730 But it turns out to be otherwise. 1719 01:30:25,730 --> 01:30:28,340 So what we have is that it's a five parameter. 1720 01:30:28,340 --> 01:30:31,040 So they are trying to build a prototype. 1721 01:30:31,040 --> 01:30:33,700 And they're trying to find what is the output. 1722 01:30:33,700 --> 01:30:38,270 So that's the last one that actually blow out [INAUDIBLE].. 1723 01:30:38,270 --> 01:30:41,810 And then yeah, so what we have is actually 1724 01:30:41,810 --> 01:30:43,450 they actually run a full DOE. 1725 01:30:47,140 --> 01:30:48,980 So that's what we have. 1726 01:30:48,980 --> 01:30:52,550 DUANE BONING: Is that data already available? 1727 01:30:52,550 --> 01:30:53,970 AUDIENCE: Yes it is. 1728 01:30:53,970 --> 01:30:56,825 So then what we have is a full DOE. 1729 01:30:56,825 --> 01:30:58,910 So that's to the power of 5. 1730 01:30:58,910 --> 01:31:02,420 So that's 32 runs. 1731 01:31:02,420 --> 01:31:06,260 And then each run, there is three radicates. 1732 01:31:06,260 --> 01:31:07,390 DUANE BONING: OK, good. 1733 01:31:07,390 --> 01:31:07,890 Yes. 1734 01:31:13,130 --> 01:31:15,890 Are all of those parameters basically continuous 1735 01:31:15,890 --> 01:31:16,610 parameters? 1736 01:31:16,610 --> 01:31:21,650 I on the diagram you mentioned different diameters 1737 01:31:21,650 --> 01:31:22,940 and heights. 1738 01:31:22,940 --> 01:31:26,150 Are all of those continuous parameters? 1739 01:31:26,150 --> 01:31:30,003 Are any of them sort of discrete choices? 1740 01:31:30,003 --> 01:31:31,670 I know they only did two levels of each. 1741 01:31:31,670 --> 01:31:33,320 So in some sense, they are discrete. 1742 01:31:33,320 --> 01:31:36,740 But if you were building a response surface model, 1743 01:31:36,740 --> 01:31:41,600 would you be able to actually have all of them 1744 01:31:41,600 --> 01:31:42,980 be continuous parameters? 1745 01:31:42,980 --> 01:31:47,870 Or would there be a mix of some arrangement 1746 01:31:47,870 --> 01:31:51,755 of holes and some diameter that was continuously varying? 1747 01:31:58,790 --> 01:32:01,550 AUDIENCE: I think it's continuous parameters. 1748 01:32:01,550 --> 01:32:06,020 And then I think maybe some of the inputs 1749 01:32:06,020 --> 01:32:08,240 are not so measurable. 1750 01:32:08,240 --> 01:32:11,100 So maybe they try to-- 1751 01:32:11,100 --> 01:32:16,550 since some of the inputs look at the noise input. 1752 01:32:16,550 --> 01:32:20,270 And then they may use the [INAUDIBLE] 1753 01:32:20,270 --> 01:32:23,780 to do the robust design. 1754 01:32:23,780 --> 01:32:29,180 And maybe it is a potential area, 1755 01:32:29,180 --> 01:32:32,390 because right now there is still some problem with the data 1756 01:32:32,390 --> 01:32:33,230 connection. 1757 01:32:33,230 --> 01:32:37,800 And then we are trying to figure the problem, all those days. 1758 01:32:37,800 --> 01:32:45,440 So actually the full area has not been exactly designed yet. 1759 01:32:45,440 --> 01:32:47,750 So that is one of our problem. 1760 01:32:47,750 --> 01:32:48,470 Yeah. 1761 01:32:48,470 --> 01:32:49,178 DUANE BONING: OK. 1762 01:32:49,178 --> 01:32:52,400 But you think you'll have data? 1763 01:32:52,400 --> 01:32:54,860 AUDIENCE: Yeah, we have several data. 1764 01:32:54,860 --> 01:33:02,540 But we do ANOVA, and then we find out that our value is not 1765 01:33:02,540 --> 01:33:03,800 exactly so high. 1766 01:33:03,800 --> 01:33:08,780 It's just a little larger than 0.5. 1767 01:33:08,780 --> 01:33:12,000 So I think maybe the data is not complete. 1768 01:33:12,000 --> 01:33:15,462 So maybe we need more data. 1769 01:33:15,462 --> 01:33:16,170 DUANE BONING: OK. 1770 01:33:16,170 --> 01:33:18,440 Well, if you get more data, that's great. 1771 01:33:18,440 --> 01:33:21,470 But it's not just the r squared, but I 1772 01:33:21,470 --> 01:33:26,600 assume your ANOVA does show some significance to a model? 1773 01:33:26,600 --> 01:33:27,500 OK. 1774 01:33:27,500 --> 01:33:28,190 Good, good. 1775 01:33:28,190 --> 01:33:32,120 Then I think you've got stuff to work with. 1776 01:33:32,120 --> 01:33:33,327 Now, in the email-- 1777 01:33:33,327 --> 01:33:34,160 AUDIENCE: Professor? 1778 01:33:34,160 --> 01:33:35,930 DUANE BONING: Yes? 1779 01:33:35,930 --> 01:33:38,180 AUDIENCE: I actually think that it may not be very 1780 01:33:38,180 --> 01:33:41,720 significant from the model. 1781 01:33:41,720 --> 01:33:45,260 Yeah, I do think that it's not that significant. 1782 01:33:45,260 --> 01:33:48,080 And I actually talked to the company 1783 01:33:48,080 --> 01:33:51,050 to see why they are not trying to find more points, 1784 01:33:51,050 --> 01:33:54,000 since they are not really able to find a good fit. 1785 01:33:54,000 --> 01:33:56,960 But the response that I got from them 1786 01:33:56,960 --> 01:34:00,680 is that to run the experiment, they actually 1787 01:34:00,680 --> 01:34:03,950 need to build a prototype, which is very expensive. 1788 01:34:03,950 --> 01:34:07,890 And as a result, they are not planning to run that. 1789 01:34:07,890 --> 01:34:12,080 Actually, I thought that is a good idea maybe 1790 01:34:12,080 --> 01:34:13,130 to run a midpoint. 1791 01:34:16,480 --> 01:34:20,180 I was not able to convince the company for doing that. 1792 01:34:24,230 --> 01:34:26,360 Yeah, so that's a problem they're facing. 1793 01:34:26,360 --> 01:34:33,450 That's a lack of data for this study. 1794 01:34:33,450 --> 01:34:37,820 So at the same time, we have also [INAUDIBLE] 1795 01:34:37,820 --> 01:34:41,370 exploring some other alternatives. 1796 01:34:41,370 --> 01:34:44,630 So the [INAUDIBLE] we got is actually-- 1797 01:34:44,630 --> 01:34:49,430 so in the parent case, it's a product design optimization. 1798 01:34:49,430 --> 01:34:51,500 So then the next one that we're looking into 1799 01:34:51,500 --> 01:34:54,210 is a process optimization. 1800 01:34:54,210 --> 01:34:59,990 So in the process optimization, basically it's 1801 01:34:59,990 --> 01:35:02,570 a process line where there is actually 1802 01:35:02,570 --> 01:35:08,300 different sole plates that are actually 1803 01:35:08,300 --> 01:35:11,120 they're produced in a different period of time. 1804 01:35:11,120 --> 01:35:14,280 And then they also manage it differently. 1805 01:35:14,280 --> 01:35:16,850 So that's also like the [INAUDIBLE] influence. 1806 01:35:16,850 --> 01:35:19,130 Like for example, we are also trying 1807 01:35:19,130 --> 01:35:20,870 to see, for example, whether we can 1808 01:35:20,870 --> 01:35:29,870 see there's any correlation between the defects. 1809 01:35:29,870 --> 01:35:32,640 The defects are [INAUDIBLE] or things like this. 1810 01:35:32,640 --> 01:35:34,070 So we're trying to see whether we 1811 01:35:34,070 --> 01:35:39,290 can find correlation between different layers or even 1812 01:35:39,290 --> 01:35:41,810 spatially. 1813 01:35:41,810 --> 01:35:44,150 Because it's arranged in terms of rows and columns, 1814 01:35:44,150 --> 01:35:49,110 I think that's five rows and seven columns. 1815 01:35:49,110 --> 01:35:52,730 So in each carrier, that's 35 units. 1816 01:35:55,430 --> 01:36:00,050 So then we thought that probably we can do a nested variance 1817 01:36:00,050 --> 01:36:02,570 analysis on that. 1818 01:36:02,570 --> 01:36:04,820 So one aspect I'm looking into is 1819 01:36:04,820 --> 01:36:07,700 because we also have the data for the different stations. 1820 01:36:07,700 --> 01:36:09,650 So one thing that we can look into 1821 01:36:09,650 --> 01:36:12,860 is which station is the main contributor. 1822 01:36:12,860 --> 01:36:14,000 So that's one. 1823 01:36:14,000 --> 01:36:17,930 So secondly is between the shift of the timing. 1824 01:36:17,930 --> 01:36:23,720 So for example, every hour, so an hour and maybe 1825 01:36:23,720 --> 01:36:25,790 the different shift, maybe one shift an hour, 1826 01:36:25,790 --> 01:36:26,750 things like this. 1827 01:36:26,750 --> 01:36:29,630 So then we can also do a nested variance 1828 01:36:29,630 --> 01:36:31,377 based on the shift timing. 1829 01:36:31,377 --> 01:36:32,210 DUANE BONING: Right. 1830 01:36:32,210 --> 01:36:33,710 This sounds very interesting. 1831 01:36:33,710 --> 01:36:37,670 I don't think we have anybody else doing a nested variance 1832 01:36:37,670 --> 01:36:39,210 kind of analysis. 1833 01:36:39,210 --> 01:36:41,030 We have several DOEs. 1834 01:36:41,030 --> 01:36:44,540 So in the overall class, I think this 1835 01:36:44,540 --> 01:36:48,530 would be very interesting to have this different project 1836 01:36:48,530 --> 01:36:50,120 you're talking about. 1837 01:36:50,120 --> 01:36:53,168 Now, for that one you've already got the data. 1838 01:36:53,168 --> 01:36:54,210 It sounds like it's good. 1839 01:36:54,210 --> 01:36:57,030 They have lots of manufacturing data in that case. 1840 01:36:57,030 --> 01:36:59,075 Is that correct? 1841 01:36:59,075 --> 01:36:59,700 AUDIENCE: Yeah. 1842 01:36:59,700 --> 01:37:01,080 There are [INAUDIBLE] data. 1843 01:37:01,080 --> 01:37:03,750 But then the person who ran the data, 1844 01:37:03,750 --> 01:37:05,700 they actually are not so careful to take 1845 01:37:05,700 --> 01:37:08,370 note of every parameter. 1846 01:37:08,370 --> 01:37:11,130 So then initially I actually talked 1847 01:37:11,130 --> 01:37:14,160 to the company, the person, I asked 1848 01:37:14,160 --> 01:37:16,050 whether we can get all the data and then we 1849 01:37:16,050 --> 01:37:17,790 can combine all the data together 1850 01:37:17,790 --> 01:37:20,560 and then we'll be able to get a very rich set of data. 1851 01:37:20,560 --> 01:37:22,050 We can do all the analysis. 1852 01:37:22,050 --> 01:37:25,690 But it seems that the data that we get is missing in pieces. 1853 01:37:25,690 --> 01:37:29,010 For example, they are doing hypothesis testing kind 1854 01:37:29,010 --> 01:37:29,850 of thing. 1855 01:37:29,850 --> 01:37:31,963 So basically, they are doing a p-test. 1856 01:37:31,963 --> 01:37:34,380 So it's basically a hypothesis testing where they actually 1857 01:37:34,380 --> 01:37:37,230 check each parameter to see whether it's 1858 01:37:37,230 --> 01:37:38,610 significant or not significant. 1859 01:37:38,610 --> 01:37:40,860 And so while they are doing that, 1860 01:37:40,860 --> 01:37:42,960 they do not consider any other parameters. 1861 01:37:42,960 --> 01:37:49,530 So although the process is controlled by many inputs, 1862 01:37:49,530 --> 01:37:51,870 but at any single, one time, they only 1863 01:37:51,870 --> 01:37:55,420 care about a particular set of a parameter. 1864 01:37:55,420 --> 01:37:58,380 So as a result, we are not able to combine all the data 1865 01:37:58,380 --> 01:38:02,670 together as one big cell of data and do as much analysis 1866 01:38:02,670 --> 01:38:06,010 as we would like to have. 1867 01:38:06,010 --> 01:38:11,220 But we thought that that would be an interesting project 1868 01:38:11,220 --> 01:38:11,820 to go about. 1869 01:38:11,820 --> 01:38:14,280 And at the same time, I also think 1870 01:38:14,280 --> 01:38:17,820 that we are trying to continue to explore a product thing 1871 01:38:17,820 --> 01:38:18,900 and be a process thing. 1872 01:38:18,900 --> 01:38:22,980 So as to get a comparison-- 1873 01:38:22,980 --> 01:38:26,040 because we thought that this real life situation, we also 1874 01:38:26,040 --> 01:38:29,190 like to see, is the point of view for a product DOE 1875 01:38:29,190 --> 01:38:30,660 and process DOE-- 1876 01:38:30,660 --> 01:38:32,550 how is it different. 1877 01:38:32,550 --> 01:38:36,750 Also we understand that there are some limitations 1878 01:38:36,750 --> 01:38:39,060 for the product DOE, because it pretty 1879 01:38:39,060 --> 01:38:41,910 difficult for the company to run, 1880 01:38:41,910 --> 01:38:44,280 to make a new prototype each time 1881 01:38:44,280 --> 01:38:46,320 compared to a process DOE where they can 1882 01:38:46,320 --> 01:38:48,960 change a setting rather easily. 1883 01:38:48,960 --> 01:38:51,690 Yeah, so there is a plan. 1884 01:38:51,690 --> 01:38:54,030 DUANE BONING: I think this sounds like a very good plan. 1885 01:38:54,030 --> 01:38:57,630 Because I wanted to come back to the product DOE, 1886 01:38:57,630 --> 01:38:59,010 because I think-- 1887 01:38:59,010 --> 01:39:02,040 so first off, I think with your team, 1888 01:39:02,040 --> 01:39:05,200 you've got a good team of four people. 1889 01:39:05,200 --> 01:39:09,900 So being able to look at both of these problems is great. 1890 01:39:09,900 --> 01:39:17,220 And that includes a case study and examples of real life 1891 01:39:17,220 --> 01:39:21,720 limitations that are there in both situations 1892 01:39:21,720 --> 01:39:25,590 and recommendations you could make in both cases 1893 01:39:25,590 --> 01:39:31,590 to the company that might help them get more value out 1894 01:39:31,590 --> 01:39:34,560 of the activities that they've got to be able to make better 1895 01:39:34,560 --> 01:39:36,250 decisions in the future. 1896 01:39:36,250 --> 01:39:40,320 So going back to the product one with the DOE, the two 1897 01:39:40,320 --> 01:39:43,020 to the fifth DOE, I think it would 1898 01:39:43,020 --> 01:39:47,130 be very interesting to somehow include, maybe 1899 01:39:47,130 --> 01:39:51,480 you just make it up, but include what more you could 1900 01:39:51,480 --> 01:39:55,680 do if you had a center point. 1901 01:39:55,680 --> 01:39:58,800 And show if the center point looks like this, 1902 01:39:58,800 --> 01:40:01,080 look how different your decision would 1903 01:40:01,080 --> 01:40:04,140 be in design optimization. 1904 01:40:04,140 --> 01:40:06,340 Versus if your center point looks like this, 1905 01:40:06,340 --> 01:40:08,110 then the model looks completely different, 1906 01:40:08,110 --> 01:40:09,790 and this is what you should do. 1907 01:40:09,790 --> 01:40:13,290 In other words, show how important that center point 1908 01:40:13,290 --> 01:40:16,050 might be to their thinking. 1909 01:40:16,050 --> 01:40:19,290 Could be a nice part of this that actually might help 1910 01:40:19,290 --> 01:40:23,430 make the case persuasively to the company 1911 01:40:23,430 --> 01:40:28,650 that in their future it might be very valuable 1912 01:40:28,650 --> 01:40:31,320 and worth the expense of building just one more 1913 01:40:31,320 --> 01:40:32,730 prototype. 1914 01:40:32,730 --> 01:40:36,360 If they're already building 32 different prototypes 1915 01:40:36,360 --> 01:40:39,690 with some missing data perhaps, one more 1916 01:40:39,690 --> 01:40:43,830 would tell them so much more potentially. 1917 01:40:43,830 --> 01:40:46,380 So I think that could be very interesting. 1918 01:40:46,380 --> 01:40:51,450 And I'm glad, I like the plan of trying to do some of both. 1919 01:40:51,450 --> 01:40:53,575 Do I have that right? 1920 01:40:53,575 --> 01:40:54,200 AUDIENCE: Yeah. 1921 01:40:54,200 --> 01:40:57,570 So I actually think that when they made the prototype, maybe 1922 01:40:57,570 --> 01:41:01,020 they choose the high and low because it's easier for them 1923 01:41:01,020 --> 01:41:05,110 to make the prototype rather than making a center point. 1924 01:41:05,110 --> 01:41:07,980 So I do understand that a center point 1925 01:41:07,980 --> 01:41:10,200 has the advantage of being able to fit 1926 01:41:10,200 --> 01:41:11,400 models and things like that. 1927 01:41:11,400 --> 01:41:13,050 In the technical aspect, maybe they 1928 01:41:13,050 --> 01:41:15,370 are limited by machines or things like these. 1929 01:41:15,370 --> 01:41:18,120 Maybe they have a template where using a template they 1930 01:41:18,120 --> 01:41:20,010 are able to get a plus or a minus. 1931 01:41:20,010 --> 01:41:22,110 But they are not able to get the midpoint. 1932 01:41:22,110 --> 01:41:23,595 So I'm not sure-- 1933 01:41:23,595 --> 01:41:27,210 perhaps, is it possible that we discuss points like this? 1934 01:41:27,210 --> 01:41:27,960 DUANE BONING: Yes. 1935 01:41:27,960 --> 01:41:28,830 Absolutely. 1936 01:41:28,830 --> 01:41:31,350 Those are very realistic limitations. 1937 01:41:31,350 --> 01:41:34,110 You might find that a couple of parameters 1938 01:41:34,110 --> 01:41:35,250 can have center points. 1939 01:41:35,250 --> 01:41:36,780 Other ones can't. 1940 01:41:36,780 --> 01:41:38,950 And so it's a mixed center point. 1941 01:41:38,950 --> 01:41:41,340 And so that's a good issue. 1942 01:41:41,340 --> 01:41:42,060 Absolutely. 1943 01:41:42,060 --> 01:41:44,685 All of these kind of realistic constraints that drive-- 1944 01:41:48,900 --> 01:41:51,810 that help limit or drive them to the decisions 1945 01:41:51,810 --> 01:41:54,180 that they made but also exploring 1946 01:41:54,180 --> 01:41:56,530 is there some room to do some more? 1947 01:41:56,530 --> 01:42:00,260 So yes, that's a very good point. 1948 01:42:00,260 --> 01:42:01,260 AUDIENCE: OK, thank you. 1949 01:42:01,260 --> 01:42:02,052 DUANE BONING: Good. 1950 01:42:02,052 --> 01:42:03,360 Well, this sounds exciting. 1951 01:42:03,360 --> 01:42:05,940 Get the data as quickly as you can 1952 01:42:05,940 --> 01:42:12,940 even with all of its ugliness and get going on that. 1953 01:42:12,940 --> 01:42:16,380 And then if issues come up, send us email 1954 01:42:16,380 --> 01:42:20,170 and we can see what we can do to help. 1955 01:42:20,170 --> 01:42:24,780 So I think I just have a short period of time now. 1956 01:42:24,780 --> 01:42:26,370 Let's see. 1957 01:42:26,370 --> 01:42:28,830 I saw you in the back while the camera 1958 01:42:28,830 --> 01:42:31,620 was focused there, Priyanka, I saw you in the back there. 1959 01:42:31,620 --> 01:42:34,620 So you guys project. 1960 01:42:34,620 --> 01:42:39,600 [INAUDIBLE] Stanley. 1961 01:42:39,600 --> 01:42:42,400 Thoughts on your project? 1962 01:42:42,400 --> 01:42:47,250 First off, I really like the ECM aspect, 1963 01:42:47,250 --> 01:42:48,960 because I've got a student working 1964 01:42:48,960 --> 01:42:52,020 on electropolishing or electrochemical 1965 01:42:52,020 --> 01:42:55,150 mechanical polishing for semiconductor stuff. 1966 01:42:55,150 --> 01:42:57,510 So the student has been learning a little bit 1967 01:42:57,510 --> 01:42:58,680 about electrochemistry. 1968 01:42:58,680 --> 01:43:01,212 So I saw this and I thought it was really interesting. 1969 01:43:05,640 --> 01:43:07,890 AUDIENCE: The story is sad but true. 1970 01:43:07,890 --> 01:43:13,020 It actually happened in my undergrad university. 1971 01:43:13,020 --> 01:43:16,200 And I got rescued on a boat, and we went to rescue the machine, 1972 01:43:16,200 --> 01:43:17,970 and it was bolted on a concrete slab 1973 01:43:17,970 --> 01:43:20,340 and it was under 10 feet of water. 1974 01:43:20,340 --> 01:43:22,440 DUANE BONING: Do you have pictures? 1975 01:43:22,440 --> 01:43:23,315 AUDIENCE: Yeah, I do. 1976 01:43:23,315 --> 01:43:24,232 DUANE BONING: Oh good. 1977 01:43:24,232 --> 01:43:24,738 Excellent. 1978 01:43:27,550 --> 01:43:28,710 Now, in terms of-- 1979 01:43:28,710 --> 01:43:32,820 I think I sent a little bit of information back in the email, 1980 01:43:32,820 --> 01:43:36,300 because it looked like, if I'm remembering, 1981 01:43:36,300 --> 01:43:38,700 you wanted to do a couple of things 1982 01:43:38,700 --> 01:43:41,340 to see was the equipment restored 1983 01:43:41,340 --> 01:43:44,680 and operating back to its original state. 1984 01:43:44,680 --> 01:43:45,700 And I understood-- 1985 01:43:45,700 --> 01:43:46,742 AUDIENCE: That's correct. 1986 01:43:46,742 --> 01:43:48,720 DUANE BONING: Yeah, I understood that part. 1987 01:43:48,720 --> 01:43:50,250 And then the other interesting piece 1988 01:43:50,250 --> 01:43:53,370 was building an analytic model. 1989 01:43:53,370 --> 01:43:56,440 And using that, it sounded like-- 1990 01:43:56,440 --> 01:43:58,590 well, tell me what you would like 1991 01:43:58,590 --> 01:44:02,730 to do with the analytic model. 1992 01:44:02,730 --> 01:44:05,250 AUDIENCE: So initially, the experiment 1993 01:44:05,250 --> 01:44:09,030 was done to get data to substantiate 1994 01:44:09,030 --> 01:44:10,450 an analytical model. 1995 01:44:10,450 --> 01:44:14,730 So the model was based on the Faraday's laws of electrolysis. 1996 01:44:14,730 --> 01:44:17,500 But that related only a few of the parameters. 1997 01:44:17,500 --> 01:44:20,610 So we were basically trying to incorporate the flow rate 1998 01:44:20,610 --> 01:44:23,130 term into that equation. 1999 01:44:23,130 --> 01:44:25,590 But the focus kind of got shifted because the machine 2000 01:44:25,590 --> 01:44:27,000 stopped running midway. 2001 01:44:27,000 --> 01:44:29,880 And we spent a very long time trying 2002 01:44:29,880 --> 01:44:31,530 to get it back on its feet. 2003 01:44:31,530 --> 01:44:33,960 And the readings that we took after that, 2004 01:44:33,960 --> 01:44:36,720 so I can sort the readings on the basis of date. 2005 01:44:36,720 --> 01:44:39,150 We were four students working on that project, 2006 01:44:39,150 --> 01:44:41,100 and we took a large set of readings. 2007 01:44:41,100 --> 01:44:42,827 So I can sort it by date and find out 2008 01:44:42,827 --> 01:44:44,910 what's before the flood and what's after the flood 2009 01:44:44,910 --> 01:44:51,120 and to find out if really a shift had occurred after that. 2010 01:44:51,120 --> 01:44:52,920 When we were using it, we never treated it 2011 01:44:52,920 --> 01:44:54,760 as a separate set of data. 2012 01:44:54,760 --> 01:44:58,380 We combined and indiscriminately averaged out the whole thing. 2013 01:44:58,380 --> 01:45:00,870 So that's something that in hindsight we probably 2014 01:45:00,870 --> 01:45:02,292 should have looked at. 2015 01:45:02,292 --> 01:45:03,000 DUANE BONING: OK. 2016 01:45:03,000 --> 01:45:07,440 I think that's sort of the core of the idea for your group. 2017 01:45:07,440 --> 01:45:09,030 And I like that. 2018 01:45:09,030 --> 01:45:12,540 What I'm thinking about is if there's one or two 2019 01:45:12,540 --> 01:45:14,370 additional ideas that you might be 2020 01:45:14,370 --> 01:45:18,810 able to explore given the fact that you've also 2021 01:45:18,810 --> 01:45:21,040 got this analytic model. 2022 01:45:21,040 --> 01:45:25,500 So it's going back a little bit to this idea of synthetic data 2023 01:45:25,500 --> 01:45:27,330 or looking at your analytic model 2024 01:45:27,330 --> 01:45:32,790 and maybe even doing things like a sensitivity 2025 01:45:32,790 --> 01:45:35,760 analysis on the model. 2026 01:45:35,760 --> 01:45:38,040 Saying based on the model, these are 2027 01:45:38,040 --> 01:45:43,720 the parameters I think we would be most sensitive to in a DOE. 2028 01:45:43,720 --> 01:45:44,970 Some of those kinds of things. 2029 01:45:44,970 --> 01:45:50,790 I think it might be neat to just brainstorm or think 2030 01:45:50,790 --> 01:45:53,340 about things you might actually use the analytic model, 2031 01:45:53,340 --> 01:45:54,930 since you've got that also. 2032 01:45:54,930 --> 01:45:59,340 Even though you only know some constants from physics and some 2033 01:45:59,340 --> 01:46:01,600 you were trying to fit to your data. 2034 01:46:01,600 --> 01:46:05,800 And that whole piece sounds very interesting. 2035 01:46:05,800 --> 01:46:07,200 AUDIENCE: OK. 2036 01:46:07,200 --> 01:46:11,070 The data we have basically consists of varying three input 2037 01:46:11,070 --> 01:46:12,150 parameters. 2038 01:46:12,150 --> 01:46:13,860 Feed rate, flow rate, and voltage. 2039 01:46:13,860 --> 01:46:16,270 And the output is the MRR. 2040 01:46:16,270 --> 01:46:18,060 So the way we measured MRR was we 2041 01:46:18,060 --> 01:46:20,220 took the difference in the weight of the slab. 2042 01:46:20,220 --> 01:46:22,530 DUANE BONING: Yes, that sounds good. 2043 01:46:22,530 --> 01:46:27,480 AUDIENCE: And the time for the machining. 2044 01:46:27,480 --> 01:46:31,620 And so we have the MRR for about-- we have about 50 to 60 2045 01:46:31,620 --> 01:46:33,288 readings that we've taken. 2046 01:46:33,288 --> 01:46:34,080 DUANE BONING: Good. 2047 01:46:34,080 --> 01:46:36,960 Yeah, that was my other question in the email was how much data 2048 01:46:36,960 --> 01:46:37,650 you had. 2049 01:46:37,650 --> 01:46:40,890 Because it sounded like each run is fairly long. 2050 01:46:40,890 --> 01:46:42,780 Yeah, so you've got lots of students working 2051 01:46:42,780 --> 01:46:43,830 on this for a long time. 2052 01:46:43,830 --> 01:46:45,510 AUDIENCE: Yeah, we have four students working on it 2053 01:46:45,510 --> 01:46:46,540 almost around the clock. 2054 01:46:46,540 --> 01:46:50,130 So quite a lot of readings. 2055 01:46:50,130 --> 01:46:52,290 DUANE BONING: OK, that sounds very interesting. 2056 01:46:52,290 --> 01:46:56,620 And it will be an interesting story to hear as well. 2057 01:46:56,620 --> 01:46:59,500 AUDIENCE: Yeah, I'll try to put in some pictures. 2058 01:46:59,500 --> 01:47:01,450 DUANE BONING: So if other questions come up, 2059 01:47:01,450 --> 01:47:04,200 let me know, especially as you think a little bit about-- 2060 01:47:04,200 --> 01:47:08,700 your group thinks of ways to use the analytic model. 2061 01:47:08,700 --> 01:47:09,420 OK. 2062 01:47:09,420 --> 01:47:12,990 AUDIENCE: A problem with the data is that the flow rate, 2063 01:47:12,990 --> 01:47:16,420 we couldn't vary the flow rate on digital control. 2064 01:47:16,420 --> 01:47:18,510 So there's this whole set of six valves 2065 01:47:18,510 --> 01:47:21,390 that we had to manipulate to set a flow. 2066 01:47:21,390 --> 01:47:27,150 And so all the other factors are set at fixed levels. 2067 01:47:27,150 --> 01:47:31,050 But the flow rate is kind of an average value 2068 01:47:31,050 --> 01:47:33,630 that it kind of oscillated about. 2069 01:47:33,630 --> 01:47:37,398 So I'm not very sure how to deal. 2070 01:47:37,398 --> 01:47:38,190 DUANE BONING: Yeah. 2071 01:47:38,190 --> 01:47:40,140 Well first off, that's an interesting question 2072 01:47:40,140 --> 01:47:42,040 we haven't talked a lot about in class. 2073 01:47:42,040 --> 01:47:45,360 So it'll be interesting to raise that and talk about it. 2074 01:47:45,360 --> 01:47:49,830 That basically, there's a spread on your input. 2075 01:47:49,830 --> 01:47:52,050 There's variation on your input. 2076 01:47:52,050 --> 01:47:54,030 And some of the methods we've talked about 2077 01:47:54,030 --> 01:47:59,430 might include just recognizing that and recognizing 2078 01:47:59,430 --> 01:48:04,140 that that might propagate through the data. 2079 01:48:04,140 --> 01:48:07,980 Because often we pretend in all of our DOEs 2080 01:48:07,980 --> 01:48:10,860 that when I pick the input, it's rock solid 2081 01:48:10,860 --> 01:48:12,960 and it's never rock solid. 2082 01:48:12,960 --> 01:48:15,450 Usually it's very well, tightly controlled. 2083 01:48:15,450 --> 01:48:19,170 But often there is variation in that. 2084 01:48:19,170 --> 01:48:24,300 And the basic approach, that might be a place where 2085 01:48:24,300 --> 01:48:28,140 looking at your analytic model might actually 2086 01:48:28,140 --> 01:48:33,180 give you a sense of how big a perturbation 2087 01:48:33,180 --> 01:48:37,540 variations on the input might produce in the output. 2088 01:48:37,540 --> 01:48:44,610 So you could actually get an estimate of roughly 2089 01:48:44,610 --> 01:48:49,800 how important it is to control more accurately to the inputs. 2090 01:48:49,800 --> 01:48:50,830 Things like that. 2091 01:48:50,830 --> 01:48:54,468 So I think that's a very interesting real life problem. 2092 01:48:54,468 --> 01:48:55,760 And that's just one suggestion. 2093 01:48:55,760 --> 01:48:59,160 AUDIENCE: There's another issue. 2094 01:48:59,160 --> 01:49:02,190 The flow rate was the cause of all our troubles throughout. 2095 01:49:02,190 --> 01:49:05,880 Because we couldn't find a proper correlation analytically 2096 01:49:05,880 --> 01:49:08,460 for the flow rate, because the voltage 2097 01:49:08,460 --> 01:49:12,270 and the other parameters could be easily incorporated 2098 01:49:12,270 --> 01:49:14,370 into the analytic model just using 2099 01:49:14,370 --> 01:49:16,080 an extension of Faraday's laws. 2100 01:49:16,080 --> 01:49:19,300 But the flow rate was a little difficult to capture. 2101 01:49:19,300 --> 01:49:21,480 DUANE BONING: So it's not in the model. 2102 01:49:21,480 --> 01:49:23,220 AUDIENCE: It's not really in the model. 2103 01:49:23,220 --> 01:49:24,110 DUANE BONING: I see. 2104 01:49:24,110 --> 01:49:24,720 OK. 2105 01:49:24,720 --> 01:49:28,405 AUDIENCE: And the way we calculated flow rate, 2106 01:49:28,405 --> 01:49:30,030 the way we've measured our flow rate is 2107 01:49:30,030 --> 01:49:32,940 we've got a maximum value, a minimum value, and a most 2108 01:49:32,940 --> 01:49:34,860 common central value. 2109 01:49:34,860 --> 01:49:38,008 That's the way we've measured the flow rate. 2110 01:49:38,008 --> 01:49:40,050 DUANE BONING: I think it'll be interesting to see 2111 01:49:40,050 --> 01:49:43,110 how you dealt with that or recommendations 2112 01:49:43,110 --> 01:49:45,040 on how to deal with that. 2113 01:49:45,040 --> 01:49:45,690 That's good. 2114 01:49:45,690 --> 01:49:51,840 I think these real life challenges are a good thing. 2115 01:49:51,840 --> 01:49:52,500 Thanks. 2116 01:49:52,500 --> 01:49:55,390 I think that one sounds good. 2117 01:49:55,390 --> 01:49:56,340 Now let's see. 2118 01:50:00,290 --> 01:50:01,040 I'm trying to see. 2119 01:50:01,040 --> 01:50:03,150 Have we covered all the projects? 2120 01:50:03,150 --> 01:50:06,175 Or I've left somebody out, right? 2121 01:50:06,175 --> 01:50:07,050 AUDIENCE: [INAUDIBLE] 2122 01:50:07,050 --> 01:50:07,842 DUANE BONING: Yeah. 2123 01:50:11,130 --> 01:50:12,645 I actually have to run. 2124 01:50:16,320 --> 01:50:16,890 Let me see. 2125 01:50:16,890 --> 01:50:20,193 Which one are you? 2126 01:50:20,193 --> 01:50:24,900 AUDIENCE: We're the team working on the injection molding data. 2127 01:50:24,900 --> 01:50:27,022 DUANE BONING: Ah, yes, OK. 2128 01:50:27,022 --> 01:50:28,230 AUDIENCE: Die casting, sorry. 2129 01:50:30,858 --> 01:50:32,400 We're working on die casting process. 2130 01:50:32,400 --> 01:50:35,553 And we got our data from a paper. 2131 01:50:35,553 --> 01:50:36,470 DUANE BONING: Ah, yes. 2132 01:50:36,470 --> 01:50:38,790 So we had some good email on that. 2133 01:50:38,790 --> 01:50:39,330 Right? 2134 01:50:39,330 --> 01:50:39,960 AUDIENCE: Right, right. 2135 01:50:39,960 --> 01:50:40,260 DUANE BONING: Yeah. 2136 01:50:40,260 --> 01:50:42,390 I think you guys are in very good shape. 2137 01:50:42,390 --> 01:50:44,940 There were some suggestions from Hayden. 2138 01:50:44,940 --> 01:50:46,532 And I sent an email also. 2139 01:50:46,532 --> 01:50:47,490 AUDIENCE: That's right. 2140 01:50:47,490 --> 01:50:50,400 DUANE BONING: So did you have follow on questions? 2141 01:50:50,400 --> 01:50:51,840 I think you guys-- 2142 01:50:51,840 --> 01:50:54,790 that's good data and you can do a lot more with it. 2143 01:50:54,790 --> 01:50:59,770 So any additional thoughts or questions? 2144 01:50:59,770 --> 01:51:02,930 AUDIENCE: So we have a question concerning inputs. 2145 01:51:02,930 --> 01:51:04,080 There are three inputs. 2146 01:51:04,080 --> 01:51:06,090 Two of them they have three levels. 2147 01:51:06,090 --> 01:51:08,990 And the other one has only two levels. 2148 01:51:08,990 --> 01:51:11,790 Does it matter when you do-- 2149 01:51:11,790 --> 01:51:15,930 when you do the ANOVA, ISM, all these analysis? 2150 01:51:15,930 --> 01:51:20,480 Because the levels are different for inputs. 2151 01:51:20,480 --> 01:51:22,970 DUANE BONING: The quick answer is no. 2152 01:51:22,970 --> 01:51:26,660 But it should be interesting for you to show why that still 2153 01:51:26,660 --> 01:51:30,670 is good and that's no problem. 2154 01:51:30,670 --> 01:51:31,520 Just think about it. 2155 01:51:31,520 --> 01:51:34,960 I mean, in terms of some model dependencies, 2156 01:51:34,960 --> 01:51:36,940 if you have three levels, you might 2157 01:51:36,940 --> 01:51:41,230 be able to do quadratic in that model parameter. 2158 01:51:41,230 --> 01:51:43,180 But if you only have two levels in others, 2159 01:51:43,180 --> 01:51:45,800 you'll only have linear terms for that. 2160 01:51:45,800 --> 01:51:46,510 So that's fine. 2161 01:51:46,510 --> 01:51:48,480 That's OK. 2162 01:51:48,480 --> 01:51:50,062 AUDIENCE: OK. 2163 01:51:50,062 --> 01:51:50,770 DUANE BONING: OK. 2164 01:51:50,770 --> 01:51:54,280 So if questions come up as you're working on that one, 2165 01:51:54,280 --> 01:51:55,600 send us more email. 2166 01:51:55,600 --> 01:51:58,030 Because Hayden had lots of ideas. 2167 01:51:58,030 --> 01:52:01,750 There may be more than you can do. 2168 01:52:01,750 --> 01:52:05,050 But I think there are some very interesting ideas, 2169 01:52:05,050 --> 01:52:06,790 because there's replicate data. 2170 01:52:06,790 --> 01:52:09,430 Variance might be interesting to model. 2171 01:52:09,430 --> 01:52:12,920 Several neat things you can do with that. 2172 01:52:12,920 --> 01:52:15,310 AUDIENCE: OK. 2173 01:52:15,310 --> 01:52:16,630 We'll follow with emails. 2174 01:52:16,630 --> 01:52:17,380 DUANE BONING: OK. 2175 01:52:17,380 --> 01:52:17,880 Good. 2176 01:52:17,880 --> 01:52:20,440 And we can also again on Thursday if there's 2177 01:52:20,440 --> 01:52:22,260 more questions that come up. 2178 01:52:22,260 --> 01:52:25,000 After class we can talk briefly. 2179 01:52:25,000 --> 01:52:26,650 Because by then you should hopefully 2180 01:52:26,650 --> 01:52:27,760 be well into the project. 2181 01:52:30,107 --> 01:52:30,940 AUDIENCE: Thank you. 2182 01:52:30,940 --> 01:52:32,050 DUANE BONING: Thank you. 2183 01:52:32,050 --> 01:52:34,530 See you guys later.