WEBVTT

00:00:00.000 --> 00:00:02.460
The following content is
provided under a Creative

00:00:02.460 --> 00:00:03.730
Commons license.

00:00:03.730 --> 00:00:06.060
Your support will help
MIT OpenCourseWare

00:00:06.060 --> 00:00:10.090
continue to offer high quality
educational resources for free.

00:00:10.090 --> 00:00:12.690
To make a donation or to
view additional materials

00:00:12.690 --> 00:00:16.560
from hundreds of MIT courses,
visit MIT OpenCourseWare

00:00:16.560 --> 00:00:17.904
at ocw.mit.edu.

00:00:20.970 --> 00:00:23.220
DUANE BONING: OK, so what
I'm going to try to do today

00:00:23.220 --> 00:00:28.290
is kind of a fun case
study on spatial modeling.

00:00:28.290 --> 00:00:31.080
And then we will
have time afterwards,

00:00:31.080 --> 00:00:32.580
especially with the
Singapore folks,

00:00:32.580 --> 00:00:34.800
to try to at least
have brief meetings

00:00:34.800 --> 00:00:36.450
on a couple of the projects.

00:00:36.450 --> 00:00:38.190
I've met with at
least one group.

00:00:38.190 --> 00:00:42.180
I think one group you
guys met with Dave Hardt.

00:00:42.180 --> 00:00:43.950
At least talked, email.

00:00:43.950 --> 00:00:46.161
Did you meet also?

00:00:46.161 --> 00:00:47.772
AUDIENCE: [INAUDIBLE]

00:00:47.772 --> 00:00:48.480
DUANE BONING: OK.

00:00:48.480 --> 00:00:52.094
Because it looked like you guys
are in good shape to, right?

00:00:52.094 --> 00:00:53.710
AUDIENCE: I don't know.

00:00:53.710 --> 00:00:55.730
DUANE BONING: He said do more.

00:00:55.730 --> 00:00:58.682
AUDIENCE: [INAUDIBLE]

00:00:58.682 --> 00:00:59.390
DUANE BONING: OK.

00:00:59.390 --> 00:01:01.940
So we can have some
follow up on that as well.

00:01:01.940 --> 00:01:04.450
But what I'm going to do is
go through this case study.

00:01:07.030 --> 00:01:08.947
My preferences
actually this is--

00:01:08.947 --> 00:01:10.780
I would have liked to
have been able to talk

00:01:10.780 --> 00:01:13.197
about this a little bit earlier,
because this is some very

00:01:13.197 --> 00:01:15.730
neat stuff on spatial
modeling that in the past

00:01:15.730 --> 00:01:17.728
has sometimes been a
part of some projects.

00:01:17.728 --> 00:01:19.270
So maybe you can
look at this and see

00:01:19.270 --> 00:01:22.960
if there's something extra on
spatial modeling you can do.

00:01:22.960 --> 00:01:25.780
Essentially, this is going to
be based on three papers that

00:01:25.780 --> 00:01:27.580
are also on the website.

00:01:27.580 --> 00:01:31.000
There's a paper by
Davis, one by Mozumder,

00:01:31.000 --> 00:01:34.070
and one by Guo and Sachs.

00:01:34.070 --> 00:01:37.070
So you can grab those
papers and learn more.

00:01:37.070 --> 00:01:40.310
But my basic agenda
is as shown here.

00:01:40.310 --> 00:01:43.600
First I want to sort of talk
a little bit about applying

00:01:43.600 --> 00:01:46.180
the same response surface
modeling and regression

00:01:46.180 --> 00:01:48.400
methodology that
we've talked about

00:01:48.400 --> 00:01:50.630
for things like
process conditions

00:01:50.630 --> 00:01:53.250
also to spatial parameters.

00:01:53.250 --> 00:01:56.050
So if you were
spatially sampling,

00:01:56.050 --> 00:02:00.130
could you build a model as a
function of spatial coordinates

00:02:00.130 --> 00:02:03.580
and get a sense of spatial
uniformity across a wafer,

00:02:03.580 --> 00:02:06.340
across a part, across
some spatial dimension?

00:02:06.340 --> 00:02:08.350
And of course,
the answer is yes.

00:02:08.350 --> 00:02:11.470
What's interesting is then when
you go and calculate things

00:02:11.470 --> 00:02:14.380
like uniformity or
uniformity metrics,

00:02:14.380 --> 00:02:16.610
there's some special
issues that come up.

00:02:16.610 --> 00:02:19.870
And in particular,
your sampling plan

00:02:19.870 --> 00:02:24.820
can affect dramatically what
number you get for uniformity.

00:02:24.820 --> 00:02:30.100
And so we'll talk about ways
to not fool yourself on that.

00:02:30.100 --> 00:02:34.660
The second part will be
then to go back and say, OK,

00:02:34.660 --> 00:02:38.670
if I can do a response surface
model on spatial coordinates

00:02:38.670 --> 00:02:42.270
and the process is changing,
how do I put those two things

00:02:42.270 --> 00:02:44.910
together to model
spatial variation

00:02:44.910 --> 00:02:46.830
as a function of the process?

00:02:46.830 --> 00:02:49.350
And there's two
different ideas, sort

00:02:49.350 --> 00:02:52.860
of opposite cuts at the
same problem, one based

00:02:52.860 --> 00:02:56.760
on the Mozumder paper and
the other by Guo and Sachs

00:02:56.760 --> 00:03:00.960
that I think are a
very interesting case

00:03:00.960 --> 00:03:06.300
study in that they show an
evolution in the thinking

00:03:06.300 --> 00:03:09.750
about how to efficiently
and effectively do that.

00:03:09.750 --> 00:03:13.080
So let me talk first about
just purely spatial modeling.

00:03:13.080 --> 00:03:14.760
And of course, we
know that there

00:03:14.760 --> 00:03:19.740
can be a spatial
trend, systematic trend

00:03:19.740 --> 00:03:22.290
in many, many processes.

00:03:22.290 --> 00:03:25.230
Most of the methods we've
been talking about so far kind

00:03:25.230 --> 00:03:27.330
of we're looking at
random variation, ways

00:03:27.330 --> 00:03:33.120
of assessing as if
we were sampling

00:03:33.120 --> 00:03:36.120
from a Gaussian or
some other process.

00:03:36.120 --> 00:03:39.330
But underlying that, you may
have that noise, measurement

00:03:39.330 --> 00:03:41.670
noise, or whatever,
but underlying that,

00:03:41.670 --> 00:03:44.640
there may be a
systematic spatial trend.

00:03:44.640 --> 00:03:50.760
And these may result from
inherent equipment or process

00:03:50.760 --> 00:03:54.970
asymmetries, be
highly repeatable.

00:03:54.970 --> 00:03:57.300
You would like to be able
to capture and express them.

00:03:57.300 --> 00:03:59.040
So I've sort of
pictured here an example

00:03:59.040 --> 00:04:04.650
where you might have a
concave or convex shape.

00:04:04.650 --> 00:04:07.710
If I were measuring some
parameter across the wafer,

00:04:07.710 --> 00:04:11.190
like a film thickness,
other geometric parameters,

00:04:11.190 --> 00:04:15.030
or some material property
like film resistivity.

00:04:15.030 --> 00:04:17.279
And very often
maybe gas flows are

00:04:17.279 --> 00:04:20.730
a little bit different in the
center of the wafer and so on.

00:04:20.730 --> 00:04:23.760
You get these sort of
bowl shaped patterns.

00:04:23.760 --> 00:04:25.890
But there may be other
kinds of spatial patterns,

00:04:25.890 --> 00:04:29.490
and you'd like to be
able to capture that.

00:04:29.490 --> 00:04:31.420
So our goal is how
do we model that?

00:04:31.420 --> 00:04:33.330
And then how do
we boil that down

00:04:33.330 --> 00:04:36.690
into an overall metric that says
how good this is and then drive

00:04:36.690 --> 00:04:38.550
the process to try
to improve that?

00:04:38.550 --> 00:04:42.430
Decrease non-uniformity,
improve uniformity.

00:04:42.430 --> 00:04:44.530
So what I'm going to
do is show an example.

00:04:44.530 --> 00:04:48.780
And what I'm going to do
here is use synthetic data.

00:04:48.780 --> 00:04:52.410
By the way, what I mean
by synthetic data is

00:04:52.410 --> 00:04:58.210
I'm going to generate data
that I know it's properties of.

00:04:58.210 --> 00:04:59.970
I am going to
generate a data with

00:04:59.970 --> 00:05:05.220
a known statistical and
systematic, a known random

00:05:05.220 --> 00:05:10.260
and a known systematic
component to it.

00:05:10.260 --> 00:05:13.680
This is a useful
technique that you

00:05:13.680 --> 00:05:18.460
might consider somewhere in one
of your team projects as well.

00:05:18.460 --> 00:05:20.760
You may be dealing
with real data.

00:05:20.760 --> 00:05:23.280
And very often you don't
know what ground truth is.

00:05:23.280 --> 00:05:25.530
And that's the whole interesting
thing about a project

00:05:25.530 --> 00:05:27.450
is you don't really
know the truth.

00:05:27.450 --> 00:05:29.940
What we're trying to do
is infer things about it

00:05:29.940 --> 00:05:31.740
from experimental data.

00:05:31.740 --> 00:05:34.810
If you're also exploring
some methodology,

00:05:34.810 --> 00:05:38.010
you want to see does this new
methodology that I've come up

00:05:38.010 --> 00:05:40.980
with that I think
solves some problem

00:05:40.980 --> 00:05:44.370
or applies to some
aspect of this problem?

00:05:44.370 --> 00:05:47.610
You can also test that
with synthetic data

00:05:47.610 --> 00:05:49.110
that you generate yourself.

00:05:49.110 --> 00:05:54.600
And it's a nice complement in
some cases to the other project

00:05:54.600 --> 00:05:56.670
data that you're doing.

00:05:56.670 --> 00:06:01.020
So the data that I'm generating
has a systematic component

00:06:01.020 --> 00:06:06.710
that is a circular or
elliptic kind of wafer map.

00:06:06.710 --> 00:06:10.840
Plus it's got some superimposed
random noise on it.

00:06:10.840 --> 00:06:14.310
And what I'm going
to do is sample it

00:06:14.310 --> 00:06:15.820
in two different ways.

00:06:15.820 --> 00:06:21.000
So I'm going to generate
two sampled data sets off

00:06:21.000 --> 00:06:24.930
of a more basic data
set that I create.

00:06:24.930 --> 00:06:26.670
And the sampling
patterns are going

00:06:26.670 --> 00:06:31.290
to be two that are commonly used
in semiconductor manufacturing.

00:06:31.290 --> 00:06:33.850
One is a circular wafer map.

00:06:33.850 --> 00:06:34.350
Let's see.

00:06:34.350 --> 00:06:35.370
Do I have a picture of it?

00:06:35.370 --> 00:06:35.870
Yeah.

00:06:35.870 --> 00:06:40.110
So here's a radial or
circular sampling plan.

00:06:40.110 --> 00:06:43.830
The basic idea that I take
a point at the center,

00:06:43.830 --> 00:06:47.340
then I take points
around some radius,

00:06:47.340 --> 00:06:50.250
I take additional points
around another radius,

00:06:50.250 --> 00:06:53.350
and I take additional
points at a third radius

00:06:53.350 --> 00:06:57.510
so that I've got these
rings of concentric circles

00:06:57.510 --> 00:06:59.470
as a sampling plan.

00:06:59.470 --> 00:07:02.220
An alternative one
is to simply use

00:07:02.220 --> 00:07:07.410
a rectangular or
square sampling plan.

00:07:07.410 --> 00:07:09.750
So those are going to be two
different sampling plans.

00:07:09.750 --> 00:07:12.840
What you would hope is you
get the same uniformity

00:07:12.840 --> 00:07:14.640
metric off of the
two, that it shouldn't

00:07:14.640 --> 00:07:16.200
be sensitive to sampling.

00:07:16.200 --> 00:07:21.480
I'm going to show you, in fact,
that it can be quite sensitive.

00:07:21.480 --> 00:07:23.940
OK, so what's my synthetic data?

00:07:23.940 --> 00:07:24.880
It's shown here.

00:07:24.880 --> 00:07:27.030
This is in fact the
elliptic component,

00:07:27.030 --> 00:07:31.120
the systematic component,
that I create some output.

00:07:31.120 --> 00:07:33.550
Call it resistivity here
just to be concrete.

00:07:36.160 --> 00:07:41.210
Plus random Gaussian
0 mean noise.

00:07:41.210 --> 00:07:42.970
So I've got, say, some
measurement noise,

00:07:42.970 --> 00:07:46.120
process noise
superimposed on this.

00:07:46.120 --> 00:07:48.740
What I'm going to try
to do is two things.

00:07:48.740 --> 00:07:52.930
One is build a
response surface model.

00:07:52.930 --> 00:07:56.770
Maybe, hopefully,
recover something close

00:07:56.770 --> 00:08:00.760
to the underlying ground
truth that I've imposed here.

00:08:00.760 --> 00:08:04.600
And we can now assess how
good a response surface model

00:08:04.600 --> 00:08:07.510
fit is, because I know
what the truth is.

00:08:07.510 --> 00:08:09.940
Second, I'm also
going to calculate

00:08:09.940 --> 00:08:11.860
a non-uniformity metric.

00:08:11.860 --> 00:08:14.440
And the basic metric is
going to be something

00:08:14.440 --> 00:08:18.880
like a noise to signal ratio.

00:08:18.880 --> 00:08:22.180
That is to say, the
normalized standard deviation

00:08:22.180 --> 00:08:24.080
across my measurement points.

00:08:24.080 --> 00:08:25.480
So I calculate the mean.

00:08:25.480 --> 00:08:28.450
I calculate the standard
deviation, normalize or divide.

00:08:28.450 --> 00:08:30.490
So I'm getting
something, if I multiply

00:08:30.490 --> 00:08:34.659
by 100, something like a
percentage non-uniformity

00:08:34.659 --> 00:08:35.710
across this wafer.

00:08:38.260 --> 00:08:41.230
Here's the data,
just so you have it.

00:08:41.230 --> 00:08:49.720
What I get in this sampling
plan is 25 total data points,

00:08:49.720 --> 00:08:53.110
one at the center and then
eight at each of 25, 50,

00:08:53.110 --> 00:08:55.930
and 75 millimeters
away from the center.

00:08:55.930 --> 00:09:00.820
This is assuming sort of a 200
millimeter wafer, if you will.

00:09:00.820 --> 00:09:09.010
And here's the actual data based
on my underlying synthetic data

00:09:09.010 --> 00:09:09.950
model.

00:09:09.950 --> 00:09:12.520
Here's the square sampling plan.

00:09:12.520 --> 00:09:15.160
In this case, you'll
notice that a few

00:09:15.160 --> 00:09:17.710
of the points,
these corner points,

00:09:17.710 --> 00:09:21.190
actually fall off the
edge of a round wafer.

00:09:21.190 --> 00:09:23.170
So they are discarded.

00:09:23.170 --> 00:09:25.390
And those are the
ones in bold here.

00:09:25.390 --> 00:09:28.030
Those ones are removed,
because they're

00:09:28.030 --> 00:09:30.670
outside of the wafer boundary.

00:09:30.670 --> 00:09:33.760
That should be a plus minus.

00:09:33.760 --> 00:09:38.530
So this actually has, in fact,
slightly fewer data points.

00:09:38.530 --> 00:09:43.060
It has only 21 data points,
whereas the previous sampling

00:09:43.060 --> 00:09:44.610
plan actually has more data.

00:09:44.610 --> 00:09:46.390
It's 25.

00:09:46.390 --> 00:09:47.170
Yeah.

00:09:47.170 --> 00:09:48.420
3 times 8 plus 1.

00:09:48.420 --> 00:09:51.310
25 data points.

00:09:51.310 --> 00:09:52.750
OK.

00:09:52.750 --> 00:09:54.610
One thing I want to--

00:09:54.610 --> 00:09:59.170
I'll be showing some jump plots,
some surface plots, contours,

00:09:59.170 --> 00:10:00.100
and so on.

00:10:00.100 --> 00:10:03.820
And the contouring does show
a little bit of difference

00:10:03.820 --> 00:10:08.140
because it interpolates
just to plot things.

00:10:08.140 --> 00:10:10.870
But in both of these
cases, they pretty much

00:10:10.870 --> 00:10:13.780
look similar except
for some adjustment

00:10:13.780 --> 00:10:16.300
because of the location
of the data points.

00:10:16.300 --> 00:10:20.530
And if anything,
the radial 25 points

00:10:20.530 --> 00:10:23.740
looks like it actually kind of
interpolates a little bit more

00:10:23.740 --> 00:10:28.240
to the roundish or
elliptic underlying pattern

00:10:28.240 --> 00:10:29.950
that I said I was imposing.

00:10:29.950 --> 00:10:31.690
This one because
of the square grid

00:10:31.690 --> 00:10:33.820
looks like it may
have some artifacts.

00:10:33.820 --> 00:10:36.160
But that gives you a
feel for what's going on.

00:10:36.160 --> 00:10:41.320
What we've got is a
not quite center high.

00:10:41.320 --> 00:10:43.960
It's not a pure bowl
right in the center.

00:10:43.960 --> 00:10:46.300
It's a little bit off
center, and it's not

00:10:46.300 --> 00:10:48.190
perfectly circular.

00:10:48.190 --> 00:10:49.690
It's kind of elliptic.

00:10:49.690 --> 00:10:53.410
So that gives you a feel for
what the underlying pattern is.

00:10:53.410 --> 00:10:56.260
So I say I want to fit a model.

00:10:56.260 --> 00:11:01.150
I'm going to fit a generic
second order model in x and y.

00:11:01.150 --> 00:11:04.990
So it's got linear coefficients
in x and y, an interaction

00:11:04.990 --> 00:11:09.330
term, and second order terms,
x squared and y squared.

00:11:09.330 --> 00:11:12.490
So six total coefficients.

00:11:12.490 --> 00:11:15.760
What I'm doing first here
is the radial pattern

00:11:15.760 --> 00:11:18.460
where I had 25 data points.

00:11:18.460 --> 00:11:21.220
And you can sort of
see what happens here.

00:11:21.220 --> 00:11:23.680
What we've got is
an r squared that

00:11:23.680 --> 00:11:26.680
looks like it's about 0.65.

00:11:26.680 --> 00:11:31.250
Doesn't sound great, but I know
there's noise in the process.

00:11:31.250 --> 00:11:34.330
So I got to look at
it in a little bit

00:11:34.330 --> 00:11:38.800
more detail at the ANOVA and
see is my model significant.

00:11:38.800 --> 00:11:40.900
The r squared is make
me a little bit worried.

00:11:40.900 --> 00:11:42.400
It's not great.

00:11:42.400 --> 00:11:46.180
But I look, and overall the
model compared to the error

00:11:46.180 --> 00:11:51.100
terms is significant.

00:11:53.620 --> 00:11:58.420
So I've got an f that
is much, much larger

00:11:58.420 --> 00:12:00.220
than I would expect
by chance alone.

00:12:00.220 --> 00:12:04.660
It's 99.94% significant.

00:12:04.660 --> 00:12:05.320
OK, great.

00:12:05.320 --> 00:12:09.520
So I've got a model that is
capturing something going on.

00:12:09.520 --> 00:12:12.160
Then I can go in and look
at the parameter estimates,

00:12:12.160 --> 00:12:15.190
look at each of the individual
terms, look at the t ratio,

00:12:15.190 --> 00:12:17.980
and look at the probability
of those being significant.

00:12:17.980 --> 00:12:20.530
And they all look
pretty significant

00:12:20.530 --> 00:12:26.785
except this one
for sure, a y term.

00:12:26.785 --> 00:12:29.540
It says no, 0.5
chance that would

00:12:29.540 --> 00:12:32.050
have occurred by chance alone.

00:12:32.050 --> 00:12:40.100
And I've only got maybe
92 %or 91% chance or level

00:12:40.100 --> 00:12:44.040
of confidence in this xy term.

00:12:44.040 --> 00:12:48.020
But if I was looking
at, say, 95% confidence

00:12:48.020 --> 00:12:52.190
interval cutoffs, I
might reject that term.

00:12:52.190 --> 00:12:55.160
And we'll see later,
that's correctly rejecting

00:12:55.160 --> 00:12:57.320
that term if you look back.

00:12:57.320 --> 00:12:59.570
So knowing ground
truth, actually this

00:12:59.570 --> 00:13:01.400
raises already a worry.

00:13:01.400 --> 00:13:05.480
How come on correctly
rejecting the xy term

00:13:05.480 --> 00:13:08.450
but incorrectly
rejecting another term?

00:13:08.450 --> 00:13:09.890
Interesting.

00:13:09.890 --> 00:13:10.390
OK.

00:13:14.030 --> 00:13:16.470
Let's look at the other case.

00:13:16.470 --> 00:13:18.710
I'm using a rectangular
pattern fitting

00:13:18.710 --> 00:13:21.870
the same model in this case.

00:13:21.870 --> 00:13:23.040
What happens now?

00:13:23.040 --> 00:13:25.350
Well, now we only
have 21 data points.

00:13:25.350 --> 00:13:28.250
Notice that the r squared
appears to be better.

00:13:28.250 --> 00:13:30.560
That's interesting.

00:13:30.560 --> 00:13:33.210
Again, the model is
highly significant.

00:13:33.210 --> 00:13:36.050
And now if I look at
the individual terms,

00:13:36.050 --> 00:13:39.450
the x is significant,
the y is significant,

00:13:39.450 --> 00:13:43.640
the xy is not significant.

00:13:43.640 --> 00:13:45.620
That's rejected.

00:13:45.620 --> 00:13:49.220
The x squared and
the y squared look--

00:13:49.220 --> 00:13:52.940
I guess the y squared is kind
of right marginal right at about

00:13:52.940 --> 00:13:54.810
95%.

00:13:54.810 --> 00:13:59.000
So it seems like it's coming
up with different terms

00:13:59.000 --> 00:14:00.350
in the model.

00:14:00.350 --> 00:14:03.500
And also we'll see in a
second when I compare them,

00:14:03.500 --> 00:14:06.680
it's coming up with some
different estimates.

00:14:06.680 --> 00:14:09.230
So maybe this isn't a surprise.

00:14:09.230 --> 00:14:14.280
The sampling plan is affecting
the model that I'm building.

00:14:14.280 --> 00:14:16.220
Doesn't sound too bad so far.

00:14:16.220 --> 00:14:18.407
But what I also want to do--

00:14:18.407 --> 00:14:19.490
let me compare them first.

00:14:19.490 --> 00:14:21.230
And then what I
want to do is also

00:14:21.230 --> 00:14:24.560
calculate a uniformity metric
across all of this data

00:14:24.560 --> 00:14:26.280
and see what happens.

00:14:26.280 --> 00:14:27.500
So here's our truth.

00:14:27.500 --> 00:14:29.930
I've expanded out
that original model

00:14:29.930 --> 00:14:32.690
of the systematic
component contaminated

00:14:32.690 --> 00:14:38.780
by noise out to the x terms,
the xy, and the x squared terms.

00:14:38.780 --> 00:14:42.050
And here is the radial sampling
plan and the square sampling

00:14:42.050 --> 00:14:43.310
plan.

00:14:43.310 --> 00:14:45.020
And the coefficients,
some of them

00:14:45.020 --> 00:14:48.110
are not radically different.

00:14:48.110 --> 00:14:50.660
They're kind of in the ballpark.

00:14:50.660 --> 00:14:56.360
But in terms of deciding
which terms are kept

00:14:56.360 --> 00:15:00.110
and which are rejected, we
know that this one, in fact,

00:15:00.110 --> 00:15:04.910
really is 0, whereas this
one tries to keep it.

00:15:04.910 --> 00:15:08.300
This one is very much smaller,
and we actually get rid of it.

00:15:08.300 --> 00:15:12.350
This one's small and this
one's small but real.

00:15:12.350 --> 00:15:17.810
Both of the models
fit parameters for it

00:15:17.810 --> 00:15:20.635
but make different cutoff
points on what's significant

00:15:20.635 --> 00:15:23.420
and what's not.

00:15:23.420 --> 00:15:26.210
Now, what's really
interesting, I think,

00:15:26.210 --> 00:15:30.585
is I now take that same data
and I go back-- ah, great.

00:15:30.585 --> 00:15:31.460
Come on in, Charlene.

00:15:33.750 --> 00:15:34.250
What?

00:15:34.250 --> 00:15:35.792
AUDIENCE: She's
setting up right now.

00:15:35.792 --> 00:15:39.740
DUANE BONING: Oh, Charlene,
you can set up in here.

00:15:39.740 --> 00:15:42.260
Thank you.

00:15:42.260 --> 00:15:44.930
Is if I take those
two sampling plans

00:15:44.930 --> 00:15:51.020
and I look back at the mean
and the standard deviation

00:15:51.020 --> 00:15:53.390
just across my sampling
points and calculate

00:15:53.390 --> 00:15:55.400
that non-uniformity metric.

00:15:55.400 --> 00:15:58.760
In one case, it's
about a factor of two

00:15:58.760 --> 00:16:00.020
different than the other.

00:16:06.160 --> 00:16:07.940
The question is why?

00:16:07.940 --> 00:16:11.600
And you can ponder that
question as you get coffee,

00:16:11.600 --> 00:16:13.606
because that's what
I'm going to do.

00:16:13.606 --> 00:16:15.620
So now I think
everybody's had a chance

00:16:15.620 --> 00:16:18.710
to ruminate on this
question of non-uniformities

00:16:18.710 --> 00:16:21.830
or at least read the next
two sub-bullets at the bottom

00:16:21.830 --> 00:16:24.200
of the page, and then
everybody has a good idea

00:16:24.200 --> 00:16:26.490
of what might be going on.

00:16:26.490 --> 00:16:30.320
One is, of course, we've
got systematic curvature.

00:16:30.320 --> 00:16:32.690
I'm not random sampling,
so I may, in fact,

00:16:32.690 --> 00:16:36.650
hit different points
on that curvature.

00:16:36.650 --> 00:16:39.590
So that's one hypothesis
is just because

00:16:39.590 --> 00:16:44.060
of the particular places where
my points happened to fall,

00:16:44.060 --> 00:16:45.170
that's one idea.

00:16:45.170 --> 00:16:48.410
What we'll find out, and I'll
show that by more densely

00:16:48.410 --> 00:16:51.710
sampling, that's actually not
what's going on necessarily

00:16:51.710 --> 00:16:54.800
in this problem.

00:16:54.800 --> 00:16:59.900
Actually, it's the structure
of the sampling plan.

00:16:59.900 --> 00:17:05.030
And underlying that is how much
data or what spatial region

00:17:05.030 --> 00:17:08.510
is each data point representing
of the underlying wafer

00:17:08.510 --> 00:17:10.040
surface.

00:17:10.040 --> 00:17:11.970
So let me try to explain that.

00:17:11.970 --> 00:17:15.950
So what I've done here is gone
back and tried to say maybe

00:17:15.950 --> 00:17:20.599
I just wasn't sampling enough
with just 25 or 21 data points.

00:17:20.599 --> 00:17:25.980
What's my best guess at
the true non-uniformity?

00:17:25.980 --> 00:17:29.090
Which one of these
two should I believe?

00:17:29.090 --> 00:17:33.140
Which one is closer,
the 1.9%, the 3.2%?

00:17:33.140 --> 00:17:34.430
What's a better sampling plan?

00:17:37.490 --> 00:17:40.810
Actually, before I go into
that, what do you think?

00:17:40.810 --> 00:17:43.640
I need a vote here.

00:17:43.640 --> 00:17:46.270
How many people think
that the radial sampling

00:17:46.270 --> 00:17:49.870
plan is probably closer
to the truth in terms

00:17:49.870 --> 00:17:53.200
of both the model,
but more importantly,

00:17:53.200 --> 00:17:55.450
the non-uniformity
number that comes out?

00:17:55.450 --> 00:17:58.210
And how many people think it's
a rectangular sampling plan?

00:17:58.210 --> 00:18:02.290
So all those think the
radial sampling plan is going

00:18:02.290 --> 00:18:05.350
to be better, raise your hand.

00:18:05.350 --> 00:18:10.370
I got I got two here
and a half. three.

00:18:10.370 --> 00:18:14.740
We got two, three, three in
Singapore, four in Singapore.

00:18:14.740 --> 00:18:16.210
Oh, we're talking it up.

00:18:16.210 --> 00:18:17.560
We're talking it up.

00:18:17.560 --> 00:18:21.730
OK, how about the
rectangular sampling plan?

00:18:21.730 --> 00:18:22.870
About evenly divided.

00:18:22.870 --> 00:18:26.590
I got three here,
one in Singapore.

00:18:26.590 --> 00:18:31.930
And OK, that must leave four
asleep people who didn't vote,

00:18:31.930 --> 00:18:33.580
something like that.

00:18:33.580 --> 00:18:34.150
OK.

00:18:34.150 --> 00:18:36.700
What I'm going to try to
do is get a close estimate

00:18:36.700 --> 00:18:38.950
to what maybe better truth is.

00:18:38.950 --> 00:18:40.640
And the way I'm
going to do that is

00:18:40.640 --> 00:18:47.650
try to take a very dense spatial
sample so that hopefully it's

00:18:47.650 --> 00:18:49.690
not just sort of the
random location of points.

00:18:49.690 --> 00:18:52.570
I'm going to kind of get
really close to almost

00:18:52.570 --> 00:18:54.670
a full representation
of the surface.

00:18:54.670 --> 00:18:58.090
And this is almost a 900 point--

00:18:58.090 --> 00:19:00.140
slightly under 900 point sample.

00:19:00.140 --> 00:19:00.640
What is it?

00:19:00.640 --> 00:19:02.710
665, I guess.

00:19:02.710 --> 00:19:05.530
A 29 by 29 spatial
sample and then the

00:19:05.530 --> 00:19:10.000
points that fall within
the radius of the wafer.

00:19:10.000 --> 00:19:13.660
And as you do that, you
actually get a, quote,

00:19:13.660 --> 00:19:18.430
a "true non-uniformity"
that's about 3%

00:19:18.430 --> 00:19:20.620
closer to the other number.

00:19:20.620 --> 00:19:25.240
And if one does a response
surface model fit,

00:19:25.240 --> 00:19:28.730
there's a couple of important
points to come up with this.

00:19:28.730 --> 00:19:31.430
I go in and I fit the model.

00:19:31.430 --> 00:19:35.080
In this case, the coefficients
that come out of the model

00:19:35.080 --> 00:19:39.490
are actually quite close to
the true systematic piece.

00:19:39.490 --> 00:19:43.630
But notice that the r squared
is still only about 0.77,

00:19:43.630 --> 00:19:45.520
something like that.

00:19:45.520 --> 00:19:47.980
Why is the r squared so bad?

00:19:47.980 --> 00:19:49.940
I got tons of data.

00:19:49.940 --> 00:19:51.190
Why is the r squared so bad?

00:19:56.110 --> 00:19:58.510
And the answer is kind of
highlighted right here.

00:19:58.510 --> 00:20:01.060
I've still got lots of
random noise in the process.

00:20:01.060 --> 00:20:09.820
I was adding magnitude variance
0.49 or 0.7 standard deviation

00:20:09.820 --> 00:20:14.050
values to all of
my measurements.

00:20:14.050 --> 00:20:16.780
I cannot model the random noise.

00:20:16.780 --> 00:20:18.820
That's my residual
that's leftover.

00:20:18.820 --> 00:20:21.640
I could include it as an
estimate of the random noise

00:20:21.640 --> 00:20:25.000
and know what that is,
but I can't pull that

00:20:25.000 --> 00:20:26.290
into the systematic model.

00:20:26.290 --> 00:20:31.210
So there's always a ground if
there's noise and randomness

00:20:31.210 --> 00:20:33.610
that you can never capture.

00:20:33.610 --> 00:20:35.830
OK, so this seems
to be suggesting

00:20:35.830 --> 00:20:39.790
that maybe that square
sample with the smaller

00:20:39.790 --> 00:20:41.680
number of points was
a little bit better.

00:20:41.680 --> 00:20:43.240
What's going on?

00:20:43.240 --> 00:20:46.270
This finally gets me
to the Davis paper.

00:20:46.270 --> 00:20:48.400
And what they're
going to propose

00:20:48.400 --> 00:20:51.850
is a method for looking
at systematic trends.

00:20:51.850 --> 00:20:53.770
They're first going
to make the point

00:20:53.770 --> 00:20:58.210
that the signal to noise ratio,
this standard deviation over mu

00:20:58.210 --> 00:21:01.150
or mu over standard
deviation, if I turn it

00:21:01.150 --> 00:21:05.290
into a signal to noise, is
sensitive to both the location

00:21:05.290 --> 00:21:07.000
and the number of measurements.

00:21:07.000 --> 00:21:11.770
And they're going to propose
a difference statistic rather

00:21:11.770 --> 00:21:14.350
than just the sample
standard deviation

00:21:14.350 --> 00:21:18.730
to try to get at what
a better, truer measure

00:21:18.730 --> 00:21:21.610
of overall
non-uniformity will be.

00:21:21.610 --> 00:21:23.620
Turns out this
integration statistic

00:21:23.620 --> 00:21:27.430
is based on fitting of
splines to your data,

00:21:27.430 --> 00:21:30.470
and not a lot of
people go in for that.

00:21:30.470 --> 00:21:34.750
What I'm going to show is that a
relatively simple approximation

00:21:34.750 --> 00:21:37.030
achieves a lot of the
goals of this integration

00:21:37.030 --> 00:21:41.680
statistic, which is worrying
about uniform sampling

00:21:41.680 --> 00:21:44.950
spatially, or
alternatively, by weighting

00:21:44.950 --> 00:21:47.200
the importance of each of
your measurement points

00:21:47.200 --> 00:21:49.330
by the amount of area
that it's representing.

00:21:49.330 --> 00:21:51.730
So that's the core
of the problem.

00:21:51.730 --> 00:21:57.460
So what Davis does is looks at,
first off, a sort of a smaller

00:21:57.460 --> 00:22:01.240
radial kind of picture
compared to the one I showed.

00:22:01.240 --> 00:22:03.980
It's only got 13 data points.

00:22:03.980 --> 00:22:07.090
Here's the 13 data points
with just linear interpolation

00:22:07.090 --> 00:22:07.940
between them.

00:22:07.940 --> 00:22:11.470
So that's the surface
example that he's using.

00:22:11.470 --> 00:22:14.810
And if you just purely linearly
interpolate between them,

00:22:14.810 --> 00:22:20.110
you can see that's a
pretty coarse approximation

00:22:20.110 --> 00:22:21.310
to that surface.

00:22:21.310 --> 00:22:25.360
One can easily be worried that
you might get bias or variance

00:22:25.360 --> 00:22:30.640
errors in trying to calculate
with just those data points.

00:22:30.640 --> 00:22:34.270
What they're proposing is using
thin plate splines, these TPS

00:22:34.270 --> 00:22:39.250
methods, which are conceptually,
basically fitting localized

00:22:39.250 --> 00:22:45.340
polynomials such that you have
minimum curvature over the data

00:22:45.340 --> 00:22:48.100
points or knots in the data.

00:22:48.100 --> 00:22:48.850
All right.

00:22:48.850 --> 00:22:51.190
I'm sure there are statistical
packages and others that

00:22:51.190 --> 00:22:53.560
can help fit those
thin plate splines.

00:22:53.560 --> 00:22:55.570
I found when I've
tried this it's

00:22:55.570 --> 00:22:59.620
very tricky, because what
localized means can be a little

00:22:59.620 --> 00:23:00.560
bit tricky.

00:23:00.560 --> 00:23:03.970
You've got other kind
of human intervention

00:23:03.970 --> 00:23:06.910
going on in trying
to fit these things.

00:23:06.910 --> 00:23:08.800
So it can be a
little bit tricky.

00:23:08.800 --> 00:23:10.750
But essentially,
what they're after is

00:23:10.750 --> 00:23:15.530
trying to get a model
for the whole surface.

00:23:15.530 --> 00:23:17.680
And then once they
have the whole surface,

00:23:17.680 --> 00:23:22.720
then calculate an
overall metric as

00:23:22.720 --> 00:23:26.200
if I had an infinite number of
measurement points representing

00:23:26.200 --> 00:23:27.400
the whole surface.

00:23:27.400 --> 00:23:32.395
So I want an integrated measure,
an integrated statistic i,

00:23:32.395 --> 00:23:35.350
that captures the
total deviation

00:23:35.350 --> 00:23:39.820
from a target or total deviation
from some nominal value.

00:23:39.820 --> 00:23:43.120
And so what they're
doing here is calculating

00:23:43.120 --> 00:23:47.530
a normalized
integral of deviation

00:23:47.530 --> 00:23:52.450
of this interpolated surface,
gr is their thin plate spline,

00:23:52.450 --> 00:23:55.390
from some target value,
and then just integrating

00:23:55.390 --> 00:23:56.980
that over the surface.

00:23:56.980 --> 00:24:00.040
Total deviation
from some target.

00:24:00.040 --> 00:24:02.530
Presumably the target
would be just one value.

00:24:02.530 --> 00:24:06.130
I guess you could extend
to whatever target

00:24:06.130 --> 00:24:08.410
spatial distribution you wanted.

00:24:08.410 --> 00:24:11.140
And then they're normalizing
it by the total volume

00:24:11.140 --> 00:24:14.230
of the surface.

00:24:14.230 --> 00:24:17.800
So it's sort of like a
standard deviation over mu

00:24:17.800 --> 00:24:25.090
except what it is is total
deviation over total volume.

00:24:25.090 --> 00:24:28.220
Now, this is just
total deviation.

00:24:28.220 --> 00:24:30.490
And if I was just
summing up deviations

00:24:30.490 --> 00:24:36.940
as a measure of variation just
in some randomly sampled data

00:24:36.940 --> 00:24:39.160
set not having to
do with space, you

00:24:39.160 --> 00:24:41.330
might be worried about that.

00:24:41.330 --> 00:24:44.230
Why would you be worried
about just summing up

00:24:44.230 --> 00:24:46.540
total deviations?

00:24:46.540 --> 00:24:50.560
Positive deviations, negative
deviations, they'd cancel out.

00:24:50.560 --> 00:24:53.470
So if I was plus and minus
huge amounts but about

00:24:53.470 --> 00:24:54.980
the same amount
plus and minus, I

00:24:54.980 --> 00:24:58.340
might fool myself and think they
might cancel out and come to 0.

00:24:58.340 --> 00:24:59.990
And I'd say, oh,
everything's great.

00:24:59.990 --> 00:25:02.030
And they're really
completely different.

00:25:02.030 --> 00:25:04.130
And that's why we
often use things

00:25:04.130 --> 00:25:07.940
like squares or
absolute value to try

00:25:07.940 --> 00:25:14.030
to account for or penalize for
both plus and minus deviations.

00:25:14.030 --> 00:25:16.370
And so in the Davis
paper, one can

00:25:16.370 --> 00:25:21.260
go in and apply some other
generalized loss transformation

00:25:21.260 --> 00:25:23.610
to that deviation.

00:25:23.610 --> 00:25:27.620
So you can do a sum
of squared deviations

00:25:27.620 --> 00:25:31.370
that looks a lot more
like a standard deviation.

00:25:31.370 --> 00:25:33.650
But again, the idea here
is they want to integrate

00:25:33.650 --> 00:25:36.020
that over the whole surface.

00:25:36.020 --> 00:25:39.140
Represent all of the surface,
not just the few data

00:25:39.140 --> 00:25:41.870
points that you had,
but do that using

00:25:41.870 --> 00:25:44.810
the interpolated function.

00:25:44.810 --> 00:25:47.090
Now, there's an approximation
to this integral

00:25:47.090 --> 00:25:52.730
that they mention, which is if
I have my lost transformation,

00:25:52.730 --> 00:25:55.050
I have my actual data points.

00:25:55.050 --> 00:25:58.580
So these are my
measured data points.

00:25:58.580 --> 00:26:01.640
And I have the deviation
of that measured point

00:26:01.640 --> 00:26:06.020
and then my h might be the
square or the absolute value.

00:26:06.020 --> 00:26:09.710
If I simply sum those
for my data points,

00:26:09.710 --> 00:26:12.650
I get something close,
again, to standard deviation,

00:26:12.650 --> 00:26:14.240
a sum of squares.

00:26:14.240 --> 00:26:17.432
But there's one big difference.

00:26:17.432 --> 00:26:18.530
And that's right here.

00:26:18.530 --> 00:26:20.480
The c sub j.

00:26:20.480 --> 00:26:24.770
And essentially the
c sub j is a weight

00:26:24.770 --> 00:26:30.080
that corresponds to if I was
doing the spatial integration,

00:26:30.080 --> 00:26:33.650
I would be integrating over
these thin little patches

00:26:33.650 --> 00:26:36.478
around each of my
measurement points.

00:26:36.478 --> 00:26:38.270
It wouldn't be just
that measurement point.

00:26:38.270 --> 00:26:40.910
It would be representing the
surface near that measurement

00:26:40.910 --> 00:26:41.720
point.

00:26:41.720 --> 00:26:44.330
And the c sub j is
a weight that says,

00:26:44.330 --> 00:26:48.410
how much area on my surface
does that measurement point

00:26:48.410 --> 00:26:50.360
represent?

00:26:50.360 --> 00:26:53.300
How much vote does
that one point have?

00:26:53.300 --> 00:26:56.840
And that can be very different
in these two sampling plans.

00:26:56.840 --> 00:27:02.500
So what's neat is that if you
weight appropriately for area

00:27:02.500 --> 00:27:05.360
that the point
represents, you actually

00:27:05.360 --> 00:27:09.830
get fairly close to something
like the integration coming

00:27:09.830 --> 00:27:12.200
from a thin plate spline.

00:27:12.200 --> 00:27:12.980
OK.

00:27:12.980 --> 00:27:17.210
So there's a couple of
plots in Davis's paper

00:27:17.210 --> 00:27:19.970
where they're taking that
typical radial sampling

00:27:19.970 --> 00:27:20.610
pattern.

00:27:20.610 --> 00:27:28.850
Here's a perfectly circularly
symmetric pattern, 15%

00:27:28.850 --> 00:27:33.380
systematic variation, 2%
random superimposed on it.

00:27:33.380 --> 00:27:37.640
And then what they're
doing is looking at an SNR.

00:27:37.640 --> 00:27:41.270
This is basically just sigma
over mu of just the data

00:27:41.270 --> 00:27:42.320
points.

00:27:42.320 --> 00:27:44.870
And they're showing
what estimates

00:27:44.870 --> 00:27:49.700
over 300 different
runs of this model

00:27:49.700 --> 00:27:51.530
with different amounts
of noise each time,

00:27:51.530 --> 00:27:58.460
just different instantiations
of the random 2% noise, what

00:27:58.460 --> 00:28:02.060
do they calculate
for sigma over mu?

00:28:02.060 --> 00:28:05.570
So each time they do
300 runs, out here

00:28:05.570 --> 00:28:11.570
they do it taking 73
different measurements

00:28:11.570 --> 00:28:16.970
around the wafer with a
typical radial spatial pattern.

00:28:16.970 --> 00:28:19.860
This is the problematic
one we saw earlier.

00:28:19.860 --> 00:28:22.370
And notice there is a
spread, as you would expect,

00:28:22.370 --> 00:28:26.100
because the noise is in there.

00:28:26.100 --> 00:28:28.310
But look what happens
as you go to smaller

00:28:28.310 --> 00:28:30.110
numbers of measurements.

00:28:30.110 --> 00:28:32.750
First off, the spread
or the variance

00:28:32.750 --> 00:28:37.530
in your estimate of
sigma over mu increases.

00:28:37.530 --> 00:28:39.170
That makes sense, right?

00:28:39.170 --> 00:28:40.130
I have less data.

00:28:40.130 --> 00:28:41.990
I'm going to have more
variance in how I'm

00:28:41.990 --> 00:28:43.910
estimating standard deviation.

00:28:43.910 --> 00:28:46.820
We already know that estimating
standard deviation in general

00:28:46.820 --> 00:28:48.770
requires a lot of data.

00:28:48.770 --> 00:28:50.210
It's really a tough thing.

00:28:50.210 --> 00:28:52.310
So it's getting worse.

00:28:52.310 --> 00:28:55.730
But even worse than that,
especially for, say,

00:28:55.730 --> 00:28:58.190
five measurements or
even 13 measurements,

00:28:58.190 --> 00:29:03.890
a little bit there, is
the average value coming

00:29:03.890 --> 00:29:12.680
from repeated evaluations is
not equal to the true underlying

00:29:12.680 --> 00:29:13.470
average?

00:29:13.470 --> 00:29:16.490
In other words, there's bias.

00:29:16.490 --> 00:29:21.170
The estimator of sigma
over mu is biased.

00:29:21.170 --> 00:29:23.840
It's not a very good estimator.

00:29:23.840 --> 00:29:27.050
And as you have smaller
numbers of parameters,

00:29:27.050 --> 00:29:29.690
that unequal weighting
of the surface

00:29:29.690 --> 00:29:32.480
is biasing or fooling you.

00:29:32.480 --> 00:29:35.810
Even on average you're going
to on average be wrong.

00:29:38.550 --> 00:29:40.220
What they're showing
is if they actually

00:29:40.220 --> 00:29:44.540
do their whole thin plate
spline I stat statistic,

00:29:44.540 --> 00:29:48.620
they argue both the variance is
smaller, but more importantly

00:29:48.620 --> 00:29:49.790
is it's unbiased.

00:29:49.790 --> 00:29:54.350
Even when you get down to
small numbers of measurements,

00:29:54.350 --> 00:29:58.430
you're at least
on average right.

00:29:58.430 --> 00:30:01.700
And that's just these two plots
that are simply boiled down

00:30:01.700 --> 00:30:04.490
into this comparison
that shows as a function

00:30:04.490 --> 00:30:10.550
of the number of measurements
how good the statistic is.

00:30:10.550 --> 00:30:15.530
So the spread of the statistic
over the mean of the statistic.

00:30:15.530 --> 00:30:18.680
And showing that the
variance of the statistic

00:30:18.680 --> 00:30:23.030
itself is getting smaller with
the integration statistic.

00:30:23.030 --> 00:30:25.310
So you're basically
doing a better job

00:30:25.310 --> 00:30:31.500
of estimating with their
proposed integration statistic.

00:30:31.500 --> 00:30:33.380
Now, it's not perfect.

00:30:33.380 --> 00:30:36.590
And in fact,
conceptually, if I only

00:30:36.590 --> 00:30:38.510
have a small number
of sampling points,

00:30:38.510 --> 00:30:41.750
I can still be prone to where
those sampling points happen

00:30:41.750 --> 00:30:44.550
to lie if I have
a complex surface.

00:30:44.550 --> 00:30:49.040
And so they're also showing
an asymmetric underlying

00:30:49.040 --> 00:30:53.960
non-uniformity, this crazy shape
down here in the lower left.

00:30:53.960 --> 00:30:56.990
And imagine I'm only sampling
with five data points.

00:30:56.990 --> 00:31:02.480
Well, if those five data
points land in one orientation

00:31:02.480 --> 00:31:08.450
or if they land then
slightly offset in angle,

00:31:08.450 --> 00:31:10.640
I am spatially sampling
different parts

00:31:10.640 --> 00:31:11.820
of that surface.

00:31:11.820 --> 00:31:13.460
So what happens then?

00:31:13.460 --> 00:31:17.180
And what they basically
show is, yes, their I stat

00:31:17.180 --> 00:31:19.460
does have some sensitivity.

00:31:19.460 --> 00:31:21.590
But because you're
trying to interpolate

00:31:21.590 --> 00:31:26.630
the rest of the surface,
you do a better job.

00:31:26.630 --> 00:31:29.540
You're not completely dependent
just on that local measurement

00:31:29.540 --> 00:31:30.350
point.

00:31:30.350 --> 00:31:33.680
Both approaches are still
highly sensitive to angle,

00:31:33.680 --> 00:31:39.110
but the pure sigma over
mu is much more sensitive.

00:31:39.110 --> 00:31:45.290
You get about 20% smaller
variation with their I stat.

00:31:45.290 --> 00:31:46.370
OK.

00:31:46.370 --> 00:31:50.570
So what are the key
lessons out of this part

00:31:50.570 --> 00:31:53.840
of the case so far, this paper?

00:31:53.840 --> 00:31:56.450
I think the key
lessons are watch out

00:31:56.450 --> 00:31:58.220
for your sampling plan.

00:31:58.220 --> 00:32:03.050
And especially if any of you go
into the semiconductor industry

00:32:03.050 --> 00:32:06.440
or I guess any industry
dealing with round substrates,

00:32:06.440 --> 00:32:10.430
be careful, because you will
run into this circular sampling

00:32:10.430 --> 00:32:12.290
plan again and again.

00:32:12.290 --> 00:32:16.340
And if you're calculating
some metric that is equally

00:32:16.340 --> 00:32:19.370
weighting each of these points
where each of these points is,

00:32:19.370 --> 00:32:22.460
in fact, points off on
the edge of the wafer

00:32:22.460 --> 00:32:25.550
are representing a bigger area.

00:32:25.550 --> 00:32:29.600
But they're equally weighted
with smaller sampling points

00:32:29.600 --> 00:32:31.130
near the center of the wafer.

00:32:31.130 --> 00:32:37.340
And this is a problem that I've
seen come up again and again

00:32:37.340 --> 00:32:41.420
and again that people,
I think, are not

00:32:41.420 --> 00:32:45.930
as aware of as they ought
to be in the industry.

00:32:45.930 --> 00:32:47.616
It's easy to fix.

00:32:47.616 --> 00:32:48.540
It's easy to fix.

00:32:48.540 --> 00:32:52.010
One fix that I recommend,
that I think is the cleanest,

00:32:52.010 --> 00:32:55.820
is simply do a more
uniform sampling.

00:32:55.820 --> 00:32:59.090
That rectangular sampling
has a great benefit.

00:32:59.090 --> 00:33:02.210
How much area does
each point represent?

00:33:02.210 --> 00:33:04.100
An equal area.

00:33:04.100 --> 00:33:11.120
It's not dependent on an r
theta kind of calculation.

00:33:11.120 --> 00:33:15.890
Second is if you do have some
non-uniform spatially sampling

00:33:15.890 --> 00:33:19.760
plan, you can kind of fix it
by doing a weighted metric

00:33:19.760 --> 00:33:21.560
or a weighted regression.

00:33:21.560 --> 00:33:22.348
Yeah?

00:33:22.348 --> 00:33:24.409
AUDIENCE: Normally, you
have a different kind

00:33:24.409 --> 00:33:29.080
of sampling plan like that for,
say, the [INAUDIBLE] terms.

00:33:29.080 --> 00:33:31.130
But it would be four
ranges of the [INAUDIBLE]

00:33:31.130 --> 00:33:33.920
and four ranges of the wafer.

00:33:33.920 --> 00:33:36.980
And then we would
applicate hundreds of times

00:33:36.980 --> 00:33:40.920
over the wafer.

00:33:40.920 --> 00:33:46.100
So in that case, does that
factor then, because you're--

00:33:46.100 --> 00:33:48.560
DUANE BONING: So Nalish's
point is often, say,

00:33:48.560 --> 00:33:50.930
if you're sampling
within each chip,

00:33:50.930 --> 00:33:54.020
so here's our big wafer with
lots and lots of these chips,

00:33:54.020 --> 00:33:56.930
you might sample
spatially the same ring

00:33:56.930 --> 00:33:59.570
oscillator in different
corners of the chip

00:33:59.570 --> 00:34:02.210
and then-- since there
are multiple chips.

00:34:02.210 --> 00:34:04.940
I think, again, you've got to
be a little bit alert to what

00:34:04.940 --> 00:34:07.070
it is you're doing
with that data

00:34:07.070 --> 00:34:08.690
and how you're calculating it.

00:34:08.690 --> 00:34:12.560
But that chip by chip
kind of sampling plan

00:34:12.560 --> 00:34:16.219
has the advantage that chips
are generally rectangular.

00:34:16.219 --> 00:34:19.489
And so maybe by
accident or by free,

00:34:19.489 --> 00:34:26.360
you're usually getting a more
equally representative sampling

00:34:26.360 --> 00:34:30.770
plan that lets you fit
or calculate metrics

00:34:30.770 --> 00:34:32.822
a little bit more fairly.

00:34:32.822 --> 00:34:36.119
AUDIENCE: [INAUDIBLE] you
had mentioned the sensitivity

00:34:36.119 --> 00:34:38.480
to the amount of area cover.

00:34:38.480 --> 00:34:42.290
But a lot of times you have
more of a sensitivity to factor,

00:34:42.290 --> 00:34:49.228
depending on if SRAM is nearby
or a populated logic is nearby.

00:34:49.228 --> 00:34:50.020
DUANE BONING: Yeah.

00:34:50.020 --> 00:34:56.150
So this is kind of good
for wafer level modeling.

00:34:56.150 --> 00:34:59.450
Nalish is also pointing
out in many cases,

00:34:59.450 --> 00:35:03.600
maybe this is an SRAM and
this is logic over here,

00:35:03.600 --> 00:35:06.020
and you're sensitive
to other, what

00:35:06.020 --> 00:35:10.370
I would call, finer
grained systematic sources

00:35:10.370 --> 00:35:13.550
of variation, like
layout pattern density,

00:35:13.550 --> 00:35:17.120
proximity to other structures,
other perturbing effects.

00:35:17.120 --> 00:35:20.660
And those are extremely
interesting and fascinating

00:35:20.660 --> 00:35:21.380
to look at.

00:35:21.380 --> 00:35:24.440
And I think the
basic strategy is

00:35:24.440 --> 00:35:26.360
if you know what
those are as factors

00:35:26.360 --> 00:35:30.020
and can represent
them as factors,

00:35:30.020 --> 00:35:33.980
you can actually do some
of the kind of ANOVA

00:35:33.980 --> 00:35:37.250
and model fitting with
those as factors in addition

00:35:37.250 --> 00:35:42.830
to or separating from
spatial xy dependencies.

00:35:42.830 --> 00:35:45.860
So it's usually better if you
know what those factors are not

00:35:45.860 --> 00:35:49.010
just trying to let x and
y be the stand in for them

00:35:49.010 --> 00:35:52.070
but actually explicitly
include those things

00:35:52.070 --> 00:35:54.140
so you know what causes what.

00:35:54.140 --> 00:35:57.980
And that's a very
interesting area.

00:35:57.980 --> 00:36:02.180
Any other questions on
this spatial sampling?

00:36:02.180 --> 00:36:05.390
I think it's kind of cool
but relatively intuitive

00:36:05.390 --> 00:36:08.910
once you think about it.

00:36:08.910 --> 00:36:12.050
So the next thing I want
to do is say, OK, now I

00:36:12.050 --> 00:36:17.480
know how to be careful in
building a spatial response

00:36:17.480 --> 00:36:19.020
surface model.

00:36:19.020 --> 00:36:21.500
But what if I'm worried
about how that surface

00:36:21.500 --> 00:36:24.500
changes as a function
of process conditions.

00:36:24.500 --> 00:36:27.230
What I'd like to do is have
both process and spatial

00:36:27.230 --> 00:36:28.890
dependencies in it.

00:36:28.890 --> 00:36:31.760
I'm going to show first
off an approach that

00:36:31.760 --> 00:36:36.320
basically builds a two layered
response surface model.

00:36:36.320 --> 00:36:39.050
This is the Mozumder
and Loewenstein paper.

00:36:39.050 --> 00:36:41.180
And then I'll come
back to it and flip it

00:36:41.180 --> 00:36:46.740
around that says
in the first case,

00:36:46.740 --> 00:36:51.080
the first approach is basically
to build the model of space

00:36:51.080 --> 00:36:52.820
and then take each
of the coefficients

00:36:52.820 --> 00:36:58.010
in your spatial model,
your a3 times the xy term,

00:36:58.010 --> 00:37:01.430
and now model that a3 term
as a function of your process

00:37:01.430 --> 00:37:03.830
conditions.

00:37:03.830 --> 00:37:05.390
Got it?

00:37:05.390 --> 00:37:08.040
The second approach
does the opposite.

00:37:08.040 --> 00:37:11.780
It says, I'm going to build a
model of this particular site

00:37:11.780 --> 00:37:15.450
location as a function
of process conditions.

00:37:15.450 --> 00:37:19.520
And then I can fit a spatial
model once I have that.

00:37:19.520 --> 00:37:22.370
So it's two completely
separate or orthogonal ways

00:37:22.370 --> 00:37:24.360
of looking at the problem.

00:37:24.360 --> 00:37:28.370
So the first one here is "Model
for Semiconductor Process

00:37:28.370 --> 00:37:31.820
Optimization Using Functional
Representations of Spatial

00:37:31.820 --> 00:37:33.800
Variations in Selectivity."

00:37:33.800 --> 00:37:38.120
And they make some of the
same kinds of observations

00:37:38.120 --> 00:37:39.960
that we've talked about.

00:37:39.960 --> 00:37:43.880
But what they're
doing is basically

00:37:43.880 --> 00:37:47.480
trying to say, OK, I have to
be careful in how I calculate

00:37:47.480 --> 00:37:53.300
my non-uniformity metric, but I
want to calculate that and then

00:37:53.300 --> 00:37:55.910
understand how that
changes as the function

00:37:55.910 --> 00:37:58.760
of different process conditions.

00:37:58.760 --> 00:38:00.290
And in particular,
they're looking

00:38:00.290 --> 00:38:02.960
at a silicon nitride
wet etch looking--

00:38:02.960 --> 00:38:06.050
or silicon nitride
plasma etch, excuse me.

00:38:06.050 --> 00:38:11.510
And looking at things like
the etch traits of the silicon

00:38:11.510 --> 00:38:14.120
nitride, etch rates
of the silicon dioxide

00:38:14.120 --> 00:38:19.370
as a function of things like
gas flows process conditions.

00:38:19.370 --> 00:38:21.920
Now, they're actually worried
about two key parameters

00:38:21.920 --> 00:38:23.400
or three key parameters.

00:38:23.400 --> 00:38:25.910
One is the etch rate.

00:38:25.910 --> 00:38:27.560
Faster rates are better.

00:38:27.560 --> 00:38:29.250
Uniformity is better.

00:38:29.250 --> 00:38:31.460
But they also worry
about the relative etch

00:38:31.460 --> 00:38:36.330
rate between nitride and oxide
and they want high selectivity.

00:38:36.330 --> 00:38:38.570
They want to be able to
etch through the oxide

00:38:38.570 --> 00:38:44.210
but stop on the underlying
nitride or vice versa, I guess.

00:38:44.210 --> 00:38:46.970
This is a nitride etch.

00:38:46.970 --> 00:38:51.830
OK, so what they do here, they
take a typical spatial map,

00:38:51.830 --> 00:38:53.990
19 measurements.

00:38:53.990 --> 00:38:56.840
The paper says two
concentric hexagons.

00:38:56.840 --> 00:39:00.590
But if I look at it, they look
like two concentric octagons,

00:39:00.590 --> 00:39:01.778
because it's eight points.

00:39:01.778 --> 00:39:03.320
So I don't know
where that came from.

00:39:03.320 --> 00:39:10.850
But it's basically these
octagons plus the center point.

00:39:10.850 --> 00:39:13.490
There may have been replicates
to get to the 19 measurements.

00:39:13.490 --> 00:39:15.110
Not entirely clear.

00:39:15.110 --> 00:39:16.640
But then their
process conditions

00:39:16.640 --> 00:39:20.960
are temperature, microwave
power, pressure in the chamber,

00:39:20.960 --> 00:39:24.350
as well as nitrogen
and hydrogen flows.

00:39:24.350 --> 00:39:26.990
So it's sort of
a five factor DOE

00:39:26.990 --> 00:39:29.360
that they're going
to be exploring.

00:39:29.360 --> 00:39:32.120
And what they note is
that the removal rates

00:39:32.120 --> 00:39:36.360
do have a spatial dependence as
a function of these conditions.

00:39:36.360 --> 00:39:39.560
Things like gas flow rate
means more of the gas

00:39:39.560 --> 00:39:44.660
is coming in contact with the
center or the edge of the wafer

00:39:44.660 --> 00:39:46.880
perhaps at different
temperatures, different process

00:39:46.880 --> 00:39:47.720
conditions.

00:39:47.720 --> 00:39:52.580
So they do have an etch
rate that varies spatially

00:39:52.580 --> 00:39:55.730
across the wafer, and they
would like to model that.

00:39:55.730 --> 00:40:00.140
Now, what they're going to
do is say if I know my rate

00:40:00.140 --> 00:40:02.780
spatially across the
wafer, I am going

00:40:02.780 --> 00:40:07.250
to then calculate non-uniformity
as a derived parameter.

00:40:07.250 --> 00:40:10.460
I'm not just going to
sum up my data points.

00:40:10.460 --> 00:40:13.760
I am actually going to try
to use kind of the model

00:40:13.760 --> 00:40:16.580
and do a ratio of standard
deviation to mean.

00:40:16.580 --> 00:40:20.450
But it is derived
from the model.

00:40:20.450 --> 00:40:23.810
And so their basic idea
is to do this two level

00:40:23.810 --> 00:40:27.890
model or two layered
model of the etch rates.

00:40:27.890 --> 00:40:32.450
So what they're going to
do is say the etch rate is

00:40:32.450 --> 00:40:37.110
a function of both
spatial terms,

00:40:37.110 --> 00:40:41.000
sort of the same thing we saw
before, xy, xy, x squared,

00:40:41.000 --> 00:40:42.950
y squared.

00:40:42.950 --> 00:40:46.650
But also of processed terms.

00:40:46.650 --> 00:40:49.490
So what they would like to
be able to do is plug in

00:40:49.490 --> 00:40:52.190
and say, OK, if I know
my hydrogen flow rate

00:40:52.190 --> 00:40:56.540
and I want to look at some
particular xy location,

00:40:56.540 --> 00:40:59.510
I would like to know at
this process condition

00:40:59.510 --> 00:41:04.340
and at this spatial location
what is my etch rate.

00:41:04.340 --> 00:41:06.170
So what they have
to do is basically

00:41:06.170 --> 00:41:09.710
fit this two layered
model that has

00:41:09.710 --> 00:41:17.030
both dependencies on the
spatial terms and on the process

00:41:17.030 --> 00:41:19.190
terms in it.

00:41:19.190 --> 00:41:21.350
And another way of looking
at this is essentially

00:41:21.350 --> 00:41:24.320
what they're doing is, as
I mentioned earlier, they

00:41:24.320 --> 00:41:26.480
are building a
spatial model where

00:41:26.480 --> 00:41:30.110
each coefficient, sort
of an a1, is also then

00:41:30.110 --> 00:41:32.710
a function of the
process conditions.

00:41:35.540 --> 00:41:38.300
So here's an example
of a regression surface

00:41:38.300 --> 00:41:39.710
that comes out.

00:41:39.710 --> 00:41:41.600
This is the spatial regression.

00:41:41.600 --> 00:41:45.200
Again, as a function of xy
and so on for the removal

00:41:45.200 --> 00:41:48.110
rate or etch rate
of silicon nitride.

00:41:48.110 --> 00:41:51.920
And then these coefficients
themselves, they go in

00:41:51.920 --> 00:41:54.350
and they build a model
of those coefficients

00:41:54.350 --> 00:41:58.987
as a function of the process.

00:41:58.987 --> 00:42:01.320
And when you do that, you can
get some pretty good fits.

00:42:01.320 --> 00:42:01.970
Look at these.

00:42:01.970 --> 00:42:09.110
These are square root of in the
range 0.93, 0.90 is the worst,

00:42:09.110 --> 00:42:16.320
0.99 for each of the
significance for each of those

00:42:16.320 --> 00:42:16.820
terms.

00:42:16.820 --> 00:42:21.980
In other words, this
is the spatial model

00:42:21.980 --> 00:42:26.720
for that c sub 2n whatever.

00:42:26.720 --> 00:42:29.590
C sub 2x.

00:42:29.590 --> 00:42:32.810
I guess c sub 2y yn.

00:42:32.810 --> 00:42:34.640
But the goodness of
fit, then, of doing

00:42:34.640 --> 00:42:37.610
that spatial coordinate model
is a function of the process.

00:42:37.610 --> 00:42:40.250
It's a pretty good fit.

00:42:40.250 --> 00:42:43.340
And then what they
do is say, OK, I'm

00:42:43.340 --> 00:42:47.150
going to do polynomial
regression, as we saw before.

00:42:47.150 --> 00:42:50.630
They actually find
kind of a nasty thing.

00:42:50.630 --> 00:42:53.690
To get good fits, they
had to go to cubic fits.

00:42:57.340 --> 00:42:58.592
That may be.

00:42:58.592 --> 00:43:00.550
Just a couple of other
points of interest here.

00:43:00.550 --> 00:43:03.250
They use the Latin hypercube
sampling rather than

00:43:03.250 --> 00:43:06.880
a central composite sampling.

00:43:06.880 --> 00:43:08.740
This is a really
cute sampling plan.

00:43:08.740 --> 00:43:13.000
Did I describe this Latin
hypercube sampling plan before?

00:43:13.000 --> 00:43:13.750
I think I did.

00:43:13.750 --> 00:43:16.120
But basically this
was saying, OK,

00:43:16.120 --> 00:43:19.690
if I have a space,
some x1 parameter

00:43:19.690 --> 00:43:21.820
and some x2
parameter, and I only

00:43:21.820 --> 00:43:25.720
have five data points
I'm going to allow myself

00:43:25.720 --> 00:43:29.380
to sample, what you do is
you basically divide both

00:43:29.380 --> 00:43:35.140
your coordinates or dimensions
up into five spaces, five

00:43:35.140 --> 00:43:35.755
equal spaces.

00:43:39.200 --> 00:43:43.430
And then you take a
sample point and make sure

00:43:43.430 --> 00:43:45.290
that every one of your
five sample points

00:43:45.290 --> 00:43:50.420
uniquely follows in
one row and one column.

00:43:50.420 --> 00:43:53.600
So you might get
something like this.

00:43:53.600 --> 00:43:55.610
What am I missing?

00:43:55.610 --> 00:43:57.680
Maybe that.

00:43:57.680 --> 00:44:00.200
And what's cool there
is if you project down

00:44:00.200 --> 00:44:03.770
onto either x1
coordinate, now I've

00:44:03.770 --> 00:44:09.080
got sort of five data points
spread throughout that space.

00:44:09.080 --> 00:44:11.060
Or if I project
down onto the x2,

00:44:11.060 --> 00:44:12.980
I similarly have
five coordinates

00:44:12.980 --> 00:44:14.992
spread throughout that space.

00:44:14.992 --> 00:44:17.450
You have to be a little bit
careful, because that algorithm

00:44:17.450 --> 00:44:19.760
could have
accidentally given you

00:44:19.760 --> 00:44:21.660
a spatial correlation
in your sampling.

00:44:21.660 --> 00:44:25.400
So for example,
those points follow

00:44:25.400 --> 00:44:27.320
my rule of one row, one column.

00:44:27.320 --> 00:44:29.688
But that has an unintentional
correlation in it.

00:44:29.688 --> 00:44:31.730
So there's a part of the
Latin hypercube sampling

00:44:31.730 --> 00:44:36.792
algorithm that swaps rows
and columns to avoid that.

00:44:36.792 --> 00:44:38.000
So that's what they do there.

00:44:38.000 --> 00:44:39.542
Actually, it's good
that you're aware

00:44:39.542 --> 00:44:41.990
of Latin hypercube
sampling, because especially

00:44:41.990 --> 00:44:46.160
for constrained cases where
you know how many data points

00:44:46.160 --> 00:44:52.400
you can sample, it's actually
quite an interesting approach.

00:44:52.400 --> 00:44:52.940
OK.

00:44:52.940 --> 00:44:56.540
Then what they go in
and do is build out

00:44:56.540 --> 00:44:59.990
of these fundamental rate
functions that they fit,

00:44:59.990 --> 00:45:03.710
they built derived functions.

00:45:03.710 --> 00:45:06.770
These are derived.

00:45:06.770 --> 00:45:09.140
It says if I know the
removal rate or etch

00:45:09.140 --> 00:45:12.680
rate at different locations,
now I can fold that together

00:45:12.680 --> 00:45:16.760
to calculate an
aggregate uniformity

00:45:16.760 --> 00:45:19.760
kind of using some
of these ideas

00:45:19.760 --> 00:45:21.230
we were talking about before.

00:45:21.230 --> 00:45:25.730
I'm not simply calculating from
my raw data uniformity number

00:45:25.730 --> 00:45:29.720
and then building a separate
model, a separate response

00:45:29.720 --> 00:45:33.920
surface model, a separate
polynomial model of uniformity.

00:45:33.920 --> 00:45:37.040
Instead I'm getting close
to the physics as I can

00:45:37.040 --> 00:45:40.580
and build this great
model and then recognize

00:45:40.580 --> 00:45:45.560
that uniformity is simply
a calculated function

00:45:45.560 --> 00:45:50.000
across that underlying
rate function.

00:45:50.000 --> 00:45:57.140
And they happen to be
using something like a rate

00:45:57.140 --> 00:45:59.420
sigma divided by rate mean.

00:45:59.420 --> 00:46:03.140
So they're integrating
that over their space.

00:46:03.140 --> 00:46:05.450
By the way, this little
n simply indicates

00:46:05.450 --> 00:46:08.630
that they're calculating
separately or building models

00:46:08.630 --> 00:46:14.240
separately for the oxide rate
and the nitride etch rate.

00:46:14.240 --> 00:46:16.160
They both vary spatially.

00:46:16.160 --> 00:46:18.410
So they built these two models.

00:46:18.410 --> 00:46:20.510
They do the same thing
for this selectivity.

00:46:20.510 --> 00:46:23.570
Again, this is the
ratio of etch rate

00:46:23.570 --> 00:46:26.660
of nitride to silicon dioxide.

00:46:26.660 --> 00:46:30.260
And that also is
a derived model.

00:46:33.270 --> 00:46:36.270
And notice that there's
something really important

00:46:36.270 --> 00:46:42.540
about these derived
models and why they help.

00:46:42.540 --> 00:46:45.240
May be less obvious,
although we've already

00:46:45.240 --> 00:46:48.360
talked about it here with
the uniformity metric.

00:46:48.360 --> 00:46:52.650
Maybe they'll focus first
here on the selectivity.

00:46:52.650 --> 00:46:58.150
Is selectivity a
linear function?

00:46:58.150 --> 00:46:58.650
No.

00:46:58.650 --> 00:46:59.670
It's kind of nasty.

00:46:59.670 --> 00:47:03.870
It's a ratio of two other rates.

00:47:03.870 --> 00:47:06.790
Why would that
necessarily be linear?

00:47:06.790 --> 00:47:09.780
This is kind of a
complicated functional form

00:47:09.780 --> 00:47:13.590
that if you were trying to
directly model selectivity,

00:47:13.590 --> 00:47:16.920
it might be very
difficult to capture

00:47:16.920 --> 00:47:20.880
by chance that complicated
functional form.

00:47:20.880 --> 00:47:24.090
And even worse, think
about the computations

00:47:24.090 --> 00:47:27.580
that go in the calculation
of standard deviation.

00:47:27.580 --> 00:47:30.060
Remember standard deviation?

00:47:30.060 --> 00:47:34.940
It's the square root of
the sum of some x sub

00:47:34.940 --> 00:47:44.940
i minus the mean squared
divided by 1 over n minus 1.

00:47:44.940 --> 00:47:46.320
So you got squaring.

00:47:46.320 --> 00:47:48.240
You got a square root in there.

00:47:48.240 --> 00:47:50.670
That's a very
nonlinear operation.

00:47:50.670 --> 00:47:56.080
Why would a linear model be
very good at capturing that?

00:47:56.080 --> 00:47:59.260
In fact, if you tried
to do it directly,

00:47:59.260 --> 00:48:04.770
you might get a
non-uniformity that

00:48:04.770 --> 00:48:07.450
needed not only cubic
terms, but who knows,

00:48:07.450 --> 00:48:10.230
fourth order terms, fifth order
terms, which would be quite

00:48:10.230 --> 00:48:12.240
complicated to actually fit it.

00:48:12.240 --> 00:48:15.510
And what they're showing in
the paper is an example where

00:48:15.510 --> 00:48:20.550
a relatively simple
rate function,

00:48:20.550 --> 00:48:22.890
they're doing it here also
as a function of, say,

00:48:22.890 --> 00:48:28.890
theta and r, not just, say,
xy, but that transformation

00:48:28.890 --> 00:48:30.370
is pretty simple.

00:48:30.370 --> 00:48:33.480
If you then feed that into
the uniformity calculation,

00:48:33.480 --> 00:48:36.330
the expansions of
squares and square roots,

00:48:36.330 --> 00:48:39.400
you get kind of
almost, if you will,

00:48:39.400 --> 00:48:45.860
for free a functional
form that is

00:48:45.860 --> 00:48:48.800
quite complicated in
capturing the true functional

00:48:48.800 --> 00:48:51.620
dependencies of uniformity.

00:48:51.620 --> 00:48:54.020
That's the important
idea here is you

00:48:54.020 --> 00:48:58.062
have a simple, direct,
underlying model of rate.

00:48:58.062 --> 00:49:00.020
And then if you've got
a complicated functional

00:49:00.020 --> 00:49:02.750
dependence, like
non-uniformity metric on that,

00:49:02.750 --> 00:49:07.370
you calculate that from the
simpler underlying model.

00:49:07.370 --> 00:49:08.480
I like that.

00:49:08.480 --> 00:49:10.070
I think that that's nice.

00:49:10.070 --> 00:49:12.470
That's a key idea in this paper.

00:49:12.470 --> 00:49:15.740
Then they go on and do, once
they've got these models,

00:49:15.740 --> 00:49:19.400
they do multiple
objective optimization

00:49:19.400 --> 00:49:23.690
trying to minimize or
maximize selectivity, minimize

00:49:23.690 --> 00:49:27.890
their non-uniformity
value, maximize the rate.

00:49:27.890 --> 00:49:29.660
They've got some constraints.

00:49:29.660 --> 00:49:34.280
They go in and kind of
do a multiple objective.

00:49:34.280 --> 00:49:39.620
Come up with some optimum values
and show how much improvement

00:49:39.620 --> 00:49:41.720
they get.

00:49:41.720 --> 00:49:45.170
But the key idea, I think,
was that contribution

00:49:45.170 --> 00:49:48.620
of this notion that you
want to start with something

00:49:48.620 --> 00:49:51.150
that's simpler model.

00:49:51.150 --> 00:49:54.890
Then if you've got a complicated
optimization objective,

00:49:54.890 --> 00:49:57.740
use the simple model, build
the complicated objective

00:49:57.740 --> 00:50:03.550
or complicated derived
function, and then use that.

00:50:03.550 --> 00:50:05.433
So any questions on that?

00:50:05.433 --> 00:50:06.600
Does that seem pretty clear?

00:50:09.320 --> 00:50:13.590
What I'm going to show next is
the last piece of this puzzle,

00:50:13.590 --> 00:50:14.780
which is--

00:50:14.780 --> 00:50:17.104
yes, before I do that?

00:50:17.104 --> 00:50:18.080
AUDIENCE: [INAUDIBLE]

00:50:18.080 --> 00:50:18.980
DUANE BONING: Yes.

00:50:18.980 --> 00:50:23.846
AUDIENCE: It's different
to [INAUDIBLE] when you

00:50:23.846 --> 00:50:26.490
compare rectangle to circle.

00:50:26.490 --> 00:50:30.646
So the circle has the minimal
area to perimeter ratio,

00:50:30.646 --> 00:50:33.470
while the rectangle
has the maximum.

00:50:33.470 --> 00:50:37.220
So it's most likely when we have
a rectangle we most likely have

00:50:37.220 --> 00:50:40.988
to have a uniform sampling.

00:50:40.988 --> 00:50:42.530
Is it correct to
look at it that way?

00:50:47.283 --> 00:50:48.950
DUANE BONING: I don't
see a relationship

00:50:48.950 --> 00:50:51.680
between the perimeter to area.

00:50:51.680 --> 00:50:56.630
AUDIENCE: For a given perimeter.

00:50:56.630 --> 00:50:58.700
DUANE BONING: And the
reason I'm not sure

00:50:58.700 --> 00:51:03.590
that works is if I
have-- here's my wafer.

00:51:03.590 --> 00:51:06.440
In the rectangular case, it's
easy to sort of tessellate

00:51:06.440 --> 00:51:12.710
the space and cover the space
with non-overlapping regions

00:51:12.710 --> 00:51:15.410
that each data point represents.

00:51:15.410 --> 00:51:20.030
In order to do that with
the circular sampling plan,

00:51:20.030 --> 00:51:21.845
each one of these
is not a circle.

00:51:24.800 --> 00:51:28.100
The way you have to sort of
do it is each one of these

00:51:28.100 --> 00:51:35.285
is this arc to cover the
space with non-overlapping.

00:51:35.285 --> 00:51:35.910
Now, who knows?

00:51:35.910 --> 00:51:38.790
If you allow yourself sort
of overlapping coverage,

00:51:38.790 --> 00:51:42.540
maybe what you're talking
about makes sense.

00:51:42.540 --> 00:51:46.800
But I don't think it's really
a perimeter that each area--

00:51:46.800 --> 00:51:49.200
I think it really
is area, just purely

00:51:49.200 --> 00:51:52.170
area that it's representing.

00:51:52.170 --> 00:51:55.710
But I haven't thought
about the perimeter issue.

00:51:55.710 --> 00:51:58.470
It's kind of interesting.

00:51:58.470 --> 00:52:00.630
And by the way, this is
the kind of calculation

00:52:00.630 --> 00:52:02.940
you would do back
in spatial modeling

00:52:02.940 --> 00:52:06.900
to figure out as a
function of your xy

00:52:06.900 --> 00:52:08.610
coordinate for your
measurement point

00:52:08.610 --> 00:52:11.280
how much area that
ought to represent.

00:52:11.280 --> 00:52:13.410
You're basically
dividing it up and you

00:52:13.410 --> 00:52:15.960
have to kind of do
those calculations.

00:52:19.910 --> 00:52:20.780
OK.

00:52:20.780 --> 00:52:24.380
The last thing I want to show
is this paper by Guo and Sachs.

00:52:24.380 --> 00:52:26.720
And again, I remind
you Elliot Sachs

00:52:26.720 --> 00:52:29.150
is a professor here in
mechanical engineering

00:52:29.150 --> 00:52:33.200
who two lives ago
did a lot of work

00:52:33.200 --> 00:52:34.880
in semiconductor
process control.

00:52:38.390 --> 00:52:41.810
His life after that
was 3D printing.

00:52:41.810 --> 00:52:46.290
And his current life
is now solar energy.

00:52:46.290 --> 00:52:49.130
So he's made some
wonderful contributions

00:52:49.130 --> 00:52:50.090
and some big moves.

00:52:50.090 --> 00:52:53.840
And I actually like
this paper quite a lot.

00:52:53.840 --> 00:52:56.630
The basic idea in this, it's
looking at this issue, again,

00:52:56.630 --> 00:53:00.260
of spatial uniformity, how one
does modeling, and then uses

00:53:00.260 --> 00:53:04.190
that for optimization and
control of uniformity.

00:53:04.190 --> 00:53:08.120
And basically, here
he builds on, I think,

00:53:08.120 --> 00:53:12.980
he builds on this notion that
we learned from the Mozumder

00:53:12.980 --> 00:53:17.840
and Loewenstein paper, which is
build simple underlying models

00:53:17.840 --> 00:53:20.720
and then combine them and
use them to solve the larger

00:53:20.720 --> 00:53:21.800
problems.

00:53:21.800 --> 00:53:25.190
But keep it as simple
as you possibly can.

00:53:25.190 --> 00:53:29.480
And he suggests,
together with Andy Guo,

00:53:29.480 --> 00:53:32.480
he suggests that we
should flip around

00:53:32.480 --> 00:53:35.330
what Loewenstein
and Mozumder did

00:53:35.330 --> 00:53:41.450
and actually build models for
each spatial location first.

00:53:41.450 --> 00:53:43.497
Don't fit the
spatial model first

00:53:43.497 --> 00:53:45.080
and then take that
coefficient and try

00:53:45.080 --> 00:53:47.240
to build a model
of that coefficient

00:53:47.240 --> 00:53:49.340
as a function of process.

00:53:49.340 --> 00:53:52.520
Instead just look at
that one spatial location

00:53:52.520 --> 00:53:56.570
and build a model of that
spatial location's response

00:53:56.570 --> 00:53:59.420
as a function of the
process conditions.

00:53:59.420 --> 00:54:04.190
And now I can combine that to
other spatial derived things,

00:54:04.190 --> 00:54:08.240
like fitting a spatial model
or not necessarily even fitting

00:54:08.240 --> 00:54:11.030
a spatial model, but
now I can basically

00:54:11.030 --> 00:54:15.170
use spatial information,
not lose it.

00:54:15.170 --> 00:54:17.540
I still know the left
side or the right side

00:54:17.540 --> 00:54:20.510
is higher or lower.

00:54:20.510 --> 00:54:23.730
What happens when you calculate
standard deviation over mu?

00:54:23.730 --> 00:54:26.450
You lose information of
left side or right side

00:54:26.450 --> 00:54:27.920
was higher or lower.

00:54:27.920 --> 00:54:30.800
You just boil it down
to they were different.

00:54:30.800 --> 00:54:32.690
He says keep the sites.

00:54:32.690 --> 00:54:33.920
Keep the sites.

00:54:33.920 --> 00:54:40.430
Use that information to
drive towards an improvement.

00:54:40.430 --> 00:54:42.720
Keep the simple
model, build those,

00:54:42.720 --> 00:54:44.340
and then you can combine them.

00:54:44.340 --> 00:54:46.440
So let's look and
see how that works.

00:54:46.440 --> 00:54:47.590
Here's the basic idea.

00:54:50.120 --> 00:54:54.050
Let's say I was measuring
three different locations

00:54:54.050 --> 00:54:58.100
on the wafer, y1, y2, y3.

00:54:58.100 --> 00:55:00.140
And I have two
equipment settings.

00:55:00.140 --> 00:55:02.990
So two different
process parameters.

00:55:02.990 --> 00:55:06.800
What you're going to do here
is build a response surface

00:55:06.800 --> 00:55:09.260
model and a DOE.

00:55:09.260 --> 00:55:12.080
And there's two different
approaches one can take.

00:55:12.080 --> 00:55:16.280
One can take what I'll
called the single response

00:55:16.280 --> 00:55:18.320
surface, the classical
approach that

00:55:18.320 --> 00:55:21.170
says I'm looking at different
combinations of my input

00:55:21.170 --> 00:55:22.220
parameters.

00:55:22.220 --> 00:55:24.200
I measure my three points.

00:55:24.200 --> 00:55:27.830
I've got three data points I can
then calculate for that process

00:55:27.830 --> 00:55:29.840
condition, that run number one.

00:55:29.840 --> 00:55:32.480
I can calculate the standard
deviation across those three

00:55:32.480 --> 00:55:35.630
data points, divide by the mean
of those three data points.

00:55:35.630 --> 00:55:38.630
That gives me what the
uniformity spatially

00:55:38.630 --> 00:55:40.910
was for that run.

00:55:40.910 --> 00:55:43.400
I can then repeat that for
all of my different process

00:55:43.400 --> 00:55:44.910
conditions.

00:55:44.910 --> 00:55:48.530
And then I can build a
single response surface

00:55:48.530 --> 00:55:52.820
at the end, SRS, Single
Response Surface,

00:55:52.820 --> 00:55:58.730
that is the response surface of
the response of non-uniformity

00:55:58.730 --> 00:56:02.760
as a function of my
process conditions.

00:56:02.760 --> 00:56:04.820
That's a classic approach,
and you will still

00:56:04.820 --> 00:56:06.620
run into that in many papers.

00:56:06.620 --> 00:56:09.230
We didn't see it in
the Loewenstein paper.

00:56:09.230 --> 00:56:11.610
They used a slightly
different approach.

00:56:11.610 --> 00:56:13.580
But this is a classic approach.

00:56:13.580 --> 00:56:18.140
And what he says is
no, don't do that.

00:56:18.140 --> 00:56:20.300
Do not do that.

00:56:20.300 --> 00:56:25.910
Instead use the same design
of experiments, same data,

00:56:25.910 --> 00:56:29.240
but do something that I think
is a little bit smarter, which

00:56:29.240 --> 00:56:32.090
says take your different site.

00:56:32.090 --> 00:56:33.890
This is a site one.

00:56:33.890 --> 00:56:35.630
I'm always measuring
the left side.

00:56:35.630 --> 00:56:36.830
Maybe this is the center.

00:56:36.830 --> 00:56:38.720
This is the right
side of the wafer.

00:56:38.720 --> 00:56:44.150
And build a response surface
for that site response,

00:56:44.150 --> 00:56:46.700
maybe it's the thickness or
the etch rate or whatever,

00:56:46.700 --> 00:56:51.110
as a function of the
process conditions.

00:56:51.110 --> 00:56:53.750
Do the same for all of
the sites that you've got.

00:56:53.750 --> 00:56:59.150
Build separate multiple
response surfaces.

00:56:59.150 --> 00:57:02.120
And now you know how each site
responds for whatever process

00:57:02.120 --> 00:57:03.710
conditions you want.

00:57:03.710 --> 00:57:06.020
And now you can combine them
if you want a uniformity

00:57:06.020 --> 00:57:07.850
metric to see which is better.

00:57:07.850 --> 00:57:09.350
But you can also
do smarter things

00:57:09.350 --> 00:57:11.090
like try to balance them.

00:57:11.090 --> 00:57:13.610
Get the left side back up
to match the right side.

00:57:13.610 --> 00:57:15.440
Do other sorts of things.

00:57:15.440 --> 00:57:18.950
OK, let's see how that works.

00:57:18.950 --> 00:57:21.740
His basic point here is
very similar to the Mozumder

00:57:21.740 --> 00:57:26.840
and Loewenstein, which
is the single response

00:57:26.840 --> 00:57:31.580
surface is a very tough
job to directly model

00:57:31.580 --> 00:57:35.390
in one response surface, this
highly nonlinear sigma over mu.

00:57:35.390 --> 00:57:38.780
And in fact, if I try to
build a second order model,

00:57:38.780 --> 00:57:41.630
you often need a
second order model

00:57:41.630 --> 00:57:45.180
as a function of the
process conditions,

00:57:45.180 --> 00:57:48.740
which means lots of sampling
in your multiple levels

00:57:48.740 --> 00:57:53.090
in your process conditions,
at least three, maybe more.

00:57:53.090 --> 00:57:55.460
And you get a very
complicated model.

00:57:55.460 --> 00:57:59.930
What he also shows
is that very often

00:57:59.930 --> 00:58:02.510
with the multiple
response surface model,

00:58:02.510 --> 00:58:08.750
you can have much simpler lower
order models for each site.

00:58:08.750 --> 00:58:11.030
And in fact, what
he will do is show

00:58:11.030 --> 00:58:15.890
if I just build a linear model
for each site as a function

00:58:15.890 --> 00:58:18.320
of my process
conditions, meaning I

00:58:18.320 --> 00:58:22.160
actually need fewer
levels in my DOE.

00:58:22.160 --> 00:58:28.010
I can actually get simple linear
models for my individual sites.

00:58:28.010 --> 00:58:30.530
And then when I
functionally calculate

00:58:30.530 --> 00:58:34.550
the mu, the mean of those
three, that's a simple formula,

00:58:34.550 --> 00:58:38.690
and standard deviation over
mu is a simple formula.

00:58:38.690 --> 00:58:43.970
But embedded in it is the
right nonlinear functional form

00:58:43.970 --> 00:58:45.860
for sigma over mu.

00:58:45.860 --> 00:58:49.700
And so you get this
non-linearity out

00:58:49.700 --> 00:58:52.970
of very simple linear models.

00:58:52.970 --> 00:58:55.040
And that's the key idea.

00:58:55.040 --> 00:58:58.070
Similar to Mozumder
and Loewenstein.

00:58:58.070 --> 00:58:59.990
Build the simple
model and then use it

00:58:59.990 --> 00:59:02.870
and combine it in whatever
complicated functional form

00:59:02.870 --> 00:59:04.670
you have.

00:59:04.670 --> 00:59:06.890
So he argues that
you can actually

00:59:06.890 --> 00:59:09.020
get a smaller number of data.

00:59:09.020 --> 00:59:12.440
Not just use the same data in
a smarter way, but actually

00:59:12.440 --> 00:59:17.370
sample less in your process
conditions in many cases.

00:59:17.370 --> 00:59:19.700
So you get a savings in the DOE.

00:59:19.700 --> 00:59:22.250
Another very nice
advantage that's

00:59:22.250 --> 00:59:26.180
articulated in the paper, I'm
not going to go into here,

00:59:26.180 --> 00:59:30.770
is that each of those
models of the process

00:59:30.770 --> 00:59:34.550
is a nice, simple linear model.

00:59:34.550 --> 00:59:37.910
And you can use simple
linear models with the cycle

00:59:37.910 --> 00:59:42.620
to cycle run by run control
that Dave Hardt talked

00:59:42.620 --> 00:59:44.540
about a couple of lectures ago.

00:59:44.540 --> 00:59:46.880
You can basically
take that as a model,

00:59:46.880 --> 00:59:50.630
adapt that model rapidly
to changing process drifts

00:59:50.630 --> 00:59:55.070
or conditions in your equipment,
use that updated model in sort

00:59:55.070 --> 00:59:59.090
of a PI or PID kind
of fashion, and use

00:59:59.090 --> 01:00:02.180
that to improve the
selection or pick

01:00:02.180 --> 01:00:04.220
the selection of the
right process condition

01:00:04.220 --> 01:00:06.230
for the next wafer run.

01:00:06.230 --> 01:00:09.410
It's a lot easier to use this
kind of a model for cycle

01:00:09.410 --> 01:00:11.010
to cycle control.

01:00:11.010 --> 01:00:14.090
He also has some other
very nice points.

01:00:14.090 --> 01:00:17.120
He does a bit of analysis in
the paper saying which one

01:00:17.120 --> 01:00:20.610
is less susceptible to noise.

01:00:20.610 --> 01:00:23.840
It turns out that the site
models kind of average

01:00:23.840 --> 01:00:26.510
out noise a little bit better
than putting everything

01:00:26.510 --> 01:00:28.250
into signal to noise.

01:00:28.250 --> 01:00:33.170
And then the last point is a
really important one, which

01:00:33.170 --> 01:00:35.690
is if you get this
really complicated

01:00:35.690 --> 01:00:39.860
functional cubic
model that tells you

01:00:39.860 --> 01:00:42.920
how non-uniformity changes
as a function of your process

01:00:42.920 --> 01:00:47.930
condition, do you as the process
engineer have any idea really

01:00:47.930 --> 01:00:51.260
intuitively what will
happen if you change a knob

01:00:51.260 --> 01:00:54.260
or what's happening
spatially across your wafer?

01:00:54.260 --> 01:00:57.620
His argument is you've lost
a lot of that information.

01:00:57.620 --> 01:00:59.930
If you actually have
your site models,

01:00:59.930 --> 01:01:02.120
you can build that
full spatial map,

01:01:02.120 --> 01:01:05.300
see how each site is changing
as you change your process

01:01:05.300 --> 01:01:08.830
condition, and actually see--

01:01:08.830 --> 01:01:10.940
build some process
knowledge, see

01:01:10.940 --> 01:01:14.280
what's going on much
closer to the process.

01:01:14.280 --> 01:01:16.860
So here's a couple of examples.

01:01:16.860 --> 01:01:19.130
This relates to,
again, the point

01:01:19.130 --> 01:01:22.490
that the complexity of
the uniformity metric

01:01:22.490 --> 01:01:25.620
has a lot in it and you
actually lose information.

01:01:25.620 --> 01:01:28.160
So here's the point where
if I measure the thickness

01:01:28.160 --> 01:01:35.180
at the left site, thickness
at the right site, so red

01:01:35.180 --> 01:01:41.000
is my left and blue here are
my right measurement points.

01:01:41.000 --> 01:01:45.020
Let's say I have
simple linear responses

01:01:45.020 --> 01:01:49.870
for how those two sides change
as a function of the input

01:01:49.870 --> 01:01:51.340
parameter.

01:01:51.340 --> 01:01:53.800
They really do change linearly.

01:01:53.800 --> 01:01:56.450
Notice what happens,
in one case,

01:01:56.450 --> 01:02:00.320
the left side is much
thicker than the right

01:02:00.320 --> 01:02:01.840
and in the other
process condition,

01:02:01.840 --> 01:02:03.940
the right side is much
thicker than the left.

01:02:03.940 --> 01:02:07.150
If you actually just directly
do a single response surface

01:02:07.150 --> 01:02:12.550
kind of idea or do our
calculation of sigma over mu,

01:02:12.550 --> 01:02:17.230
you can easily get a
little bit fooled here.

01:02:17.230 --> 01:02:20.440
They're both equal in terms
of a uniformity value,

01:02:20.440 --> 01:02:23.620
but you've lost track just
looking at these values

01:02:23.620 --> 01:02:24.620
by themselves.

01:02:24.620 --> 01:02:28.780
You don't know what's
going on on the wafer.

01:02:28.780 --> 01:02:30.520
So looking back
here, you can see

01:02:30.520 --> 01:02:31.930
what's going on in the wafer.

01:02:31.930 --> 01:02:34.960
The second point is if
one wants to actually go

01:02:34.960 --> 01:02:38.680
in and do a medium
input and interpolate

01:02:38.680 --> 01:02:41.320
if I have these site
models, I could actually

01:02:41.320 --> 01:02:45.400
project and guess
and say, I think

01:02:45.400 --> 01:02:47.380
that if I were to
run at the middle,

01:02:47.380 --> 01:02:48.970
I would have very
good uniformity.

01:02:48.970 --> 01:02:51.350
The thickness would be
the same in both cases.

01:02:51.350 --> 01:02:55.120
And if I calculate a value based
on these underlying models,

01:02:55.120 --> 01:02:57.310
I can actually
project what happens

01:02:57.310 --> 01:02:59.590
at the intermediate values.

01:02:59.590 --> 01:03:03.760
This would suggest for a control
problem or an optimization

01:03:03.760 --> 01:03:06.400
try the center point.

01:03:06.400 --> 01:03:09.010
Whereas if all I had were
these two data points,

01:03:09.010 --> 01:03:11.290
I would say they're the same.

01:03:11.290 --> 01:03:13.870
I can't really
improve things much.

01:03:13.870 --> 01:03:15.050
Yeah?

01:03:15.050 --> 01:03:17.655
AUDIENCE: Left or right
with respect to [INAUDIBLE]

01:03:17.655 --> 01:03:22.343
if you have a wafer and you
have a knot and then [INAUDIBLE]

01:03:22.343 --> 01:03:26.470
wafer is in a certain way
with respect to the notch,

01:03:26.470 --> 01:03:28.870
but you still have a reticle.

01:03:28.870 --> 01:03:31.390
And that can be with any
direction with respect

01:03:31.390 --> 01:03:32.020
to the notch.

01:03:32.020 --> 01:03:34.240
So any information
that you get, you

01:03:34.240 --> 01:03:39.353
would never be sure for
the friendships how--

01:03:39.353 --> 01:03:41.770
DUANE BONING: Yeah, I'm not
projecting down to chip level.

01:03:41.770 --> 01:03:43.790
I'm thinking here wafer level.

01:03:43.790 --> 01:03:45.790
And I think you always
know-- you can always

01:03:45.790 --> 01:03:50.170
refer to something with the
wafer notch or the wafer flat.

01:03:50.170 --> 01:03:52.000
So I think that's--

01:03:52.000 --> 01:03:55.150
right and left, these
are conceptual here,

01:03:55.150 --> 01:03:57.580
but I think they can
be very well defined.

01:03:57.580 --> 01:04:01.270
If you are also worried
about repeated chip scale

01:04:01.270 --> 01:04:03.070
things coming from
lithography fields,

01:04:03.070 --> 01:04:08.390
that's an extra layer
of spatial concerns.

01:04:08.390 --> 01:04:08.890
OK.

01:04:08.890 --> 01:04:11.620
And then this is essentially
the implications for control

01:04:11.620 --> 01:04:15.280
are pretty much what
I was just describing.

01:04:15.280 --> 01:04:20.800
If you just use sort of
the high and the low,

01:04:20.800 --> 01:04:23.290
and this is your
intended input, you

01:04:23.290 --> 01:04:25.990
can now predict a
little bit better

01:04:25.990 --> 01:04:30.260
what your actual output would
be with these simplified cases.

01:04:30.260 --> 01:04:33.520
So in the paper,
they do some examples

01:04:33.520 --> 01:04:36.790
actually using some experimental
data that they generated.

01:04:36.790 --> 01:04:39.370
This was for the low
pressure chemical vapor

01:04:39.370 --> 01:04:41.500
deposition of polysilicon.

01:04:41.500 --> 01:04:46.060
And it has spatial uniformity
in it not just on the wafer.

01:04:46.060 --> 01:04:49.990
Actually what they're doing
here is spatial uniformity

01:04:49.990 --> 01:04:52.000
across the tube.

01:04:52.000 --> 01:04:54.610
So as a function
of wafer position,

01:04:54.610 --> 01:04:59.260
these are big multi-wafer tubes
that might actually process 24

01:04:59.260 --> 01:05:03.040
or 48 or 96 wafers all at once.

01:05:03.040 --> 01:05:04.630
And there's a gas flow.

01:05:04.630 --> 01:05:07.750
There's an injector for
the gas, and the gas flows.

01:05:07.750 --> 01:05:11.330
We've got a center
injector, source injector.

01:05:11.330 --> 01:05:13.720
The gas flows are
somewhat non-uniform.

01:05:13.720 --> 01:05:17.500
And you may, in fact, have
systematic spatial dependencies

01:05:17.500 --> 01:05:22.540
as a function of temperatures,
gas flows, and so on.

01:05:22.540 --> 01:05:24.670
And what they basically
went in and did

01:05:24.670 --> 01:05:28.120
is compared single
response surface models

01:05:28.120 --> 01:05:31.210
to multiple response
surface models.

01:05:31.210 --> 01:05:34.960
Pictured here is
basically showing

01:05:34.960 --> 01:05:37.540
this complicated
dependence of signal

01:05:37.540 --> 01:05:42.610
to noise ratio, the single
response surface model,

01:05:42.610 --> 01:05:45.940
but the simple dependence
of deposition rates

01:05:45.940 --> 01:05:48.230
on these two parameters.

01:05:48.230 --> 01:05:53.890
So down here, this is
basically saying my site here

01:05:53.890 --> 01:05:55.240
is wafer number 26.

01:05:55.240 --> 01:05:58.450
I want to know what the
average thickness on wafer 26

01:05:58.450 --> 01:06:01.210
is as a function of
these two flow rates.

01:06:01.210 --> 01:06:05.530
And it's a nice, simple
dependence on the flow rates.

01:06:05.530 --> 01:06:08.500
If I look at a different
site, a little bit

01:06:08.500 --> 01:06:11.080
further down the tube
at wafer number 124,

01:06:11.080 --> 01:06:14.240
I also get a simple model.

01:06:14.240 --> 01:06:19.150
But if I now combine these
into a sigma over mu,

01:06:19.150 --> 01:06:23.470
it has a very complicated shape.

01:06:23.470 --> 01:06:26.620
Built out of these simple
dependencies, simple responses

01:06:26.620 --> 01:06:29.830
spatially, they combine
to a very complicated

01:06:29.830 --> 01:06:34.000
non-uniformity signature.

01:06:34.000 --> 01:06:37.150
Then what they did
is say, OK, I'm

01:06:37.150 --> 01:06:42.550
going to compare SRS to
MRS. And I'm, furthermore,

01:06:42.550 --> 01:06:45.340
going to tie one
arm behind my back

01:06:45.340 --> 01:06:49.480
when I do MRS. The arm I'm
going to tie behind my back

01:06:49.480 --> 01:06:54.810
is I'm going to let SRS
do three level DOEs.

01:06:54.810 --> 01:06:57.090
Two parameters, flow rate
one and flow rate two.

01:06:57.090 --> 01:07:00.990
But I'm doing a full
factorial three level.

01:07:00.990 --> 01:07:03.750
I do nine different
process experiments.

01:07:03.750 --> 01:07:07.620
And I fit the model for
non-uniformity for that.

01:07:07.620 --> 01:07:10.020
For MRS, I'm just
going to let myself

01:07:10.020 --> 01:07:12.060
pick the high and the low.

01:07:12.060 --> 01:07:16.770
I'm just going to do less than
half the number of experiments

01:07:16.770 --> 01:07:19.350
and build my site
models and then

01:07:19.350 --> 01:07:24.000
see what kind of predicted
signal to noise ratio

01:07:24.000 --> 01:07:25.380
I will get.

01:07:25.380 --> 01:07:27.180
And I'm going to
look now at also

01:07:27.180 --> 01:07:31.320
how noise factors into this and
see, do I do better with SRS

01:07:31.320 --> 01:07:33.450
or do I do better with MRS?

01:07:33.450 --> 01:07:36.210
And this is a picture that
shows, hey, in these two cases,

01:07:36.210 --> 01:07:40.170
they're kind of suggesting
different non-uniformity

01:07:40.170 --> 01:07:41.400
signatures.

01:07:41.400 --> 01:07:44.040
But what happens
now if I repeat that

01:07:44.040 --> 01:07:48.510
with different amounts
of injected noise?

01:07:48.510 --> 01:07:51.420
And it's basically
showing that the MRS

01:07:51.420 --> 01:07:54.660
in four repeats of
this kind of experiment

01:07:54.660 --> 01:07:56.820
with different amounts
of injected noise

01:07:56.820 --> 01:08:00.480
pretty much gives the
same surface each time.

01:08:00.480 --> 01:08:06.180
Look what happens over
here in the SRS case.

01:08:06.180 --> 01:08:07.920
Three out of the
four times, they're

01:08:07.920 --> 01:08:10.290
kind of the same bowl shaped.

01:08:10.290 --> 01:08:14.760
But one case, you actually get
this weird hyperbolic surface.

01:08:14.760 --> 01:08:17.310
More susceptible, this is
just kind of an example

01:08:17.310 --> 01:08:20.700
not proving anything, but
just showing an example

01:08:20.700 --> 01:08:23.580
that actually the models
you get with the SRS

01:08:23.580 --> 01:08:26.819
can actually vary and be
much more sensitive to noise

01:08:26.819 --> 01:08:29.760
even though you got
more measurements, more

01:08:29.760 --> 01:08:34.200
spatial sample or more
levels of your DOE.

01:08:34.200 --> 01:08:36.240
And then they went
in and did sort

01:08:36.240 --> 01:08:40.740
of a more thorough example,
building lots of models,

01:08:40.740 --> 01:08:45.029
and then using that
model to drive or select

01:08:45.029 --> 01:08:46.740
what an optimum point is.

01:08:46.740 --> 01:08:50.910
So maybe you were trying to
maximize the removal rate,

01:08:50.910 --> 01:08:55.229
find the optimum point
in each of those cases

01:08:55.229 --> 01:08:59.010
and see what the spread
is with the SRS and MRS

01:08:59.010 --> 01:09:01.170
in the predicted optimal points.

01:09:01.170 --> 01:09:03.210
And you can see over
here on the right

01:09:03.210 --> 01:09:05.100
the basic conclusion
that they come

01:09:05.100 --> 01:09:09.899
to is based on the
single response surface

01:09:09.899 --> 01:09:13.170
model and the multiple
response surface model.

01:09:13.170 --> 01:09:16.590
They're spread in the two
in terms of the optimum.

01:09:16.590 --> 01:09:20.700
But the single response surface
model is basically biased.

01:09:20.700 --> 01:09:23.760
It drives you to
an optimum point

01:09:23.760 --> 01:09:27.510
that is actually not close
to the measured optimum point

01:09:27.510 --> 01:09:29.450
on average.

01:09:29.450 --> 01:09:33.450
They basically say single
response surface modeling

01:09:33.450 --> 01:09:34.800
is dangerous.

01:09:34.800 --> 01:09:36.720
Not only is it
inefficient, but it

01:09:36.720 --> 01:09:39.819
can drive you to make errors.

01:09:39.819 --> 01:09:42.970
So that's a very
interesting observation.

01:09:42.970 --> 01:09:45.227
So I think I'm going to
wrap it up here so we have

01:09:45.227 --> 01:09:47.560
a little bit of time to talk
about some of the projects,

01:09:47.560 --> 01:09:50.130
especially with Singapore people
while we've got the video.

01:09:50.130 --> 01:09:53.340
But I think these are really
neat papers, really neat ideas.

01:09:53.340 --> 01:09:58.200
Basically first idea is spatial
sampling, how you sample,

01:09:58.200 --> 01:10:00.120
and what area each of
those sampling points

01:10:00.120 --> 01:10:02.340
represents matters.

01:10:02.340 --> 01:10:04.830
So you can be smart about
that, using either weighting

01:10:04.830 --> 01:10:06.360
or uniform sampling.

01:10:06.360 --> 01:10:09.300
And then there's some neat
ideas about combined process

01:10:09.300 --> 01:10:10.860
and spatial modeling.

01:10:10.860 --> 01:10:13.620
I basically really like the
multiple response surface

01:10:13.620 --> 01:10:14.850
modeling approach.

01:10:14.850 --> 01:10:17.130
I think the Mozumder
and Loewenstein

01:10:17.130 --> 01:10:19.020
was a nice stepping
stone on the way

01:10:19.020 --> 01:10:21.390
to recognize that you'd
like to build simple models

01:10:21.390 --> 01:10:22.740
and then derive on them.

01:10:22.740 --> 01:10:26.190
I like the Guo and Sachs because
it carries it one step further.

01:10:26.190 --> 01:10:28.680
It says build simple
models of your response

01:10:28.680 --> 01:10:30.720
at each of your
spatial locations

01:10:30.720 --> 01:10:32.820
and then combine
those or use those

01:10:32.820 --> 01:10:37.200
as you see fit to either
boil down and come up

01:10:37.200 --> 01:10:40.560
with an overall uniformity or
make other kinds of control

01:10:40.560 --> 01:10:42.730
or optimization decisions.

01:10:42.730 --> 01:10:45.540
So any questions on that
before we switch over

01:10:45.540 --> 01:10:47.010
to talking about some projects?

01:10:50.880 --> 01:10:51.940
Think about it.

01:10:51.940 --> 01:10:53.440
Maybe some of your
data is actually

01:10:53.440 --> 01:10:55.470
a spatially sampled for
one of your projects.

01:10:55.470 --> 01:10:56.880
It'd be cool.

01:10:56.880 --> 01:10:58.710
Cool to add some spatial
modeling in there.

01:11:02.320 --> 01:11:06.040
How finely refined can you
probe that monkey brain?

01:11:08.740 --> 01:11:10.690
OK, that's it.

01:11:10.690 --> 01:11:15.490
I know some of the
folks here in Cambridge,

01:11:15.490 --> 01:11:17.230
I've met with one group.

01:11:17.230 --> 01:11:20.830
I think Dave has talked to
Matt and you guys are thinking.

01:11:20.830 --> 01:11:25.390
I talked also-- will want
to try to set up a meeting.

01:11:25.390 --> 01:11:28.000
I think I sent email
pretty much to everybody

01:11:28.000 --> 01:11:29.770
saying I think the
projects look good.

01:11:29.770 --> 01:11:31.810
Here's some ideas you
might think about.

01:11:31.810 --> 01:11:34.960
The purpose of the meeting
is not to get approval.

01:11:34.960 --> 01:11:36.610
Consider this now.

01:11:36.610 --> 01:11:37.660
You're approved.

01:11:37.660 --> 01:11:38.680
Go, run.

01:11:38.680 --> 01:11:40.390
You've only got a week.

01:11:40.390 --> 01:11:41.620
Less than a week.

01:11:41.620 --> 01:11:42.700
Go full boar.

01:11:42.700 --> 01:11:46.930
The purpose of the meeting is
to answer questions, help out

01:11:46.930 --> 01:11:49.870
if you've got
additional inquiries.

01:11:49.870 --> 01:11:54.400
Feel free to contact me, Dave
Hardt, Hayden, at any time.

01:11:54.400 --> 01:11:58.040
But I would like, if we haven't
had a chance to talk yet,

01:11:58.040 --> 01:12:00.960
just try to set something up.

01:12:00.960 --> 01:12:07.570
AUDIENCE: [INAUDIBLE]

01:12:07.570 --> 01:12:12.400
DUANE BONING: Well, one and a
half hours less than one week

01:12:12.400 --> 01:12:14.080
from today will be the--

01:12:14.080 --> 01:12:18.250
so what I will also be
doing is making assignments

01:12:18.250 --> 01:12:21.760
for what groups will present
on Tuesday for the next week

01:12:21.760 --> 01:12:25.750
and what groups will present
on Thursday of next week.

01:12:25.750 --> 01:12:29.170
So it'll be kind of
random luck of the draw.

01:12:29.170 --> 01:12:33.760
But I'll let you know
by the end of this week

01:12:33.760 --> 01:12:35.140
when your time will be.

01:12:35.140 --> 01:12:38.560
And then everybody's
reports are due on Friday

01:12:38.560 --> 01:12:41.300
of next week, the end of class.

01:12:41.300 --> 01:12:43.030
OK?

01:12:43.030 --> 01:12:45.190
I didn't mean to panic you.

01:12:45.190 --> 01:12:49.150
You've got almost at least a
full week for the presentation.

01:12:49.150 --> 01:12:53.920
And then in fact, one idea is
if you get some quick questions

01:12:53.920 --> 01:12:57.910
or feedback from the class, from
us, during the presentation,

01:12:57.910 --> 01:13:00.640
it gives you a quick chance
to maybe add a little bit

01:13:00.640 --> 01:13:02.680
or do a quick
additional analysis

01:13:02.680 --> 01:13:05.120
before the report is due.

01:13:05.120 --> 01:13:07.630
So I actually do
recommend that you

01:13:07.630 --> 01:13:10.930
take that opportunity
to fold in some feedback

01:13:10.930 --> 01:13:13.030
from the presentation.

01:13:13.030 --> 01:13:16.990
Do not think that you're
locked in completely

01:13:16.990 --> 01:13:17.890
at the presentation.

01:13:17.890 --> 01:13:20.110
That's most of the
way there, but it

01:13:20.110 --> 01:13:25.960
does give a chance for a
little bit of additional work

01:13:25.960 --> 01:13:27.940
in the last couple of days.

01:13:27.940 --> 01:13:29.310
OK?

01:13:29.310 --> 01:13:30.870
Thanks.

01:13:30.870 --> 01:13:31.980
So let's see.

01:13:31.980 --> 01:13:32.970
Singapore folks.

01:13:36.570 --> 01:13:39.350
First I think I have--

01:13:39.350 --> 01:13:42.570
let me get out of this.

01:13:42.570 --> 01:13:44.243
Somebody sent me slides.

01:13:44.243 --> 01:13:45.660
Let me talk with
that group first.

01:13:50.620 --> 01:13:51.190
Yeah.

01:13:51.190 --> 01:13:53.560
So David, Stephen, Jenny.

01:13:53.560 --> 01:13:56.020
These were the
slides you guys sent.

01:13:59.100 --> 01:14:01.170
So somebody want to--

01:14:01.170 --> 01:14:02.670
somebody talk me through these.

01:14:02.670 --> 01:14:05.685
I can push buttons if you
want or you can come up.

01:14:05.685 --> 01:14:07.680
It's probably easier
to do this than try

01:14:07.680 --> 01:14:11.520
to switch control to Singapore.

01:14:11.520 --> 01:14:16.200
AUDIENCE: OK, so our project try
to apply the [INAUDIBLE] method

01:14:16.200 --> 01:14:18.520
to supply chain case study.

01:14:18.520 --> 01:14:22.770
So we can click to next slide.

01:14:22.770 --> 01:14:24.300
Next slide.

01:14:24.300 --> 01:14:26.200
I think we can skip this.

01:14:26.200 --> 01:14:28.110
Also this will be
basically the model.

01:14:28.110 --> 01:14:32.700
We can consider the
supplier and Company X

01:14:32.700 --> 01:14:35.260
and can consider the customers.

01:14:35.260 --> 01:14:38.310
So following other parameters
that we will consider,

01:14:38.310 --> 01:14:43.170
[INAUDIBLE] service level that
your supplier promised to you

01:14:43.170 --> 01:14:48.810
will be the buffer size in the
company between the machines.

01:14:48.810 --> 01:14:51.000
And I have TDF and TDR.

01:14:51.000 --> 01:14:53.640
And p, the production
rate of the flow line.

01:14:53.640 --> 01:14:58.110
LT, the transportation lead
time from company to a customer.

01:14:58.110 --> 01:15:00.570
LC is the lead time
that the company

01:15:00.570 --> 01:15:02.100
promised to the customer.

01:15:02.100 --> 01:15:04.517
And mu and sigma for the demand.

01:15:04.517 --> 01:15:05.850
DUANE BONING: So quick question.

01:15:05.850 --> 01:15:08.640
Are all of these sort
of input parameters

01:15:08.640 --> 01:15:11.550
or are some of these
output parameters?

01:15:16.780 --> 01:15:19.800
AUDIENCE: These are the
input parameters I think.

01:15:19.800 --> 01:15:23.482
All of these parameters
are the DOE parameters.

01:15:23.482 --> 01:15:24.190
DUANE BONING: OK.

01:15:24.190 --> 01:15:31.080
So some will be sort of set that
essentially define the setup.

01:15:31.080 --> 01:15:35.400
And some will be DOE.

01:15:35.400 --> 01:15:37.025
That's what you're saying?

01:15:37.025 --> 01:15:37.650
AUDIENCE: Yeah.

01:15:37.650 --> 01:15:40.290
DUANE BONING: OK, OK.

01:15:40.290 --> 01:15:43.350
AUDIENCE: And next slide.

01:15:43.350 --> 01:15:49.590
It's about the problem
that is for the company.

01:15:49.590 --> 01:15:54.790
Sometimes the service level
will out of the range.

01:15:54.790 --> 01:15:58.100
And also using the data
for the service level

01:15:58.100 --> 01:16:02.220
is unlike other parameters.

01:16:02.220 --> 01:16:03.390
The data is quite less.

01:16:03.390 --> 01:16:07.900
So we need to detect the
shift as soon as possible.

01:16:07.900 --> 01:16:11.340
And after that, you may want
to improve some parameters

01:16:11.340 --> 01:16:15.600
to improve the service level.

01:16:15.600 --> 01:16:18.780
So the objective-- yeah.

01:16:18.780 --> 01:16:22.350
DUANE BONING: So you've got sort
of two branches here in part.

01:16:22.350 --> 01:16:25.890
One is essentially SPC, right?

01:16:25.890 --> 01:16:27.360
So that's what you're saying?

01:16:27.360 --> 01:16:30.532
And the other is optimization.

01:16:30.532 --> 01:16:31.740
AUDIENCE: Yeah, that's right.

01:16:31.740 --> 01:16:32.448
DUANE BONING: OK.

01:16:32.448 --> 01:16:33.960
Got it.

01:16:33.960 --> 01:16:34.590
AUDIENCE: OK.

01:16:34.590 --> 01:16:37.920
And you click again.

01:16:37.920 --> 01:16:40.700
This objective.

01:16:40.700 --> 01:16:45.230
To find the first
step is apply the SPC.

01:16:45.230 --> 01:16:48.050
And you find out what
are the main effects

01:16:48.050 --> 01:16:51.560
in the second stage that
you apply the RSM to do

01:16:51.560 --> 01:16:56.570
optimization to give
advice for the company that

01:16:56.570 --> 01:16:59.240
how can you
distribute the budget

01:16:59.240 --> 01:17:03.840
to adjust some of the
parameters to get the best

01:17:03.840 --> 01:17:06.410
result for improving
the service level.

01:17:06.410 --> 01:17:07.250
DUANE BONING: OK.

01:17:07.250 --> 01:17:10.580
So one thought here,
just going back.

01:17:10.580 --> 01:17:11.630
Let's see.

01:17:11.630 --> 01:17:15.560
Back to this slide.

01:17:15.560 --> 01:17:19.860
Some of these parameters
might be discrete,

01:17:19.860 --> 01:17:23.120
which is interesting, I think.

01:17:23.120 --> 01:17:26.090
So it looks like buffer
size is probably discrete.

01:17:26.090 --> 01:17:27.690
Is that the plan?

01:17:27.690 --> 01:17:30.770
AUDIENCE: Yeah, yeah.

01:17:30.770 --> 01:17:32.930
DUANE BONING: Is that the
only discrete parameter?

01:17:36.357 --> 01:17:38.190
AUDIENCE: Yeah, I think
that's the only one.

01:17:38.190 --> 01:17:40.523
DUANE BONING: Because that
might be interesting in terms

01:17:40.523 --> 01:17:41.160
of the DOE.

01:17:41.160 --> 01:17:43.530
Does the discreetness matter?

01:17:43.530 --> 01:17:47.880
You have only choice A, B,
C, and it might be hard.

01:17:47.880 --> 01:17:51.420
Here it might
still be monotonic,

01:17:51.420 --> 01:17:54.180
meaning there really
is a progression.

01:17:54.180 --> 01:17:55.770
So you could have
discrete levels,

01:17:55.770 --> 01:17:58.200
and they might have
three levels that are

01:17:58.200 --> 01:18:02.022
ordered in some meaningful way.

01:18:02.022 --> 01:18:03.480
So that'll be an
interesting-- that

01:18:03.480 --> 01:18:08.190
might be an interesting twist
both on the DOE, but more

01:18:08.190 --> 01:18:12.030
interestingly, for
response surface

01:18:12.030 --> 01:18:14.850
modeling and optimization.

01:18:14.850 --> 01:18:17.250
Because what do you do
if your optimization

01:18:17.250 --> 01:18:21.078
says pick a buffer size of 1.2?

01:18:21.078 --> 01:18:22.510
AUDIENCE: Oh, I see.

01:18:22.510 --> 01:18:24.240
DUANE BONING: So I think that's
an interesting twist you'll

01:18:24.240 --> 01:18:24.990
have to deal with.

01:18:28.020 --> 01:18:30.150
So you might actually
have to sort of come up

01:18:30.150 --> 01:18:35.640
with some, on that point,
some strategies for rounding

01:18:35.640 --> 01:18:40.170
off and maybe assessing both the
high and low round off points

01:18:40.170 --> 01:18:43.730
to see which one's
actually better.

01:18:43.730 --> 01:18:46.370
That might be an
example strategy.

01:18:46.370 --> 01:18:47.410
OK, keep going.

01:18:47.410 --> 01:18:48.410
AUDIENCE: OK, thank you.

01:18:48.410 --> 01:18:49.040
OK.

01:18:49.040 --> 01:18:52.940
That's the three stages
that we summarize.

01:18:52.940 --> 01:18:57.680
First stage is the advanced
SPC. The second stage the DOE.

01:18:57.680 --> 01:19:02.010
And the last stage is
response of this model.

01:19:02.010 --> 01:19:04.220
DUANE BONING: When
you say stage,

01:19:04.220 --> 01:19:06.530
does it have to
be in that order?

01:19:10.110 --> 01:19:11.850
AUDIENCE: I think yeah.

01:19:11.850 --> 01:19:16.920
First because you try
to plot the control

01:19:16.920 --> 01:19:18.660
chart for the service level.

01:19:18.660 --> 01:19:21.090
Then you find the problem.

01:19:21.090 --> 01:19:23.640
Then you start to
focus on the effects.

01:19:23.640 --> 01:19:25.470
That's stage two.

01:19:25.470 --> 01:19:27.610
And after that you
do the optimization.

01:19:30.550 --> 01:19:34.300
That's what we think
should be order.

01:19:34.300 --> 01:19:36.880
DUANE BONING: Because one
thought maybe to think about

01:19:36.880 --> 01:19:41.680
is whether switching that
order might make sense.

01:19:41.680 --> 01:19:44.710
I mean, in one scenario,
I think you're right.

01:19:44.710 --> 01:19:47.440
And in one actual sort
of real life scenario,

01:19:47.440 --> 01:19:51.010
you may have SPC
charts already up.

01:19:51.010 --> 01:19:52.990
And then you go out of control.

01:19:52.990 --> 01:19:56.710
You see you're out
of spec somewhere.

01:19:56.710 --> 01:19:59.080
And then if I understand
your approach here,

01:19:59.080 --> 01:20:02.440
you're saying to figure
out what the problem was,

01:20:02.440 --> 01:20:05.365
you want to do a DOE
and solve the problem.

01:20:09.070 --> 01:20:10.870
That might be an
interesting scenario,

01:20:10.870 --> 01:20:18.280
but often the other way
around is do a DOE up front

01:20:18.280 --> 01:20:21.310
so you actually know what
you should be monitoring.

01:20:21.310 --> 01:20:25.510
You actually understand larger
the basic relationships,

01:20:25.510 --> 01:20:30.820
what's significant, what
you should be monitoring,

01:20:30.820 --> 01:20:34.330
where there's a
likely actual effect.

01:20:34.330 --> 01:20:43.330
And then help use the
DOE to set up the SPC.

01:20:43.330 --> 01:20:45.970
Or maybe the sequence--
maybe what I would do,

01:20:45.970 --> 01:20:48.010
maybe it goes the
other way around.

01:20:48.010 --> 01:20:56.290
Stage four, revise
SPC. Maybe that

01:20:56.290 --> 01:20:58.180
would be a good compromise.

01:20:58.180 --> 01:21:00.260
Because I think once
you've built a DOE,

01:21:00.260 --> 01:21:05.170
now you've got a lot more
information about what the--

01:21:05.170 --> 01:21:09.130
you've got a more solid model,
a more functional model.

01:21:09.130 --> 01:21:15.175
And you can use that to perhaps
set better limits on SPC

01:21:15.175 --> 01:21:18.290
and decide what parameters
are most important.

01:21:18.290 --> 01:21:18.940
So I like that.

01:21:18.940 --> 01:21:21.910
AUDIENCE: That's right, we
can revise the control chart

01:21:21.910 --> 01:21:22.460
parameters.

01:21:22.460 --> 01:21:23.830
Yeah, that's right.

01:21:23.830 --> 01:21:27.070
And I think we'll use CellSim.

01:21:27.070 --> 01:21:29.980
It's an Excel based
simulation software

01:21:29.980 --> 01:21:32.140
to build a model,
supply chain model,

01:21:32.140 --> 01:21:35.510
and used Minitab to
follow data analysis.

01:21:35.510 --> 01:21:36.220
OK?

01:21:36.220 --> 01:21:37.730
DUANE BONING: OK.

01:21:37.730 --> 01:21:38.230
Great.

01:21:38.230 --> 01:21:39.640
So this is all going to be--

01:21:42.150 --> 01:21:44.410
do you actually have
some company data

01:21:44.410 --> 01:21:48.130
or are you going to pretty much
mostly use the CellSim model?

01:21:53.675 --> 01:21:54.550
AUDIENCE: Next slide.

01:21:54.550 --> 01:21:56.110
You can change to next slide.

01:21:56.110 --> 01:21:58.090
DUANE BONING: Ah, good, yeah.

01:21:58.090 --> 01:21:59.200
AUDIENCE: Yeah.

01:21:59.200 --> 01:22:02.500
So for the service
level, I think

01:22:02.500 --> 01:22:05.050
we will get from the simulation.

01:22:05.050 --> 01:22:07.600
But the input from the company.

01:22:07.600 --> 01:22:11.212
Because we can't get the service
level data from the company.

01:22:11.212 --> 01:22:11.920
DUANE BONING: OK.

01:22:15.010 --> 01:22:17.080
So if I understand
what you're saying

01:22:17.080 --> 01:22:20.890
is you might actually try
to model kind of a real line

01:22:20.890 --> 01:22:24.160
with some of the parameters
that tell you how that line is

01:22:24.160 --> 01:22:29.290
structured and then use that
to do simulations and generate

01:22:29.290 --> 01:22:30.932
service level?

01:22:30.932 --> 01:22:32.140
AUDIENCE: Yeah, that's right.

01:22:32.140 --> 01:22:32.740
DUANE BONING: I see.

01:22:32.740 --> 01:22:34.720
AUDIENCE: So the input
from company, yeah.

01:22:34.720 --> 01:22:35.428
DUANE BONING: OK.

01:22:35.428 --> 01:22:38.290
But you have no output
data from the company?

01:22:38.290 --> 01:22:40.532
It's all going to be simulation?

01:22:40.532 --> 01:22:41.740
AUDIENCE: Yeah, that's right.

01:22:41.740 --> 01:22:43.990
DUANE BONING: OK, OK.

01:22:43.990 --> 01:22:45.520
That certainly
gives the advantage

01:22:45.520 --> 01:22:50.260
I talked about today
of synthetic data.

01:22:50.260 --> 01:22:54.730
I mean, it's nice that you've
got a realistic scenario based

01:22:54.730 --> 01:23:00.250
on a company scenario for
setting up your simulation.

01:23:00.250 --> 01:23:04.480
But you have lots of latitude
in your synthetic data

01:23:04.480 --> 01:23:06.910
to actually--

01:23:06.910 --> 01:23:12.860
I would encourage you
to consider exploring

01:23:12.860 --> 01:23:15.860
intentional perturbations.

01:23:15.860 --> 01:23:20.120
You can introduce different
faults into your line

01:23:20.120 --> 01:23:25.250
and see what happens both
in the SPC and in the model.

01:23:25.250 --> 01:23:27.320
You've got a lot of
flexibility because you're

01:23:27.320 --> 01:23:28.800
doing the simulation.

01:23:28.800 --> 01:23:31.850
And you can play
with and say what

01:23:31.850 --> 01:23:35.360
happens if this piece
of equipment goes down

01:23:35.360 --> 01:23:38.660
or the mean time
to failure triples.

01:23:38.660 --> 01:23:41.810
And I didn't know that.

01:23:41.810 --> 01:23:43.190
How does things improve?

01:23:43.190 --> 01:23:47.750
Or mean time to failure
becomes 1/10 what it was.

01:23:47.750 --> 01:23:49.950
I'm failing much more often.

01:23:49.950 --> 01:23:52.280
So I think you can
construct very nicely

01:23:52.280 --> 01:23:58.280
a lot of different scenarios
based on your knowledge

01:23:58.280 --> 01:24:03.350
from sort of factory modeling
that might be relevant ones

01:24:03.350 --> 01:24:06.990
to study with a DOE approach.

01:24:06.990 --> 01:24:08.720
So I think that's rich.

01:24:08.720 --> 01:24:10.500
That gives you a
lot of opportunity.

01:24:10.500 --> 01:24:11.520
So that's good.

01:24:14.178 --> 01:24:14.720
AUDIENCE: OK.

01:24:14.720 --> 01:24:16.538
I think that's the
end of the slides.

01:24:16.538 --> 01:24:17.330
DUANE BONING: Yeah.

01:24:17.330 --> 01:24:18.770
Hayden, do you have--

01:24:18.770 --> 01:24:20.740
you want to-- here,
why don't you?

01:24:23.235 --> 01:24:24.860
GUEST SPEAKER: We're
just thinking more

01:24:24.860 --> 01:24:27.110
about additional things
you might explore.

01:24:27.110 --> 01:24:30.320
So let me just make
sure I understand

01:24:30.320 --> 01:24:34.200
how the Excel model works.

01:24:34.200 --> 01:24:36.800
I mean, it sounds
like you actually

01:24:36.800 --> 01:24:41.270
have a function that determines
y as a function of all

01:24:41.270 --> 01:24:42.470
these input variables.

01:24:42.470 --> 01:24:44.180
And you know what
that function is.

01:24:44.180 --> 01:24:46.975
DUANE BONING: Well,
it's very complicated.

01:24:46.975 --> 01:24:48.600
GUEST SPEAKER: But
you know what it is.

01:24:48.600 --> 01:24:53.540
So if you're doing DOE, you're
trying to build a model,

01:24:53.540 --> 01:24:55.650
but you already have the model.

01:24:55.650 --> 01:24:59.990
So I'm not sure the timeline.

01:24:59.990 --> 01:25:03.290
Basically, you're going to be
trying to get to the model--

01:25:03.290 --> 01:25:06.830
AUDIENCE: Actually you don't
have to the exact model.

01:25:06.830 --> 01:25:10.670
The service level relates
to all of the parameters.

01:25:10.670 --> 01:25:14.960
But you cannot derive a model.

01:25:14.960 --> 01:25:18.590
GUEST SPEAKER: You can't derive
a-- so how is this exact model?

01:25:18.590 --> 01:25:21.050
Is it a lookup
table or something?

01:25:21.050 --> 01:25:24.230
AUDIENCE: We will
simulate the supply chain

01:25:24.230 --> 01:25:29.540
and we run the simulation
to get a production lead

01:25:29.540 --> 01:25:31.940
time, a production cycle time.

01:25:31.940 --> 01:25:37.730
That is influenced by the
buffer size, by the MTTF, MTTR.

01:25:37.730 --> 01:25:41.660
And we will check for
the production lead time

01:25:41.660 --> 01:25:43.580
plus transportation lead time.

01:25:43.580 --> 01:25:49.730
If this is less
than the lead time

01:25:49.730 --> 01:25:52.370
that you promised your
customer, then that

01:25:52.370 --> 01:25:55.280
means you satisfy the customer.

01:25:55.280 --> 01:25:59.350
And if the production lead
time plus the transportation

01:25:59.350 --> 01:26:01.430
lead time larger
than LC, then it

01:26:01.430 --> 01:26:05.690
will account for
one [INAUDIBLE]..

01:26:05.690 --> 01:26:07.790
That's what we generate
the service level.

01:26:07.790 --> 01:26:09.200
GUEST SPEAKER: OK, I see.

01:26:09.200 --> 01:26:11.913
So you want some approximate
way of predicting

01:26:11.913 --> 01:26:12.830
what it's going to be.

01:26:12.830 --> 01:26:13.760
OK, fair enough.

01:26:13.760 --> 01:26:15.260
DUANE BONING: I
think Hayden's point

01:26:15.260 --> 01:26:18.710
is if you have, in
general, very often you

01:26:18.710 --> 01:26:21.530
have very complicated
models that

01:26:21.530 --> 01:26:24.540
may take a long time to
actually run any one point.

01:26:24.540 --> 01:26:26.450
And so I think that's
lurking in your mind.

01:26:26.450 --> 01:26:28.180
What you would
like to do is build

01:26:28.180 --> 01:26:30.950
simplified approximate
derived models

01:26:30.950 --> 01:26:35.240
off of a complicated
long simulation time

01:26:35.240 --> 01:26:38.390
sophisticated CellSim model.

01:26:38.390 --> 01:26:38.980
Is that right?

01:26:44.000 --> 01:26:44.750
GUEST SPEAKER: OK.

01:26:44.750 --> 01:26:45.080
Great.

01:26:45.080 --> 01:26:47.330
DUANE BONING: I mean, even
though CellSim may actually

01:26:47.330 --> 01:26:48.200
be fairly fast.

01:26:48.200 --> 01:26:51.770
In what I've seen in many of
these factory simulations,

01:26:51.770 --> 01:26:56.350
running the factory simulation
itself may take hours or days.

01:26:56.350 --> 01:26:58.968
And that's where you
might actually be building

01:26:58.968 --> 01:27:00.260
a simplified approximate model.

01:27:03.910 --> 01:27:07.760
AUDIENCE: I have
one question here.

01:27:07.760 --> 01:27:09.830
I'm not very sure
about the demand data.

01:27:09.830 --> 01:27:12.710
But seems like the
CellSim can only

01:27:12.710 --> 01:27:16.620
have a normal distribution, an
exponential distribution input.

01:27:16.620 --> 01:27:19.220
So if the normal
distribution is not

01:27:19.220 --> 01:27:22.700
very appropriate for the
demand, what can we do?

01:27:28.103 --> 01:27:29.270
DUANE BONING: Good question.

01:27:29.270 --> 01:27:34.520
I think if it can at least do
two different distributions,

01:27:34.520 --> 01:27:37.310
one interesting thing would
be to see how sensitive

01:27:37.310 --> 01:27:40.880
the simulation is to whether
it's-- you said a normal

01:27:40.880 --> 01:27:42.590
distribution and the
other was a Poisson?

01:27:42.590 --> 01:27:44.270
Or what was the other one?

01:27:44.270 --> 01:27:45.845
AUDIENCE: It's a exponential.

01:27:45.845 --> 01:27:48.380
DUANE BONING: Exponential.

01:27:48.380 --> 01:27:51.560
I think usually
an exponential is

01:27:51.560 --> 01:27:55.400
what's often assumed for these
kinds of arrival processes.

01:27:55.400 --> 01:27:59.870
But it would be interesting,
actually, to see how much--

01:27:59.870 --> 01:28:02.060
that might end up being
a nice question for you

01:28:02.060 --> 01:28:07.070
to ask you and try with your
data and your simulation.

01:28:07.070 --> 01:28:11.930
How do things differ as
a function of not just

01:28:11.930 --> 01:28:13.730
the parameters of
the distribution

01:28:13.730 --> 01:28:17.540
but the type of distribution?

01:28:17.540 --> 01:28:19.320
It might not matter at all.

01:28:19.320 --> 01:28:21.500
That might be crucial.

01:28:21.500 --> 01:28:23.690
That might be the
most important thing.

01:28:23.690 --> 01:28:24.260
I don't know.

01:28:29.240 --> 01:28:29.820
Good.

01:28:29.820 --> 01:28:33.300
Any other questions on this one?

01:28:33.300 --> 01:28:34.440
AUDIENCE: No, we're fine.

01:28:34.440 --> 01:28:35.273
DUANE BONING: Great.

01:28:35.273 --> 01:28:35.790
Go for it.

01:28:35.790 --> 01:28:38.010
It sounds interesting.

01:28:38.010 --> 01:28:41.220
I like how this integrates
your other classes

01:28:41.220 --> 01:28:44.070
and your other learning too.

01:28:44.070 --> 01:28:47.010
AUDIENCE: That's the
inspiration from another class.

01:28:47.010 --> 01:28:48.500
DUANE BONING: Exactly.

01:28:48.500 --> 01:28:49.000
OK.

01:28:57.060 --> 01:28:57.560
All right.

01:28:57.560 --> 01:28:58.400
So let's see.

01:28:58.400 --> 01:29:01.250
I had talked with--

01:29:01.250 --> 01:29:06.560
is there-- who wants to go next?

01:29:06.560 --> 01:29:09.770
Other people have not sent
slides, but let's see.

01:29:09.770 --> 01:29:11.570
Who else have--

01:29:11.570 --> 01:29:13.490
I sent email to
I think everybody

01:29:13.490 --> 01:29:16.460
kind of late last night or
early morning for you guys.

01:29:21.850 --> 01:29:23.055
They're all mixing together.

01:29:23.055 --> 01:29:25.180
Who wants to talk about
one of their projects next?

01:29:25.180 --> 01:29:27.790
Because we still have about
20 minutes on the video time.

01:29:33.860 --> 01:29:36.200
Just talk me through your
written thing and some

01:29:36.200 --> 01:29:37.280
of the comments I sent.

01:29:46.100 --> 01:29:49.280
How about the-- let's see.

01:29:49.280 --> 01:29:56.750
I'm looking at the one
for the iron sole plate.

01:29:56.750 --> 01:30:00.110
Who's kind of a
lead person on that?

01:30:00.110 --> 01:30:02.510
By the way, I liked
this one, because I

01:30:02.510 --> 01:30:05.720
got to go and tour the Phillips
factory when we were there

01:30:05.720 --> 01:30:07.440
and see some of these
things being made.

01:30:07.440 --> 01:30:09.030
So I'm like, oh, this is cool.

01:30:09.030 --> 01:30:09.650
This is great.

01:30:12.716 --> 01:30:16.405
AUDIENCE: Well, so actually
one we have sent to you.

01:30:16.405 --> 01:30:17.780
The other part we
have in mind is

01:30:17.780 --> 01:30:21.350
that the company is going to
run some intermediate points.

01:30:21.350 --> 01:30:25.730
But it turns out
to be otherwise.

01:30:25.730 --> 01:30:28.340
So what we have is that
it's a five parameter.

01:30:28.340 --> 01:30:31.040
So they are trying
to build a prototype.

01:30:31.040 --> 01:30:33.700
And they're trying to
find what is the output.

01:30:33.700 --> 01:30:38.270
So that's the last one that
actually blow out [INAUDIBLE]..

01:30:38.270 --> 01:30:41.810
And then yeah, so what
we have is actually

01:30:41.810 --> 01:30:43.450
they actually run a full DOE.

01:30:47.140 --> 01:30:48.980
So that's what we have.

01:30:48.980 --> 01:30:52.550
DUANE BONING: Is that
data already available?

01:30:52.550 --> 01:30:53.970
AUDIENCE: Yes it is.

01:30:53.970 --> 01:30:56.825
So then what we
have is a full DOE.

01:30:56.825 --> 01:30:58.910
So that's to the power of 5.

01:30:58.910 --> 01:31:02.420
So that's 32 runs.

01:31:02.420 --> 01:31:06.260
And then each run, there
is three radicates.

01:31:06.260 --> 01:31:07.390
DUANE BONING: OK, good.

01:31:07.390 --> 01:31:07.890
Yes.

01:31:13.130 --> 01:31:15.890
Are all of those parameters
basically continuous

01:31:15.890 --> 01:31:16.610
parameters?

01:31:16.610 --> 01:31:21.650
I on the diagram you
mentioned different diameters

01:31:21.650 --> 01:31:22.940
and heights.

01:31:22.940 --> 01:31:26.150
Are all of those
continuous parameters?

01:31:26.150 --> 01:31:30.003
Are any of them sort
of discrete choices?

01:31:30.003 --> 01:31:31.670
I know they only did
two levels of each.

01:31:31.670 --> 01:31:33.320
So in some sense,
they are discrete.

01:31:33.320 --> 01:31:36.740
But if you were building
a response surface model,

01:31:36.740 --> 01:31:41.600
would you be able to
actually have all of them

01:31:41.600 --> 01:31:42.980
be continuous parameters?

01:31:42.980 --> 01:31:47.870
Or would there be a
mix of some arrangement

01:31:47.870 --> 01:31:51.755
of holes and some diameter
that was continuously varying?

01:31:58.790 --> 01:32:01.550
AUDIENCE: I think it's
continuous parameters.

01:32:01.550 --> 01:32:06.020
And then I think maybe
some of the inputs

01:32:06.020 --> 01:32:08.240
are not so measurable.

01:32:08.240 --> 01:32:11.100
So maybe they try to--

01:32:11.100 --> 01:32:16.550
since some of the inputs
look at the noise input.

01:32:16.550 --> 01:32:20.270
And then they may
use the [INAUDIBLE]

01:32:20.270 --> 01:32:23.780
to do the robust design.

01:32:23.780 --> 01:32:29.180
And maybe it is
a potential area,

01:32:29.180 --> 01:32:32.390
because right now there is
still some problem with the data

01:32:32.390 --> 01:32:33.230
connection.

01:32:33.230 --> 01:32:37.800
And then we are trying to figure
the problem, all those days.

01:32:37.800 --> 01:32:45.440
So actually the full area has
not been exactly designed yet.

01:32:45.440 --> 01:32:47.750
So that is one of our problem.

01:32:47.750 --> 01:32:48.470
Yeah.

01:32:48.470 --> 01:32:49.178
DUANE BONING: OK.

01:32:49.178 --> 01:32:52.400
But you think you'll have data?

01:32:52.400 --> 01:32:54.860
AUDIENCE: Yeah, we
have several data.

01:32:54.860 --> 01:33:02.540
But we do ANOVA, and then we
find out that our value is not

01:33:02.540 --> 01:33:03.800
exactly so high.

01:33:03.800 --> 01:33:08.780
It's just a little
larger than 0.5.

01:33:08.780 --> 01:33:12.000
So I think maybe the
data is not complete.

01:33:12.000 --> 01:33:15.462
So maybe we need more data.

01:33:15.462 --> 01:33:16.170
DUANE BONING: OK.

01:33:16.170 --> 01:33:18.440
Well, if you get more
data, that's great.

01:33:18.440 --> 01:33:21.470
But it's not just
the r squared, but I

01:33:21.470 --> 01:33:26.600
assume your ANOVA does show
some significance to a model?

01:33:26.600 --> 01:33:27.500
OK.

01:33:27.500 --> 01:33:28.190
Good, good.

01:33:28.190 --> 01:33:32.120
Then I think you've
got stuff to work with.

01:33:32.120 --> 01:33:33.327
Now, in the email--

01:33:33.327 --> 01:33:34.160
AUDIENCE: Professor?

01:33:34.160 --> 01:33:35.930
DUANE BONING: Yes?

01:33:35.930 --> 01:33:38.180
AUDIENCE: I actually think
that it may not be very

01:33:38.180 --> 01:33:41.720
significant from the model.

01:33:41.720 --> 01:33:45.260
Yeah, I do think that
it's not that significant.

01:33:45.260 --> 01:33:48.080
And I actually
talked to the company

01:33:48.080 --> 01:33:51.050
to see why they are not
trying to find more points,

01:33:51.050 --> 01:33:54.000
since they are not really
able to find a good fit.

01:33:54.000 --> 01:33:56.960
But the response
that I got from them

01:33:56.960 --> 01:34:00.680
is that to run the
experiment, they actually

01:34:00.680 --> 01:34:03.950
need to build a prototype,
which is very expensive.

01:34:03.950 --> 01:34:07.890
And as a result, they are
not planning to run that.

01:34:07.890 --> 01:34:12.080
Actually, I thought that
is a good idea maybe

01:34:12.080 --> 01:34:13.130
to run a midpoint.

01:34:16.480 --> 01:34:20.180
I was not able to convince
the company for doing that.

01:34:24.230 --> 01:34:26.360
Yeah, so that's a
problem they're facing.

01:34:26.360 --> 01:34:33.450
That's a lack of
data for this study.

01:34:33.450 --> 01:34:37.820
So at the same time, we
have also [INAUDIBLE]

01:34:37.820 --> 01:34:41.370
exploring some
other alternatives.

01:34:41.370 --> 01:34:44.630
So the [INAUDIBLE]
we got is actually--

01:34:44.630 --> 01:34:49.430
so in the parent case, it's a
product design optimization.

01:34:49.430 --> 01:34:51.500
So then the next one
that we're looking into

01:34:51.500 --> 01:34:54.210
is a process optimization.

01:34:54.210 --> 01:34:59.990
So in the process
optimization, basically it's

01:34:59.990 --> 01:35:02.570
a process line where
there is actually

01:35:02.570 --> 01:35:08.300
different sole plates
that are actually

01:35:08.300 --> 01:35:11.120
they're produced in a
different period of time.

01:35:11.120 --> 01:35:14.280
And then they also
manage it differently.

01:35:14.280 --> 01:35:16.850
So that's also like the
[INAUDIBLE] influence.

01:35:16.850 --> 01:35:19.130
Like for example,
we are also trying

01:35:19.130 --> 01:35:20.870
to see, for example,
whether we can

01:35:20.870 --> 01:35:29.870
see there's any correlation
between the defects.

01:35:29.870 --> 01:35:32.640
The defects are [INAUDIBLE]
or things like this.

01:35:32.640 --> 01:35:34.070
So we're trying
to see whether we

01:35:34.070 --> 01:35:39.290
can find correlation between
different layers or even

01:35:39.290 --> 01:35:41.810
spatially.

01:35:41.810 --> 01:35:44.150
Because it's arranged in
terms of rows and columns,

01:35:44.150 --> 01:35:49.110
I think that's five
rows and seven columns.

01:35:49.110 --> 01:35:52.730
So in each carrier,
that's 35 units.

01:35:55.430 --> 01:36:00.050
So then we thought that probably
we can do a nested variance

01:36:00.050 --> 01:36:02.570
analysis on that.

01:36:02.570 --> 01:36:04.820
So one aspect I'm
looking into is

01:36:04.820 --> 01:36:07.700
because we also have the data
for the different stations.

01:36:07.700 --> 01:36:09.650
So one thing that
we can look into

01:36:09.650 --> 01:36:12.860
is which station is
the main contributor.

01:36:12.860 --> 01:36:14.000
So that's one.

01:36:14.000 --> 01:36:17.930
So secondly is between
the shift of the timing.

01:36:17.930 --> 01:36:23.720
So for example, every
hour, so an hour and maybe

01:36:23.720 --> 01:36:25.790
the different shift,
maybe one shift an hour,

01:36:25.790 --> 01:36:26.750
things like this.

01:36:26.750 --> 01:36:29.630
So then we can also
do a nested variance

01:36:29.630 --> 01:36:31.377
based on the shift timing.

01:36:31.377 --> 01:36:32.210
DUANE BONING: Right.

01:36:32.210 --> 01:36:33.710
This sounds very interesting.

01:36:33.710 --> 01:36:37.670
I don't think we have anybody
else doing a nested variance

01:36:37.670 --> 01:36:39.210
kind of analysis.

01:36:39.210 --> 01:36:41.030
We have several DOEs.

01:36:41.030 --> 01:36:44.540
So in the overall
class, I think this

01:36:44.540 --> 01:36:48.530
would be very interesting to
have this different project

01:36:48.530 --> 01:36:50.120
you're talking about.

01:36:50.120 --> 01:36:53.168
Now, for that one you've
already got the data.

01:36:53.168 --> 01:36:54.210
It sounds like it's good.

01:36:54.210 --> 01:36:57.030
They have lots of manufacturing
data in that case.

01:36:57.030 --> 01:36:59.075
Is that correct?

01:36:59.075 --> 01:36:59.700
AUDIENCE: Yeah.

01:36:59.700 --> 01:37:01.080
There are [INAUDIBLE] data.

01:37:01.080 --> 01:37:03.750
But then the person
who ran the data,

01:37:03.750 --> 01:37:05.700
they actually are not
so careful to take

01:37:05.700 --> 01:37:08.370
note of every parameter.

01:37:08.370 --> 01:37:11.130
So then initially
I actually talked

01:37:11.130 --> 01:37:14.160
to the company,
the person, I asked

01:37:14.160 --> 01:37:16.050
whether we can get all
the data and then we

01:37:16.050 --> 01:37:17.790
can combine all
the data together

01:37:17.790 --> 01:37:20.560
and then we'll be able to
get a very rich set of data.

01:37:20.560 --> 01:37:22.050
We can do all the analysis.

01:37:22.050 --> 01:37:25.690
But it seems that the data that
we get is missing in pieces.

01:37:25.690 --> 01:37:29.010
For example, they are doing
hypothesis testing kind

01:37:29.010 --> 01:37:29.850
of thing.

01:37:29.850 --> 01:37:31.963
So basically, they
are doing a p-test.

01:37:31.963 --> 01:37:34.380
So it's basically a hypothesis
testing where they actually

01:37:34.380 --> 01:37:37.230
check each parameter
to see whether it's

01:37:37.230 --> 01:37:38.610
significant or not significant.

01:37:38.610 --> 01:37:40.860
And so while they
are doing that,

01:37:40.860 --> 01:37:42.960
they do not consider
any other parameters.

01:37:42.960 --> 01:37:49.530
So although the process is
controlled by many inputs,

01:37:49.530 --> 01:37:51.870
but at any single,
one time, they only

01:37:51.870 --> 01:37:55.420
care about a particular
set of a parameter.

01:37:55.420 --> 01:37:58.380
So as a result, we are not
able to combine all the data

01:37:58.380 --> 01:38:02.670
together as one big cell of
data and do as much analysis

01:38:02.670 --> 01:38:06.010
as we would like to have.

01:38:06.010 --> 01:38:11.220
But we thought that that would
be an interesting project

01:38:11.220 --> 01:38:11.820
to go about.

01:38:11.820 --> 01:38:14.280
And at the same
time, I also think

01:38:14.280 --> 01:38:17.820
that we are trying to continue
to explore a product thing

01:38:17.820 --> 01:38:18.900
and be a process thing.

01:38:18.900 --> 01:38:22.980
So as to get a comparison--

01:38:22.980 --> 01:38:26.040
because we thought that this
real life situation, we also

01:38:26.040 --> 01:38:29.190
like to see, is the point
of view for a product DOE

01:38:29.190 --> 01:38:30.660
and process DOE--

01:38:30.660 --> 01:38:32.550
how is it different.

01:38:32.550 --> 01:38:36.750
Also we understand that
there are some limitations

01:38:36.750 --> 01:38:39.060
for the product DOE,
because it pretty

01:38:39.060 --> 01:38:41.910
difficult for the
company to run,

01:38:41.910 --> 01:38:44.280
to make a new
prototype each time

01:38:44.280 --> 01:38:46.320
compared to a process
DOE where they can

01:38:46.320 --> 01:38:48.960
change a setting rather easily.

01:38:48.960 --> 01:38:51.690
Yeah, so there is a plan.

01:38:51.690 --> 01:38:54.030
DUANE BONING: I think this
sounds like a very good plan.

01:38:54.030 --> 01:38:57.630
Because I wanted to come
back to the product DOE,

01:38:57.630 --> 01:38:59.010
because I think--

01:38:59.010 --> 01:39:02.040
so first off, I
think with your team,

01:39:02.040 --> 01:39:05.200
you've got a good
team of four people.

01:39:05.200 --> 01:39:09.900
So being able to look at both
of these problems is great.

01:39:09.900 --> 01:39:17.220
And that includes a case study
and examples of real life

01:39:17.220 --> 01:39:21.720
limitations that are
there in both situations

01:39:21.720 --> 01:39:25.590
and recommendations you
could make in both cases

01:39:25.590 --> 01:39:31.590
to the company that might
help them get more value out

01:39:31.590 --> 01:39:34.560
of the activities that they've
got to be able to make better

01:39:34.560 --> 01:39:36.250
decisions in the future.

01:39:36.250 --> 01:39:40.320
So going back to the product
one with the DOE, the two

01:39:40.320 --> 01:39:43.020
to the fifth DOE,
I think it would

01:39:43.020 --> 01:39:47.130
be very interesting to
somehow include, maybe

01:39:47.130 --> 01:39:51.480
you just make it up, but
include what more you could

01:39:51.480 --> 01:39:55.680
do if you had a center point.

01:39:55.680 --> 01:39:58.800
And show if the center
point looks like this,

01:39:58.800 --> 01:40:01.080
look how different
your decision would

01:40:01.080 --> 01:40:04.140
be in design optimization.

01:40:04.140 --> 01:40:06.340
Versus if your center
point looks like this,

01:40:06.340 --> 01:40:08.110
then the model looks
completely different,

01:40:08.110 --> 01:40:09.790
and this is what you should do.

01:40:09.790 --> 01:40:13.290
In other words, show how
important that center point

01:40:13.290 --> 01:40:16.050
might be to their thinking.

01:40:16.050 --> 01:40:19.290
Could be a nice part of this
that actually might help

01:40:19.290 --> 01:40:23.430
make the case persuasively
to the company

01:40:23.430 --> 01:40:28.650
that in their future it
might be very valuable

01:40:28.650 --> 01:40:31.320
and worth the expense of
building just one more

01:40:31.320 --> 01:40:32.730
prototype.

01:40:32.730 --> 01:40:36.360
If they're already building
32 different prototypes

01:40:36.360 --> 01:40:39.690
with some missing
data perhaps, one more

01:40:39.690 --> 01:40:43.830
would tell them so
much more potentially.

01:40:43.830 --> 01:40:46.380
So I think that could
be very interesting.

01:40:46.380 --> 01:40:51.450
And I'm glad, I like the plan
of trying to do some of both.

01:40:51.450 --> 01:40:53.575
Do I have that right?

01:40:53.575 --> 01:40:54.200
AUDIENCE: Yeah.

01:40:54.200 --> 01:40:57.570
So I actually think that when
they made the prototype, maybe

01:40:57.570 --> 01:41:01.020
they choose the high and low
because it's easier for them

01:41:01.020 --> 01:41:05.110
to make the prototype rather
than making a center point.

01:41:05.110 --> 01:41:07.980
So I do understand
that a center point

01:41:07.980 --> 01:41:10.200
has the advantage
of being able to fit

01:41:10.200 --> 01:41:11.400
models and things like that.

01:41:11.400 --> 01:41:13.050
In the technical
aspect, maybe they

01:41:13.050 --> 01:41:15.370
are limited by machines
or things like these.

01:41:15.370 --> 01:41:18.120
Maybe they have a template
where using a template they

01:41:18.120 --> 01:41:20.010
are able to get a
plus or a minus.

01:41:20.010 --> 01:41:22.110
But they are not able
to get the midpoint.

01:41:22.110 --> 01:41:23.595
So I'm not sure--

01:41:23.595 --> 01:41:27.210
perhaps, is it possible that
we discuss points like this?

01:41:27.210 --> 01:41:27.960
DUANE BONING: Yes.

01:41:27.960 --> 01:41:28.830
Absolutely.

01:41:28.830 --> 01:41:31.350
Those are very
realistic limitations.

01:41:31.350 --> 01:41:34.110
You might find that a
couple of parameters

01:41:34.110 --> 01:41:35.250
can have center points.

01:41:35.250 --> 01:41:36.780
Other ones can't.

01:41:36.780 --> 01:41:38.950
And so it's a
mixed center point.

01:41:38.950 --> 01:41:41.340
And so that's a good issue.

01:41:41.340 --> 01:41:42.060
Absolutely.

01:41:42.060 --> 01:41:44.685
All of these kind of realistic
constraints that drive--

01:41:48.900 --> 01:41:51.810
that help limit or drive
them to the decisions

01:41:51.810 --> 01:41:54.180
that they made
but also exploring

01:41:54.180 --> 01:41:56.530
is there some room
to do some more?

01:41:56.530 --> 01:42:00.260
So yes, that's a
very good point.

01:42:00.260 --> 01:42:01.260
AUDIENCE: OK, thank you.

01:42:01.260 --> 01:42:02.052
DUANE BONING: Good.

01:42:02.052 --> 01:42:03.360
Well, this sounds exciting.

01:42:03.360 --> 01:42:05.940
Get the data as
quickly as you can

01:42:05.940 --> 01:42:12.940
even with all of its ugliness
and get going on that.

01:42:12.940 --> 01:42:16.380
And then if issues
come up, send us email

01:42:16.380 --> 01:42:20.170
and we can see what
we can do to help.

01:42:20.170 --> 01:42:24.780
So I think I just have a
short period of time now.

01:42:24.780 --> 01:42:26.370
Let's see.

01:42:26.370 --> 01:42:28.830
I saw you in the
back while the camera

01:42:28.830 --> 01:42:31.620
was focused there, Priyanka,
I saw you in the back there.

01:42:31.620 --> 01:42:34.620
So you guys project.

01:42:34.620 --> 01:42:39.600
[INAUDIBLE] Stanley.

01:42:39.600 --> 01:42:42.400
Thoughts on your project?

01:42:42.400 --> 01:42:47.250
First off, I really
like the ECM aspect,

01:42:47.250 --> 01:42:48.960
because I've got
a student working

01:42:48.960 --> 01:42:52.020
on electropolishing
or electrochemical

01:42:52.020 --> 01:42:55.150
mechanical polishing
for semiconductor stuff.

01:42:55.150 --> 01:42:57.510
So the student has been
learning a little bit

01:42:57.510 --> 01:42:58.680
about electrochemistry.

01:42:58.680 --> 01:43:01.212
So I saw this and I thought
it was really interesting.

01:43:05.640 --> 01:43:07.890
AUDIENCE: The story
is sad but true.

01:43:07.890 --> 01:43:13.020
It actually happened in
my undergrad university.

01:43:13.020 --> 01:43:16.200
And I got rescued on a boat, and
we went to rescue the machine,

01:43:16.200 --> 01:43:17.970
and it was bolted
on a concrete slab

01:43:17.970 --> 01:43:20.340
and it was under
10 feet of water.

01:43:20.340 --> 01:43:22.440
DUANE BONING: Do
you have pictures?

01:43:22.440 --> 01:43:23.315
AUDIENCE: Yeah, I do.

01:43:23.315 --> 01:43:24.232
DUANE BONING: Oh good.

01:43:24.232 --> 01:43:24.738
Excellent.

01:43:27.550 --> 01:43:28.710
Now, in terms of--

01:43:28.710 --> 01:43:32.820
I think I sent a little bit of
information back in the email,

01:43:32.820 --> 01:43:36.300
because it looked like,
if I'm remembering,

01:43:36.300 --> 01:43:38.700
you wanted to do
a couple of things

01:43:38.700 --> 01:43:41.340
to see was the
equipment restored

01:43:41.340 --> 01:43:44.680
and operating back to
its original state.

01:43:44.680 --> 01:43:45.700
And I understood--

01:43:45.700 --> 01:43:46.742
AUDIENCE: That's correct.

01:43:46.742 --> 01:43:48.720
DUANE BONING: Yeah, I
understood that part.

01:43:48.720 --> 01:43:50.250
And then the other
interesting piece

01:43:50.250 --> 01:43:53.370
was building an analytic model.

01:43:53.370 --> 01:43:56.440
And using that,
it sounded like--

01:43:56.440 --> 01:43:58.590
well, tell me what
you would like

01:43:58.590 --> 01:44:02.730
to do with the analytic model.

01:44:02.730 --> 01:44:05.250
AUDIENCE: So initially,
the experiment

01:44:05.250 --> 01:44:09.030
was done to get
data to substantiate

01:44:09.030 --> 01:44:10.450
an analytical model.

01:44:10.450 --> 01:44:14.730
So the model was based on the
Faraday's laws of electrolysis.

01:44:14.730 --> 01:44:17.500
But that related only a
few of the parameters.

01:44:17.500 --> 01:44:20.610
So we were basically trying
to incorporate the flow rate

01:44:20.610 --> 01:44:23.130
term into that equation.

01:44:23.130 --> 01:44:25.590
But the focus kind of got
shifted because the machine

01:44:25.590 --> 01:44:27.000
stopped running midway.

01:44:27.000 --> 01:44:29.880
And we spent a very
long time trying

01:44:29.880 --> 01:44:31.530
to get it back on its feet.

01:44:31.530 --> 01:44:33.960
And the readings that
we took after that,

01:44:33.960 --> 01:44:36.720
so I can sort the readings
on the basis of date.

01:44:36.720 --> 01:44:39.150
We were four students
working on that project,

01:44:39.150 --> 01:44:41.100
and we took a large
set of readings.

01:44:41.100 --> 01:44:42.827
So I can sort it by
date and find out

01:44:42.827 --> 01:44:44.910
what's before the flood
and what's after the flood

01:44:44.910 --> 01:44:51.120
and to find out if really a
shift had occurred after that.

01:44:51.120 --> 01:44:52.920
When we were using it,
we never treated it

01:44:52.920 --> 01:44:54.760
as a separate set of data.

01:44:54.760 --> 01:44:58.380
We combined and indiscriminately
averaged out the whole thing.

01:44:58.380 --> 01:45:00.870
So that's something that
in hindsight we probably

01:45:00.870 --> 01:45:02.292
should have looked at.

01:45:02.292 --> 01:45:03.000
DUANE BONING: OK.

01:45:03.000 --> 01:45:07.440
I think that's sort of the core
of the idea for your group.

01:45:07.440 --> 01:45:09.030
And I like that.

01:45:09.030 --> 01:45:12.540
What I'm thinking about
is if there's one or two

01:45:12.540 --> 01:45:14.370
additional ideas
that you might be

01:45:14.370 --> 01:45:18.810
able to explore given
the fact that you've also

01:45:18.810 --> 01:45:21.040
got this analytic model.

01:45:21.040 --> 01:45:25.500
So it's going back a little bit
to this idea of synthetic data

01:45:25.500 --> 01:45:27.330
or looking at your
analytic model

01:45:27.330 --> 01:45:32.790
and maybe even doing
things like a sensitivity

01:45:32.790 --> 01:45:35.760
analysis on the model.

01:45:35.760 --> 01:45:38.040
Saying based on the
model, these are

01:45:38.040 --> 01:45:43.720
the parameters I think we would
be most sensitive to in a DOE.

01:45:43.720 --> 01:45:44.970
Some of those kinds of things.

01:45:44.970 --> 01:45:50.790
I think it might be neat
to just brainstorm or think

01:45:50.790 --> 01:45:53.340
about things you might actually
use the analytic model,

01:45:53.340 --> 01:45:54.930
since you've got that also.

01:45:54.930 --> 01:45:59.340
Even though you only know some
constants from physics and some

01:45:59.340 --> 01:46:01.600
you were trying to
fit to your data.

01:46:01.600 --> 01:46:05.800
And that whole piece
sounds very interesting.

01:46:05.800 --> 01:46:07.200
AUDIENCE: OK.

01:46:07.200 --> 01:46:11.070
The data we have basically
consists of varying three input

01:46:11.070 --> 01:46:12.150
parameters.

01:46:12.150 --> 01:46:13.860
Feed rate, flow
rate, and voltage.

01:46:13.860 --> 01:46:16.270
And the output is the MRR.

01:46:16.270 --> 01:46:18.060
So the way we
measured MRR was we

01:46:18.060 --> 01:46:20.220
took the difference in
the weight of the slab.

01:46:20.220 --> 01:46:22.530
DUANE BONING: Yes,
that sounds good.

01:46:22.530 --> 01:46:27.480
AUDIENCE: And the time
for the machining.

01:46:27.480 --> 01:46:31.620
And so we have the MRR for
about-- we have about 50 to 60

01:46:31.620 --> 01:46:33.288
readings that we've taken.

01:46:33.288 --> 01:46:34.080
DUANE BONING: Good.

01:46:34.080 --> 01:46:36.960
Yeah, that was my other question
in the email was how much data

01:46:36.960 --> 01:46:37.650
you had.

01:46:37.650 --> 01:46:40.890
Because it sounded like
each run is fairly long.

01:46:40.890 --> 01:46:42.780
Yeah, so you've got
lots of students working

01:46:42.780 --> 01:46:43.830
on this for a long time.

01:46:43.830 --> 01:46:45.510
AUDIENCE: Yeah, we have
four students working on it

01:46:45.510 --> 01:46:46.540
almost around the clock.

01:46:46.540 --> 01:46:50.130
So quite a lot of readings.

01:46:50.130 --> 01:46:52.290
DUANE BONING: OK, that
sounds very interesting.

01:46:52.290 --> 01:46:56.620
And it will be an interesting
story to hear as well.

01:46:56.620 --> 01:46:59.500
AUDIENCE: Yeah, I'll try
to put in some pictures.

01:46:59.500 --> 01:47:01.450
DUANE BONING: So if
other questions come up,

01:47:01.450 --> 01:47:04.200
let me know, especially as
you think a little bit about--

01:47:04.200 --> 01:47:08.700
your group thinks of ways
to use the analytic model.

01:47:08.700 --> 01:47:09.420
OK.

01:47:09.420 --> 01:47:12.990
AUDIENCE: A problem with the
data is that the flow rate,

01:47:12.990 --> 01:47:16.420
we couldn't vary the flow
rate on digital control.

01:47:16.420 --> 01:47:18.510
So there's this whole
set of six valves

01:47:18.510 --> 01:47:21.390
that we had to
manipulate to set a flow.

01:47:21.390 --> 01:47:27.150
And so all the other factors
are set at fixed levels.

01:47:27.150 --> 01:47:31.050
But the flow rate is
kind of an average value

01:47:31.050 --> 01:47:33.630
that it kind of
oscillated about.

01:47:33.630 --> 01:47:37.398
So I'm not very
sure how to deal.

01:47:37.398 --> 01:47:38.190
DUANE BONING: Yeah.

01:47:38.190 --> 01:47:40.140
Well first off, that's
an interesting question

01:47:40.140 --> 01:47:42.040
we haven't talked a
lot about in class.

01:47:42.040 --> 01:47:45.360
So it'll be interesting to
raise that and talk about it.

01:47:45.360 --> 01:47:49.830
That basically, there's
a spread on your input.

01:47:49.830 --> 01:47:52.050
There's variation on your input.

01:47:52.050 --> 01:47:54.030
And some of the methods
we've talked about

01:47:54.030 --> 01:47:59.430
might include just recognizing
that and recognizing

01:47:59.430 --> 01:48:04.140
that that might propagate
through the data.

01:48:04.140 --> 01:48:07.980
Because often we pretend
in all of our DOEs

01:48:07.980 --> 01:48:10.860
that when I pick the
input, it's rock solid

01:48:10.860 --> 01:48:12.960
and it's never rock solid.

01:48:12.960 --> 01:48:15.450
Usually it's very well,
tightly controlled.

01:48:15.450 --> 01:48:19.170
But often there is
variation in that.

01:48:19.170 --> 01:48:24.300
And the basic approach,
that might be a place where

01:48:24.300 --> 01:48:28.140
looking at your analytic
model might actually

01:48:28.140 --> 01:48:33.180
give you a sense of
how big a perturbation

01:48:33.180 --> 01:48:37.540
variations on the input
might produce in the output.

01:48:37.540 --> 01:48:44.610
So you could actually get
an estimate of roughly

01:48:44.610 --> 01:48:49.800
how important it is to control
more accurately to the inputs.

01:48:49.800 --> 01:48:50.830
Things like that.

01:48:50.830 --> 01:48:54.468
So I think that's a very
interesting real life problem.

01:48:54.468 --> 01:48:55.760
And that's just one suggestion.

01:48:55.760 --> 01:48:59.160
AUDIENCE: There's another issue.

01:48:59.160 --> 01:49:02.190
The flow rate was the cause of
all our troubles throughout.

01:49:02.190 --> 01:49:05.880
Because we couldn't find a
proper correlation analytically

01:49:05.880 --> 01:49:08.460
for the flow rate,
because the voltage

01:49:08.460 --> 01:49:12.270
and the other parameters
could be easily incorporated

01:49:12.270 --> 01:49:14.370
into the analytic
model just using

01:49:14.370 --> 01:49:16.080
an extension of Faraday's laws.

01:49:16.080 --> 01:49:19.300
But the flow rate was a
little difficult to capture.

01:49:19.300 --> 01:49:21.480
DUANE BONING: So it's
not in the model.

01:49:21.480 --> 01:49:23.220
AUDIENCE: It's not
really in the model.

01:49:23.220 --> 01:49:24.110
DUANE BONING: I see.

01:49:24.110 --> 01:49:24.720
OK.

01:49:24.720 --> 01:49:28.405
AUDIENCE: And the way
we calculated flow rate,

01:49:28.405 --> 01:49:30.030
the way we've measured
our flow rate is

01:49:30.030 --> 01:49:32.940
we've got a maximum value,
a minimum value, and a most

01:49:32.940 --> 01:49:34.860
common central value.

01:49:34.860 --> 01:49:38.008
That's the way we've
measured the flow rate.

01:49:38.008 --> 01:49:40.050
DUANE BONING: I think
it'll be interesting to see

01:49:40.050 --> 01:49:43.110
how you dealt with
that or recommendations

01:49:43.110 --> 01:49:45.040
on how to deal with that.

01:49:45.040 --> 01:49:45.690
That's good.

01:49:45.690 --> 01:49:51.840
I think these real life
challenges are a good thing.

01:49:51.840 --> 01:49:52.500
Thanks.

01:49:52.500 --> 01:49:55.390
I think that one sounds good.

01:49:55.390 --> 01:49:56.340
Now let's see.

01:50:00.290 --> 01:50:01.040
I'm trying to see.

01:50:01.040 --> 01:50:03.150
Have we covered
all the projects?

01:50:03.150 --> 01:50:06.175
Or I've left
somebody out, right?

01:50:06.175 --> 01:50:07.050
AUDIENCE: [INAUDIBLE]

01:50:07.050 --> 01:50:07.842
DUANE BONING: Yeah.

01:50:11.130 --> 01:50:12.645
I actually have to run.

01:50:16.320 --> 01:50:16.890
Let me see.

01:50:16.890 --> 01:50:20.193
Which one are you?

01:50:20.193 --> 01:50:24.900
AUDIENCE: We're the team working
on the injection molding data.

01:50:24.900 --> 01:50:27.022
DUANE BONING: Ah, yes, OK.

01:50:27.022 --> 01:50:28.230
AUDIENCE: Die casting, sorry.

01:50:30.858 --> 01:50:32.400
We're working on
die casting process.

01:50:32.400 --> 01:50:35.553
And we got our
data from a paper.

01:50:35.553 --> 01:50:36.470
DUANE BONING: Ah, yes.

01:50:36.470 --> 01:50:38.790
So we had some
good email on that.

01:50:38.790 --> 01:50:39.330
Right?

01:50:39.330 --> 01:50:39.960
AUDIENCE: Right, right.

01:50:39.960 --> 01:50:40.260
DUANE BONING: Yeah.

01:50:40.260 --> 01:50:42.390
I think you guys are
in very good shape.

01:50:42.390 --> 01:50:44.940
There were some
suggestions from Hayden.

01:50:44.940 --> 01:50:46.532
And I sent an email also.

01:50:46.532 --> 01:50:47.490
AUDIENCE: That's right.

01:50:47.490 --> 01:50:50.400
DUANE BONING: So did you
have follow on questions?

01:50:50.400 --> 01:50:51.840
I think you guys--

01:50:51.840 --> 01:50:54.790
that's good data and you
can do a lot more with it.

01:50:54.790 --> 01:50:59.770
So any additional
thoughts or questions?

01:50:59.770 --> 01:51:02.930
AUDIENCE: So we have a
question concerning inputs.

01:51:02.930 --> 01:51:04.080
There are three inputs.

01:51:04.080 --> 01:51:06.090
Two of them they
have three levels.

01:51:06.090 --> 01:51:08.990
And the other one
has only two levels.

01:51:08.990 --> 01:51:11.790
Does it matter when you do--

01:51:11.790 --> 01:51:15.930
when you do the ANOVA,
ISM, all these analysis?

01:51:15.930 --> 01:51:20.480
Because the levels are
different for inputs.

01:51:20.480 --> 01:51:22.970
DUANE BONING: The
quick answer is no.

01:51:22.970 --> 01:51:26.660
But it should be interesting
for you to show why that still

01:51:26.660 --> 01:51:30.670
is good and that's no problem.

01:51:30.670 --> 01:51:31.520
Just think about it.

01:51:31.520 --> 01:51:34.960
I mean, in terms of
some model dependencies,

01:51:34.960 --> 01:51:36.940
if you have three
levels, you might

01:51:36.940 --> 01:51:41.230
be able to do quadratic
in that model parameter.

01:51:41.230 --> 01:51:43.180
But if you only have
two levels in others,

01:51:43.180 --> 01:51:45.800
you'll only have
linear terms for that.

01:51:45.800 --> 01:51:46.510
So that's fine.

01:51:46.510 --> 01:51:48.480
That's OK.

01:51:48.480 --> 01:51:50.062
AUDIENCE: OK.

01:51:50.062 --> 01:51:50.770
DUANE BONING: OK.

01:51:50.770 --> 01:51:54.280
So if questions come up as
you're working on that one,

01:51:54.280 --> 01:51:55.600
send us more email.

01:51:55.600 --> 01:51:58.030
Because Hayden
had lots of ideas.

01:51:58.030 --> 01:52:01.750
There may be more
than you can do.

01:52:01.750 --> 01:52:05.050
But I think there are some
very interesting ideas,

01:52:05.050 --> 01:52:06.790
because there's replicate data.

01:52:06.790 --> 01:52:09.430
Variance might be
interesting to model.

01:52:09.430 --> 01:52:12.920
Several neat things
you can do with that.

01:52:12.920 --> 01:52:15.310
AUDIENCE: OK.

01:52:15.310 --> 01:52:16.630
We'll follow with emails.

01:52:16.630 --> 01:52:17.380
DUANE BONING: OK.

01:52:17.380 --> 01:52:17.880
Good.

01:52:17.880 --> 01:52:20.440
And we can also again
on Thursday if there's

01:52:20.440 --> 01:52:22.260
more questions that come up.

01:52:22.260 --> 01:52:25.000
After class we can talk briefly.

01:52:25.000 --> 01:52:26.650
Because by then you
should hopefully

01:52:26.650 --> 01:52:27.760
be well into the project.

01:52:30.107 --> 01:52:30.940
AUDIENCE: Thank you.

01:52:30.940 --> 01:52:32.050
DUANE BONING: Thank you.

01:52:32.050 --> 01:52:34.530
See you guys later.