WEBVTT

00:00:00.000 --> 00:00:02.470 align:middle line:90%
[SQUEAKING]

00:00:02.470 --> 00:00:03.952 align:middle line:90%
[RUSTLING]

00:00:03.952 --> 00:00:05.434 align:middle line:90%
[CLICKING]

00:00:05.434 --> 00:00:15.662 align:middle line:90%


00:00:15.662 --> 00:00:17.120 align:middle line:84%
RICHARD DE NEUFVILLE:
Hello, again.

00:00:17.120 --> 00:00:21.200 align:middle line:84%
Now, the next section that we'll
be talking about is this topic,

00:00:21.200 --> 00:00:24.200 align:middle line:90%
the forecast is always wrong.

00:00:24.200 --> 00:00:28.520 align:middle line:84%
Now, I put it in quotes
there because it is possible

00:00:28.520 --> 00:00:31.250 align:middle line:84%
that the forecast
actually matches what

00:00:31.250 --> 00:00:33.020 align:middle line:90%
happens later on in the world.

00:00:33.020 --> 00:00:37.010 align:middle line:84%
It is not as if there's some
devil that if you say yes,

00:00:37.010 --> 00:00:39.270 align:middle line:90%
it will say no or vice versa.

00:00:39.270 --> 00:00:42.440 align:middle line:84%
But our experience
is that the forecasts

00:00:42.440 --> 00:00:46.370 align:middle line:84%
we make about the future
tend to differ substantially

00:00:46.370 --> 00:00:48.120 align:middle line:90%
from what happens.

00:00:48.120 --> 00:00:50.540 align:middle line:84%
So the objective
of this exercise

00:00:50.540 --> 00:00:54.830 align:middle line:84%
is to give us a realistic
assessment of our ability

00:00:54.830 --> 00:00:56.870 align:middle line:90%
to forecast accurately.

00:00:56.870 --> 00:00:59.900 align:middle line:84%
You will come away
with the evidence

00:00:59.900 --> 00:01:03.470 align:middle line:84%
that, in general,
the forecast that we

00:01:03.470 --> 00:01:06.740 align:middle line:84%
make for the
performance of a system

00:01:06.740 --> 00:01:10.550 align:middle line:84%
five, 10, 15 years in the
future, or even a year

00:01:10.550 --> 00:01:15.110 align:middle line:84%
or six months, if it's a fast
turnaround kind of technology,

00:01:15.110 --> 00:01:18.840 align:middle line:84%
are different from
what we anticipated.

00:01:18.840 --> 00:01:23.000 align:middle line:84%
Let's look at some evidence
and, first of all, the simplest

00:01:23.000 --> 00:01:27.590 align:middle line:84%
possible case, projection
of project costs.

00:01:27.590 --> 00:01:31.090 align:middle line:84%
So here is a ratio, a
graph showing the ratio,

00:01:31.090 --> 00:01:32.290 align:middle line:90%
of real cost--

00:01:32.290 --> 00:01:33.550 align:middle line:90%
what it actually cost--

00:01:33.550 --> 00:01:35.230 align:middle line:90%
to the estimated cost.

00:01:35.230 --> 00:01:38.710 align:middle line:84%
And on the vertical
axis here, it

00:01:38.710 --> 00:01:43.210 align:middle line:84%
is a percent of observations,
going from 0% up to about 15%

00:01:43.210 --> 00:01:47.800 align:middle line:84%
here, of the whole observations
on a very simple job, which

00:01:47.800 --> 00:01:49.330 align:middle line:90%
I'll explain in a moment.

00:01:49.330 --> 00:01:52.700 align:middle line:84%
So this is how many
observations here.

00:01:52.700 --> 00:01:57.550 align:middle line:84%
And this is the ratio between
the real cost and the estimated

00:01:57.550 --> 00:01:58.430 align:middle line:90%
cost.

00:01:58.430 --> 00:02:01.990 align:middle line:84%
So in this case,
if you had actually

00:02:01.990 --> 00:02:05.240 align:middle line:84%
gotten exactly the
forecast correctly,

00:02:05.240 --> 00:02:08.419 align:middle line:84%
you would have 1
being the ratio.

00:02:08.419 --> 00:02:11.440 align:middle line:84%
And what you see
is that it's very

00:02:11.440 --> 00:02:14.200 align:middle line:84%
different over a
range, as evidenced

00:02:14.200 --> 00:02:18.370 align:middle line:84%
by this histogram of
occurrences, frequency

00:02:18.370 --> 00:02:19.880 align:middle line:90%
of occurrences.

00:02:19.880 --> 00:02:23.040 align:middle line:84%
Now, let me say a little
bit about what this is.

00:02:23.040 --> 00:02:27.320 align:middle line:84%
This represents what possibly
is the simplest possible

00:02:27.320 --> 00:02:29.780 align:middle line:90%
technological activity.

00:02:29.780 --> 00:02:35.690 align:middle line:84%
It represents the cost
of resurfacing runways.

00:02:35.690 --> 00:02:37.460 align:middle line:84%
I told you I was
interested in airports.

00:02:37.460 --> 00:02:39.530 align:middle line:90%
This is an airport example.

00:02:39.530 --> 00:02:43.130 align:middle line:84%
Now, resurfacing runways
is that you basically

00:02:43.130 --> 00:02:46.670 align:middle line:84%
put down a layer of asphalt
over the top of the runway.

00:02:46.670 --> 00:02:50.210 align:middle line:84%
Because after aircraft
pound on it and pound on it

00:02:50.210 --> 00:02:54.590 align:middle line:84%
and pound on it through the
many landings that occur,

00:02:54.590 --> 00:02:56.040 align:middle line:90%
it breaks up a bit.

00:02:56.040 --> 00:03:01.760 align:middle line:84%
So the task is you put down a
determined amount of asphalt,

00:03:01.760 --> 00:03:04.190 align:middle line:84%
a depth, over the
width of the air

00:03:04.190 --> 00:03:06.890 align:middle line:90%
of the runway times the length.

00:03:06.890 --> 00:03:10.730 align:middle line:90%
And the volume is pretty clear.

00:03:10.730 --> 00:03:13.340 align:middle line:84%
The technology is basically
you drive up a truck,

00:03:13.340 --> 00:03:16.900 align:middle line:84%
you dump out the asphalt, you
roll it down, and that it is.

00:03:16.900 --> 00:03:20.760 align:middle line:84%
It could hardly be
more simple than that.

00:03:20.760 --> 00:03:26.780 align:middle line:84%
So you might ask, why is it that
this ratio of the real cost,

00:03:26.780 --> 00:03:29.150 align:middle line:90%
the actual cost, is so large?

00:03:29.150 --> 00:03:31.190 align:middle line:84%
Let me emphasize
how large it is.

00:03:31.190 --> 00:03:37.060 align:middle line:84%
It goes down from about 0.5 on
one scale to about 2 and 1/2

00:03:37.060 --> 00:03:38.660 align:middle line:90%
at the other end of the scale.

00:03:38.660 --> 00:03:44.150 align:middle line:84%
And all this is around a median
value of about 1 and 1/4.

00:03:44.150 --> 00:03:47.090 align:middle line:90%
1.25 is the highest point.

00:03:47.090 --> 00:03:49.880 align:middle line:84%
Now it'd be easy
to understand why

00:03:49.880 --> 00:03:56.540 align:middle line:84%
the median, the most frequent,
value is higher than 1.

00:03:56.540 --> 00:03:59.900 align:middle line:84%
Because while you're
there, you may decide, oh,

00:03:59.900 --> 00:04:02.930 align:middle line:84%
I'll do a little extra taxiway,
I'll do a little extra here.

00:04:02.930 --> 00:04:06.590 align:middle line:84%
And it's our common experience
that their estimates

00:04:06.590 --> 00:04:10.100 align:middle line:84%
are undershooting
what actually happens.

00:04:10.100 --> 00:04:13.400 align:middle line:84%
But what's interesting about
this case, which is so simple,

00:04:13.400 --> 00:04:19.050 align:middle line:84%
is that we go from 1.25 as
a median down to about half

00:04:19.050 --> 00:04:19.550 align:middle line:90%
of it--

00:04:19.550 --> 00:04:22.850 align:middle line:84%
0.5 or 0.6-- to
about double of it.

00:04:22.850 --> 00:04:30.400 align:middle line:84%
That is, our ability to
predict the actual cost

00:04:30.400 --> 00:04:34.000 align:middle line:84%
is off by a factor
of 2-- half or double

00:04:34.000 --> 00:04:36.490 align:middle line:84%
what actually is the
most frequent cost.

00:04:36.490 --> 00:04:38.180 align:middle line:90%
This is really remarkable.

00:04:38.180 --> 00:04:40.930 align:middle line:84%
This is the kind of thing I'd
like you to keep in your mind

00:04:40.930 --> 00:04:42.940 align:middle line:90%
about how difficult it is.

00:04:42.940 --> 00:04:46.400 align:middle line:90%
Now, why does this occur?

00:04:46.400 --> 00:04:49.570 align:middle line:84%
It's not because the designers
couldn't do the math,

00:04:49.570 --> 00:04:52.930 align:middle line:84%
that width times
length times depth

00:04:52.930 --> 00:04:56.540 align:middle line:84%
is a certain amount of asphalt,
and there's how to do it.

00:04:56.540 --> 00:04:58.900 align:middle line:84%
What happens here in
this particular case,

00:04:58.900 --> 00:05:04.960 align:middle line:84%
as in many cases, is that
one of the major determinants

00:05:04.960 --> 00:05:09.130 align:middle line:84%
of this distribution is the
fact that energy prices--

00:05:09.130 --> 00:05:11.290 align:middle line:84%
in this case, petroleum
prices, but they

00:05:11.290 --> 00:05:14.050 align:middle line:84%
tie in with other
energy prices--

00:05:14.050 --> 00:05:15.610 align:middle line:90%
vary enormously.

00:05:15.610 --> 00:05:19.720 align:middle line:84%
That is, the price of oil
during the last 20 years

00:05:19.720 --> 00:05:24.220 align:middle line:84%
or so has gone from about
$15 to $150 per barrel,

00:05:24.220 --> 00:05:27.070 align:middle line:84%
by a factor of 8
or 9 or 10, however

00:05:27.070 --> 00:05:29.980 align:middle line:84%
you wish to talk about
it, but a huge variation.

00:05:29.980 --> 00:05:31.600 align:middle line:84%
And that's what's
reflected here.

00:05:31.600 --> 00:05:33.880 align:middle line:84%
That's one of the things
that's reflected here.

00:05:33.880 --> 00:05:36.970 align:middle line:84%
But it's not just asphalt.
If you're making cement,

00:05:36.970 --> 00:05:40.870 align:middle line:84%
if you are heating steel
and creating steel,

00:05:40.870 --> 00:05:43.210 align:middle line:90%
all these are energy intensive.

00:05:43.210 --> 00:05:46.670 align:middle line:84%
Electricity prices
change the same way.

00:05:46.670 --> 00:05:54.040 align:middle line:84%
So this is largely driven by
factors beyond our control.

00:05:54.040 --> 00:05:57.130 align:middle line:84%
In addition to
these is the factors

00:05:57.130 --> 00:05:58.960 align:middle line:90%
that their labor changes.

00:05:58.960 --> 00:06:03.550 align:middle line:84%
That is, during periods
of high economic activity,

00:06:03.550 --> 00:06:07.360 align:middle line:84%
you'll have to pay
overtime for the workers.

00:06:07.360 --> 00:06:10.990 align:middle line:84%
When there are recessions
or bad economic times,

00:06:10.990 --> 00:06:12.290 align:middle line:90%
you'll get low bids.

00:06:12.290 --> 00:06:15.400 align:middle line:84%
And so in addition to
the materials changes,

00:06:15.400 --> 00:06:17.350 align:middle line:90%
there are the economic changes.

00:06:17.350 --> 00:06:19.900 align:middle line:84%
But in short, this
particular example

00:06:19.900 --> 00:06:23.830 align:middle line:84%
gives you an illustration
of the way that,

00:06:23.830 --> 00:06:26.440 align:middle line:84%
even in the simplest
possible situations,

00:06:26.440 --> 00:06:29.890 align:middle line:84%
that our ability to
predict the actual cost

00:06:29.890 --> 00:06:32.980 align:middle line:90%
is not very good at all.

00:06:32.980 --> 00:06:35.620 align:middle line:84%
This is the kind of
changes that happen.

00:06:35.620 --> 00:06:40.030 align:middle line:84%
Now, to amplify this, I
have a similar kind of graph

00:06:40.030 --> 00:06:44.050 align:middle line:90%
here for all kinds of projects.

00:06:44.050 --> 00:06:47.590 align:middle line:84%
Again, this is on the
vertical axis up here.

00:06:47.590 --> 00:06:51.700 align:middle line:84%
And we're talking
about the average cost

00:06:51.700 --> 00:06:55.610 align:middle line:90%
compared to the predicted cost.

00:06:55.610 --> 00:06:58.060 align:middle line:84%
So the predicted cost
is 1, about here.

00:06:58.060 --> 00:07:00.340 align:middle line:84%
And the averages are
whatever is on here.

00:07:00.340 --> 00:07:03.220 align:middle line:84%
It goes up to about 2 and
1/2 in this particular graph.

00:07:03.220 --> 00:07:06.400 align:middle line:84%
And it's for a whole
variety of projects.

00:07:06.400 --> 00:07:08.410 align:middle line:90%
This was surveyed some time ago.

00:07:08.410 --> 00:07:12.040 align:middle line:84%
And the quality
of this is defined

00:07:12.040 --> 00:07:14.230 align:middle line:84%
by that little bar
over the far side,

00:07:14.230 --> 00:07:17.170 align:middle line:84%
over about in here,
where it talks

00:07:17.170 --> 00:07:19.150 align:middle line:90%
about the standard deviation.

00:07:19.150 --> 00:07:21.020 align:middle line:84%
What's that, the
standard deviation?

00:07:21.020 --> 00:07:23.770 align:middle line:84%
The standard
deviation is a measure

00:07:23.770 --> 00:07:28.550 align:middle line:84%
of how the dispersion
is around the average.

00:07:28.550 --> 00:07:31.300 align:middle line:84%
So in the previous
graph, we saw that there

00:07:31.300 --> 00:07:35.530 align:middle line:84%
was a very wide dispersion
of about a factor of 1.

00:07:35.530 --> 00:07:39.340 align:middle line:84%
But in all these ones here,
the standard deviation

00:07:39.340 --> 00:07:42.070 align:middle line:84%
shows the dispersion,
which is this little bar

00:07:42.070 --> 00:07:45.820 align:middle line:84%
on the far right-hand
side for you, which

00:07:45.820 --> 00:07:50.230 align:middle line:84%
is about a standard
deviation of close to 1/2.

00:07:50.230 --> 00:07:52.540 align:middle line:84%
And these are for
ordinary projects--

00:07:52.540 --> 00:07:57.860 align:middle line:84%
building highways, all kinds of
ordinary kinds of activities.

00:07:57.860 --> 00:08:01.480 align:middle line:84%
So what we see here,
the common experience

00:08:01.480 --> 00:08:06.340 align:middle line:84%
is that the projected
costs in advance

00:08:06.340 --> 00:08:13.090 align:middle line:84%
are under what actually happens,
and under by large amounts,

00:08:13.090 --> 00:08:15.610 align:middle line:84%
both up and down,
high variations,

00:08:15.610 --> 00:08:18.220 align:middle line:84%
the standard
deviation about 50%.

00:08:18.220 --> 00:08:21.550 align:middle line:84%
Plus or minus 50%
is what we get.

00:08:21.550 --> 00:08:25.960 align:middle line:84%
Now, this is for the same
kind of figure but for NASA.

00:08:25.960 --> 00:08:29.650 align:middle line:84%
NASA's the National Aeronautics
and Space Administration.

00:08:29.650 --> 00:08:33.020 align:middle line:84%
And it shows these now
for high-tech groups.

00:08:33.020 --> 00:08:35.409 align:middle line:84%
And what you observe
when we're talking

00:08:35.409 --> 00:08:39.190 align:middle line:84%
about high-tech and more
innovative and less usual

00:08:39.190 --> 00:08:43.809 align:middle line:84%
kinds of activities that the
cost variation is very much

00:08:43.809 --> 00:08:44.890 align:middle line:90%
higher here.

00:08:44.890 --> 00:08:46.930 align:middle line:90%
And now we have--

00:08:46.930 --> 00:08:52.630 align:middle line:84%
a lot of these different
projects went up to about twice

00:08:52.630 --> 00:08:54.190 align:middle line:90%
the estimated cost.

00:08:54.190 --> 00:08:58.820 align:middle line:84%
And over on the right, you
see the standard deviation,

00:08:58.820 --> 00:09:01.730 align:middle line:90%
which is now over 60%.

00:09:01.730 --> 00:09:03.770 align:middle line:90%
And it's more variation.

00:09:03.770 --> 00:09:08.080 align:middle line:84%
So what we observe in each
case that I've shown so far

00:09:08.080 --> 00:09:12.880 align:middle line:84%
is that we don't get it
right, and we don't even

00:09:12.880 --> 00:09:15.050 align:middle line:90%
get it right within a range.

00:09:15.050 --> 00:09:17.470 align:middle line:84%
There's a large range
of possibilities.

00:09:17.470 --> 00:09:21.000 align:middle line:84%
Our forecasts are, as I
suggested at the beginning,

00:09:21.000 --> 00:09:24.220 align:middle line:84%
our forecasts are wrong,
wrong in the sense

00:09:24.220 --> 00:09:28.690 align:middle line:84%
that what actually happens is
not what you said was happening

00:09:28.690 --> 00:09:30.550 align:middle line:84%
or what the expert
said was happening.

00:09:30.550 --> 00:09:34.580 align:middle line:84%
And it's not saying the experts
aren't good at what they do.

00:09:34.580 --> 00:09:36.340 align:middle line:90%
It is that the world changes.

00:09:36.340 --> 00:09:37.970 align:middle line:90%
Things happen.

00:09:37.970 --> 00:09:41.020 align:middle line:84%
So we need to take
this into account

00:09:41.020 --> 00:09:45.340 align:middle line:84%
as we think about our ability
to estimate and manage

00:09:45.340 --> 00:09:48.120 align:middle line:90%
a set of projects.

00:09:48.120 --> 00:09:54.000 align:middle line:90%