WEBVTT

00:00:00.000 --> 00:00:01.950 align:middle line:90%
[SQUEAKING]

00:00:01.950 --> 00:00:02.450 align:middle line:90%


00:00:02.450 --> 00:00:03.910 align:middle line:90%
[RUSTLING]

00:00:03.910 --> 00:00:04.410 align:middle line:90%


00:00:04.410 --> 00:00:05.880 align:middle line:90%
[CLICKING]

00:00:05.880 --> 00:00:13.050 align:middle line:90%


00:00:13.050 --> 00:00:15.405 align:middle line:84%
RICHARD DE NEUFVILLE: So
what do we do in practice?

00:00:15.405 --> 00:00:18.100 align:middle line:90%


00:00:18.100 --> 00:00:21.640 align:middle line:84%
The simple approach is to
calculate the upper bound.

00:00:21.640 --> 00:00:26.280 align:middle line:84%
This is known as the expected
value of perfect information.

00:00:26.280 --> 00:00:32.960 align:middle line:84%
So this is where I'm going to
be leading you in this exercise.

00:00:32.960 --> 00:00:35.350 align:middle line:84%
So the expected value
of perfect information

00:00:35.350 --> 00:00:37.480 align:middle line:90%
is totally hypothetical.

00:00:37.480 --> 00:00:39.490 align:middle line:90%
It does not exist.

00:00:39.490 --> 00:00:42.370 align:middle line:84%
But it simplifies the
analysis and gives you

00:00:42.370 --> 00:00:45.970 align:middle line:84%
an upper bound, which is useful
because we can definitely

00:00:45.970 --> 00:00:49.330 align:middle line:84%
exclude some tests
if they don't meet--

00:00:49.330 --> 00:00:54.500 align:middle line:84%
if they exceed the upper
bound of the possible value.

00:00:54.500 --> 00:01:00.910 align:middle line:84%
So the concept is that there is
a perfect test, which exactly

00:01:00.910 --> 00:01:03.830 align:middle line:90%
which event is going to occur.

00:01:03.830 --> 00:01:04.860 align:middle line:90%
I will call this--

00:01:04.860 --> 00:01:08.080 align:middle line:84%
I'm calling this a
Cassandra machine.

00:01:08.080 --> 00:01:10.750 align:middle line:84%
The Greek mythology, not that
anybody should necessarily

00:01:10.750 --> 00:01:15.910 align:middle line:84%
know it, is that this
lady was Cassandra

00:01:15.910 --> 00:01:18.940 align:middle line:84%
who had the gift, so to
speak, of always seeing

00:01:18.940 --> 00:01:24.190 align:middle line:84%
what the future would be and
with the curse that nobody

00:01:24.190 --> 00:01:25.210 align:middle line:90%
would ever believe her.

00:01:25.210 --> 00:01:30.010 align:middle line:84%
It was all about
the battle for Troy,

00:01:30.010 --> 00:01:33.550 align:middle line:84%
and she could predict
that somebody would die,

00:01:33.550 --> 00:01:36.590 align:middle line:84%
but nobody would
believe her, and so on.

00:01:36.590 --> 00:01:40.360 align:middle line:84%
So that was the thing, that
she could predict absolutely.

00:01:40.360 --> 00:01:43.360 align:middle line:84%
So this is the
Cassandra machine.

00:01:43.360 --> 00:01:45.550 align:middle line:90%
It predicts absolutely.

00:01:45.550 --> 00:01:50.090 align:middle line:84%
Now, of course,
that does not exist,

00:01:50.090 --> 00:01:52.780 align:middle line:90%
but what is the benefit of it?

00:01:52.780 --> 00:01:57.420 align:middle line:84%
So the Cassandra machine
is this black box

00:01:57.420 --> 00:02:01.530 align:middle line:84%
which predicts exactly what
test result will occur.

00:02:01.530 --> 00:02:04.940 align:middle line:84%
It produces the best
possible information.

00:02:04.940 --> 00:02:07.650 align:middle line:84%
Therefore, I mean,
it's absolutely true.

00:02:07.650 --> 00:02:10.520 align:middle line:84%
So if you had the best
possible information-- that is,

00:02:10.520 --> 00:02:13.670 align:middle line:84%
if you ended up with
no uncertainty--

00:02:13.670 --> 00:02:17.100 align:middle line:84%
you could make the
best decisions,

00:02:17.100 --> 00:02:21.290 align:middle line:84%
and you would have the maximum
gain over the decision you'd

00:02:21.290 --> 00:02:24.920 align:middle line:90%
make without the information.

00:02:24.920 --> 00:02:27.320 align:middle line:84%
Therefore, it gives
you the upper limit

00:02:27.320 --> 00:02:29.550 align:middle line:90%
on the value of the test.

00:02:29.550 --> 00:02:32.810 align:middle line:84%
So what we can do if we
can get the real value,

00:02:32.810 --> 00:02:35.070 align:middle line:84%
we can at least set
some limits on it.

00:02:35.070 --> 00:02:37.370 align:middle line:84%
And this is what the
Cassandra machine,

00:02:37.370 --> 00:02:40.040 align:middle line:90%
this hypothetical thing, does.

00:02:40.040 --> 00:02:43.400 align:middle line:84%
The further beauty of it
is that the calculation

00:02:43.400 --> 00:02:46.220 align:middle line:84%
for this expected value
of perfect information

00:02:46.220 --> 00:02:48.570 align:middle line:90%
is almost trivial.

00:02:48.570 --> 00:02:51.350 align:middle line:84%
The first time you try to do
it, you'll flounder around a bit

00:02:51.350 --> 00:02:52.740 align:middle line:90%
because you're not used to it.

00:02:52.740 --> 00:02:56.840 align:middle line:84%
But basically, it's
a very simple device

00:02:56.840 --> 00:03:00.720 align:middle line:90%
and is pretty transparent.

00:03:00.720 --> 00:03:03.690 align:middle line:90%
So let me give you an example.

00:03:03.690 --> 00:03:07.377 align:middle line:84%
So remember, this is
the raincoat problem,

00:03:07.377 --> 00:03:08.960 align:middle line:84%
that you're getting
up in the morning,

00:03:08.960 --> 00:03:11.570 align:middle line:84%
and should you decide to
take the raincoat or not.

00:03:11.570 --> 00:03:16.250 align:middle line:84%
There may be a chance of
rain, rain or no rain,

00:03:16.250 --> 00:03:18.210 align:middle line:90%
as in the diagram.

00:03:18.210 --> 00:03:21.260 align:middle line:84%
There's a prior assessment
that the probability is

00:03:21.260 --> 00:03:24.740 align:middle line:90%
0.4 or 0.6 of rain or no rain.

00:03:24.740 --> 00:03:29.810 align:middle line:84%
And there is these values of
the situation in the end of 5,

00:03:29.810 --> 00:03:34.220 align:middle line:84%
minus 2, minus 10, and 4, values
that I've invented just to give

00:03:34.220 --> 00:03:35.630 align:middle line:90%
us numbers to play with.

00:03:35.630 --> 00:03:40.260 align:middle line:90%
But that's the original problem.

00:03:40.260 --> 00:03:46.400 align:middle line:84%
Now, just to set ourselves
back into this context,

00:03:46.400 --> 00:03:49.750 align:middle line:84%
the better choice at
this point is what?

00:03:49.750 --> 00:03:52.180 align:middle line:84%
So how do we remind
you how it is?

00:03:52.180 --> 00:03:55.330 align:middle line:84%
If we take the raincoat,
which is a top decision,

00:03:55.330 --> 00:03:59.740 align:middle line:84%
we have a 0.4 chance of
having 5, which averages to 2.

00:03:59.740 --> 00:04:05.410 align:middle line:84%
We have a 0.6 chance
of getting a minus 2,

00:04:05.410 --> 00:04:08.410 align:middle line:90%
which is a minus 1.2.

00:04:08.410 --> 00:04:12.370 align:middle line:90%
2 minus 1.2 is 0.8.

00:04:12.370 --> 00:04:15.490 align:middle line:84%
Conversely, if I don't
take the raincoat,

00:04:15.490 --> 00:04:18.680 align:middle line:84%
I have a probability
of 4 of the minus 10.

00:04:18.680 --> 00:04:19.870 align:middle line:90%
That's minus 4.

00:04:19.870 --> 00:04:24.760 align:middle line:90%
A probability of 0.6 of 4, 2.4.

00:04:24.760 --> 00:04:26.390 align:middle line:90%
So it's a negative number.

00:04:26.390 --> 00:04:30.880 align:middle line:84%
So my best choice is to
take the raincoat, right?

00:04:30.880 --> 00:04:33.580 align:middle line:84%
So I hope that
you're all now back

00:04:33.580 --> 00:04:35.200 align:middle line:90%
into this particular groove.

00:04:35.200 --> 00:04:38.680 align:middle line:90%


00:04:38.680 --> 00:04:46.450 align:middle line:84%
So the setup for the
probabilities for the EVPI.

00:04:46.450 --> 00:04:50.440 align:middle line:84%
It's either going to say it's
going to rain or not rain.

00:04:50.440 --> 00:04:51.820 align:middle line:90%
That is the perfect machine.

00:04:51.820 --> 00:04:54.400 align:middle line:84%
You press the button,
say truth please,

00:04:54.400 --> 00:04:58.690 align:middle line:84%
and it'll say Cassandra
machine says rain.

00:04:58.690 --> 00:05:01.600 align:middle line:84%
Or it will say the
contrary because I've only

00:05:01.600 --> 00:05:03.600 align:middle line:90%
given it two choices.

00:05:03.600 --> 00:05:07.350 align:middle line:84%
Now, what are the probabilities
it's gonna say that?

00:05:07.350 --> 00:05:10.310 align:middle line:84%
Well, your best estimate
of these probabilities

00:05:10.310 --> 00:05:13.010 align:middle line:84%
is what you have
started with before.

00:05:13.010 --> 00:05:16.940 align:middle line:84%
You thought there was a 40%
chance of getting rain, so 40%

00:05:16.940 --> 00:05:20.270 align:middle line:84%
of the time, it's going to
tell you it's going to rain.

00:05:20.270 --> 00:05:22.130 align:middle line:90%
And conversely, so that you--

00:05:22.130 --> 00:05:25.220 align:middle line:84%
a priori, you have
your best estimates

00:05:25.220 --> 00:05:31.220 align:middle line:84%
for the value of the outcome
is what your priors were.

00:05:31.220 --> 00:05:35.660 align:middle line:84%
Now, in general,
once you know what

00:05:35.660 --> 00:05:41.490 align:middle line:84%
the resolution of
the uncertainty is,

00:05:41.490 --> 00:05:43.710 align:middle line:84%
the best decision
is pretty obvious.

00:05:43.710 --> 00:05:45.930 align:middle line:90%
If you knew it would rain--

00:05:45.930 --> 00:05:48.900 align:middle line:84%
going back here, if you
knew it was going to rain,

00:05:48.900 --> 00:05:51.690 align:middle line:90%
you're comparing 5 versus 10.

00:05:51.690 --> 00:05:53.310 align:middle line:90%
Well, you'd take the raincoat.

00:05:53.310 --> 00:05:56.130 align:middle line:84%
If you knew for sure it
wasn't going to rain,

00:05:56.130 --> 00:06:00.900 align:middle line:84%
you compare minus 2 and 4, and
you don't take the raincoat.

00:06:00.900 --> 00:06:06.300 align:middle line:84%
So you don't have to do any
complicated calculations

00:06:06.300 --> 00:06:08.440 align:middle line:90%
of the possibilities.

00:06:08.440 --> 00:06:12.150 align:middle line:84%
You would have a
probability of 0.4

00:06:12.150 --> 00:06:14.340 align:middle line:84%
that would take the
raincoat to get it to 5,

00:06:14.340 --> 00:06:19.560 align:middle line:84%
and the probability of 0.6
of not taking the raincoat

00:06:19.560 --> 00:06:20.970 align:middle line:90%
and getting a result of 4.

00:06:20.970 --> 00:06:27.840 align:middle line:84%
So you can write down your best
decisions pretty automatically.

00:06:27.840 --> 00:06:32.240 align:middle line:84%
Now, the expected
value after this case,

00:06:32.240 --> 00:06:36.330 align:middle line:84%
as I'm running this
calculation, doing the 0.4 times

00:06:36.330 --> 00:06:43.070 align:middle line:84%
the result of 5 and the 0.6 of
4, your perfect value is 4.4.

00:06:43.070 --> 00:06:46.640 align:middle line:84%
Now, the expected value
of perfect information

00:06:46.640 --> 00:06:50.720 align:middle line:84%
is the value of the increment,
not the value at the end of it.

00:06:50.720 --> 00:06:52.220 align:middle line:90%
It's the value of the increment.

00:06:52.220 --> 00:06:54.090 align:middle line:90%
What did you get from it?

00:06:54.090 --> 00:06:57.320 align:middle line:84%
So the expected value
after the test was 4.4.

00:06:57.320 --> 00:07:00.590 align:middle line:84%
The expected value
before the test was 0.8.

00:07:00.590 --> 00:07:03.380 align:middle line:84%
The expected value of perfect
information in this case

00:07:03.380 --> 00:07:09.220 align:middle line:90%
is 3.6.

00:07:09.220 --> 00:07:15.910 align:middle line:84%
All right, so the nice thing
about it is the prior--

00:07:15.910 --> 00:07:18.640 align:middle line:90%
the expected value of simple--

00:07:18.640 --> 00:07:21.530 align:middle line:84%
perfect information is
simple to calculate.

00:07:21.530 --> 00:07:25.740 align:middle line:84%
The prior probability
must equal the probability

00:07:25.740 --> 00:07:30.360 align:middle line:84%
of the perfect test result. Once
you've run the perfect test,

00:07:30.360 --> 00:07:33.660 align:middle line:84%
the posterior probabilities
are either 0 or 1.

00:07:33.660 --> 00:07:35.880 align:middle line:84%
Either this happens,
or it doesn't happen.

00:07:35.880 --> 00:07:37.950 align:middle line:90%
There's no doubt about that.

00:07:37.950 --> 00:07:41.340 align:middle line:84%
Once you know that there is a
whole lot of 0 probabilities

00:07:41.340 --> 00:07:43.800 align:middle line:84%
and there's a one
probability, you

00:07:43.800 --> 00:07:45.390 align:middle line:84%
know what you're
going to choose,

00:07:45.390 --> 00:07:50.325 align:middle line:84%
and pretty much you can
write the expected value

00:07:50.325 --> 00:07:53.100 align:middle line:84%
of perfect information
formula directly.

00:07:53.100 --> 00:07:55.770 align:middle line:84%
There's no need to
use Bayes' Theorem.

00:07:55.770 --> 00:07:58.850 align:middle line:90%


00:07:58.850 --> 00:08:01.550 align:middle line:84%
So Robert, since you asked
the question before--

00:08:01.550 --> 00:08:03.450 align:middle line:84%
I'm not trying to
put you on the spot.

00:08:03.450 --> 00:08:06.950 align:middle line:84%
But since you've been doing
it, and I'm showing you this.

00:08:06.950 --> 00:08:09.570 align:middle line:90%


00:08:09.570 --> 00:08:13.040 align:middle line:84%
Why did you not use this
approach, if you knew about it?

00:08:13.040 --> 00:08:14.930 align:middle line:84%
Or maybe you didn't
know about it.

00:08:14.930 --> 00:08:17.480 align:middle line:90%
So what is the--

00:08:17.480 --> 00:08:19.070 align:middle line:84%
can you have a
discussion about this,

00:08:19.070 --> 00:08:21.560 align:middle line:84%
please, or your
response to this,

00:08:21.560 --> 00:08:24.110 align:middle line:84%
if I'm not embarrassing you or
putting you on the spot, which

00:08:24.110 --> 00:08:25.721 align:middle line:90%
I don't mean.

00:08:25.721 --> 00:08:27.170 align:middle line:90%
AUDIENCE: No, that's fine.

00:08:27.170 --> 00:08:29.945 align:middle line:84%
I actually haven't heard this
perfect information idea.

00:08:29.945 --> 00:08:32.520 align:middle line:90%


00:08:32.520 --> 00:08:35.039 align:middle line:84%
Yeah, so that's probably the
main reason I didn't use this.

00:08:35.039 --> 00:08:38.039 align:middle line:84%
But the other one
was at Chevron,

00:08:38.039 --> 00:08:41.250 align:middle line:84%
we kind of have a
defined procedure

00:08:41.250 --> 00:08:45.150 align:middle line:84%
in terms of how we justify
value of information.

00:08:45.150 --> 00:08:50.100 align:middle line:84%
So that procedure was utilizing
the conditional probabilities.

00:08:50.100 --> 00:08:53.333 align:middle line:84%
But myself, I was personally
a bit uncomfortable

00:08:53.333 --> 00:08:54.750 align:middle line:84%
because like you
were saying, it's

00:08:54.750 --> 00:08:59.010 align:middle line:84%
hard to figure out what the
percentages would be for either

00:08:59.010 --> 00:09:02.430 align:middle line:90%
a good, medium, or bad outcome.

00:09:02.430 --> 00:09:05.370 align:middle line:84%
So it was pretty challenging
to actually apply,

00:09:05.370 --> 00:09:09.630 align:middle line:84%
and I think at the end, people
weren't super confident in it,

00:09:09.630 --> 00:09:10.880 align:middle line:90%
like were saying.

00:09:10.880 --> 00:09:13.130 align:middle line:84%
RICHARD DE NEUFVILLE: Well,
I think that that's right.

00:09:13.130 --> 00:09:17.580 align:middle line:90%
So about 30 or 40 years ago--

00:09:17.580 --> 00:09:19.760 align:middle line:84%
I mean, I'm old enough
to remember that--

00:09:19.760 --> 00:09:24.735 align:middle line:84%
there was a phase where
decision analysis was really

00:09:24.735 --> 00:09:26.360 align:middle line:84%
taught a lot in
business schools, which

00:09:26.360 --> 00:09:28.610 align:middle line:90%
has since gone by.

00:09:28.610 --> 00:09:32.060 align:middle line:84%
And companies picked up on that
because it was certainly better

00:09:32.060 --> 00:09:34.340 align:middle line:90%
than not looking at uncertainty.

00:09:34.340 --> 00:09:39.380 align:middle line:84%
And it was nice to have
this notion of a Bayes'

00:09:39.380 --> 00:09:41.300 align:middle line:90%
rule, a Bayes' Theorem.

00:09:41.300 --> 00:09:47.100 align:middle line:84%
And people thought of themselves
as being Bayesian or not.

00:09:47.100 --> 00:09:49.100 align:middle line:84%
And so there's a whole
history about why

00:09:49.100 --> 00:09:54.090 align:middle line:84%
this was the correct
improvement in approach,

00:09:54.090 --> 00:09:58.980 align:middle line:84%
not thinking about the fact
that these other issues,

00:09:58.980 --> 00:10:00.990 align:middle line:84%
the probabilities-- the
associated probabilities

00:10:00.990 --> 00:10:02.365 align:middle line:84%
and the probability
of occurrence

00:10:02.365 --> 00:10:04.980 align:middle line:84%
and so forth, which are the
part of that special factor

00:10:04.980 --> 00:10:08.010 align:middle line:84%
for the operation of
the Bayes' Theorem

00:10:08.010 --> 00:10:10.620 align:middle line:84%
are just basically hard to
get and hard to document

00:10:10.620 --> 00:10:11.950 align:middle line:90%
and hard to prove.

00:10:11.950 --> 00:10:15.420 align:middle line:84%
So I think that
you are reflecting,

00:10:15.420 --> 00:10:16.770 align:middle line:90%
which may exist elsewhere--

00:10:16.770 --> 00:10:19.110 align:middle line:84%
I'm not pointing a
finger at Chevron--

00:10:19.110 --> 00:10:24.060 align:middle line:84%
but that they have a
decision analysis manual.

00:10:24.060 --> 00:10:26.850 align:middle line:84%
They have an approach, a
procedure, and so forth.

00:10:26.850 --> 00:10:30.690 align:middle line:90%
But it doesn't really fly.

00:10:30.690 --> 00:10:35.530 align:middle line:84%
And as you suggest, that a lot
of people don't believe in it.

00:10:35.530 --> 00:10:39.930 align:middle line:84%
And I would be one of
those, if I were in Chevron,

00:10:39.930 --> 00:10:43.020 align:middle line:90%
because it's such a spongy--

00:10:43.020 --> 00:10:48.690 align:middle line:84%
the calculation is correct,
but the data are spongy.

00:10:48.690 --> 00:10:52.620 align:middle line:84%
And it's the GIGO, garbage in,
garbage out kind of situation,

00:10:52.620 --> 00:10:53.340 align:middle line:90%
I think.

00:10:53.340 --> 00:10:56.880 align:middle line:84%
So I understand that you're
encouraged to do that,

00:10:56.880 --> 00:11:03.030 align:middle line:84%
and this EVPI is
trivial mathematically.

00:11:03.030 --> 00:11:07.010 align:middle line:84%
I mean, you don't have
to be smart or think

00:11:07.010 --> 00:11:09.650 align:middle line:84%
through the complexities, but
it gives you an upper bound.

00:11:09.650 --> 00:11:12.650 align:middle line:84%
And that is the
useful aspect of it.

00:11:12.650 --> 00:11:15.470 align:middle line:84%
And I thank you for
noting that it is not

00:11:15.470 --> 00:11:17.330 align:middle line:84%
the usual way of
approaching things.

00:11:17.330 --> 00:11:19.860 align:middle line:90%


00:11:19.860 --> 00:11:23.190 align:middle line:84%
But I think that it's a
reasonable way to do it,

00:11:23.190 --> 00:11:23.930 align:middle line:90%
and productive.

00:11:23.930 --> 00:11:26.900 align:middle line:84%
And I would encourage you
to take the message back.

00:11:26.900 --> 00:11:29.000 align:middle line:84%
Now, I can see that
they're not going to change

00:11:29.000 --> 00:11:30.750 align:middle line:90%
the procedures for you.

00:11:30.750 --> 00:11:34.140 align:middle line:84%
And so what people do in the
case if the procedure is wrong,

00:11:34.140 --> 00:11:38.200 align:middle line:84%
they sort of try to ignore it,
which is what you reflected.

00:11:38.200 --> 00:11:43.000 align:middle line:90%