WEBVTT

00:00:20.150 --> 00:00:23.600
PETER SZOLOVITS: So
today's topic is workflow,

00:00:23.600 --> 00:00:27.010
and this is something that--

00:00:27.010 --> 00:00:30.820
a topic that I didn't
realize existed

00:00:30.820 --> 00:00:33.970
when I started
working in this area,

00:00:33.970 --> 00:00:37.270
but I've had my nose
ground and ground into it

00:00:37.270 --> 00:00:39.030
for many decades.

00:00:39.030 --> 00:00:42.100
And so finally, it has
become obvious to me

00:00:42.100 --> 00:00:44.000
that it's something
to pay attention to.

00:00:49.100 --> 00:00:51.100
So here's an
interesting question.

00:00:51.100 --> 00:00:54.790
Suppose that your goal
in the kind of work

00:00:54.790 --> 00:00:59.230
that we're doing in this class
is to improve medical care--

00:00:59.230 --> 00:01:02.570
not an unreasonable goal.

00:01:02.570 --> 00:01:04.640
So how do you do it?

00:01:04.640 --> 00:01:08.390
Well, we had an idea
back in the 1970s

00:01:08.390 --> 00:01:10.880
when I was getting
started on this, which

00:01:10.880 --> 00:01:15.620
was that we wanted to
understand what the world's best

00:01:15.620 --> 00:01:19.640
experts did and to
create decision support

00:01:19.640 --> 00:01:24.560
systems by encapsulating
their knowledge about how

00:01:24.560 --> 00:01:27.680
to do diagnosis, how to
do prognosis and treatment

00:01:27.680 --> 00:01:32.330
selection, in order to
improve the performance

00:01:32.330 --> 00:01:35.480
of every other doctor
who was not a world class

00:01:35.480 --> 00:01:40.430
expert by allowing the world
class expertise captured

00:01:40.430 --> 00:01:44.060
in a computer system to
help people figure out

00:01:44.060 --> 00:01:45.350
how to do better--

00:01:45.350 --> 00:01:50.000
so to make them more accurate
diagnosticians, more efficient

00:01:50.000 --> 00:01:52.850
therapists, et cetera.

00:01:52.850 --> 00:01:56.030
And the goal here was
really to bring up

00:01:56.030 --> 00:01:58.490
the average performance
of everybody

00:01:58.490 --> 00:02:00.300
in the health care system.

00:02:00.300 --> 00:02:04.580
So we used to say things
like, bring everybody

00:02:04.580 --> 00:02:08.389
practicing medicine closer
to the level of practice

00:02:08.389 --> 00:02:09.679
of the world class experts.

00:02:13.690 --> 00:02:20.380
Now, that turned out not
to be what was important.

00:02:20.380 --> 00:02:24.610
And so there was another idea
that came along a little bit

00:02:24.610 --> 00:02:28.810
later that said, well,
it's not really so much

00:02:28.810 --> 00:02:32.890
the average performance
of doctors that's bad.

00:02:32.890 --> 00:02:37.340
It's the subaverage performance
that's really terrible.

00:02:37.340 --> 00:02:42.580
And so if you're subaverage
performance leads

00:02:42.580 --> 00:02:48.440
to your patients dying, but your
above average performance only

00:02:48.440 --> 00:02:53.510
makes a moderate difference
in their outcomes,

00:02:53.510 --> 00:02:58.150
then it's clearly more important
to focus on the people who

00:02:58.150 --> 00:03:03.590
are the worst doctors and to
get them to act in a better way.

00:03:03.590 --> 00:03:06.470
And thus, was born the
idea of a protocol that

00:03:06.470 --> 00:03:11.120
says, let's treat similar
patients in similar ways.

00:03:11.120 --> 00:03:15.200
And the value of that is
to reduce the variance--

00:03:15.200 --> 00:03:19.220
so improve average
versus reduce variance.

00:03:19.220 --> 00:03:21.780
So which of these is better?

00:03:21.780 --> 00:03:23.930
Well it depends on
your loss function.

00:03:23.930 --> 00:03:28.550
So as I was suggesting, if your
loss function is a symmetric so

00:03:28.550 --> 00:03:33.050
that doing badly or
doing below average

00:03:33.050 --> 00:03:36.080
is much worse than
doing above average

00:03:36.080 --> 00:03:43.040
is much better, than this
protocol idea of reducing

00:03:43.040 --> 00:03:46.620
variance is really important.

00:03:46.620 --> 00:03:51.050
And this is pretty much what
the medical system has adopted.

00:03:51.050 --> 00:03:54.140
So I wanted to try to
help you visualize this.

00:03:54.140 --> 00:03:59.780
Suppose that on some
arbitrary scale of 0 to 8,

00:03:59.780 --> 00:04:04.670
we have an usual, normal
distribution, of on the left

00:04:04.670 --> 00:04:07.080
the base behaviors--

00:04:07.080 --> 00:04:11.750
so this is how people, on
average, normally behave--

00:04:11.750 --> 00:04:15.210
we assume that there's something
like a normal distribution.

00:04:15.210 --> 00:04:18.940
So here is a world class
expert whose performance

00:04:18.940 --> 00:04:23.090
is up at 6 or 7
and here's the dud

00:04:23.090 --> 00:04:27.330
of a doctor whose performance
is down between 0 and 1.

00:04:27.330 --> 00:04:32.460
And the average doctor
is just shy of 4.

00:04:32.460 --> 00:04:34.230
So here are two scenarios.

00:04:34.230 --> 00:04:38.210
Scenario one is that we
improve these guys performance

00:04:38.210 --> 00:04:39.710
by just a little bit.

00:04:39.710 --> 00:04:45.530
So we improve it by
0.1 performance points,

00:04:45.530 --> 00:04:49.190
I think is what I've
done in this model.

00:04:49.190 --> 00:04:51.500
versus another
approach, which is

00:04:51.500 --> 00:04:54.050
suppose we could cut
down the variance

00:04:54.050 --> 00:04:58.400
dramatically so that this same
normal distribution becomes

00:04:58.400 --> 00:04:59.480
narrower.

00:04:59.480 --> 00:05:02.780
Its average is still in
exactly the same place,

00:05:02.780 --> 00:05:06.800
but now there are
no distant outliers.

00:05:06.800 --> 00:05:10.280
So there aren't doctors
who perform a lot worse,

00:05:10.280 --> 00:05:15.220
and there aren't doctors who
perform a lot better either.

00:05:15.220 --> 00:05:17.530
Well, what happens in that case?

00:05:17.530 --> 00:05:20.050
Well, you have to look
at the cost function.

00:05:20.050 --> 00:05:24.070
So if you have a cost
function like this that says,

00:05:24.070 --> 00:05:29.380
that somebody's performing at
the 0 level has a cost of 1.

00:05:29.380 --> 00:05:32.450
Whereas somebody
performing at the 8 level

00:05:32.450 --> 00:05:38.170
has a cost of almost 0, and
it's exponentially declining

00:05:38.170 --> 00:05:42.220
like this, so that the
average performance has

00:05:42.220 --> 00:05:47.080
a much lower cost than the
average between the worst

00:05:47.080 --> 00:05:50.920
performance and the
best performance.

00:05:50.920 --> 00:05:54.900
So this suggests that,
if you could bunch people

00:05:54.900 --> 00:05:58.950
into this region of performance,
that your overall costs

00:05:58.950 --> 00:06:00.420
would go down.

00:06:00.420 --> 00:06:04.710
And, in fact-- this is a
purely hypothetical model that

00:06:04.710 --> 00:06:05.830
I've built--

00:06:05.830 --> 00:06:08.130
but if you do the
calculations, you

00:06:08.130 --> 00:06:11.610
discover that for the
base distribution,

00:06:11.610 --> 00:06:14.460
here is the
distribution of costs.

00:06:14.460 --> 00:06:17.790
For the slightly
improved distribution,

00:06:17.790 --> 00:06:23.410
you get a cost, which is
1,694 versus 781, again,

00:06:23.410 --> 00:06:25.270
in arbitrary units.

00:06:25.270 --> 00:06:28.140
But if you manage to
narrow the distribution,

00:06:28.140 --> 00:06:31.380
you can get the total cost
down to less than what

00:06:31.380 --> 00:06:34.530
you do by improving the average.

00:06:34.530 --> 00:06:38.550
Now, this is not a proof,
but this is the right idea.

00:06:38.550 --> 00:06:41.160
The proof is
probably in the fact

00:06:41.160 --> 00:06:44.220
that medical systems
have adopted this,

00:06:44.220 --> 00:06:48.690
and have decided that getting
all doctors to behave more

00:06:48.690 --> 00:06:54.150
like the average doctor is the
best practical way of improving

00:06:54.150 --> 00:06:54.960
medical care.

00:06:59.780 --> 00:07:03.230
Well, how do we narrow the
performance distribution?

00:07:03.230 --> 00:07:08.680
So one way is by having
guidelines and protocols where

00:07:08.680 --> 00:07:12.880
you have some learned body who
prescribes appropriate methods

00:07:12.880 --> 00:07:15.680
to diagnose and treat patients.

00:07:15.680 --> 00:07:19.780
So what happens is, for example,
the article here from November

00:07:19.780 --> 00:07:24.610
of 2018, a report of the
American College of Cardiology,

00:07:24.610 --> 00:07:28.960
the American Heart Association
Task Force on Clinical Practice

00:07:28.960 --> 00:07:32.230
Guidelines, and this
has been adopted

00:07:32.230 --> 00:07:36.850
by this cornucopia of
three and four letter

00:07:36.850 --> 00:07:39.820
abbreviated organizations.

00:07:39.820 --> 00:07:42.450
And it's a guideline
on the management

00:07:42.450 --> 00:07:44.240
of blood cholesterol.

00:07:44.240 --> 00:07:49.150
So as you know, having high
cholesterol is dangerous.

00:07:49.150 --> 00:07:52.480
It can lead to heart
attacks and strokes,

00:07:52.480 --> 00:07:55.420
and so there is a
consensus that it would be

00:07:55.420 --> 00:07:58.160
good to lower that in people.

00:07:58.160 --> 00:08:03.100
So these guys went about
this by gathering together

00:08:03.100 --> 00:08:05.600
a bunch of world
experts and saying,

00:08:05.600 --> 00:08:07.170
well, how do we do this?

00:08:07.170 --> 00:08:11.620
What do we promulgate
as the appropriate way

00:08:11.620 --> 00:08:14.780
to care for patients
with this condition?

00:08:14.780 --> 00:08:16.280
And the first thing
they did is they

00:08:16.280 --> 00:08:22.820
came up with a
color coded notion

00:08:22.820 --> 00:08:27.440
of how strong the recommendation
a certain recommendation

00:08:27.440 --> 00:08:31.060
should be And
another color coded

00:08:31.060 --> 00:08:39.659
or shaded level of certainty
in that recommendation.

00:08:39.659 --> 00:08:43.940
So, for example, if you say
something is in class 1,

00:08:43.940 --> 00:08:47.150
so it's a strong
recommendation, then

00:08:47.150 --> 00:08:51.260
you use words like is
recommended, or is indicated,

00:08:51.260 --> 00:08:56.150
useful, effective, beneficial,
should be performed, et cetera.

00:08:56.150 --> 00:09:00.470
If it's in class 2,
where the benefit is

00:09:00.470 --> 00:09:02.900
much greater than
the risk, then you

00:09:02.900 --> 00:09:07.340
say things like it's reasonable,
it can be useful, et cetera.

00:09:07.340 --> 00:09:12.380
If the benefit is maybe equal
to or a little bit better

00:09:12.380 --> 00:09:15.800
than the risk, you say
waffle words, like might be

00:09:15.800 --> 00:09:18.950
reasonable, may be considered.

00:09:18.950 --> 00:09:22.500
If there is no benefit,
in other words,

00:09:22.500 --> 00:09:25.340
if it roughly equals
the risk, then you say,

00:09:25.340 --> 00:09:27.080
it's not recommended.

00:09:27.080 --> 00:09:29.630
And if the risk is
greater than the benefit,

00:09:29.630 --> 00:09:32.870
then you say things like
it's potentially harmful,

00:09:32.870 --> 00:09:34.820
causes harm, et cetera.

00:09:34.820 --> 00:09:37.400
So if you were giving
a recommendation

00:09:37.400 --> 00:09:42.110
on whether to spray
disinfectant down your lungs,

00:09:42.110 --> 00:09:47.390
you might put that in red and
say, this is not recommended.

00:09:47.390 --> 00:09:51.500
And then here,
this shading coding

00:09:51.500 --> 00:09:54.140
is basically how
good is the evidence

00:09:54.140 --> 00:09:56.060
for this recommendation.

00:09:56.060 --> 00:10:02.150
So the best evidence,
the level A,

00:10:02.150 --> 00:10:06.260
is high-quality evidence from
multiple randomized controlled

00:10:06.260 --> 00:10:09.740
clinical trials,
or a meta-analyses

00:10:09.740 --> 00:10:18.560
of a high-quality RCTs,
or RCTs corroborated

00:10:18.560 --> 00:10:21.020
by high-quality
registry studies.

00:10:21.020 --> 00:10:24.320
And then we go down
to level C, which

00:10:24.320 --> 00:10:29.000
is consensus of expert opinion
based on clinical experience,

00:10:29.000 --> 00:10:33.330
but without any sort
of formal analysis.

00:10:33.330 --> 00:10:36.350
So if you look at this
particular document

00:10:36.350 --> 00:10:40.280
on cholesterol it
says, well, here

00:10:40.280 --> 00:10:43.250
are the recommendations
on the measurement

00:10:43.250 --> 00:10:49.420
of LDL and non-HDL cholesterol.

00:10:49.420 --> 00:10:53.630
And they say here,
the confidence

00:10:53.630 --> 00:11:01.940
and the recommendation
is one, and it's based

00:11:01.940 --> 00:11:05.690
on B and our level of evidence.

00:11:05.690 --> 00:11:09.260
And it says, in adults who
are 20 years or older and not

00:11:09.260 --> 00:11:12.980
on lipid-lowering
therapy, measurements

00:11:12.980 --> 00:11:16.040
of either a fasting or a
non-fasting blood-- dot,

00:11:16.040 --> 00:11:17.000
dot, dot.

00:11:17.000 --> 00:11:20.120
So you could read this
in the notes later.

00:11:20.120 --> 00:11:25.820
But notice that there are
high force recommendations.

00:11:25.820 --> 00:11:28.490
There are lower force
recommendations,

00:11:28.490 --> 00:11:31.130
and each recommendation
is also shading

00:11:31.130 --> 00:11:34.040
coded to tell you what
the strength of evidence

00:11:34.040 --> 00:11:36.870
is for this kind
of recommendation.

00:11:36.870 --> 00:11:39.050
Here's just another example.

00:11:39.050 --> 00:11:43.700
This is secondary
atherosclerotic cardiovascular

00:11:43.700 --> 00:11:45.810
disease prevention.

00:11:45.810 --> 00:11:50.310
So this is for somebody
who's already ill,

00:11:50.310 --> 00:11:52.770
and it's a bunch
of recommendations.

00:11:52.770 --> 00:11:57.200
If you're over 75 years
of age, or younger

00:11:57.200 --> 00:12:03.020
with a clinical case of
coronary vascular disease,

00:12:03.020 --> 00:12:07.220
then high intensity
statin therapy

00:12:07.220 --> 00:12:10.820
should be initiated or
continued with the aim

00:12:10.820 --> 00:12:16.760
of achieving a 50% or greater
reduction in LDLC and et

00:12:16.760 --> 00:12:17.370
cetera.

00:12:17.370 --> 00:12:20.870
So again, a whole bunch of
different recommendations.

00:12:20.870 --> 00:12:24.460
Once again, the strength of the
recommendation-- by the way,

00:12:24.460 --> 00:12:28.100
this is just the first
page of a couple of pages--

00:12:28.100 --> 00:12:31.230
and the quality of
evidence for it.

00:12:31.230 --> 00:12:34.940
So this is very
much the way that

00:12:34.940 --> 00:12:37.820
learned societies are
now trying to influence

00:12:37.820 --> 00:12:42.380
the practice of medicine in
order to reduce the variance

00:12:42.380 --> 00:12:47.640
and get everybody to
behave in a normal way.

00:12:47.640 --> 00:12:52.070
You've probably seen articles
about Atul Gawande, who's

00:12:52.070 --> 00:12:56.600
a surgeon here in Boston, and
he's gotten publicly famous

00:12:56.600 --> 00:12:59.360
for advocating checklists.

00:12:59.360 --> 00:13:01.100
And he says, for
example, if you're

00:13:01.100 --> 00:13:04.820
a surgeon, you should act
like an airline pilot,

00:13:04.820 --> 00:13:07.490
that before you take
off in the airplane,

00:13:07.490 --> 00:13:09.950
you go through a
sanity checklist

00:13:09.950 --> 00:13:13.880
to make sure that all the
systems are working properly,

00:13:13.880 --> 00:13:16.730
that all the switches
are set correctly,

00:13:16.730 --> 00:13:18.950
which in a surgical
setting would be things

00:13:18.950 --> 00:13:23.720
like you have all the right
necessary equipment available,

00:13:23.720 --> 00:13:27.320
that you know what to do in
various potential emergencies,

00:13:27.320 --> 00:13:27.910
et cetera.

00:13:33.010 --> 00:13:37.770
So here are their take-home
messages, which makes sense.

00:13:37.770 --> 00:13:41.520
Here, I've abstracted
these from the paper

00:13:41.520 --> 00:13:44.010
that has all of these details.

00:13:44.010 --> 00:13:49.200
So number one, you
go, well, duh--

00:13:49.200 --> 00:13:53.400
in all individuals, emphasize
a heart healthy lifestyle

00:13:53.400 --> 00:13:54.810
across the life course.

00:13:54.810 --> 00:13:59.830
That seems not
terribly controversial,

00:13:59.830 --> 00:14:03.070
and in people who
are already diseased,

00:14:03.070 --> 00:14:08.235
reduce low-density lipoprotein
with high-intensity therapy

00:14:08.235 --> 00:14:10.290
by statins.

00:14:10.290 --> 00:14:16.590
And in very high risk
ASCVD, use a threshold

00:14:16.590 --> 00:14:19.710
of 70 milligrams per
deciliter, et cetera.

00:14:19.710 --> 00:14:22.890
So these are the
summary recommendations.

00:14:22.890 --> 00:14:27.480
And the hope is that doctors
reading these sorts of articles

00:14:27.480 --> 00:14:32.040
come away from them convinced
and will remember that they're

00:14:32.040 --> 00:14:34.530
supposed to act this way
when they're interacting

00:14:34.530 --> 00:14:36.910
with their patients.

00:14:36.910 --> 00:14:41.430
This is a flow chart,
again, abstracted

00:14:41.430 --> 00:14:47.340
from that paper by them
which says, everybody,

00:14:47.340 --> 00:14:51.210
you should emphasize
a healthy lifestyle.

00:14:51.210 --> 00:14:54.420
And then depending on
your age, depending

00:14:54.420 --> 00:15:01.380
on what your estimate
of lifetime risk is,

00:15:01.380 --> 00:15:04.230
you wind up in
different categories.

00:15:04.230 --> 00:15:09.000
And these different categories
have different recommendations

00:15:09.000 --> 00:15:12.530
for what you ought to
do with your patients.

00:15:12.530 --> 00:15:14.870
This is for
secondary prevention.

00:15:14.870 --> 00:15:19.170
So it's a similar flow chart for
people who are already diseased

00:15:19.170 --> 00:15:20.380
and not just at risk.

00:15:23.980 --> 00:15:30.050
And then for people at very
high risk for future events,

00:15:30.050 --> 00:15:33.910
which is defined
by these histories

00:15:33.910 --> 00:15:36.820
and these high-risk
conditions, these

00:15:36.820 --> 00:15:41.170
are the people who fall
into that second flow chart

00:15:41.170 --> 00:15:43.570
and should be treated that way.

00:15:43.570 --> 00:15:49.290
Now, by the way, I
didn't make a poll,

00:15:49.290 --> 00:15:50.850
so I'll give you the answer.

00:15:50.850 --> 00:15:53.870
But it's interesting to ask.

00:15:53.870 --> 00:15:56.960
So when papers like
this get published,

00:15:56.960 --> 00:16:00.870
how well do doctors
actually adhere to these?

00:16:00.870 --> 00:16:03.680
And the answer turns
out to be not very well,

00:16:03.680 --> 00:16:06.710
and it takes many, many
years before these kinds

00:16:06.710 --> 00:16:10.340
of recommendations are
taken up by the majority

00:16:10.340 --> 00:16:17.570
of the community, so even
very, very uncontroversial

00:16:17.570 --> 00:16:19.100
recommendations.

00:16:19.100 --> 00:16:22.280
For example, I think
20 years ago there

00:16:22.280 --> 00:16:26.090
was a recommendation that said
that anybody who's had a heart

00:16:26.090 --> 00:16:30.830
attack should be treated, even
if they're now asymptomatic,

00:16:30.830 --> 00:16:32.100
with beta blockers.

00:16:32.100 --> 00:16:35.750
Because in various
trials, they showed

00:16:35.750 --> 00:16:40.850
that there was a 35% reduction
in repeat heart attacks

00:16:40.850 --> 00:16:44.060
as a result of this treatment.

00:16:44.060 --> 00:16:51.140
It took, I think, over a dozen
years before most doctors

00:16:51.140 --> 00:16:54.440
were aware of this
and started making

00:16:54.440 --> 00:16:57.005
that kind of recommendation
to their patients.

00:17:02.320 --> 00:17:05.060
There's something
called the AHRQ,

00:17:05.060 --> 00:17:08.240
the Agency for Health
Research and Quality.

00:17:08.240 --> 00:17:11.630
And until the current
administration,

00:17:11.630 --> 00:17:15.260
they ran a national
guideline clearinghouse

00:17:15.260 --> 00:17:21.800
that contained myriad of
these guidelines, published

00:17:21.800 --> 00:17:25.069
by different authorities,
and was available for people

00:17:25.069 --> 00:17:29.000
to download and use.

00:17:29.000 --> 00:17:31.700
There's been an attempt
by Guideline Central

00:17:31.700 --> 00:17:36.740
to take over some of these roles
since the government shutdown

00:17:36.740 --> 00:17:42.380
the government run one, and
they have about 2,000 guidelines

00:17:42.380 --> 00:17:45.950
that are posted on their site.

00:17:45.950 --> 00:17:48.360
And these are some
of the examples.

00:17:48.360 --> 00:17:51.650
So risk reduction
of prostate cancer

00:17:51.650 --> 00:17:54.320
with drugs or
nutritional supplements,

00:17:54.320 --> 00:17:57.980
stem cell transplantation
in multiple myeloma,

00:17:57.980 --> 00:18:02.030
stem cell transplantation
in myelodysplastic syndromes

00:18:02.030 --> 00:18:06.770
and acute myeloid
leukemia, et cetera.

00:18:06.770 --> 00:18:10.340
And then they also publish
a bunch of risk calculators

00:18:10.340 --> 00:18:11.660
that say--

00:18:11.660 --> 00:18:15.950
I don't know what the 4T
score is for heparin-induced

00:18:15.950 --> 00:18:17.720
thrombocytopenia--

00:18:17.720 --> 00:18:20.280
but there are tons
of these as well.

00:18:20.280 --> 00:18:22.850
So there's a clearinghouse
of these things.

00:18:22.850 --> 00:18:25.970
And you, as a practicing
doctor, can go to these.

00:18:25.970 --> 00:18:28.610
Or your hospital can
decide that they're

00:18:28.610 --> 00:18:33.260
going to provide these
guidelines to their doctors,

00:18:33.260 --> 00:18:35.930
and either encourage,
or in some cases,

00:18:35.930 --> 00:18:39.320
coerce them to use the
guidelines in order

00:18:39.320 --> 00:18:43.220
to determine what
their activity is.

00:18:43.220 --> 00:18:47.150
Now, notice that this is a
very top-down kind of activity.

00:18:47.150 --> 00:18:51.650
So it's typically done by
these learned societies that

00:18:51.650 --> 00:18:54.560
bring together
experts to cogitate

00:18:54.560 --> 00:18:57.140
on what the right thing
to do is, and then they

00:18:57.140 --> 00:18:59.870
tell the rest of the
world how to do it.

00:18:59.870 --> 00:19:03.690
But there's also a kind
of bottom-up activity.

00:19:03.690 --> 00:19:09.250
So there is something
called a "care plan."

00:19:09.250 --> 00:19:14.100
Now, a care plan is
really a nursing term.

00:19:14.100 --> 00:19:16.850
So if you hang
out at a hospital,

00:19:16.850 --> 00:19:21.230
the thing you discover is that
the doctors are evanescent.

00:19:21.230 --> 00:19:23.430
They appear and disappear.

00:19:23.430 --> 00:19:27.800
They're like
elementary particles,

00:19:27.800 --> 00:19:30.110
and they're not
around all the time.

00:19:30.110 --> 00:19:32.900
The people who are
actually taking care of you

00:19:32.900 --> 00:19:34.610
are the nurses.

00:19:34.610 --> 00:19:39.440
And so the nurses have
developed a set of methodologies

00:19:39.440 --> 00:19:42.900
for how to ensure that
they take good care of you,

00:19:42.900 --> 00:19:44.780
and one of them
is the development

00:19:44.780 --> 00:19:46.580
of these care plans.

00:19:46.580 --> 00:19:48.740
And then what
clinical pathways are

00:19:48.740 --> 00:19:51.680
is an attempt to take the
care plans that nurses

00:19:51.680 --> 00:19:54.770
use in taking care
of individuals

00:19:54.770 --> 00:19:57.350
and to generalize from
those and say, well,

00:19:57.350 --> 00:20:02.150
what are the typical ways in
which we take care of patients

00:20:02.150 --> 00:20:04.070
in a particular cohort?

00:20:04.070 --> 00:20:06.180
So I'm going to talk a
little bit about that,

00:20:06.180 --> 00:20:09.650
and one of the papers I gave
you as an optional reading

00:20:09.650 --> 00:20:12.650
for today is about
cow paths, which

00:20:12.650 --> 00:20:19.310
are these attempts to build
generalizations of care plans.

00:20:19.310 --> 00:20:26.460
So this is a care plan from the
Michigan Center for Nursing,

00:20:26.460 --> 00:20:28.730
which is an educational
organization that

00:20:28.730 --> 00:20:32.660
tries to help nurses figure
out how to be good nurses.

00:20:32.660 --> 00:20:35.300
I was very amused when
I was looking for this.

00:20:35.300 --> 00:20:39.590
I ran across a video, which is
some experienced nurse talking

00:20:39.590 --> 00:20:42.650
about how you build
these care plans.

00:20:42.650 --> 00:20:46.520
And she sort of says, well,
when you're in nursing school,

00:20:46.520 --> 00:20:50.630
you learn how to build these
very elaborate carefully

00:20:50.630 --> 00:20:52.490
constructed care plans.

00:20:52.490 --> 00:20:54.680
When you're actually
practicing as a nurse,

00:20:54.680 --> 00:20:57.210
you'll never have
time to do this.

00:20:57.210 --> 00:21:00.510
And so you're going to do a
rough approximation to this.

00:21:00.510 --> 00:21:01.830
And don't worry about it.

00:21:01.830 --> 00:21:05.450
But for now, satisfy your
professors by doing these

00:21:05.450 --> 00:21:07.880
exercises correctly.

00:21:07.880 --> 00:21:10.500
So take a look at this.

00:21:10.500 --> 00:21:12.260
So there are a bunch of columns.

00:21:12.260 --> 00:21:14.430
The leftmost one
says assessment.

00:21:14.430 --> 00:21:17.330
So this is objective,
subjective,

00:21:17.330 --> 00:21:19.620
and medical diagnostic data.

00:21:19.620 --> 00:21:23.720
So the objective data is this
patient has gangrene-infected

00:21:23.720 --> 00:21:24.740
left foot--

00:21:24.740 --> 00:21:28.680
not a good thing, an open
wound, et cetera, et cetera.

00:21:28.680 --> 00:21:32.150
Subjective data, the patient
said the pain is worse

00:21:32.150 --> 00:21:34.490
when walking and turning.

00:21:34.490 --> 00:21:37.190
She dreads physical
therapy, and she

00:21:37.190 --> 00:21:41.220
wishes she did not have
to be in this situation--

00:21:41.220 --> 00:21:42.990
surprise.

00:21:42.990 --> 00:21:44.850
But that's definitely
subjective.

00:21:44.850 --> 00:21:48.360
You can't see external
evidence of that.

00:21:48.360 --> 00:21:51.510
The nursing diagnosis
is that this patient

00:21:51.510 --> 00:21:56.190
has impaired tissue
integrity in reference

00:21:56.190 --> 00:21:59.890
to the wound and the
presence of an infection.

00:21:59.890 --> 00:22:06.900
Now, that diagnosis actually
comes with a kind of guideline

00:22:06.900 --> 00:22:09.120
about how to make
that diagnosis.

00:22:09.120 --> 00:22:12.270
In other words, in order
to be able to put that down

00:22:12.270 --> 00:22:14.910
on the care plan,
she has to make sure

00:22:14.910 --> 00:22:16.920
that characteristics
of the patient

00:22:16.920 --> 00:22:21.060
satisfy certain criteria
which are the definition

00:22:21.060 --> 00:22:24.060
of that diagnosis.

00:22:24.060 --> 00:22:28.290
The patient outcomes--
so this is the goals

00:22:28.290 --> 00:22:30.660
that the nurse is
trying to achieve.

00:22:30.660 --> 00:22:33.220
And notice, there
are five goals here.

00:22:33.220 --> 00:22:36.750
One is that the patient will
report any altered sensation

00:22:36.750 --> 00:22:43.470
of pain at the tissue impairment
between January 23 and 24.

00:22:43.470 --> 00:22:46.770
So this is a very specific goal.

00:22:46.770 --> 00:22:49.410
It says, the
patient will tell me

00:22:49.410 --> 00:22:53.380
that they feel
better, that there's

00:22:53.380 --> 00:22:59.950
a change in their feeling
in their infected left foot.

00:22:59.950 --> 00:23:07.090
They will understand the plan to
heal tissue and prevent injury.

00:23:07.090 --> 00:23:09.870
So there's a patient
education component.

00:23:09.870 --> 00:23:13.590
They will describe measures to
protect and heal the tissue,

00:23:13.590 --> 00:23:16.230
including wound care by 124.

00:23:16.230 --> 00:23:19.770
So notice, this is the
patient describing to you

00:23:19.770 --> 00:23:22.440
what you are planning to do
for them, in other words,

00:23:22.440 --> 00:23:26.130
demonstrating an understanding
of what the plan is

00:23:26.130 --> 00:23:29.230
and what's likely
to happen with them.

00:23:29.230 --> 00:23:33.060
Experience a wound decrease
that decreases in size

00:23:33.060 --> 00:23:36.000
and has increased
granulation tissue,

00:23:36.000 --> 00:23:38.820
and achieve functional
pain goal of 0

00:23:38.820 --> 00:23:41.790
by 124 per the
patient's verbalization.

00:23:41.790 --> 00:23:44.070
So when they come
in and they ask you

00:23:44.070 --> 00:23:50.790
on that pain scale, are
you at a 0, or a 10,

00:23:50.790 --> 00:23:53.400
or somewhere in
between, the goal

00:23:53.400 --> 00:23:55.770
is that the patient
will say, I'm at a 0,

00:23:55.770 --> 00:23:58.380
in other words, no pain.

00:23:58.380 --> 00:24:00.210
Now, what are the interventions?

00:24:00.210 --> 00:24:02.580
Well, these are the
things that the nurse

00:24:02.580 --> 00:24:06.930
plans to do in order to
try to achieve those goals.

00:24:06.930 --> 00:24:09.720
And then the rationale
is an explanation

00:24:09.720 --> 00:24:14.640
of why it's reasonable to expect
those interventions to achieve

00:24:14.640 --> 00:24:16.110
those goals.

00:24:16.110 --> 00:24:18.480
And the evaluation
of outcomes says,

00:24:18.480 --> 00:24:25.860
what criteria or what are
the actual outcomes for what

00:24:25.860 --> 00:24:27.670
we're trying to achieve?

00:24:27.670 --> 00:24:30.480
So that gets filled
in later, obviously,

00:24:30.480 --> 00:24:33.300
then when the plan is made.

00:24:33.300 --> 00:24:35.790
So if you look at a
website like this,

00:24:35.790 --> 00:24:41.340
there are templated care plans
for many, many conditions.

00:24:41.340 --> 00:24:44.490
You can see that I'm
only up to C in an A to Z

00:24:44.490 --> 00:24:48.960
listing from this one website,
and there are plenty of others.

00:24:48.960 --> 00:24:53.520
But there is an admission care
plan, adult failure to thrive,

00:24:53.520 --> 00:24:59.790
alcohol withdrawal, runny
nose, altered cardiac output,

00:24:59.790 --> 00:25:01.260
amputation.

00:25:01.260 --> 00:25:03.900
I don't know what
an anasarca is--

00:25:03.900 --> 00:25:06.910
anemia, angina, anticoagulant
care, et cetera.

00:25:06.910 --> 00:25:09.750
So there are tons of
different conditions

00:25:09.750 --> 00:25:11.670
that different
patients fall into,

00:25:11.670 --> 00:25:17.910
and this is a way of trying to
list the template care plans.

00:25:17.910 --> 00:25:20.760
Now, this paper is
kind of interesting,

00:25:20.760 --> 00:25:24.390
by Yiye Zhang and colleagues.

00:25:24.390 --> 00:25:30.100
And what they did
is they said, well,

00:25:30.100 --> 00:25:33.090
let's take all these
care plans and let's

00:25:33.090 --> 00:25:36.660
try to build a machine learning
system that learns what

00:25:36.660 --> 00:25:40.230
are the typical patterns that
are embedded in those care

00:25:40.230 --> 00:25:41.490
plans.

00:25:41.490 --> 00:25:43.410
But they didn't
start with the plans.

00:25:43.410 --> 00:25:45.080
This is retrospective analysis.

00:25:45.080 --> 00:25:48.480
So what they started with
is the actual records

00:25:48.480 --> 00:25:51.450
of what was done
to each patient.

00:25:51.450 --> 00:25:54.480
And so the idea is that
you get treatment data

00:25:54.480 --> 00:25:57.240
from the electronic
health record.

00:25:57.240 --> 00:26:01.440
Then you identify patient
subgroups from that data,

00:26:01.440 --> 00:26:04.680
and then you mine for
common treatment patterns.

00:26:04.680 --> 00:26:07.770
And you have medical
experts evaluate these,

00:26:07.770 --> 00:26:10.680
and these then become
clinical pathways,

00:26:10.680 --> 00:26:13.730
which are this generalization
of the care plans

00:26:13.730 --> 00:26:17.680
to particular
subpopulations of patients.

00:26:17.680 --> 00:26:22.080
So the idea is that they
define a bunch of abstractions.

00:26:22.080 --> 00:26:25.080
So they say, look,
an event is a visit.

00:26:25.080 --> 00:26:27.510
So, for example,
for an outpatient,

00:26:27.510 --> 00:26:31.080
anything that happens to you
during one visit to a doctor

00:26:31.080 --> 00:26:33.840
or to a hospital.

00:26:33.840 --> 00:26:37.320
So it's a set of procedures,
a set of medications,

00:26:37.320 --> 00:26:38.440
a set of diagnoses.

00:26:41.910 --> 00:26:43.740
And by the way,
they were focusing

00:26:43.740 --> 00:26:50.580
on people with kidney disease
as the target population

00:26:50.580 --> 00:26:52.530
that they were looking at.

00:26:52.530 --> 00:26:56.970
So then they say,
OK, individual events

00:26:56.970 --> 00:27:01.230
are going to be abstracted into
these supernodes, which capture

00:27:01.230 --> 00:27:05.400
a unique combination of
associations of events

00:27:05.400 --> 00:27:08.070
associated with some visit.

00:27:08.070 --> 00:27:12.660
So you might worry that this
is going to be combinatorial,

00:27:12.660 --> 00:27:16.140
because there are many possible
combinations of things.

00:27:16.140 --> 00:27:18.330
And that is, in fact,
a bit of a problem,

00:27:18.330 --> 00:27:20.740
I think, in their analysis.

00:27:20.740 --> 00:27:23.070
So now, you have
these supernodes,

00:27:23.070 --> 00:27:26.580
and then each patient
has a visit sequence,

00:27:26.580 --> 00:27:29.970
which is a time-ordered
list of the supernodes.

00:27:29.970 --> 00:27:32.520
So every time you
go see your doctor,

00:27:32.520 --> 00:27:35.010
you have one new supernode.

00:27:35.010 --> 00:27:37.860
And so you have a
time series of these.

00:27:37.860 --> 00:27:39.790
And then they do
the following thing.

00:27:39.790 --> 00:27:44.580
They say, gee, when we talk
to our doctors and nurses,

00:27:44.580 --> 00:27:47.550
they tell us that
they care mostly

00:27:47.550 --> 00:27:52.080
about what happened at the last
visit that the patient had.

00:27:52.080 --> 00:27:54.660
But they also care
a little bit less,

00:27:54.660 --> 00:27:57.750
but they still care about what
happened at the visit previous

00:27:57.750 --> 00:28:03.120
to that, but not so much about
history going further back.

00:28:03.120 --> 00:28:06.210
And so they say, well,
in a Markov chain,

00:28:06.210 --> 00:28:11.520
we only have things depend on
the last node in the Markov

00:28:11.520 --> 00:28:12.610
chain.

00:28:12.610 --> 00:28:15.060
So let's change
the model here so

00:28:15.060 --> 00:28:17.430
that we will combine
pairs of visits

00:28:17.430 --> 00:28:22.110
into nodes so that each
node in the Markov chain

00:28:22.110 --> 00:28:27.300
will represent the last two
visits that the patient had.

00:28:27.300 --> 00:28:30.450
So this could, again, cause
some combinatorial problems.

00:28:30.450 --> 00:28:34.020
But here's the image
that they come up with.

00:28:34.020 --> 00:28:36.150
So there are individual items.

00:28:36.150 --> 00:28:39.240
Is it a hospital visit,
an office visit, a visit

00:28:39.240 --> 00:28:41.860
for the purpose of education?

00:28:41.860 --> 00:28:45.530
Are you in chronic kidney
disease stage four?

00:28:48.150 --> 00:28:50.730
Was an ultrasound done?

00:28:50.730 --> 00:28:54.630
Were you given ACE inhibitors?

00:28:54.630 --> 00:28:56.800
Were you given
diuretics, et cetera?

00:28:56.800 --> 00:28:59.340
So these are all the
data that we mentioned.

00:28:59.340 --> 00:29:02.160
They treat that as a bag.

00:29:02.160 --> 00:29:04.860
And then they say,
OK, we're going

00:29:04.860 --> 00:29:10.650
to identify all the bags that
have the same exact content.

00:29:10.650 --> 00:29:15.400
An asterisk, they didn't
look, for example,

00:29:15.400 --> 00:29:17.880
at the dose of medication
that you were given,

00:29:17.880 --> 00:29:20.250
only which medication it was.

00:29:20.250 --> 00:29:23.080
So there are some
collapsing that way.

00:29:23.080 --> 00:29:26.430
Then the supernodes
are these combinations

00:29:26.430 --> 00:29:31.230
where we say, OK, you
had a particular purpose,

00:29:31.230 --> 00:29:34.470
a particular diagnosis,
a particular set

00:29:34.470 --> 00:29:38.320
of interventions, a
particular set of procedures.

00:29:38.320 --> 00:29:41.940
And again, we list all
possible combinations of those,

00:29:41.940 --> 00:29:46.120
and then that sequence
represents your sequence.

00:29:46.120 --> 00:29:48.990
These are aggregated
into supernodes.

00:29:48.990 --> 00:29:51.540
That represents
your visit sequence,

00:29:51.540 --> 00:29:54.540
and then these super
pairs are this hack

00:29:54.540 --> 00:29:58.800
to let you look two steps
back in the Markov chain.

00:29:58.800 --> 00:30:03.270
And so they wind up with
about 3,500 different

00:30:03.270 --> 00:30:05.220
of these super pair nodes.

00:30:05.220 --> 00:30:07.860
So it is combinatorial,
but it's not terribly

00:30:07.860 --> 00:30:09.660
combinatorial in their data.

00:30:14.350 --> 00:30:18.570
They then compute the
maximum of the length

00:30:18.570 --> 00:30:22.740
of common subsequences between
each pair of visit sequences.

00:30:22.740 --> 00:30:25.350
So they're going to
cluster these sequences.

00:30:25.350 --> 00:30:28.620
They define a
distance function that

00:30:28.620 --> 00:30:34.140
says that the more they
share a common sequence,

00:30:34.140 --> 00:30:36.810
the less distant they
are from each other.

00:30:36.810 --> 00:30:38.610
And the particular
distance function

00:30:38.610 --> 00:30:43.140
they used is the length of
each sequence minus twice

00:30:43.140 --> 00:30:46.560
the length of the common
subsequence, the longest

00:30:46.560 --> 00:30:50.190
common subsequence, which
seems pretty reasonable.

00:30:50.190 --> 00:30:54.330
And then hierarchical clustering
into distinct subgroups,

00:30:54.330 --> 00:30:58.740
they came up with 31 groups
for this group of patients,

00:30:58.740 --> 00:31:01.350
and here they are.

00:31:01.350 --> 00:31:06.360
And what you see is
that some of them

00:31:06.360 --> 00:31:08.830
don't differ a whole
lot from each other.

00:31:08.830 --> 00:31:12.270
So, for example,
these two differ only

00:31:12.270 --> 00:31:17.220
in that the patient got some
medication and diuretics

00:31:17.220 --> 00:31:21.190
in one case and just that
medication in the other case.

00:31:21.190 --> 00:31:24.360
So these are-- it is a
hierarchical cluster,

00:31:24.360 --> 00:31:28.050
and the things lower
down in the clustering

00:31:28.050 --> 00:31:30.600
are probably fairly
close to each other.

00:31:30.600 --> 00:31:32.490
Nevertheless, what
they're able to do,

00:31:32.490 --> 00:31:35.910
then, is to estimate
a transition matrix

00:31:35.910 --> 00:31:42.180
among these supernode
pair states,

00:31:42.180 --> 00:31:45.180
and they can look at
different trajectories

00:31:45.180 --> 00:31:49.510
depending on the degree
of support for the data.

00:31:49.510 --> 00:31:52.020
So you can set
different thresholds

00:31:52.020 --> 00:31:57.390
on how many cases have to be
in a particular state in order

00:31:57.390 --> 00:32:02.910
for you to take transitions to
or from that state seriously.

00:32:02.910 --> 00:32:05.310
One of the critiques I
would make of the study

00:32:05.310 --> 00:32:10.410
is that they had way too little
data, and so many of the groups

00:32:10.410 --> 00:32:12.420
that they came up
with had relatively

00:32:12.420 --> 00:32:18.810
small numbers of patients in
them, which is unfortunate.

00:32:18.810 --> 00:32:21.990
Now, once you have these
transition matrices,

00:32:21.990 --> 00:32:26.280
then you can say, OK,
for cluster 29, which

00:32:26.280 --> 00:32:34.580
was this cluster, so
there were a grand total

00:32:34.580 --> 00:32:38.360
of 14 patients in this cluster.

00:32:38.360 --> 00:32:40.760
They were all at chronic
kidney disease stage

00:32:40.760 --> 00:32:42.770
4, so quite severe.

00:32:42.770 --> 00:32:44.300
They were all hypertensive.

00:32:44.300 --> 00:32:49.450
They were all on ACE
inhibitors and statins,

00:32:49.450 --> 00:32:54.200
and everybody in that group
had that categorization.

00:32:54.200 --> 00:32:57.720
So if you look there
then you can say, OK,

00:32:57.720 --> 00:33:01.020
for all the things we
know about that patient,

00:33:01.020 --> 00:33:06.650
what are the probabilistic
relationships between them?

00:33:06.650 --> 00:33:09.840
And what we find is that--

00:33:09.840 --> 00:33:11.990
man, I can't read these.

00:33:11.990 --> 00:33:16.340
So these nodes
imply other nodes,

00:33:16.340 --> 00:33:22.490
and the strength of the arrows
is proportional to their width.

00:33:22.490 --> 00:33:26.180
And so this is a
representation of everything

00:33:26.180 --> 00:33:28.640
that we've learned
about that cluster,

00:33:28.640 --> 00:33:31.490
but remember, only
from those 14 patients.

00:33:31.490 --> 00:33:34.490
So I'm not sure I would
take this to the bank

00:33:34.490 --> 00:33:37.340
and rely on it too intensely.

00:33:37.340 --> 00:33:40.640
But they then, by
hand, abstract it

00:33:40.640 --> 00:33:46.610
and say, well, let's look at
an interpretation of this.

00:33:46.610 --> 00:33:49.610
And so if they look
in typical patterns

00:33:49.610 --> 00:33:51.920
that they see in
that cluster, they

00:33:51.920 --> 00:33:58.130
say, hmm, we see an
office visit in which

00:33:58.130 --> 00:34:02.060
the patient is on
these medications

00:34:02.060 --> 00:34:04.070
and has these procedures.

00:34:04.070 --> 00:34:05.870
Then they're hospitalized.

00:34:05.870 --> 00:34:12.500
Then there's
another-- let's see.

00:34:12.500 --> 00:34:13.340
No, I'm sorry.

00:34:17.290 --> 00:34:19.139
Yeah, yellow node
is an office visit.

00:34:19.139 --> 00:34:21.230
So they're hospitalized.

00:34:21.230 --> 00:34:23.210
They then get an
education visit,

00:34:23.210 --> 00:34:26.540
so that's typically with the
nurse or nurse practitioner

00:34:26.540 --> 00:34:29.130
to explain to them what
they ought to be doing.

00:34:29.130 --> 00:34:30.679
They have another hospital--

00:34:30.679 --> 00:34:32.600
they have another office visit.

00:34:32.600 --> 00:34:34.280
They have a hospital visit.

00:34:34.280 --> 00:34:38.210
They have another hospital
visit, and then they die.

00:34:38.210 --> 00:34:43.130
So that, unfortunately,
is a not atypical pattern

00:34:43.130 --> 00:34:47.210
that you see in patients who
are at a pretty severe state

00:34:47.210 --> 00:34:49.639
of chronic kidney disease.

00:34:49.639 --> 00:34:52.219
And we don't know
from this diagram

00:34:52.219 --> 00:34:58.830
how long this process
takes to take place.

00:34:58.830 --> 00:35:01.520
So I have some questions.

00:35:01.520 --> 00:35:04.190
There are a lot of subgroups.

00:35:04.190 --> 00:35:08.220
Some of them were fairly
similar to others.

00:35:08.220 --> 00:35:13.810
They have between 10 and 158
patients in each subgroup.

00:35:13.810 --> 00:35:15.720
So I would feel
much better if they

00:35:15.720 --> 00:35:22.590
had between 1,000 and
15,000 or something

00:35:22.590 --> 00:35:27.220
patients in each group, or
150,000 patients in each group.

00:35:27.220 --> 00:35:32.850
I would feel much more
believing in the representations

00:35:32.850 --> 00:35:34.200
that they found.

00:35:34.200 --> 00:35:36.090
And the other
problem is that even

00:35:36.090 --> 00:35:38.190
within an individual
subgroup, you can

00:35:38.190 --> 00:35:40.360
find very different patterns.

00:35:40.360 --> 00:35:46.350
So, for example, here is a
pattern where, again, a person

00:35:46.350 --> 00:35:48.360
has a couple of office visits.

00:35:48.360 --> 00:35:50.160
They go to the hospital.

00:35:50.160 --> 00:36:01.515
Or they go to the hospital
twice with slightly different--

00:36:01.515 --> 00:36:02.015
yes.

00:36:02.015 --> 00:36:06.970
So this person at this point
is in acute kidney injury.

00:36:06.970 --> 00:36:10.170
So you can get there either
directly from the office visit

00:36:10.170 --> 00:36:12.960
or from an earlier
hospitalization,

00:36:12.960 --> 00:36:14.710
and then they die.

00:36:14.710 --> 00:36:17.440
And so this is part
of that pattern.

00:36:17.440 --> 00:36:22.380
But here's another pattern mined
from exactly the same subgroup.

00:36:22.380 --> 00:36:25.840
Now, this subgroup has
122 patients in it,

00:36:25.840 --> 00:36:28.290
so there's a little
bit more heterogeneity.

00:36:28.290 --> 00:36:30.570
But what you see here
is that a patient

00:36:30.570 --> 00:36:35.550
is going back and forth between
education visits and doctor's

00:36:35.550 --> 00:36:38.370
visits, back and
forth between doctors

00:36:38.370 --> 00:36:42.300
visits and hospitalizations,
then a hospitalization, then

00:36:42.300 --> 00:36:46.750
another hospitalization,
but they're surviving.

00:36:46.750 --> 00:36:52.360
So it's a little bit tricky,
but I think this is a good idea,

00:36:52.360 --> 00:36:53.970
but there are
probably improvements

00:36:53.970 --> 00:36:56.370
that are possible on
the technique that's

00:36:56.370 --> 00:36:57.720
being used here.

00:36:57.720 --> 00:37:00.780
And, of course, much more
data would be very helpful

00:37:00.780 --> 00:37:02.940
in order to really
delineate what's

00:37:02.940 --> 00:37:04.200
going on in these patients.

00:37:10.670 --> 00:37:14.570
Here's a similar idea
that I was involved.

00:37:14.570 --> 00:37:18.710
Jeff Klann did his
PhD at Regenstrief,

00:37:18.710 --> 00:37:22.550
which is a very well-known,
very early adopter

00:37:22.550 --> 00:37:27.240
of computerized information
systems in Indiana.

00:37:27.240 --> 00:37:31.712
And so what he started
off-- and he said, hmm.

00:37:34.310 --> 00:37:36.770
You know the Amazon
recommendation system

00:37:36.770 --> 00:37:42.230
that says you just
bought this camera lends,

00:37:42.230 --> 00:37:45.800
and other people who bought
this camera lens also

00:37:45.800 --> 00:37:49.460
bought a cleaning kit
and a battery that goes

00:37:49.460 --> 00:37:52.100
with that camera, and so on?

00:37:52.100 --> 00:37:54.650
So he said, why don't
we apply that same idea

00:37:54.650 --> 00:37:56.630
to medical orders?

00:37:56.630 --> 00:38:01.400
And so he took the record of
all the orders at Regenstrief,

00:38:01.400 --> 00:38:04.100
and he basically
built an approximation

00:38:04.100 --> 00:38:07.740
to the Amazon recommendation
system that said,

00:38:07.740 --> 00:38:11.300
hey, other doctors who have
ordered the following set

00:38:11.300 --> 00:38:14.750
of tests have also ordered
this additional test

00:38:14.750 --> 00:38:16.400
that you didn't order.

00:38:16.400 --> 00:38:19.130
Maybe you should
consider doing it.

00:38:19.130 --> 00:38:23.840
Or conversely, other doctors who
have ordered this set of tests

00:38:23.840 --> 00:38:28.400
have never ordered this
other one in addition.

00:38:28.400 --> 00:38:30.650
And so are you sure
you really need it?

00:38:30.650 --> 00:38:33.860
So that was the idea.

00:38:33.860 --> 00:38:37.430
And what he did was
he focused on four

00:38:37.430 --> 00:38:39.540
different clinical issues.

00:38:39.540 --> 00:38:42.470
So one of them was an
emergency department visit

00:38:42.470 --> 00:38:48.110
for back pain, pregnancy,
so labor and delivery,

00:38:48.110 --> 00:38:50.790
hypertension in the
urgent visit clinic--

00:38:50.790 --> 00:38:54.870
so the urgent visit clinic
is one of these lower-level

00:38:54.870 --> 00:38:59.180
non-emergency department,
cheaper, lower level of care,

00:38:59.180 --> 00:39:03.140
but still urgent care kinds
of clinics that many hospitals

00:39:03.140 --> 00:39:06.890
have established in order to try
to keep people who are not that

00:39:06.890 --> 00:39:10.430
sick out of the emergency
department and in this

00:39:10.430 --> 00:39:13.700
lower-intensity clinic--

00:39:13.700 --> 00:39:16.250
and hypertension, and
high blood pressure,

00:39:16.250 --> 00:39:19.340
and then altered mental state
in the intensive care unit.

00:39:19.340 --> 00:39:23.270
So people in the ICU
are often medicated,

00:39:23.270 --> 00:39:26.930
and they become wacko,
and so this is trying

00:39:26.930 --> 00:39:28.970
to take care of such patients.

00:39:28.970 --> 00:39:31.250
They used three years
of encountered data

00:39:31.250 --> 00:39:33.440
from Regenstrief.

00:39:33.440 --> 00:39:37.670
And for each domain,
they limited themselves

00:39:37.670 --> 00:39:42.710
to the 40 most frequent orders,
and, again, low granularity.

00:39:42.710 --> 00:39:45.050
So, for example,
drug, but not the dose

00:39:45.050 --> 00:39:49.400
of the drug for
medications, and the 10 most

00:39:49.400 --> 00:39:56.240
frequent comorbidities or
co-occurring diagnoses.

00:39:56.240 --> 00:40:00.770
So this is an example of wisdom
of the crowd kind of approach

00:40:00.770 --> 00:40:04.970
that says, well, what
your colleagues do

00:40:04.970 --> 00:40:07.490
is probably a good
representation of what

00:40:07.490 --> 00:40:09.470
you ought to be doing.

00:40:09.470 --> 00:40:13.745
Now, what's an obvious
pitfall of this approach?

00:40:16.586 --> 00:40:18.810
I'm just checking to
see if you're awake.

00:40:18.810 --> 00:40:20.540
Yeah?

00:40:20.540 --> 00:40:23.517
AUDIENCE: Just reinforce
whatever's [INAUDIBLE]..

00:40:23.517 --> 00:40:25.350
PETER SZOLOVITS: Yeah,
if they're all bozos,

00:40:25.350 --> 00:40:27.460
they're going to train
you to be a bozo too.

00:40:29.970 --> 00:40:32.220
And there's a lot
of stuff in medicine

00:40:32.220 --> 00:40:35.310
that is not very
well-supported by evidence,

00:40:35.310 --> 00:40:38.640
where, in fact, people
have developed traditions

00:40:38.640 --> 00:40:41.400
of doing things a certain way
that may not be the right way

00:40:41.400 --> 00:40:42.460
to do it.

00:40:42.460 --> 00:40:45.290
And this just reinforces that.

00:40:45.290 --> 00:40:48.500
On the other hand, it
probably does reduce variance

00:40:48.500 --> 00:40:51.380
in the sense that we talked
about at the beginning.

00:40:51.380 --> 00:40:54.650
And so, as a result, it may
be a reasonable approach,

00:40:54.650 --> 00:40:58.010
if you're willing to
tolerate some exceptions.

00:40:58.010 --> 00:41:02.180
My favorite story is
Semmelweiss figured out

00:41:02.180 --> 00:41:07.580
that having a baby in
a hospital in Vienna

00:41:07.580 --> 00:41:12.020
was extremely dangerous
for the mother,

00:41:12.020 --> 00:41:13.730
because they would
die of what was

00:41:13.730 --> 00:41:19.070
called "child bed fever," which
was basically an infection.

00:41:19.070 --> 00:41:22.160
And Semmelweiss figured
out that maybe there

00:41:22.160 --> 00:41:24.440
was-- this was before Pasteur.

00:41:24.440 --> 00:41:26.180
But he figured out
that maybe there

00:41:26.180 --> 00:41:29.180
was something that was being
transmitted from one woman

00:41:29.180 --> 00:41:33.560
to the next that was causing
this child bed fever,

00:41:33.560 --> 00:41:35.030
and, of course, he was right.

00:41:35.030 --> 00:41:39.620
And he did an experiment,
where on his maternity ward,

00:41:39.620 --> 00:41:43.370
he had all of the
younger doctors

00:41:43.370 --> 00:41:47.450
wash their hands with some
sort of alcohol or something

00:41:47.450 --> 00:41:50.930
to kill whatever they
were transmitting.

00:41:50.930 --> 00:41:55.220
And their death rate
from this child bed fever

00:41:55.220 --> 00:41:57.730
dropped to almost 0.

00:41:57.730 --> 00:42:01.800
And he went to his colleagues
and he said, hey, guys, we

00:42:01.800 --> 00:42:04.650
could really make the
world a better place

00:42:04.650 --> 00:42:06.720
and stop killing women.

00:42:06.720 --> 00:42:10.200
And they looked at
him, and they said,

00:42:10.200 --> 00:42:15.900
you know, these hands
heal, they don't kill.

00:42:15.900 --> 00:42:20.490
Many of them were upper
class or noblemen who

00:42:20.490 --> 00:42:22.500
had gone into this profession.

00:42:22.500 --> 00:42:26.460
The idea that somehow they were
responsible for transmitting

00:42:26.460 --> 00:42:29.520
what turns out to
be bacteria was just

00:42:29.520 --> 00:42:31.380
a non-starter for them.

00:42:31.380 --> 00:42:33.930
And Semmelweiss wound
up ending his days

00:42:33.930 --> 00:42:37.630
in a mental institution,
because he went nuts.

00:42:37.630 --> 00:42:40.680
He was unable to
change practice even

00:42:40.680 --> 00:42:45.810
though he had done an experiment
to demonstrate that it worked.

00:42:45.810 --> 00:42:48.780
So this is a case where
the wisdom of the crowd

00:42:48.780 --> 00:42:52.560
was not so good and
led to bad outcomes.

00:42:55.230 --> 00:42:59.040
So like Amazon's
recommendation system,

00:42:59.040 --> 00:43:02.670
it automates the learning
of decision support rules.

00:43:02.670 --> 00:43:07.440
And what's attractive about
this is that because it's

00:43:07.440 --> 00:43:12.810
induced from real data, it tends
to deal with more complex cases

00:43:12.810 --> 00:43:17.010
than the sort of simple,
stereotypical cases

00:43:17.010 --> 00:43:19.620
for which people can
develop guidelines,

00:43:19.620 --> 00:43:22.110
for example, where
they can anticipate

00:43:22.110 --> 00:43:24.960
what's going to happen
in various circumstances.

00:43:24.960 --> 00:43:28.320
So he used the Bayesian
networking model

00:43:28.320 --> 00:43:32.760
that used diagnoses possible
orders and evidence, which

00:43:32.760 --> 00:43:36.790
is the results from orders
that were already completed.

00:43:36.790 --> 00:43:39.840
There's a system out of
University of Pittsburgh,

00:43:39.840 --> 00:43:43.950
called Tetrad, that implements
a nice version of something

00:43:43.950 --> 00:43:46.470
called Greedy
Equivalent Search, which

00:43:46.470 --> 00:43:51.190
is a faster way of
searching through the space

00:43:51.190 --> 00:43:55.290
of Bayesian networks for
an appropriate network that

00:43:55.290 --> 00:43:57.420
represents your data.

00:43:57.420 --> 00:44:01.800
So it's a highly
combinatorial problem,

00:44:01.800 --> 00:44:05.460
and the cleverness in this
is that it figures out

00:44:05.460 --> 00:44:09.630
classes of Bayesian networks
that, by definition, would

00:44:09.630 --> 00:44:11.790
fit the data equally well.

00:44:11.790 --> 00:44:15.960
And it does it by class rather
than by individual network,

00:44:15.960 --> 00:44:20.110
and so it gets a nice
combinatorial reduction.

00:44:20.110 --> 00:44:26.250
And what Jeff found is, for
example, in the pregnancy

00:44:26.250 --> 00:44:29.670
network, these
are the nodes that

00:44:29.670 --> 00:44:33.150
correspond to
various interventions

00:44:33.150 --> 00:44:35.710
and various conditions.

00:44:35.710 --> 00:44:41.070
And this is the Bayesian network
that best fits that data.

00:44:41.070 --> 00:44:44.220
It's reasonably complicated.

00:44:44.220 --> 00:44:46.230
Here are some others.

00:44:46.230 --> 00:44:50.230
This is for the emergency
department case.

00:44:50.230 --> 00:44:54.660
So you see that you have things
like chest pain and abdominal

00:44:54.660 --> 00:44:58.140
pain presenting
diagnoses, and then

00:44:58.140 --> 00:45:00.750
you have various
procedures, like an abdomen

00:45:00.750 --> 00:45:06.060
CT, or a pelvic CT, or
a chest CT, or a head

00:45:06.060 --> 00:45:10.080
CT, or a basic metabolic
panel, et cetera,

00:45:10.080 --> 00:45:12.630
and this gives you the
probabilistic relationships

00:45:12.630 --> 00:45:14.560
between them.

00:45:14.560 --> 00:45:21.240
And so what they were able to
do is to take this Bayesian

00:45:21.240 --> 00:45:24.510
network representation,
and then if you

00:45:24.510 --> 00:45:29.700
lay a particular patient's
data on that representation,

00:45:29.700 --> 00:45:33.850
that corresponds to fixing
the value of certain nodes.

00:45:33.850 --> 00:45:36.720
And then you do Bayesian
inference to figure out

00:45:36.720 --> 00:45:39.780
the probabilities of
the unobserved nodes,

00:45:39.780 --> 00:45:43.440
and you recommend the highest
probability interventions

00:45:43.440 --> 00:45:46.330
that have not yet been done.

00:45:46.330 --> 00:45:48.240
So it's a little bit
like, if you remember,

00:45:48.240 --> 00:45:51.100
we talked about
sequential diagnosis.

00:45:51.100 --> 00:45:53.020
This is a little
bit in that spirit,

00:45:53.020 --> 00:45:57.030
but it's a much more complicated
Bayesian network model rather

00:45:57.030 --> 00:46:00.490
than a naive-based model.

00:46:00.490 --> 00:46:03.550
And so the interface
looks like this.

00:46:03.550 --> 00:46:07.680
You have-- it's called the
Iterative Treatment Suggestions

00:46:07.680 --> 00:46:11.400
algorithm, and it
shows the doctor

00:46:11.400 --> 00:46:15.050
that these are the
problems of the patient,

00:46:15.050 --> 00:46:18.000
and the current orders, and
the probability that you

00:46:18.000 --> 00:46:23.580
might ask to have any
one of these orders done.

00:46:23.580 --> 00:46:30.690
And what they're able to show is
that this does reasonably well.

00:46:30.690 --> 00:46:33.300
Obviously, it wouldn't
have been published if they

00:46:33.300 --> 00:46:35.680
hadn't been able to show that.

00:46:35.680 --> 00:46:42.300
And so what you see is that, for
example, the next order that's

00:46:42.300 --> 00:46:47.010
done in an inpatient pregnancy
using this Bayesian network

00:46:47.010 --> 00:46:53.070
formalism has a position of
about fourth on the list.

00:46:53.070 --> 00:46:57.220
So their criterion for
judging this algorithm

00:46:57.220 --> 00:47:01.270
is, is it raising the
things that people actually

00:47:01.270 --> 00:47:04.420
do too high on the
list of the recommended

00:47:04.420 --> 00:47:07.510
list, on the recommended
set of actions

00:47:07.510 --> 00:47:09.250
that you consider doing?

00:47:09.250 --> 00:47:12.400
And you see that it's
fourth, on average,

00:47:12.400 --> 00:47:16.420
in inpatient pregnancy,
about sixth in the ICU,

00:47:16.420 --> 00:47:19.330
about sixth in the
emergency department,

00:47:19.330 --> 00:47:22.630
and about fifth in the
urgent care clinic.

00:47:22.630 --> 00:47:24.370
So that's pretty
good, because that

00:47:24.370 --> 00:47:27.520
means that even if you're
looking at an iPhone,

00:47:27.520 --> 00:47:30.730
there's enough screen
real estate that it'll

00:47:30.730 --> 00:47:34.540
be on the so-called first
page of Google hits,

00:47:34.540 --> 00:47:39.280
which is the only thing
people ever pay attention to.

00:47:39.280 --> 00:47:43.390
And, in fact, they can
show that the average list

00:47:43.390 --> 00:47:48.320
position corresponds to the
order rank by frequency,

00:47:48.320 --> 00:47:53.770
but that their model does a
reasonably good job of keeping

00:47:53.770 --> 00:48:01.510
you within the first 10 or
so for much of this range.

00:48:05.750 --> 00:48:08.300
I'm going to shift gears again.

00:48:08.300 --> 00:48:10.610
So Adam Right, you've met.

00:48:10.610 --> 00:48:14.570
He was discussant in one
of our earlier classes.

00:48:14.570 --> 00:48:18.290
And Adam's been very active
in trying to deploy decision

00:48:18.290 --> 00:48:19.880
support systems.

00:48:19.880 --> 00:48:25.420
And he had an interesting
episode back in--

00:48:25.420 --> 00:48:28.140
when was this--

00:48:28.140 --> 00:48:29.100
2016.

00:48:29.100 --> 00:48:32.040
So it must have been
a little before 2016.

00:48:32.040 --> 00:48:36.000
He went to demonstrate this
great decision support system

00:48:36.000 --> 00:48:39.780
that they had implemented
at the Brigham,

00:48:39.780 --> 00:48:45.030
and he put in a fake case where
an alert should have gone off

00:48:45.030 --> 00:48:51.960
for a patient who has been on
a particular drug for more than

00:48:51.960 --> 00:48:58.260
a year and needs to have their
thyroid stimulating hormone

00:48:58.260 --> 00:49:01.590
measured in order to check
for a potential side effect

00:49:01.590 --> 00:49:07.120
of long-term use of amiodarone,
as well as to have their--

00:49:07.120 --> 00:49:11.100
ALT is a liver test,
liver enzyme test.

00:49:11.100 --> 00:49:13.410
So they needed both
of those tests.

00:49:13.410 --> 00:49:16.710
He was demonstrating
this wonderful system.

00:49:16.710 --> 00:49:20.340
He put in a fake patient
who had these conditions,

00:49:20.340 --> 00:49:22.980
and the alert didn't go off.

00:49:22.980 --> 00:49:27.870
So he goes, hmm,
what's going on?

00:49:27.870 --> 00:49:30.930
And they went back,
and they discovered

00:49:30.930 --> 00:49:39.240
that in 2009 the system's
internal code for amiodarone

00:49:39.240 --> 00:49:43.040
had been changed
from 40 to 70-99.

00:49:43.040 --> 00:49:45.160
Who knows why?

00:49:45.160 --> 00:49:47.340
But the rule logic in
the system was never

00:49:47.340 --> 00:49:49.810
updated to reflect this change.

00:49:49.810 --> 00:49:55.230
And so, in fact, if
you look at the history

00:49:55.230 --> 00:49:57.907
of the use of amiodarone--

00:49:57.907 --> 00:49:59.490
by the way, it's an
interesting graph.

00:49:59.490 --> 00:50:03.780
The blue dots are weekdays, and
the black dots are weekends.

00:50:03.780 --> 00:50:07.950
So not a lot goes on in the
hospital during the weekend.

00:50:07.950 --> 00:50:11.250
But what you see is that--

00:50:11.250 --> 00:50:15.060
I don't know what happened
before about the end of 2009.

00:50:15.060 --> 00:50:18.240
They probably weren't running
that rule or something.

00:50:18.240 --> 00:50:21.210
But what you see is sort
of a gradual increase

00:50:21.210 --> 00:50:26.010
in the use of this rule, and
then you see a long decrease

00:50:26.010 --> 00:50:32.640
from 2010 up through 2013 when
they discovered this problem.

00:50:32.640 --> 00:50:34.350
Now, why a decrease?

00:50:34.350 --> 00:50:36.420
I mean, it's not a
sudden jump to 0.

00:50:39.030 --> 00:50:43.290
And the reason was
that this came about--

00:50:43.290 --> 00:50:45.750
first of all, it
came about gradually,

00:50:45.750 --> 00:50:50.700
because the people who had had
this drug before that change

00:50:50.700 --> 00:50:55.290
in the software had
gotten the old code, which

00:50:55.290 --> 00:50:57.630
was still triggering the rule.

00:50:57.630 --> 00:51:01.050
It's just that as time went
on, more and more people

00:51:01.050 --> 00:51:06.810
who needed the test had gotten
the drug with its new code.

00:51:06.810 --> 00:51:12.360
And with that new code, it was
no longer triggering the rule.

00:51:12.360 --> 00:51:15.900
And then this is the point at
which they discovered the bug,

00:51:15.900 --> 00:51:17.040
and then they fixed it.

00:51:17.040 --> 00:51:18.840
Of course, it came
right back up again.

00:51:27.280 --> 00:51:28.270
Oh.

00:51:28.270 --> 00:51:31.550
Well, I'll talk about some
of the others as well.

00:51:31.550 --> 00:51:36.670
So this was the amiodarone case.

00:51:36.670 --> 00:51:40.510
So it fell suddenly, as
some patients were taken off

00:51:40.510 --> 00:51:44.530
the drug and others were started
with this new internal code.

00:51:47.760 --> 00:51:52.270
And as I said, the alert
logic was fixed back in 2013.

00:51:52.270 --> 00:51:52.770
Yeah?

00:51:52.770 --> 00:51:55.062
AUDIENCE: So I don't know
how hospital IT systems work,

00:51:55.062 --> 00:51:56.705
and it might vary
from place to place.

00:51:56.705 --> 00:51:59.020
But is there ever a notion
of like this computer needs

00:51:59.020 --> 00:52:00.890
to be updated for the
software, but that one already

00:52:00.890 --> 00:52:01.416
got updated?

00:52:01.416 --> 00:52:03.915
Or are they all synced
up so that they all

00:52:03.915 --> 00:52:05.380
get updated at the same time?

00:52:05.380 --> 00:52:06.922
PETER SZOLOVITS:
They tend to all get

00:52:06.922 --> 00:52:08.680
updated at the same time.

00:52:08.680 --> 00:52:11.650
There are disasters
that have happened

00:52:11.650 --> 00:52:13.990
in that updating process.

00:52:13.990 --> 00:52:18.220
Famously, the Beth Israel was
down for about three days.

00:52:18.220 --> 00:52:21.430
Their computer
system just crashed.

00:52:21.430 --> 00:52:24.760
And what they discovered is that
they had this very complicated

00:52:24.760 --> 00:52:31.330
network in which there were
cyclic dependencies in order

00:52:31.330 --> 00:52:33.580
to boot up different systems.

00:52:33.580 --> 00:52:35.830
So some system had
to be up in order

00:52:35.830 --> 00:52:38.440
to let some other
system be up, which

00:52:38.440 --> 00:52:42.040
had to be up in order to
let the first system be up.

00:52:42.040 --> 00:52:45.310
And, of course, in
normal operation,

00:52:45.310 --> 00:52:47.270
they never take down
the whole system,

00:52:47.270 --> 00:52:50.860
and so nobody had discovered
this until there was--

00:52:50.860 --> 00:52:53.380
Cisco screwed them.

00:52:53.380 --> 00:52:58.360
There was some fix in the
routers that caused everything

00:52:58.360 --> 00:53:01.700
to crash, and then they
couldn't bring it back up again.

00:53:01.700 --> 00:53:04.330
And so that was a big panic.

00:53:04.330 --> 00:53:06.900
John Halamka, who's
the CIO there,

00:53:06.900 --> 00:53:08.590
is a former student of mine.

00:53:08.590 --> 00:53:13.450
And after this all
played out, I asked John,

00:53:13.450 --> 00:53:16.610
so what's the first thing
you did when this happened?

00:53:16.610 --> 00:53:19.690
And he said, I sent a
couple of panel trucks

00:53:19.690 --> 00:53:28.180
down to the Staples warehouse
to buy pads of paper,

00:53:28.180 --> 00:53:29.230
which is pretty smart.

00:53:33.190 --> 00:53:34.630
So here's another example.

00:53:34.630 --> 00:53:36.980
This is lead screening.

00:53:36.980 --> 00:53:40.720
And so this was a case where
there is a lead screening

00:53:40.720 --> 00:53:42.010
rule for two-year-olds.

00:53:42.010 --> 00:53:45.970
There is also one for one-,
three-, and four-year-olds.

00:53:45.970 --> 00:53:49.180
And there was no change in
screening for one-, three-,

00:53:49.180 --> 00:53:53.260
and four-year-olds, but the
screening for two-year-olds

00:53:53.260 --> 00:54:01.000
went from 300 or 400 a day down
to 0 for several years before

00:54:01.000 --> 00:54:06.850
they noticed it, and then went
back up to the previous level.

00:54:06.850 --> 00:54:10.960
And they never did quite
figure out what happened here,

00:54:10.960 --> 00:54:17.140
but something added two
incomplete clauses to the rule

00:54:17.140 --> 00:54:20.920
having to do with gender
and smoking status.

00:54:20.920 --> 00:54:25.510
But the clauses were incomplete,
and so they were actually

00:54:25.510 --> 00:54:31.150
looking for the case of neither
the gender nor the smoking

00:54:31.150 --> 00:54:33.880
status having been specified.

00:54:33.880 --> 00:54:36.160
So smoking status
for a two-year-old,

00:54:36.160 --> 00:54:39.580
you could imagine, is
not often specified,

00:54:39.580 --> 00:54:42.550
but gender typically is.

00:54:42.550 --> 00:54:47.170
And so the rule never
fired because of that,

00:54:47.170 --> 00:54:50.780
and they have no idea how
these changes were made.

00:54:50.780 --> 00:54:53.560
There's a complicated
logging system

00:54:53.560 --> 00:54:58.100
that logs all the changes, and
it crashed and lost its logging

00:54:58.100 --> 00:54:58.600
data.

00:54:58.600 --> 00:55:03.830
And it's a just so story.

00:55:03.830 --> 00:55:09.570
Chlamydia screen--
this was human error.

00:55:09.570 --> 00:55:13.430
And so they wound
up-- they found

00:55:13.430 --> 00:55:18.230
this very quickly, because they
had a two-month-old boy who

00:55:18.230 --> 00:55:21.440
had numerous duplicate
reminders, including

00:55:21.440 --> 00:55:25.490
suggestions for
mammograms, pap smears,

00:55:25.490 --> 00:55:28.670
pneumococcal vaccination,
and cholesterol

00:55:28.670 --> 00:55:32.090
screening, and a
suggestion to start

00:55:32.090 --> 00:55:34.190
the patient on various meds.

00:55:34.190 --> 00:55:38.330
So this was just a human
error in revising the rule,

00:55:38.330 --> 00:55:40.640
and that one they
found pretty quickly.

00:55:40.640 --> 00:55:44.120
So that's amusing.

00:55:44.120 --> 00:55:48.110
But what's interesting is these
guys went on to say, well, how

00:55:48.110 --> 00:55:52.100
could we monitor for this
in some ongoing fashion?

00:55:52.100 --> 00:55:54.530
And so they said, well,
there's this notion

00:55:54.530 --> 00:55:59.150
of change point detection,
which is an interesting machine

00:55:59.150 --> 00:56:01.280
learning problem, again.

00:56:01.280 --> 00:56:06.110
And so they said, well, suppose
we built a dynamic linear model

00:56:06.110 --> 00:56:08.480
that includes seasonality,
because we have

00:56:08.480 --> 00:56:11.420
to deal with the fact
that a lot of stuff

00:56:11.420 --> 00:56:14.030
happens Monday through
Friday and nothing happens

00:56:14.030 --> 00:56:15.510
on weekends?

00:56:15.510 --> 00:56:19.370
And so they created
a model that says

00:56:19.370 --> 00:56:24.590
that your output is some
function, f, of your inputs,

00:56:24.590 --> 00:56:27.540
plus some noise.

00:56:27.540 --> 00:56:31.760
The noise is Gaussian with
some variance, capital V,

00:56:31.760 --> 00:56:36.440
and that x evolves according
to some evolution that

00:56:36.440 --> 00:56:40.310
says it depends on the
previous value of x,

00:56:40.310 --> 00:56:43.800
plus some other noise,
which is also Gaussian.

00:56:43.800 --> 00:56:48.080
So that's the general sort of
time series modeling approach

00:56:48.080 --> 00:56:49.940
that people often take.

00:56:49.940 --> 00:56:53.120
And then they said, well, we
have to deal with seasonality.

00:56:53.120 --> 00:56:57.200
So what we're going to do is
define a period, namely a week,

00:56:57.200 --> 00:56:59.540
and then we're going
to separate out

00:56:59.540 --> 00:57:03.200
the states on different
days of the week

00:57:03.200 --> 00:57:07.850
in order to give us the ability
to model that seasonality.

00:57:07.850 --> 00:57:09.980
I worked on a different
project having

00:57:09.980 --> 00:57:15.280
to do with outbreak detection
for infectious diseases,

00:57:15.280 --> 00:57:18.140
and there the
periodicity was a year,

00:57:18.140 --> 00:57:21.260
because things like the
flu come in yearly cycles

00:57:21.260 --> 00:57:23.360
rather than in weekly cycles.

00:57:23.360 --> 00:57:26.930
And so that idea
is pretty common.

00:57:26.930 --> 00:57:29.810
And then they built this
multiprocess dynamic linear

00:57:29.810 --> 00:57:34.580
model that says,
basically, imagine

00:57:34.580 --> 00:57:38.240
that our data is
being generated by one

00:57:38.240 --> 00:57:43.380
of a set of these
dynamic linear models.

00:57:43.380 --> 00:57:45.770
And so we have an
additional state variable

00:57:45.770 --> 00:57:48.610
at each time that says
which of the models

00:57:48.610 --> 00:57:54.110
is in control to generate
the data at this point.

00:57:54.110 --> 00:58:00.110
And so if you have the set of
observations up to some time,

00:58:00.110 --> 00:58:03.650
t, then you can
compute the probability

00:58:03.650 --> 00:58:08.940
that model i is driving the
generator at this point.

00:58:08.940 --> 00:58:11.480
And so you can have
three basic models.

00:58:11.480 --> 00:58:14.030
You can have a
model that says it's

00:58:14.030 --> 00:58:16.190
a stable model, in
other words, what

00:58:16.190 --> 00:58:18.770
you expect is the steady state.

00:58:18.770 --> 00:58:22.550
So that would be the normal
weekly variation in volume

00:58:22.550 --> 00:58:24.500
for any of these alerts.

00:58:24.500 --> 00:58:27.710
You can have a model which
is an additive outlier.

00:58:27.710 --> 00:58:30.950
So that's something that says,
all of a sudden, something

00:58:30.950 --> 00:58:33.350
happened, like that
chlamydia screen

00:58:33.350 --> 00:58:37.610
or one of the other things
that had a very quick blip.

00:58:37.610 --> 00:58:40.010
Or you can have a
level shift change,

00:58:40.010 --> 00:58:45.530
like the change that
happened when the screening

00:58:45.530 --> 00:58:48.590
rules or the alert
rule for amiodarone

00:58:48.590 --> 00:58:51.740
stopped firing, because
it went from one level

00:58:51.740 --> 00:58:56.120
to a very different level
over a period of a relatively

00:58:56.120 --> 00:58:58.110
short period of time.

00:58:58.110 --> 00:59:02.120
And then what you can do is
calculate the probability

00:59:02.120 --> 00:59:08.040
of any of these models being
in control at the next time,

00:59:08.040 --> 00:59:10.160
and that's called the
change point score.

00:59:10.160 --> 00:59:14.750
And you can calculate this from
the data that you're given.

00:59:14.750 --> 00:59:17.030
And of course, they
have tons of data.

00:59:17.030 --> 00:59:20.970
It's a big hospital and
lots of these alerts go on.

00:59:20.970 --> 00:59:26.670
And if you plot this, there's
the data for a time series.

00:59:26.670 --> 00:59:29.690
So you see the weekly variation.

00:59:29.690 --> 00:59:32.480
But what you see is
that the probability

00:59:32.480 --> 00:59:40.730
of the steady behavior is quite
high except at certain points

00:59:40.730 --> 00:59:43.130
where it all of a sudden dips.

00:59:43.130 --> 00:59:45.980
And so those are places where
you suspect that something

00:59:45.980 --> 00:59:47.960
interesting is going on.

00:59:47.960 --> 00:59:53.960
And similarly, the probability
of a temporary offset

00:59:53.960 --> 01:00:00.110
goes up at these various points,
and the probability of a level

01:00:00.110 --> 01:00:02.740
shift goes up at this point.

01:00:02.740 --> 01:00:04.230
And you can see
that, indeed, there

01:00:04.230 --> 01:00:07.440
is a level shift
from essentially 0 up

01:00:07.440 --> 01:00:13.440
to this periodic behavior in
the original data sequence.

01:00:13.440 --> 01:00:16.740
And so they actually implemented
this in the hospital,

01:00:16.740 --> 01:00:19.320
and so now you get
not just alerts,

01:00:19.320 --> 01:00:24.090
but you get
meta-alerts that say,

01:00:24.090 --> 01:00:27.510
this kid ought to be screened
for their lead levels,

01:00:27.510 --> 01:00:30.480
but also the lead
level screening rule

01:00:30.480 --> 01:00:33.570
hasn't fired as often as
we expected it to fire.

01:00:41.790 --> 01:00:43.680
Yeah, so there are
a lot of details

01:00:43.680 --> 01:00:46.710
in the paper that you can
look up, if you're interested.

01:00:46.710 --> 01:00:50.250
And what they find
is that, if you

01:00:50.250 --> 01:00:54.480
look at the area under the
delay false positive rate curve,

01:00:54.480 --> 01:00:58.410
so you're trading off how
long it takes to be certain

01:00:58.410 --> 01:01:02.670
that one of these conditions has
occurred versus how often you

01:01:02.670 --> 01:01:09.180
cry wolf, and you see
that their algorithm does

01:01:09.180 --> 01:01:11.580
much better than a
bunch of other things

01:01:11.580 --> 01:01:15.030
that they tried it against,
which are earlier attempts

01:01:15.030 --> 01:01:16.680
to do this.

01:01:16.680 --> 01:01:19.690
And these are all highly
statistically significant,

01:01:19.690 --> 01:01:22.770
so they got a nice
paper out of it.

01:01:25.710 --> 01:01:28.440
In the remaining
time, I wanted to talk

01:01:28.440 --> 01:01:32.550
about a number of other
issues that really

01:01:32.550 --> 01:01:34.290
have to do with workflow.

01:01:34.290 --> 01:01:37.680
So we've talked about
alerting, but there

01:01:37.680 --> 01:01:39.840
are an interesting
set of studies

01:01:39.840 --> 01:01:43.060
about how these alerting
systems actually work.

01:01:43.060 --> 01:01:47.040
So there was a cool idea from
the Beth Israel Deaconess

01:01:47.040 --> 01:01:52.440
Hospital here in Boston
where they said, well,

01:01:52.440 --> 01:01:56.580
what we really need to
do is to escalate alerts.

01:01:56.580 --> 01:02:00.780
So, for example, it's
quite typical in a hospital

01:02:00.780 --> 01:02:04.950
that, if you're a doctor and
you have a patient who you have

01:02:04.950 --> 01:02:07.620
just sent their
blood to the lab,

01:02:07.620 --> 01:02:15.330
and let's say there serum
potassium comes back as 7 or 8,

01:02:15.330 --> 01:02:18.990
that patient is at high risk of
going into cardiac arrhythmia

01:02:18.990 --> 01:02:20.340
and dying.

01:02:20.340 --> 01:02:23.640
And so your pager, in
those days, goes off,

01:02:23.640 --> 01:02:25.800
and you read this text
message that says,

01:02:25.800 --> 01:02:28.920
Mr. Jones has a
serum potassium of 8.

01:02:28.920 --> 01:02:32.580
You'd better look in on him.

01:02:32.580 --> 01:02:34.660
So what they did
was very clever.

01:02:34.660 --> 01:02:38.430
They said, well, the problem is
busy doctors might ignore this.

01:02:38.430 --> 01:02:42.030
And so we'll then start
a countdown timer.

01:02:42.030 --> 01:02:47.100
And we'll say, did Dr.
Smith actually come and look

01:02:47.100 --> 01:02:52.060
at Mr. Jones within 20 minutes?

01:02:52.060 --> 01:02:54.060
And if the answer
is no, then they

01:02:54.060 --> 01:02:58.420
send the page to the
doctor's boss that says,

01:02:58.420 --> 01:03:01.980
hey, we sent this guy a
page, and within 20 minutes

01:03:01.980 --> 01:03:05.460
he didn't look in
on the patient.

01:03:05.460 --> 01:03:08.340
And then they start
another timer.

01:03:08.340 --> 01:03:14.850
And they say, if that boss
doesn't respond within an hour,

01:03:14.850 --> 01:03:18.900
then they send a page to
the head of the hospital

01:03:18.900 --> 01:03:22.440
saying, you're her
infectious disease people

01:03:22.440 --> 01:03:25.120
are doing a lousy job,
because they're not--

01:03:25.120 --> 01:03:28.680
or in this case, you're
endocrine people, or whatever,

01:03:28.680 --> 01:03:31.080
are doing a lousy job,
because they're not

01:03:31.080 --> 01:03:33.820
responding to these alerts.

01:03:33.820 --> 01:03:38.470
Now, how do you think
the doctors liked this?

01:03:38.470 --> 01:03:40.320
Not much.

01:03:40.320 --> 01:03:44.430
And there is a real
problem with overalerting.

01:03:44.430 --> 01:03:46.890
And there is no
general rule that

01:03:46.890 --> 01:03:51.300
says, how often can you bug
the head of the hospital

01:03:51.300 --> 01:03:55.810
with an alert like this before
he or she just says, well,

01:03:55.810 --> 01:03:59.070
turn off the damn thing,
I don't want to see these?

01:03:59.070 --> 01:04:02.130
And clearly, if you set the
thresholds at different places,

01:04:02.130 --> 01:04:04.090
you get different results.

01:04:04.090 --> 01:04:07.650
So, for example, I remember
Tufts implemented a system

01:04:07.650 --> 01:04:12.715
like this back in the 1980s,
but they would send a page

01:04:12.715 --> 01:04:19.240
on every order where any of
the lab results were abnormal,

01:04:19.240 --> 01:04:21.240
and that was way too much.

01:04:21.240 --> 01:04:27.060
Because a lot of these
tests generate 20 results.

01:04:27.060 --> 01:04:31.650
Normal is defined as the
95% confidence interval.

01:04:31.650 --> 01:04:34.110
What are the chances
that out of 20 tests,

01:04:34.110 --> 01:04:38.310
which aren't really independent,
but if they were, one of them

01:04:38.310 --> 01:04:40.740
would be pretty guaranteed
to be out of range

01:04:40.740 --> 01:04:42.790
for most of the patients?

01:04:42.790 --> 01:04:46.890
And so basically every
test generated an alert

01:04:46.890 --> 01:04:48.120
to the doctor.

01:04:48.120 --> 01:04:50.250
And the doctors did
threaten to kill

01:04:50.250 --> 01:04:52.800
the people who had
implemented the system,

01:04:52.800 --> 01:04:55.110
and it got turned off.

01:04:55.110 --> 01:04:58.050
A system like this, if
you set the threshold

01:04:58.050 --> 01:05:03.180
to be not abnormal, but
life-threateningly abnormal,

01:05:03.180 --> 01:05:08.730
and if you set the
rate and the time

01:05:08.730 --> 01:05:11.970
durations such that it's
reasonable for people

01:05:11.970 --> 01:05:16.320
to respond to it, then
maybe it can be acceptable.

01:05:16.320 --> 01:05:21.420
When we did this project on
looking at how an emergency

01:05:21.420 --> 01:05:24.810
department could anticipate
a flood of patients

01:05:24.810 --> 01:05:27.480
because it looked like
flu season was starting,

01:05:27.480 --> 01:05:31.200
for example, the
question we asked is,

01:05:31.200 --> 01:05:36.390
how many false alarms a
month can you guys tolerate?

01:05:36.390 --> 01:05:37.890
And they thought about it.

01:05:37.890 --> 01:05:42.720
And the ED docs got together
and said, three times a month

01:05:42.720 --> 01:05:46.370
you can cry wolf,
because we really

01:05:46.370 --> 01:05:49.530
want to know when
it actually happens.

01:05:49.530 --> 01:05:53.990
And we'd rather be prepared,
and we can tolerate a 10% error

01:05:53.990 --> 01:05:56.690
rate on this prediction.

01:05:56.690 --> 01:05:59.270
But I don't know what
it is in this domain.

01:06:04.270 --> 01:06:06.700
Another interesting study was--

01:06:06.700 --> 01:06:08.490
it's become quite popular.

01:06:08.490 --> 01:06:10.570
I got a bunch of
emails from my doctor

01:06:10.570 --> 01:06:15.010
today, because I had ordered
a refill on some prescription,

01:06:15.010 --> 01:06:17.980
and he wanted to know how it's
going, and blah, blah, blah.

01:06:17.980 --> 01:06:24.100
So the BI asked the question,
what fraction of those messages

01:06:24.100 --> 01:06:27.400
are never read by the
patients that they're sent to?

01:06:27.400 --> 01:06:29.620
Which is an important
question, because if you're

01:06:29.620 --> 01:06:32.650
relying on that mode
of communication

01:06:32.650 --> 01:06:36.160
as part of your workflow,
you'd like it to be 0.

01:06:36.160 --> 01:06:40.490
It turned out only to be 3%,
which is remarkably good.

01:06:40.490 --> 01:06:43.630
That means that most people
are actually paying attention

01:06:43.630 --> 01:06:44.800
to those kinds of messages.

01:06:47.350 --> 01:06:50.680
Then I wanted to say a few
words about the importance

01:06:50.680 --> 01:06:54.490
of communication
and then finish up

01:06:54.490 --> 01:06:58.150
by mentioning some so
far failed attempts

01:06:58.150 --> 01:07:03.260
at really good integration of
all different data sources.

01:07:03.260 --> 01:07:08.590
So as I said, the
BI started in 1994

01:07:08.590 --> 01:07:11.380
with a system that
said, if you're

01:07:11.380 --> 01:07:15.160
taking a renally-excreted
or a nephrotoxic drug,

01:07:15.160 --> 01:07:18.280
then we're going to
warn people if there

01:07:18.280 --> 01:07:20.350
is a rising creatinine
level, which

01:07:20.350 --> 01:07:22.930
is an indication that
your kidneys are not

01:07:22.930 --> 01:07:24.820
functioning so well.

01:07:24.820 --> 01:07:28.090
Because, of course, if the
drug is renally excreted,

01:07:28.090 --> 01:07:31.060
that means that if your kidneys
are not excreting things

01:07:31.060 --> 01:07:33.130
at the rate they're
supposed to, you're

01:07:33.130 --> 01:07:36.760
going to wind up building up
the amount of drug in your body,

01:07:36.760 --> 01:07:39.340
and that can become toxic.

01:07:39.340 --> 01:07:42.880
So they saw a 21-hour,
so almost a full day,

01:07:42.880 --> 01:07:48.160
reduction in response time
from the medical staff

01:07:48.160 --> 01:07:51.610
given these alerts versus
what happened before.

01:07:51.610 --> 01:07:52.970
That's remarkable.

01:07:52.970 --> 01:07:55.690
I mean, saving a
day in responding

01:07:55.690 --> 01:07:58.780
to a condition like
this is really quite

01:07:58.780 --> 01:08:01.540
an impressive
result. And they also

01:08:01.540 --> 01:08:04.150
saw, in terms of
clinical outcome,

01:08:04.150 --> 01:08:07.180
that the risk of
renal impairment

01:08:07.180 --> 01:08:11.410
was reduced to about half of
the preintervention level.

01:08:11.410 --> 01:08:15.130
So that earlier
response actually

01:08:15.130 --> 01:08:17.620
was saving people's
kidney function

01:08:17.620 --> 01:08:21.910
by getting people to
intervene earlier.

01:08:21.910 --> 01:08:24.939
I found it interesting
they said 44% of doctors

01:08:24.939 --> 01:08:29.950
found these alerts helpful,
28% found them annoying,

01:08:29.950 --> 01:08:33.580
but 65% of them
wanted them continued

01:08:33.580 --> 01:08:36.114
to be used in a survey.

01:08:40.819 --> 01:08:43.180
Enrico Carrera is
one of my heroes.

01:08:43.180 --> 01:08:45.609
He used to be in the UK.

01:08:45.609 --> 01:08:47.720
He's now in Australia.

01:08:47.720 --> 01:08:53.240
And he had this very deep
insight back in the 1980s.

01:08:53.240 --> 01:08:56.890
He said, you know,
all you computer guys

01:08:56.890 --> 01:09:01.060
who are treading on
this medical field

01:09:01.060 --> 01:09:06.590
think that all of the action
is about decision-making,

01:09:06.590 --> 01:09:08.240
but it's not.

01:09:08.240 --> 01:09:11.740
All of the action is
really about communication,

01:09:11.740 --> 01:09:16.130
that health care is
basically a team sport.

01:09:16.130 --> 01:09:19.069
And unless we spend
much more time

01:09:19.069 --> 01:09:21.990
studying what goes
on in communication,

01:09:21.990 --> 01:09:23.689
we're going to miss the boat.

01:09:23.689 --> 01:09:30.080
And then mostly, we didn't
pay any attention to him,

01:09:30.080 --> 01:09:31.430
but he's kept at it.

01:09:31.430 --> 01:09:35.180
So he said, well, how big
is the communication space?

01:09:35.180 --> 01:09:44.630
So he cited a 1985 study
that said that about 50%

01:09:44.630 --> 01:09:50.210
of requests for information
are ones that people

01:09:50.210 --> 01:09:54.020
ask their colleague
for versus 26%

01:09:54.020 --> 01:09:57.180
that they look up
in their own notes.

01:09:57.180 --> 01:10:05.120
So if a doctor is on rounds,
walks into a patient's room

01:10:05.120 --> 01:10:09.200
and says, I want to know has
this guy's temperature been

01:10:09.200 --> 01:10:15.230
going up or down, a quarter of
the time he'll look at notes.

01:10:15.230 --> 01:10:18.560
And half the time, he'll
turn to the nurse and say,

01:10:18.560 --> 01:10:24.860
is this patient's
temperature going up or down?

01:10:24.860 --> 01:10:27.860
So he says that's interesting.

01:10:27.860 --> 01:10:31.430
Paul Tang did a study
in the '90s that

01:10:31.430 --> 01:10:35.390
said that in a clinic,
about 60% of the time

01:10:35.390 --> 01:10:41.450
is spent talking among the
staff, not doing anything else.

01:10:41.450 --> 01:10:49.180
Enrico and one of his
colleagues said that almost 100%

01:10:49.180 --> 01:10:51.890
of non-patient
record information,

01:10:51.890 --> 01:10:54.730
in other words, the thing
that's not in the written health

01:10:54.730 --> 01:10:58.480
record, is done by talking.

01:10:58.480 --> 01:11:02.980
That's almost tautological,
because where else would you

01:11:02.980 --> 01:11:04.420
get it?

01:11:04.420 --> 01:11:10.060
And then Charlie Saffron at the
BI did a time and motion study

01:11:10.060 --> 01:11:13.090
and was looking at, I think,
nursing behavior, and saying

01:11:13.090 --> 01:11:17.200
that about half their time was
face-to-face communication,

01:11:17.200 --> 01:11:21.070
about 10% with electronic
medical records,

01:11:21.070 --> 01:11:25.570
and also a lot of email,
and voicemail, and paper

01:11:25.570 --> 01:11:30.410
reminders as ways of
communicating among people.

01:11:30.410 --> 01:11:37.120
So this was a study looking at--

01:11:37.120 --> 01:11:41.920
this is that 1998 study
by Colera and Tombs.

01:11:41.920 --> 01:11:44.410
And they're looking at
a consultant, the house

01:11:44.410 --> 01:11:46.480
officer, another consultant.

01:11:46.480 --> 01:11:48.190
These are British
titles, because this

01:11:48.190 --> 01:11:49.920
was done in Australia--

01:11:49.920 --> 01:11:51.700
a nurse, et cetera.

01:11:51.700 --> 01:11:56.770
And they say, OK,
among hospital staff--

01:11:56.770 --> 01:12:04.180
I think this was in
one shift, I believe,

01:12:04.180 --> 01:12:07.450
I should have had
that on the slide--

01:12:07.450 --> 01:12:11.180
this is the number of pages
that they sent and received.

01:12:11.180 --> 01:12:14.260
So they range from
0 up to about 4.

01:12:14.260 --> 01:12:17.410
The number of telephone
calls made and received--

01:12:17.410 --> 01:12:20.890
this ranges from 0 up to 13.

01:12:20.890 --> 01:12:22.700
Oh, here's the length
of observation.

01:12:22.700 --> 01:12:25.600
So this was over a period
of about three hours

01:12:25.600 --> 01:12:27.950
for each of these patients.

01:12:27.950 --> 01:12:30.250
And this is the total
number of events.

01:12:30.250 --> 01:12:31.180
So think about it.

01:12:31.180 --> 01:12:35.380
In 3 and 1/2 hours, the
senior house officer

01:12:35.380 --> 01:12:41.060
had 24 distinct communication
events happen to that person.

01:12:41.060 --> 01:12:46.940
So that means, what,
that's like 7--

01:12:46.940 --> 01:12:50.030
yeah, like 7 an hour.

01:12:50.030 --> 01:12:57.550
So that's like 1 every
10 minutes, roughly.

01:12:57.550 --> 01:13:01.307
So it's an interrupt-driven
kind of environment.

01:13:05.210 --> 01:13:09.230
Here's one particular
subject that they looked at,

01:13:09.230 --> 01:13:11.900
three and a quarter
hours of observation.

01:13:11.900 --> 01:13:15.420
This person spent 86%
of their time talking.

01:13:15.420 --> 01:13:20.060
31% were taken up
with 28 interruptions.

01:13:20.060 --> 01:13:24.910
So even the interruptions
were being interrupted.

01:13:24.910 --> 01:13:29.990
25% were multitasking with
two or more conversations.

01:13:29.990 --> 01:13:34.290
87%, face-to-face or
on a phone or a pager.

01:13:34.290 --> 01:13:36.890
So most of that is talk time.

01:13:36.890 --> 01:13:40.970
And 13% dealing with
computers and patient notes.

01:13:40.970 --> 01:13:44.810
So the communication
function is really important.

01:13:44.810 --> 01:13:49.730
And I don't have anything
profound to say about it other

01:13:49.730 --> 01:13:53.580
than I'll put up a pointer
to some of these papers.

01:13:53.580 --> 01:13:56.490
But the kinds of things
they're considering

01:13:56.490 --> 01:13:58.850
are, well, we could
introduce new channels,

01:13:58.850 --> 01:14:02.900
or new types of messages, or
new communication policies

01:14:02.900 --> 01:14:05.580
that say, you know
you may not interrupt

01:14:05.580 --> 01:14:08.660
the person who's
taking care of patients

01:14:08.660 --> 01:14:11.910
while they're doing it,
or something like that.

01:14:11.910 --> 01:14:16.160
And then moving from synchronous
to asynchronous methods,

01:14:16.160 --> 01:14:20.300
like voicemail, or
email, or Slack,

01:14:20.300 --> 01:14:24.590
or some modern
communication mechanism.

01:14:27.380 --> 01:14:28.790
Let me skip by these.

01:14:34.510 --> 01:14:36.790
Next to the last
topic, quickly, how

01:14:36.790 --> 01:14:38.710
do you keep from
dropping the ball?

01:14:38.710 --> 01:14:40.540
So there are a lot
of analyses that

01:14:40.540 --> 01:14:44.920
say that the biggest
mistakes in health care

01:14:44.920 --> 01:14:47.770
are made not because somebody
makes the wrong decision,

01:14:47.770 --> 01:14:50.920
but it's because somebody
fails to make a decision.

01:14:50.920 --> 01:14:52.960
They just forget
about something.

01:14:52.960 --> 01:14:55.795
They don't follow-up on
something that they ought to.

01:14:55.795 --> 01:15:00.670
The patient is going along,
and you think everything's OK,

01:15:00.670 --> 01:15:02.470
and you don't deal with it.

01:15:02.470 --> 01:15:08.290
So inspired partly by
that escalation of pagers

01:15:08.290 --> 01:15:11.110
that I read about
at the Beth Israel,

01:15:11.110 --> 01:15:13.810
I said, well, this sounds
like what we really need

01:15:13.810 --> 01:15:17.380
is a workflow engine that's
approximately a discrete event

01:15:17.380 --> 01:15:18.640
simulator.

01:15:18.640 --> 01:15:23.810
So has anybody built a discrete
events simulator in this class?

01:15:23.810 --> 01:15:26.320
It's a fairly standard sort
of programming problem,

01:15:26.320 --> 01:15:29.500
and it's useful in simulating
all kinds of things

01:15:29.500 --> 01:15:32.980
that involve discrete events.

01:15:32.980 --> 01:15:37.120
And the idea is that
you have a timeline,

01:15:37.120 --> 01:15:39.730
and you run down the
timeline, and you

01:15:39.730 --> 01:15:43.840
execute the next
activity that comes up.

01:15:43.840 --> 01:15:47.260
And that activity
does something.

01:15:47.260 --> 01:15:52.510
It sends an email, or it shoots
a rocket, or whatever field

01:15:52.510 --> 01:15:54.460
you're doing the simulation in.

01:15:54.460 --> 01:15:57.370
But most importantly, what
it does is-- the last thing

01:15:57.370 --> 01:16:01.090
it does is it schedules
something else to happen later

01:16:01.090 --> 01:16:02.740
in the timeline.

01:16:02.740 --> 01:16:06.610
So, for example, for something
that happens once a day, when

01:16:06.610 --> 01:16:09.880
it happens, the task
that runs schedules it

01:16:09.880 --> 01:16:12.290
to happen again the next day.

01:16:12.290 --> 01:16:15.130
And that means that it's going
to be continually operating

01:16:15.130 --> 01:16:16.480
all the time.

01:16:16.480 --> 01:16:20.770
So the idea I had was that what
you'd like to do is to say,

01:16:20.770 --> 01:16:24.100
if at some time, t,
I have a task that

01:16:24.100 --> 01:16:28.330
says do x or asks
z to do y, or both,

01:16:28.330 --> 01:16:30.970
then the last thing
should be at some time

01:16:30.970 --> 01:16:36.910
in the future schedule another
task that says, is y done?

01:16:36.910 --> 01:16:41.860
And if not, then go notify
somebody or go remind somebody.

01:16:41.860 --> 01:16:44.230
And as far as I
know, no hospital

01:16:44.230 --> 01:16:48.580
and no electronic record system
has any capability like this,

01:16:48.580 --> 01:16:52.270
but I still think
it's a terrific idea.

01:16:52.270 --> 01:16:56.620
And then I wanted to
finish with a pointer

01:16:56.620 --> 01:17:01.790
to a problem that is
still very much with us.

01:17:01.790 --> 01:17:06.310
So in 1994, some
colleagues and I

01:17:06.310 --> 01:17:11.020
wrote this thing we called
"The Guardian Angel Manifesto."

01:17:11.020 --> 01:17:14.320
And the idea was that we
should engage patients

01:17:14.320 --> 01:17:16.900
more in their own
care, because they

01:17:16.900 --> 01:17:18.940
can keep track of
a lot of the things

01:17:18.940 --> 01:17:23.140
that systems didn't do a very
good job of keeping track of.

01:17:23.140 --> 01:17:27.370
And the idea was that you would
have a computational process

01:17:27.370 --> 01:17:31.390
that would start off at the
time your parents conceived you

01:17:31.390 --> 01:17:36.430
and run until your
autopsy after you died.

01:17:36.430 --> 01:17:38.650
And during this
time, it would be

01:17:38.650 --> 01:17:42.760
responsible for collecting
all the relevant health care

01:17:42.760 --> 01:17:43.750
data about you.

01:17:43.750 --> 01:17:46.660
So it would be your
electronic medical record,

01:17:46.660 --> 01:17:48.190
but it would also be active.

01:17:48.190 --> 01:17:51.340
So it would help you
communicate with your providers.

01:17:51.340 --> 01:17:54.310
It would help educate you
about any conditions you have.

01:17:54.310 --> 01:17:56.440
It would remind
you about things.

01:17:56.440 --> 01:17:58.970
It would schedule stuff
for you, et cetera.

01:17:58.970 --> 01:18:02.050
So this was a nice
science fiction vision.

01:18:02.050 --> 01:18:06.100
And in the mid-2000s,
Adam Bosworth,

01:18:06.100 --> 01:18:09.012
who was a VP of
Google, came to me.

01:18:09.012 --> 01:18:10.720
And he said, you know,
I read your thing.

01:18:10.720 --> 01:18:11.580
It's a good idea.

01:18:11.580 --> 01:18:13.960
I'm going to do it.

01:18:13.960 --> 01:18:18.610
So Google started up this thing
called Google Health, which

01:18:18.610 --> 01:18:21.460
was more focused on being
at least the personal health

01:18:21.460 --> 01:18:22.690
record.

01:18:22.690 --> 01:18:26.590
They did a pilot with 1,600
people at Cleveland Clinic,

01:18:26.590 --> 01:18:30.730
and then they went
public as a beta.

01:18:30.730 --> 01:18:33.360
And three years
later, they killed it.

01:18:36.340 --> 01:18:37.790
And they had a
bunch of partners.

01:18:37.790 --> 01:18:42.460
So they had Allscripts,
and Beth Israel,

01:18:42.460 --> 01:18:45.760
and Blue Cross of Massachusetts,
and the Cleveland Clinic,

01:18:45.760 --> 01:18:47.660
and CVS, and so on.

01:18:47.660 --> 01:18:49.960
So they did their job
of trying to connect

01:18:49.960 --> 01:18:52.330
to a bunch of important players.

01:18:52.330 --> 01:18:54.760
But, of course, they
didn't have everybody.

01:18:54.760 --> 01:18:56.740
And so, for example,
I, of course,

01:18:56.740 --> 01:19:00.130
immediately signed
up for an account,

01:19:00.130 --> 01:19:06.040
and the only company that I had
ever dealt with out of that set

01:19:06.040 --> 01:19:08.650
was Walgreens, where
I had bought a skin

01:19:08.650 --> 01:19:11.440
cream one time for a skin rash.

01:19:11.440 --> 01:19:14.470
And so my total medical
record consisted

01:19:14.470 --> 01:19:16.390
of a skin rash
and a cream that I

01:19:16.390 --> 01:19:19.120
had bought to take care of it--

01:19:19.120 --> 01:19:22.250
not very helpful.

01:19:22.250 --> 01:19:25.660
And so nobody, other
than these partners,

01:19:25.660 --> 01:19:28.270
could enter data
automatically, which

01:19:28.270 --> 01:19:31.720
meant that you had to be even
more anal compulsive than I

01:19:31.720 --> 01:19:34.450
am in order to
sit there and type

01:19:34.450 --> 01:19:38.410
in my entire medical
history into the system,

01:19:38.410 --> 01:19:45.070
especially, because if I did so,
nobody would ever look at it.

01:19:45.070 --> 01:19:48.100
Because if I go to
my doctor and say,

01:19:48.100 --> 01:19:52.300
hey, Doc, here's the Google
URL for my medical record,

01:19:52.300 --> 01:19:55.623
and here's the password by
which you can access it,

01:19:55.623 --> 01:19:57.790
what do you think are the
odds that they're actually

01:19:57.790 --> 01:19:59.274
going to look?

01:19:59.274 --> 01:20:00.160
AUDIENCE: 0.

01:20:00.160 --> 01:20:02.510
PETER SZOLOVITS: 0.

01:20:02.510 --> 01:20:06.650
So the thing was an
absolute abject failure.

01:20:06.650 --> 01:20:08.700
And people keep trying it.

01:20:08.700 --> 01:20:11.360
And so far, nobody has
figured out how to do it,

01:20:11.360 --> 01:20:13.520
but it's still a good idea.

01:20:13.520 --> 01:20:16.840
With that, we'll
stop on workflow.