WEBVTT

00:00:15.580 --> 00:00:17.680
PROFESSOR: So welcome, everyone.

00:00:17.680 --> 00:00:20.740
Today is the first
of what will be

00:00:20.740 --> 00:00:25.070
a series of four guest lectures
throughout the semester.

00:00:25.070 --> 00:00:28.412
There will be two
guest lectures,

00:00:28.412 --> 00:00:30.370
starting the week from
today, and then there'll

00:00:30.370 --> 00:00:32.350
be another one towards
the end of the semester.

00:00:32.350 --> 00:00:34.930
And what Pete and
I decided to do

00:00:34.930 --> 00:00:37.120
is to bring in people who
know a lot more than us

00:00:37.120 --> 00:00:39.298
about some area of expertise.

00:00:39.298 --> 00:00:40.840
In today's instance,
it's going to be

00:00:40.840 --> 00:00:44.590
about cardiovascular
medicine, in particular

00:00:44.590 --> 00:00:47.260
about how to use imaging
and machine learning

00:00:47.260 --> 00:00:49.240
on images in that context.

00:00:49.240 --> 00:00:52.150
And for today's
lecture, we're very

00:00:52.150 --> 00:00:57.220
excited to have professor
Rahul Deo to speak.

00:00:57.220 --> 00:01:00.240
Rahul's name kept on showing
up, as I did research

00:01:00.240 --> 00:01:01.660
over the last couple of years.

00:01:01.660 --> 00:01:04.660
First, my group was
starting to get interested

00:01:04.660 --> 00:01:07.315
in echocardiography,
and we said, oh, here's

00:01:07.315 --> 00:01:08.920
an interesting
paper to read on it.

00:01:08.920 --> 00:01:12.610
We read it, and then
we read another paper

00:01:12.610 --> 00:01:17.050
on doing subtyping
of ejection fraction

00:01:17.050 --> 00:01:19.840
which is a type of heart
failure, and we read it.

00:01:19.840 --> 00:01:22.450
I wasn't really paying attention
to the names on the papers,

00:01:22.450 --> 00:01:23.908
and then suddenly,
someone told me,

00:01:23.908 --> 00:01:26.878
there's this guy moving
to Boston next month who's

00:01:26.878 --> 00:01:29.170
doing a lot of interesting
work and interesting machine

00:01:29.170 --> 00:01:29.950
learning.

00:01:29.950 --> 00:01:31.762
You should go meet him.

00:01:31.762 --> 00:01:33.220
And of course, I
meet him, and then

00:01:33.220 --> 00:01:34.762
I tell him about
these papers I read,

00:01:34.762 --> 00:01:38.270
and he said, oh, I wrote
all of those papers.

00:01:38.270 --> 00:01:40.360
He was a senior author on them.

00:01:40.360 --> 00:01:42.410
So Rahul's been
around for a while.

00:01:42.410 --> 00:01:48.440
He is already a
senior in his field.

00:01:48.440 --> 00:01:52.900
He started out doing his medical
school training at Cornell,

00:01:52.900 --> 00:01:55.450
in Cornell Medical
School, in New York

00:01:55.450 --> 00:01:58.630
City, at the same time as
doing his PhD at Rockefeller

00:01:58.630 --> 00:01:59.860
University.

00:01:59.860 --> 00:02:01.690
And then he spent the
first large chunk--

00:02:01.690 --> 00:02:05.770
after his post-doctoral
training, up here in Boston,

00:02:05.770 --> 00:02:07.270
at Harvard Medical
School-- he spent

00:02:07.270 --> 00:02:12.910
a large chunk of his career as
faculty at UCSF, in California.

00:02:12.910 --> 00:02:15.880
And just moved back this
past year to take a position

00:02:15.880 --> 00:02:18.900
as the chief data
scientist-- is that right--

00:02:18.900 --> 00:02:21.070
for the One Brave
Idea project which

00:02:21.070 --> 00:02:24.685
is a very large initiative
joint between MIT and Brigham

00:02:24.685 --> 00:02:27.805
and Women's Hospital to study
cardiovascular medicine.

00:02:27.805 --> 00:02:29.830
He'll tell you more maybe.

00:02:29.830 --> 00:02:34.360
And Rahul's research has
really gone the full spectrum,

00:02:34.360 --> 00:02:36.407
but the type of things
you'll hear about today

00:02:36.407 --> 00:02:38.740
is actually not what he's
been doing most of his career,

00:02:38.740 --> 00:02:39.820
amazingly so.

00:02:39.820 --> 00:02:40.990
Most of his career,
he's been thinking more

00:02:40.990 --> 00:02:42.640
about genotype and
how to really bridge

00:02:42.640 --> 00:02:48.803
that genotype-phenotype branch,
but I asked him specifically

00:02:48.803 --> 00:02:49.720
to talk about imaging.

00:02:49.720 --> 00:02:51.190
So that's what he'll be focusing
on today in his lecture.

00:02:51.190 --> 00:02:53.648
And without further ado, thank
you, Rahul, for coming here.

00:02:53.648 --> 00:02:56.150
[APPLAUSE]

00:02:56.650 --> 00:02:58.663
RAHUL DEO: So I'm
used to lecturing

00:02:58.663 --> 00:03:00.580
the clinical audiences,
so you guys are by far

00:03:00.580 --> 00:03:02.190
the most technical audience.

00:03:02.190 --> 00:03:04.510
So please spare me a
little bit, but I actually

00:03:04.510 --> 00:03:08.530
want to encourage
interruptions, questions.

00:03:08.530 --> 00:03:10.120
This is a very
opinionated lecture,

00:03:10.120 --> 00:03:13.780
so that if anybody has sort of
any questions, reservations,

00:03:13.780 --> 00:03:15.280
please bring them
up during lecture.

00:03:15.280 --> 00:03:17.200
Don't wait till the end.

00:03:17.200 --> 00:03:22.150
And in part, it's opinionated
because I feel passionately

00:03:22.150 --> 00:03:27.370
that the stuff we're doing needs
to make its way into practice.

00:03:27.370 --> 00:03:30.083
It's not by itself purely
academically interesting.

00:03:30.083 --> 00:03:31.750
We need to study the
things we're doing.

00:03:31.750 --> 00:03:33.970
We're already picking up
what everybody else here

00:03:33.970 --> 00:03:35.380
is already doing.

00:03:35.380 --> 00:03:38.740
So it's OK from that
standpoint, but it really

00:03:38.740 --> 00:03:39.648
has to make its way.

00:03:39.648 --> 00:03:42.190
And that means that we have to
have some mature understanding

00:03:42.190 --> 00:03:44.042
of what makes its
way into practice,

00:03:44.042 --> 00:03:45.250
where the resistance will be.

00:03:45.250 --> 00:03:48.850
So the lecture will be peppered
throughout with some opinions

00:03:48.850 --> 00:03:52.070
and comments in that, and
hopefully, that will be useful.

00:03:52.070 --> 00:03:53.778
So just a quick
outline, just going

00:03:53.778 --> 00:03:56.320
to introduce cardiac structure
and function which is probably

00:03:56.320 --> 00:03:59.350
not part of the regular
undergraduate and graduate

00:03:59.350 --> 00:04:00.730
training here at MIT.

00:04:00.730 --> 00:04:03.683
Talk a little bit about what the
major cardiac diagnostics are

00:04:03.683 --> 00:04:04.600
and how they use them.

00:04:04.600 --> 00:04:09.032
And all this is really
to help guide the thought

00:04:09.032 --> 00:04:10.990
and the decision making
about how we would ever

00:04:10.990 --> 00:04:12.850
automate and bring this into--

00:04:12.850 --> 00:04:15.280
how to bring machine learning,
artificial intelligence,

00:04:15.280 --> 00:04:16.530
into actual clinical practice.

00:04:16.530 --> 00:04:18.220
Because I need to
give enough background

00:04:18.220 --> 00:04:20.568
so you realize what
the challenges are,

00:04:20.568 --> 00:04:23.110
and then the question probably
every has is where's the data?

00:04:23.110 --> 00:04:24.657
How would how would
one get access

00:04:24.657 --> 00:04:26.740
to some of this stuff to
be able to potentially do

00:04:26.740 --> 00:04:28.210
work in this area?

00:04:28.210 --> 00:04:31.270
And then, I'm going to venture a
little bit into computer vision

00:04:31.270 --> 00:04:33.130
and just talk about
some of the topics

00:04:33.130 --> 00:04:35.088
that at least I've been
thinking about that are

00:04:35.088 --> 00:04:36.328
relevant to what we're doing.

00:04:36.328 --> 00:04:37.870
And then talk about
some of this work

00:04:37.870 --> 00:04:40.948
around an automated pipeline for
echocardiogram, not as by any

00:04:40.948 --> 00:04:42.490
means a gold standard
but really just

00:04:42.490 --> 00:04:44.350
as sort of an initial
foray into trying

00:04:44.350 --> 00:04:47.350
to make a dent into this.

00:04:47.350 --> 00:04:50.158
And then thinking a little
bit about what lessons--

00:04:50.158 --> 00:04:52.450
David mentioned that you
talked about electrocardiogram

00:04:52.450 --> 00:04:56.110
last week or last class,
and so a little bit of some

00:04:56.110 --> 00:04:59.050
of the ideas from there, and
how they would lend themselves

00:04:59.050 --> 00:05:01.240
to insights about future
types of approaches

00:05:01.240 --> 00:05:02.950
with automated interpretation.

00:05:02.950 --> 00:05:05.380
And then my background is
actually more in biology.

00:05:05.380 --> 00:05:07.392
So I'm going to come
back and say, OK,

00:05:07.392 --> 00:05:09.850
enough with all this imaging
stuff, what about the biology?

00:05:09.850 --> 00:05:12.450
How can we make
some insights there?

00:05:12.450 --> 00:05:13.330
OK.

00:05:13.330 --> 00:05:17.320
So every time people
try to get funding

00:05:17.320 --> 00:05:19.870
for coronary heart disease,
they try to talk up

00:05:19.870 --> 00:05:21.340
just how important it is.

00:05:21.340 --> 00:05:23.380
So this is still--

00:05:23.380 --> 00:05:25.570
we have some battles with
the oncology people--

00:05:25.570 --> 00:05:30.340
but this is still the leading
cause of death in the world.

00:05:30.340 --> 00:05:33.610
And then people like I,
you're just emphasizing

00:05:33.610 --> 00:05:34.560
the developed world.

00:05:34.560 --> 00:05:37.127
There's lots of communicable
diseases that matter much more.

00:05:37.127 --> 00:05:39.460
So even if you look at those,
and you look at the bottom

00:05:39.460 --> 00:05:44.560
here, this still, if this is all
causes of death age-adjusted,

00:05:44.560 --> 00:05:46.920
cardiovascular disease is
still number one amongst that.

00:05:46.920 --> 00:05:52.180
So certainly it remains
important and increasingly so

00:05:52.180 --> 00:05:54.550
in some of the
developing world also.

00:05:54.550 --> 00:05:57.598
So it's important to think
a little bit about what

00:05:57.598 --> 00:05:59.140
the heart does,
because this is going

00:05:59.140 --> 00:06:01.740
to guide at least the way that
diseases have been classified.

00:06:01.740 --> 00:06:03.740
So the main thing the
heart does is it's a pump,

00:06:03.740 --> 00:06:06.593
and it delivers oxygenated
blood throughout the circulatory

00:06:06.593 --> 00:06:08.260
system to all the
tissues that need it--

00:06:08.260 --> 00:06:11.830
the brain, the kidneys, the
muscles, and oxygen, of course,

00:06:11.830 --> 00:06:14.440
is required for ATP production.

00:06:14.440 --> 00:06:16.240
So it's a pretty
impressive organ.

00:06:16.240 --> 00:06:18.160
It pumps about five
liters of blood a minute,

00:06:18.160 --> 00:06:21.660
and with exercise, that can go
up five to seven-fold or so,

00:06:21.660 --> 00:06:24.280
with conditioned athletes,
not me, but other people

00:06:24.280 --> 00:06:26.660
can ramp that up substantially.

00:06:26.660 --> 00:06:29.890
And we have this need to keep
a very, very regular beat,

00:06:29.890 --> 00:06:33.340
so if you pause for
about three seconds,

00:06:33.340 --> 00:06:36.310
you are likely to get
lightheaded or pass out.

00:06:36.310 --> 00:06:40.330
So you have to maintain this
rhythmic beating of your heart,

00:06:40.330 --> 00:06:42.310
and you can compute
what that would be,

00:06:42.310 --> 00:06:45.370
and somewhere around two billion
beats in a typical lifetime.

00:06:45.370 --> 00:06:49.353
So I'm going to show a
lot of pictures and videos

00:06:49.353 --> 00:06:50.020
throughout this.

00:06:50.020 --> 00:06:52.562
So it's probably worthwhile just
to take a pause a little bit

00:06:52.562 --> 00:06:54.790
and talk about what the
anatomy of the heart is.

00:06:54.790 --> 00:06:57.880
So the heart sits like
this, so the pointy part

00:06:57.880 --> 00:07:01.200
is kind of sitting out
to the side, like that.

00:07:01.200 --> 00:07:04.540
And so I'm going to just sort
of describe the flow of blood.

00:07:04.540 --> 00:07:07.180
So the blood comes in something
called the inferior vena

00:07:07.180 --> 00:07:10.510
cava or the superior vena cava,
that's draining from the brain.

00:07:10.510 --> 00:07:12.880
This is draining
from the lower body,

00:07:12.880 --> 00:07:16.312
and then enters into a chamber
called the right atrium.

00:07:16.312 --> 00:07:18.520
It moves through something
called the tricuspid valve

00:07:18.520 --> 00:07:20.080
into what's called
the right ventricle.

00:07:20.080 --> 00:07:21.997
The right ventricle has
got some muscle to it.

00:07:21.997 --> 00:07:23.935
It pumps into the lungs.

00:07:23.935 --> 00:07:25.850
There, the blood
picks up oxygen,

00:07:25.850 --> 00:07:29.060
so that's why it's
shown as being red here.

00:07:29.060 --> 00:07:31.647
The oxygenated native blood
comes through the left atrium

00:07:31.647 --> 00:07:33.730
and then into the left
ventricle through something

00:07:33.730 --> 00:07:34.840
called the mitral valve.

00:07:34.840 --> 00:07:37.630
We'll show you some pictures
of the mitral valve later on.

00:07:37.630 --> 00:07:39.400
And then the left
ventricle, which

00:07:39.400 --> 00:07:41.170
is the big workhorse
of the heart,

00:07:41.170 --> 00:07:44.080
pumps blood through
the rest of the body,

00:07:44.080 --> 00:07:46.510
through a structure
of the aorta.

00:07:46.510 --> 00:07:48.970
So in through the right
heart, through the lungs,

00:07:48.970 --> 00:07:51.148
through the left heart,
to the rest of the body.

00:07:51.148 --> 00:07:53.440
And then shown here in yellow
is the conduction system.

00:07:53.440 --> 00:07:56.400
So you guys got a little bit
of a conversation last class

00:07:56.400 --> 00:07:57.810
on the electrical system.

00:07:57.810 --> 00:08:02.165
So the sinoatrial node is
up here in the right atrium,

00:08:02.165 --> 00:08:03.540
and then conduction
goes through.

00:08:03.540 --> 00:08:06.660
So the P wave on an EKG
represents the conduction

00:08:06.660 --> 00:08:07.410
through there.

00:08:07.410 --> 00:08:08.827
You get through
the AV node, where

00:08:08.827 --> 00:08:10.530
there's a delay which
is a PR interval,

00:08:10.530 --> 00:08:12.822
and then you get spreading
through the ventricles which

00:08:12.822 --> 00:08:16.290
is the QRS complex, and then
repolarization is the T wave.

00:08:16.290 --> 00:08:18.840
So that's the electrical system,
and of course, these things

00:08:18.840 --> 00:08:20.910
have to work
intimately together.

00:08:24.570 --> 00:08:27.080
Every single basic kind
of cardiac physiology

00:08:27.080 --> 00:08:29.850
will show this diagram called
the Wiggers diagram which

00:08:29.850 --> 00:08:31.838
really just shows the
interconnectedness

00:08:31.838 --> 00:08:32.880
of the electrical system.

00:08:32.880 --> 00:08:34.590
So there's the EKG up there.

00:08:34.590 --> 00:08:37.929
These are the heart sounds
that a provider would listen to

00:08:37.929 --> 00:08:39.720
with the stethoscope,
and this is

00:08:39.720 --> 00:08:43.530
capturing the flow of sort
of the changes in pressure

00:08:43.530 --> 00:08:45.120
in the heart and in the aorta.

00:08:45.120 --> 00:08:49.050
So heart fills during a period
of time called diastole.

00:08:49.050 --> 00:08:50.940
The mitral valve closes.

00:08:50.940 --> 00:08:52.020
The ventricle contracts.

00:08:52.020 --> 00:08:53.230
The pressure increases.

00:08:53.230 --> 00:08:54.907
This is a period of
time called systole.

00:08:54.907 --> 00:08:57.240
Eventually, something called
the aortic valve pops open,

00:08:57.240 --> 00:08:59.073
and blood goes through
the rest of the body.

00:08:59.073 --> 00:09:01.230
The heart finally
starts to relax.

00:09:01.230 --> 00:09:03.025
The atrioventricular
valve closes.

00:09:03.025 --> 00:09:03.900
Then, you fill again.

00:09:03.900 --> 00:09:06.960
So this happens again and again
and again in a cyclical way,

00:09:06.960 --> 00:09:09.450
and you have this combination
of electrical and mechanical

00:09:09.450 --> 00:09:11.300
properties.

00:09:11.300 --> 00:09:11.920
OK.

00:09:11.920 --> 00:09:12.890
So I have some pictures here.

00:09:12.890 --> 00:09:13.520
These are all MRIs.

00:09:13.520 --> 00:09:15.437
I'm going to talk about
echocardiography which

00:09:15.437 --> 00:09:17.990
is these very ugly, grainy
things that I unfortunately

00:09:17.990 --> 00:09:18.860
have to work with.

00:09:18.860 --> 00:09:20.952
MRIs are beautiful
but very expensive.

00:09:20.952 --> 00:09:22.160
So there's a reason for that.

00:09:22.160 --> 00:09:26.160
So this is something called the
long axis view of the heart.

00:09:26.160 --> 00:09:28.340
So this is the thick walled
left ventricle there.

00:09:28.340 --> 00:09:29.923
This is the left
atrium there, and you

00:09:29.923 --> 00:09:32.840
can see this beautiful turbulent
flow of blood in there,

00:09:32.840 --> 00:09:35.150
and it's flowing from the
atrium to the ventricle.

00:09:35.150 --> 00:09:37.190
This is another patient's.

00:09:37.190 --> 00:09:38.570
It's called the short axis view.

00:09:38.570 --> 00:09:41.090
There is the left ventricle
and the right ventricle there.

00:09:41.090 --> 00:09:43.402
So we're kind of looking
at it somewhat obliquely,

00:09:43.402 --> 00:09:45.485
and then this is another
view called the physical.

00:09:45.485 --> 00:09:46.340
It's a little bit dull there.

00:09:46.340 --> 00:09:47.150
I'm sorry.

00:09:47.150 --> 00:09:48.650
We can brighten it a little bit.

00:09:48.650 --> 00:09:52.022
This is the what's called
the four chamber view.

00:09:52.022 --> 00:09:54.230
So you can see the left
ventricle and right ventricle

00:09:54.230 --> 00:09:54.980
here.

00:09:54.980 --> 00:09:57.500
So the reason for
these different views

00:09:57.500 --> 00:10:01.550
is, ultimately, that
people have measures

00:10:01.550 --> 00:10:04.007
of function and measures
of disease that go along

00:10:04.007 --> 00:10:05.090
with these specific views.

00:10:05.090 --> 00:10:08.190
So you're going to see them
coming back again and again.

00:10:08.190 --> 00:10:08.690
OK.

00:10:08.690 --> 00:10:14.290
So the way that physicians like
to organize disease definitions

00:10:14.290 --> 00:10:16.540
really around some of these
same kind of functions.

00:10:16.540 --> 00:10:20.710
So failures of the
heart to pump properly

00:10:20.710 --> 00:10:23.380
causes a disease
called heart failure,

00:10:23.380 --> 00:10:26.380
and this shows up in terms of
being out of breath, having

00:10:26.380 --> 00:10:28.780
fluid buildup in the
belly and in the legs,

00:10:28.780 --> 00:10:30.812
and this is treated
with medications.

00:10:30.812 --> 00:10:32.770
Sometimes, you can have
some artificial devices

00:10:32.770 --> 00:10:34.562
to help the heart pump,
and ultimately, you

00:10:34.562 --> 00:10:37.310
could even have a transplant,
depending on how severe it is.

00:10:37.310 --> 00:10:38.920
So that's the pump.

00:10:38.920 --> 00:10:42.220
Blood supply to the heart
ultimately can also be blocked,

00:10:42.220 --> 00:10:44.830
and that causes a disease
called coronary artery disease.

00:10:44.830 --> 00:10:46.410
If blood is completely
blocked, you

00:10:46.410 --> 00:10:48.618
can get something called a
heart attack or myocardial

00:10:48.618 --> 00:10:49.210
infarction.

00:10:49.210 --> 00:10:51.490
That's chest pain, sometimes
shortness of breath,

00:10:51.490 --> 00:10:54.370
and we open up those blocked
vessels by angioplasty,

00:10:54.370 --> 00:10:57.790
stick a stent in there,
or bypass them altogether.

00:10:57.790 --> 00:11:02.350
And then the flow of
blood has to be one way.

00:11:02.350 --> 00:11:05.980
So abnormalities of flow
of the blood through valves

00:11:05.980 --> 00:11:09.030
is valvular disease, and so
you can have either two type

00:11:09.030 --> 00:11:10.480
valves, so that's
called stenosis.

00:11:10.480 --> 00:11:11.688
Or you can have leaky valves.

00:11:11.688 --> 00:11:12.855
That's called regurgitation.

00:11:12.855 --> 00:11:14.688
That shows up as
light-headedness, shortness

00:11:14.688 --> 00:11:17.290
of breath, fainting, and then
you've got to fix those valves.

00:11:17.290 --> 00:11:19.510
And finally, there's
abnormalities of rhythm.

00:11:19.510 --> 00:11:21.170
So something like
atrial fibrillation

00:11:21.170 --> 00:11:24.640
which is a quivering of the
atrium, so too slow heartbeats,

00:11:24.640 --> 00:11:27.100
which would look like cardiac,
can present as palpitations,

00:11:27.100 --> 00:11:28.925
fainting, or even sudden death.

00:11:28.925 --> 00:11:31.550
And you can stick a pacemaker in
there, defibrillator in there,

00:11:31.550 --> 00:11:34.190
or try to burn off
the arrhythmia.

00:11:34.190 --> 00:11:34.690
OK.

00:11:34.690 --> 00:11:38.440
So this is like the very
physiology-centric view,

00:11:38.440 --> 00:11:41.042
but the truth is that the
heart has a whole lot of cells.

00:11:41.042 --> 00:11:43.000
So there's a lot more
biology there than simply

00:11:43.000 --> 00:11:46.120
just thinking about the pumping
and the electrical function.

00:11:46.120 --> 00:11:50.000
Only 30% of the cells or so
are these cardiomyocytes.

00:11:50.000 --> 00:11:52.640
So these are the cells that
are involved in contraction.

00:11:52.640 --> 00:11:55.148
These are cells that are
excitable, but that's only 30%

00:11:55.148 --> 00:11:55.690
of the cells.

00:11:55.690 --> 00:11:57.850
There is endothelials
in the cell.

00:11:57.850 --> 00:11:58.780
There's fibroblasts.

00:11:58.780 --> 00:12:00.850
There's a bunch of
blood cells in there

00:12:00.850 --> 00:12:02.560
too, certainly a lot of red
blood cells in there too.

00:12:02.560 --> 00:12:03.710
So you have lots
of other things.

00:12:03.710 --> 00:12:05.360
So we're going to come
back to here a little bit

00:12:05.360 --> 00:12:07.818
when talking about how should
we be thinking about disease?

00:12:07.818 --> 00:12:10.750
The historic way is
to think about pumping

00:12:10.750 --> 00:12:12.820
and electrical activation,
but really, there's

00:12:12.820 --> 00:12:14.485
maybe a little bit
more complexity here

00:12:14.485 --> 00:12:15.610
that needs to be addressed.

00:12:15.610 --> 00:12:16.330
OK.

00:12:16.330 --> 00:12:20.290
So there's a lot of different--

00:12:20.290 --> 00:12:23.560
so cardiology is
very imaging-centric,

00:12:23.560 --> 00:12:25.930
and as a result,
it's very expensive.

00:12:25.930 --> 00:12:28.630
Because imaging costs
a lot of money to do,

00:12:28.630 --> 00:12:31.120
and so I have dollar
signs here reflecting

00:12:31.120 --> 00:12:32.650
the sorts of
different tests we do.

00:12:32.650 --> 00:12:36.410
So you saw the
cheapest one last week,

00:12:36.410 --> 00:12:38.990
electrocardiogram,
so one dollar sign,

00:12:38.990 --> 00:12:41.950
and that has lots of utility.

00:12:41.950 --> 00:12:44.170
For example, one could
diagnose an acute heart attack

00:12:44.170 --> 00:12:46.250
with that.

00:12:46.250 --> 00:12:48.680
Echocardiography, which
involves sound waves,

00:12:48.680 --> 00:12:51.280
is ultimately more used
for quantifying structure

00:12:51.280 --> 00:12:54.740
and function, can pick up heart
failure, valvular disease,

00:12:54.740 --> 00:12:56.280
high blood pressure
in the lungs.

00:12:56.280 --> 00:12:57.760
So that's another modality.

00:12:57.760 --> 00:13:00.760
MRI, which is just not used
all that much in this country,

00:13:00.760 --> 00:13:01.910
is very expensive.

00:13:01.910 --> 00:13:04.160
It does largely the same
things, and you can imagine,

00:13:04.160 --> 00:13:05.620
even though it's
beautiful, people

00:13:05.620 --> 00:13:07.990
have not had an easy
time and able to justify

00:13:07.990 --> 00:13:11.980
why it's any better than this
slightly cheaper modality.

00:13:11.980 --> 00:13:15.010
And then you have angiography
which can either be by CAT scan

00:13:15.010 --> 00:13:16.240
or by X-ray.

00:13:16.240 --> 00:13:19.870
And that visualizes the flow
of blood through the heart

00:13:19.870 --> 00:13:22.990
and looks for blockages which
are going to be stented,

00:13:22.990 --> 00:13:24.760
ballooned up and stented.

00:13:24.760 --> 00:13:28.840
And then you had these kind
of non-invasive technologies,

00:13:28.840 --> 00:13:32.970
like PET and SPECT that
use radionucleotides,

00:13:32.970 --> 00:13:35.200
like technetium,
rubidium, and they

00:13:35.200 --> 00:13:37.030
look for abnormalities
in blood flow

00:13:37.030 --> 00:13:38.980
to detect whether
or non-invasively

00:13:38.980 --> 00:13:40.480
there's some patch
of the heart that

00:13:40.480 --> 00:13:41.787
isn't getting enough blood.

00:13:41.787 --> 00:13:43.870
If you get one of these,
and it's abnormal, often,

00:13:43.870 --> 00:13:45.790
you go over there, and you
take a trip to the movies--

00:13:45.790 --> 00:13:47.350
as my old teachers used to say--

00:13:47.350 --> 00:13:50.800
and then you may find yourself
with an angioplasty or stent

00:13:50.800 --> 00:13:52.420
or bypass.

00:13:52.420 --> 00:13:54.708
So one of the sad
things about cardiology

00:13:54.708 --> 00:13:56.500
is we don't define our
diseases by biology.

00:13:56.500 --> 00:13:58.853
We define our
diseases often related

00:13:58.853 --> 00:14:00.520
to whether the anatomy
of the physiology

00:14:00.520 --> 00:14:03.520
is abnormal or normal, usually
based on some of these images

00:14:03.520 --> 00:14:04.860
or some of these numbers.

00:14:04.860 --> 00:14:05.990
OK.

00:14:05.990 --> 00:14:07.840
So we have to make
decisions, and we often

00:14:07.840 --> 00:14:09.443
use these very same
things too to be

00:14:09.443 --> 00:14:10.610
able to make some decisions.

00:14:10.610 --> 00:14:13.510
So we have to decide whether
we want to put a defibrillator,

00:14:13.510 --> 00:14:16.745
and to do so, you often need
to get an echocardiogram

00:14:16.745 --> 00:14:18.620
to look at the pumping
function of the heart.

00:14:18.620 --> 00:14:21.120
If you want to decide on whether
somebody needs angioplasty,

00:14:21.120 --> 00:14:22.647
you have to get an angiogram.

00:14:22.647 --> 00:14:24.730
If you want to decided to
get a valve replacement,

00:14:24.730 --> 00:14:26.470
you need an echo.

00:14:26.470 --> 00:14:28.102
But some of these
other ones actually

00:14:28.102 --> 00:14:29.560
don't involve any
imaging, and this

00:14:29.560 --> 00:14:31.268
is sort of one of the
challenges that I'm

00:14:31.268 --> 00:14:34.300
going to talk about is
that all of the future--

00:14:34.300 --> 00:14:36.680
you can imagine building
brand new risk models,

00:14:36.680 --> 00:14:38.140
new classification models.

00:14:38.140 --> 00:14:40.510
You're stuck with the
data that's out there,

00:14:40.510 --> 00:14:42.310
and the data that's
out there is ultimately

00:14:42.310 --> 00:14:44.890
being collected because
somebody feels like it's worth

00:14:44.890 --> 00:14:46.360
paying for it already.

00:14:46.360 --> 00:14:48.790
So if you want to
build a brand new risk

00:14:48.790 --> 00:14:51.550
model for who's going to
have a myocardial infarction,

00:14:51.550 --> 00:14:54.010
you're probably not going to
have any echocardiograms to be

00:14:54.010 --> 00:14:55.510
able to use for
that, because nobody

00:14:55.510 --> 00:14:57.460
is going to have paid
for that to be collected

00:14:57.460 --> 00:14:58.450
in the first place.

00:14:58.450 --> 00:14:59.350
So this is a problem.

00:14:59.350 --> 00:15:01.570
To be able to innovate, I've got
to keep on coming back to that,

00:15:01.570 --> 00:15:04.153
because I think you're going to
be shocked by the small sample

00:15:04.153 --> 00:15:05.950
sizes that we face in
some of these things.

00:15:05.950 --> 00:15:06.970
And part of it is
because if you just

00:15:06.970 --> 00:15:08.803
want to piggyback on
what insurers are going

00:15:08.803 --> 00:15:10.973
to be willing to pay
for to get your data,

00:15:10.973 --> 00:15:12.390
you're going to
be stuck with only

00:15:12.390 --> 00:15:14.340
being able to work off
the stuff we already

00:15:14.340 --> 00:15:15.215
know something about.

00:15:15.215 --> 00:15:17.280
So much of my work
has been really trying

00:15:17.280 --> 00:15:20.070
to think about how
we can change that.

00:15:20.070 --> 00:15:22.670
OK, so just a little
bit more, and then we

00:15:22.670 --> 00:15:24.520
can get into a
little bit more meat.

00:15:24.520 --> 00:15:26.640
So sort of the
universal standard

00:15:26.640 --> 00:15:29.870
for how imaging data is stored
is something called DICOMs,

00:15:29.870 --> 00:15:33.058
or Digital Imaging and
Communications standard,

00:15:33.058 --> 00:15:34.600
and really, the end
of the day, there

00:15:34.600 --> 00:15:36.457
is some compressed
data for the images.

00:15:36.457 --> 00:15:38.790
There's a DICOM header, which
I'll show you in a moment.

00:15:38.790 --> 00:15:40.282
It's lots of nice
Python libraries

00:15:40.282 --> 00:15:42.490
that are available to be
able to work with this data,

00:15:42.490 --> 00:15:45.180
and there's a free
viewer you could use too.

00:15:45.180 --> 00:15:46.555
OK.

00:15:46.555 --> 00:15:47.930
So where do I get
access to this?

00:15:47.930 --> 00:15:49.805
So this has actually
been an incredible pain.

00:15:49.805 --> 00:15:52.800
So hospitals are set up
to be clinical operations.

00:15:52.800 --> 00:15:54.300
They're not set
up to make it easy

00:15:54.300 --> 00:15:56.623
for you to get gobs
of data for being

00:15:56.623 --> 00:15:57.790
able to do machine learning.

00:15:57.790 --> 00:16:00.840
It's just not really there.

00:16:00.840 --> 00:16:04.230
And so sometimes, you have
some of these data archives

00:16:04.230 --> 00:16:05.730
that store this
data, but there's

00:16:05.730 --> 00:16:08.710
lots of reasons for why
people make that difficult.

00:16:08.710 --> 00:16:10.530
And one of them is
because often images

00:16:10.530 --> 00:16:13.100
have these burned in pixels
with identifiable information.

00:16:13.100 --> 00:16:16.077
So you'll have a patient's
name emblazoned in the image.

00:16:16.077 --> 00:16:17.160
You'll have date of birth.

00:16:17.160 --> 00:16:18.520
You'll have kind of
other attributes.

00:16:18.520 --> 00:16:20.100
So you're stuck
with that, and not

00:16:20.100 --> 00:16:22.133
only is it a problem
that they're there,

00:16:22.133 --> 00:16:24.300
the vendors don't make it
easy to be able to get rid

00:16:24.300 --> 00:16:25.133
of that information.

00:16:25.133 --> 00:16:28.170
So you actually have a
problem that they don't really

00:16:28.170 --> 00:16:31.202
make it easy to download in
bulk or de-identify this.

00:16:31.202 --> 00:16:32.910
And part of the reason
is because then it

00:16:32.910 --> 00:16:35.118
would make it easy for you
to switch vendors and have

00:16:35.118 --> 00:16:36.200
somebody else take over.

00:16:36.200 --> 00:16:37.950
So they make it a
little bit hard for you.

00:16:37.950 --> 00:16:40.530
Once it's in there, it's
hard for you to get it out,

00:16:40.530 --> 00:16:42.480
and people are
selling their data.

00:16:42.480 --> 00:16:43.900
That's certainly happening too.

00:16:43.900 --> 00:16:45.570
So there's a little
bit of attempts

00:16:45.570 --> 00:16:49.050
to try to control things that
way, and many of the labels

00:16:49.050 --> 00:16:50.710
you want are stored separately.

00:16:50.710 --> 00:16:52.390
So you want to know what the
diseases of these people.

00:16:52.390 --> 00:16:53.730
So you have the
raw imaging data,

00:16:53.730 --> 00:16:55.605
but all the clinical
stuff is somewhere else.

00:16:55.605 --> 00:16:57.130
So you have to
sometimes link that,

00:16:57.130 --> 00:16:58.740
and so you need to
get access there.

00:16:58.740 --> 00:17:01.115
And so just to give you a
little bit of an idea of scale,

00:17:01.115 --> 00:17:03.660
so we're about to get all the
ECGs from Brigham and Women's

00:17:03.660 --> 00:17:06.960
which is about 30
million historically,

00:17:06.960 --> 00:17:08.410
and this is all related to cost.

00:17:08.410 --> 00:17:11.609
So positron emission tomography,
you can get about 8,000 or so,

00:17:11.609 --> 00:17:13.800
and we're one of the
busiest centers for that.

00:17:13.800 --> 00:17:16.510
Echocardiograms are in
the 300,000 to 500,000

00:17:16.510 --> 00:17:17.260
range in archives.

00:17:17.260 --> 00:17:19.060
So that gets a little
bit more interesting.

00:17:19.060 --> 00:17:19.680
OK.

00:17:19.680 --> 00:17:21.599
This is what a DICOM
header looks like.

00:17:21.599 --> 00:17:23.849
You have some sort
of identifiers,

00:17:23.849 --> 00:17:25.980
and then you have some
information there,

00:17:25.980 --> 00:17:27.660
attributes of the
images, patient

00:17:27.660 --> 00:17:29.468
name, date of birth, frame rate.

00:17:29.468 --> 00:17:32.010
These kind of things are there,
and there's some variability.

00:17:32.010 --> 00:17:34.880
So it's never quite easy.

00:17:34.880 --> 00:17:36.170
OK.

00:17:36.170 --> 00:17:40.585
So these different modalities
have some different benefits

00:17:40.585 --> 00:17:41.960
to them which is
why they're used

00:17:41.960 --> 00:17:44.580
for one disease or the other.

00:17:44.580 --> 00:17:47.330
And so one of the real headaches
is that the heart moves.

00:17:47.330 --> 00:17:49.600
So the chest wall moves,
because we breathe,

00:17:49.600 --> 00:17:50.600
and the heart moves too.

00:17:50.600 --> 00:17:52.580
So you have to
image something that

00:17:52.580 --> 00:17:55.880
has enough temporal frequency
that you're not overwhelmed

00:17:55.880 --> 00:17:58.752
by the basic movement
of the heart itself,

00:17:58.752 --> 00:18:00.460
and so some of these
things aren't great.

00:18:00.460 --> 00:18:03.110
So SPECT or PET
acquire their images,

00:18:03.110 --> 00:18:05.443
which are radioactive
counts, over minutes.

00:18:05.443 --> 00:18:06.860
So that's certainly
a problem when

00:18:06.860 --> 00:18:08.880
it comes to something
that's moving like that,

00:18:08.880 --> 00:18:10.547
and if you want to
have high resolution.

00:18:10.547 --> 00:18:13.130
So typically, you have very
poor spatial resolution

00:18:13.130 --> 00:18:16.010
for something that
ultimately doesn't deal well

00:18:16.010 --> 00:18:17.570
with the moving aspect.

00:18:17.570 --> 00:18:20.160
So coronary angiography has
very, very fast frame rates.

00:18:20.160 --> 00:18:22.510
So that's X-ray, and
that's sort of very fast.

00:18:22.510 --> 00:18:24.740
Echocardiography
can be quite fast.

00:18:24.740 --> 00:18:26.765
MRI and CT are
not quite as good,

00:18:26.765 --> 00:18:28.640
and so there's some
degradation of the image.

00:18:28.640 --> 00:18:30.770
As a result, people do
something called gating,

00:18:30.770 --> 00:18:33.920
where they'll take the
electrocardiogram, the ECG,

00:18:33.920 --> 00:18:36.610
and try to line up
different portions

00:18:36.610 --> 00:18:37.610
of different heartbeats.

00:18:37.610 --> 00:18:40.332
And say, well, we'll take
this image from here,

00:18:40.332 --> 00:18:42.290
line it up with this one
from there, this one--

00:18:42.290 --> 00:18:44.873
I'm going to talk a little bit
about that, about registration,

00:18:44.873 --> 00:18:47.573
but ultimately, that's a problem
that people have to deal with.

00:18:47.573 --> 00:18:49.490
So it's a computer vision
problem of interest.

00:18:49.490 --> 00:18:50.750
OK.

00:18:50.750 --> 00:18:52.730
Preamble is almost done.

00:18:52.730 --> 00:18:54.770
OK.

00:18:54.770 --> 00:18:56.983
So why do we even
imagine any of this stuff

00:18:56.983 --> 00:18:57.900
is going to be useful?

00:18:57.900 --> 00:19:02.330
So it turns out that the
practice of interpreting

00:19:02.330 --> 00:19:04.520
involves a lot of
manual measurements.

00:19:04.520 --> 00:19:07.010
So people like
me, and people who

00:19:07.010 --> 00:19:08.660
have trained for
way too long, find

00:19:08.660 --> 00:19:11.550
themselves getting little rulers
and measuring various things.

00:19:11.550 --> 00:19:14.600
So for example, this is
a narrowing of an artery.

00:19:14.600 --> 00:19:16.850
So you could take a little
bit of calipers and measure

00:19:16.850 --> 00:19:19.190
across that and
compare it to here

00:19:19.190 --> 00:19:22.220
and say, ah, this
is 80% narrowed.

00:19:22.220 --> 00:19:24.440
You could measure the
area of this chamber,

00:19:24.440 --> 00:19:27.200
the left ventricle, and you
can measure its area is,

00:19:27.200 --> 00:19:29.660
and you can see, ah,
its peak area is this.

00:19:29.660 --> 00:19:31.125
It's minimum area is this.

00:19:31.125 --> 00:19:33.000
Therefore, it's contracting
a certain amount.

00:19:33.000 --> 00:19:33.917
So we do those things.

00:19:33.917 --> 00:19:36.410
We measure those things by hand.

00:19:36.410 --> 00:19:39.230
And the other thing we do is we
actually diagnose things just

00:19:39.230 --> 00:19:40.080
by looking at them.

00:19:40.080 --> 00:19:43.040
So this is a disease called
cardiac amyloid characterized

00:19:43.040 --> 00:19:44.030
by some thickening.

00:19:44.030 --> 00:19:45.110
I'll show you a little
bit more about that

00:19:45.110 --> 00:19:46.130
and some sparkling here.

00:19:46.130 --> 00:19:48.440
So people do look and say,
ah, this is what this is.

00:19:48.440 --> 00:19:50.703
So there's kind of a
classification problem

00:19:50.703 --> 00:19:52.620
that comes either at the
image or video level.

00:19:52.620 --> 00:19:54.828
So we'll talk about whether
this is even worth doing.

00:19:54.828 --> 00:19:56.020
AUDIENCE: I have a question.

00:19:56.020 --> 00:19:56.820
RAHUL DEO: Yes.

00:19:56.820 --> 00:19:58.987
AUDIENCE: Is this with
software, or do you literally

00:19:58.987 --> 00:20:00.340
take a ruler and measure?

00:20:00.340 --> 00:20:03.500
RAHUL DEO: So the software
involves clicking at one point,

00:20:03.500 --> 00:20:05.710
stretching something, and
clicking another point.

00:20:05.710 --> 00:20:07.793
So it's a little better
than pulling the ruler out

00:20:07.793 --> 00:20:11.140
of your back pocket, but
not that much better.

00:20:11.140 --> 00:20:11.690
OK.

00:20:11.690 --> 00:20:14.580
So we're going to talk
about or three little areas,

00:20:14.580 --> 00:20:15.770
and again, this is not--

00:20:15.770 --> 00:20:18.187
I got involved in this really
in the last two years or so.

00:20:18.187 --> 00:20:19.978
It's nice of David to
ask me to speak here,

00:20:19.978 --> 00:20:21.560
but I think there
are probably people

00:20:21.560 --> 00:20:24.295
in this room who have a lot
more experience in this space.

00:20:24.295 --> 00:20:26.420
But the areas that have
been relevant to what we've

00:20:26.420 --> 00:20:29.870
been doing has been image
classification and then

00:20:29.870 --> 00:20:30.850
semantic segmentation.

00:20:30.850 --> 00:20:33.350
So image classification being
assigning a label to an image,

00:20:33.350 --> 00:20:34.340
very great.

00:20:34.340 --> 00:20:37.820
Semantic segmentation, assigning
each pixel to a class label,

00:20:37.820 --> 00:20:40.278
and we haven't done anything
around the image registration,

00:20:40.278 --> 00:20:41.903
but there are some
interesting problems

00:20:41.903 --> 00:20:43.260
I've been thinking about there.

00:20:43.260 --> 00:20:45.343
And that's really mapping
different sets of images

00:20:45.343 --> 00:20:46.830
onto one coordinate system.

00:20:46.830 --> 00:20:47.330
OK.

00:20:47.330 --> 00:20:50.580
So seems obvious that
image classification would

00:20:50.580 --> 00:20:53.458
be something that you would
imagine a physician does,

00:20:53.458 --> 00:20:54.750
and so maybe we can mimic that.

00:20:54.750 --> 00:20:56.542
Seems like a reasonable
thing that happens.

00:20:56.542 --> 00:20:59.610
So lots of things that
radiologists, people

00:20:59.610 --> 00:21:03.690
who interpret images, do
involve terms of recognition,

00:21:03.690 --> 00:21:04.890
and they're really fast.

00:21:04.890 --> 00:21:08.400
So it takes them a couple of
minutes to often do things

00:21:08.400 --> 00:21:10.860
like detect if there's
cancer, detect if somebody has

00:21:10.860 --> 00:21:13.530
pneumonia, detect if there's
breast cancer in a mammogram,

00:21:13.530 --> 00:21:14.580
tells there's
fluid in the heart,

00:21:14.580 --> 00:21:17.038
and then even less than that,
one minute often, 30 seconds,

00:21:17.038 --> 00:21:19.300
they can very, very fast.

00:21:19.300 --> 00:21:23.160
So you can imagine
the wave of excitement

00:21:23.160 --> 00:21:26.730
around image classification
was really post-image net,

00:21:26.730 --> 00:21:28.830
so maybe about three years,
four years, or so ago.

00:21:28.830 --> 00:21:30.455
We're always a little
slow in medicine,

00:21:30.455 --> 00:21:32.985
so a little bit
behind other fields.

00:21:32.985 --> 00:21:34.860
And the places that they
went were the places

00:21:34.860 --> 00:21:36.810
where there are huge
data sets already,

00:21:36.810 --> 00:21:38.620
and where there's simple
recognition tests.

00:21:38.620 --> 00:21:40.710
So chest X-rays and
mammograms are both places

00:21:40.710 --> 00:21:43.410
that had a lot of
attention, and other places

00:21:43.410 --> 00:21:46.310
have been slowed down by just
how hard it is to get data.

00:21:46.310 --> 00:21:48.060
So if you can't get a
big enough data set,

00:21:48.060 --> 00:21:49.893
then you're not going
to be able to do much.

00:21:49.893 --> 00:21:50.460
OK.

00:21:50.460 --> 00:21:54.378
So David mentioned, you guys
already covered very nicely,

00:21:54.378 --> 00:21:55.920
and this is probably
kind of old hat.

00:21:55.920 --> 00:21:58.650
But I would say that prior to
convolutional neural networks,

00:21:58.650 --> 00:22:00.840
nothing was happening in
the image classification

00:22:00.840 --> 00:22:01.600
space in medicine.

00:22:01.600 --> 00:22:02.593
It was just not.

00:22:02.593 --> 00:22:05.010
People weren't even thinking
that it was even worth doing.

00:22:05.010 --> 00:22:07.110
Now, there's a lot
of interest, and so I

00:22:07.110 --> 00:22:10.650
have many different companies
coming and asking for help

00:22:10.650 --> 00:22:11.780
with some of these things.

00:22:11.780 --> 00:22:16.290
And so it is now a
very attractive thing

00:22:16.290 --> 00:22:17.713
in terms of
thinking, and I think

00:22:17.713 --> 00:22:19.380
people haven't thought
out all that well

00:22:19.380 --> 00:22:21.630
how we're going to use that.

00:22:21.630 --> 00:22:23.885
So for example, if it takes
a radiologist a minute

00:22:23.885 --> 00:22:25.260
to two minutes to
read something,

00:22:25.260 --> 00:22:28.360
how much benefit are you
going to get to automate it?

00:22:28.360 --> 00:22:30.600
And the real
problem is you can't

00:22:30.600 --> 00:22:31.860
take that radiologist away.

00:22:31.860 --> 00:22:33.360
They're still there,
because they're

00:22:33.360 --> 00:22:34.740
the ones who are on the hook.

00:22:34.740 --> 00:22:36.365
And they're going to
get sued, and it's

00:22:36.365 --> 00:22:38.460
among the most sued
profession in medicine.

00:22:38.460 --> 00:22:42.000
So there's lots of people
who can read an X-ray.

00:22:42.000 --> 00:22:44.140
You don't need to have
all that training.

00:22:44.140 --> 00:22:46.350
But if you're the one
who's going to be sued,

00:22:46.350 --> 00:22:48.070
it ends up being that
there really isn't

00:22:48.070 --> 00:22:49.320
any task shifting in medicine.

00:22:49.320 --> 00:22:51.150
There isn't that
kind of, oh, I'm

00:22:51.150 --> 00:22:53.940
going to let such
and such take on 99%,

00:22:53.940 --> 00:22:55.690
and just tell me when
there is a problem.

00:22:55.690 --> 00:22:58.260
It just doesn't happen, because
they ultimately don't feel

00:22:58.260 --> 00:22:59.610
comfortable passing that on.

00:22:59.610 --> 00:23:01.540
So that's something
to think about.

00:23:01.540 --> 00:23:03.540
So you have a task
that's relatively

00:23:03.540 --> 00:23:06.240
easy for a very, very expensive
and skilled person to do,

00:23:06.240 --> 00:23:07.990
and they refuse to give it up.

00:23:07.990 --> 00:23:08.490
OK.

00:23:08.490 --> 00:23:10.603
So that's a problem,
but you can imagine

00:23:10.603 --> 00:23:13.020
there is some scenarios-- and
we'll talk more about this--

00:23:13.020 --> 00:23:14.103
as to where that could be.

00:23:14.103 --> 00:23:16.050
So let's say it's overnight.

00:23:16.050 --> 00:23:18.660
The radiologist is sleeping
comfortably at home,

00:23:18.660 --> 00:23:20.790
and you have a bunch
of studies being

00:23:20.790 --> 00:23:22.235
done in the emergency room.

00:23:22.235 --> 00:23:24.360
And you want to figure out,
OK, which one should we

00:23:24.360 --> 00:23:25.050
call them about?

00:23:25.050 --> 00:23:26.760
So you can imagine
there could be triage,

00:23:26.760 --> 00:23:30.300
because the status quo would
be, we'll take them one by one.

00:23:30.300 --> 00:23:32.850
Maybe you could imagine sifting
through them quickly and then

00:23:32.850 --> 00:23:34.230
re-prioritizing them.

00:23:34.230 --> 00:23:35.412
They'll still be looked at.

00:23:35.412 --> 00:23:37.120
Every single one will
still be looked at.

00:23:37.120 --> 00:23:38.412
It's just the order may change.

00:23:38.412 --> 00:23:39.960
So that's an example,
and you could

00:23:39.960 --> 00:23:42.450
imagine there could be
separate-- someone else could

00:23:42.450 --> 00:23:43.540
read at the same time.

00:23:43.540 --> 00:23:44.910
And we'll come back
to this in terms of

00:23:44.910 --> 00:23:46.285
whether or not
you could have two

00:23:46.285 --> 00:23:49.500
streams and whether or not
that is a scenario that

00:23:49.500 --> 00:23:50.478
would make some sense.

00:23:50.478 --> 00:23:52.020
And maybe, in
resource-poor settings,

00:23:52.020 --> 00:23:53.670
where we're not teaming
with the radiologist,

00:23:53.670 --> 00:23:54.795
maybe that makes sense too.

00:23:54.795 --> 00:23:57.210
So we'll come back to that too.

00:23:57.210 --> 00:23:57.810
OK.

00:23:57.810 --> 00:23:59.460
So here's another problem.

00:23:59.460 --> 00:24:03.420
So almost everything in
medicine requires some element

00:24:03.420 --> 00:24:06.090
of confirmation of
a visual finding,

00:24:06.090 --> 00:24:08.100
and some of the reasons
are very simple.

00:24:08.100 --> 00:24:11.248
So let's say you want to talk
about there being a tumor.

00:24:11.248 --> 00:24:13.290
So if you're going to ask
a surgeon to biopsy it,

00:24:13.290 --> 00:24:15.440
you better tell
them where it is.

00:24:15.440 --> 00:24:17.220
It's not enough to
just say, this image

00:24:17.220 --> 00:24:18.990
has a tumor somewhere on it.

00:24:18.990 --> 00:24:20.893
So there is some element
of that that you're

00:24:20.893 --> 00:24:23.310
going to need to be a little
bit more detailed than simply

00:24:23.310 --> 00:24:26.100
making a classification
with a level one image,

00:24:26.100 --> 00:24:28.702
but I would say beyond that.

00:24:28.702 --> 00:24:30.910
Let's say, I'm going to try
to get one of my patients

00:24:30.910 --> 00:24:33.600
to go for valve surgery.

00:24:33.600 --> 00:24:36.060
I'll sit with them,
bring up their echo,

00:24:36.060 --> 00:24:39.120
sit side by side with them,
and point them to where it is.

00:24:39.120 --> 00:24:41.090
Bring up a normal
one and compare,

00:24:41.090 --> 00:24:43.240
because I want them to be
involved in the decision.

00:24:43.240 --> 00:24:45.387
I want them to feel like
they're not just trust--

00:24:45.387 --> 00:24:46.470
and they have to trust me.

00:24:46.470 --> 00:24:47.370
At the end of the
day, they don't even

00:24:47.370 --> 00:24:48.210
know that I'm showing--

00:24:48.210 --> 00:24:50.210
I'll show them their name,
but ultimately, there

00:24:50.210 --> 00:24:51.310
is some element of trust.

00:24:51.310 --> 00:24:53.440
They're not able to do
this, but at the same time,

00:24:53.440 --> 00:24:55.560
there is this sense of
shared decision making.

00:24:55.560 --> 00:24:58.920
You're trying to communicate to
somebody, whose life is really

00:24:58.920 --> 00:25:02.380
at risk here, that this is
why we're doing this decision.

00:25:02.380 --> 00:25:04.698
So the more you could imagine
that there is obscuring,

00:25:04.698 --> 00:25:06.490
the more difficult it
is to make that case.

00:25:06.490 --> 00:25:08.460
So medicine is this--

00:25:08.460 --> 00:25:12.560
I found this review by Bin Yu
from Berkeley, just came out,

00:25:12.560 --> 00:25:15.900
and it talks about this tension
between predictive accuracy

00:25:15.900 --> 00:25:17.510
and descriptive accuracy.

00:25:17.510 --> 00:25:21.005
So this is of the typical thing
we think about that matters,

00:25:21.005 --> 00:25:22.380
and there's lots
of people who've

00:25:22.380 --> 00:25:24.340
written about this thing.

00:25:24.340 --> 00:25:28.470
Medicine is tough in that it's
very demanding in this space

00:25:28.470 --> 00:25:32.340
here, and it's almost
inflexible in this space here.

00:25:32.340 --> 00:25:34.260
So it's a tough nut
to crack in terms

00:25:34.260 --> 00:25:35.760
of being able to
make some progress,

00:25:35.760 --> 00:25:38.460
and so we'll talk more about
when that's likely to happen.

00:25:38.460 --> 00:25:38.960
OK.

00:25:38.960 --> 00:25:42.060
So this again may be something
that's very familiar to you.

00:25:42.060 --> 00:25:44.910
So we had this problem in
terms of some of the disease

00:25:44.910 --> 00:25:46.620
detection models,
and I didn't find

00:25:46.620 --> 00:25:48.150
this all that
satisfying in terms

00:25:48.150 --> 00:25:50.010
being able to
successfully localize.

00:25:50.010 --> 00:25:51.635
So just digging
through the literature,

00:25:51.635 --> 00:25:55.230
it looks like this idea
of being able to explain

00:25:55.230 --> 00:25:57.690
what part of the
image is driving

00:25:57.690 --> 00:26:01.510
a certain classification.

00:26:01.510 --> 00:26:03.690
That field is modestly old.

00:26:03.690 --> 00:26:05.400
Maybe it goes back before that.

00:26:05.400 --> 00:26:07.110
But ultimately,
there's two broad ways.

00:26:07.110 --> 00:26:10.260
You can imagine finding an
exemplary image that maximally

00:26:10.260 --> 00:26:12.960
activates the classical work,
or you can take a given image

00:26:12.960 --> 00:26:17.010
and say, what aspect of it is
driving the classification?

00:26:17.010 --> 00:26:20.040
And so in this paper here
did both those things.

00:26:20.040 --> 00:26:23.135
They either went
through and optimized--

00:26:23.135 --> 00:26:25.260
starting from an average
of all the training data--

00:26:25.260 --> 00:26:28.020
they optimized the intensities
until they maximized

00:26:28.020 --> 00:26:29.580
the score for a given class.

00:26:29.580 --> 00:26:31.270
So that's what's shown here.

00:26:31.270 --> 00:26:33.660
And then another way to do
it is in some sense you could

00:26:33.660 --> 00:26:36.240
take a derivative of
the score function

00:26:36.240 --> 00:26:38.430
relative to the intensities
of all the pixels

00:26:38.430 --> 00:26:39.210
and come up with
something like this.

00:26:39.210 --> 00:26:41.070
But you could imagine, if
you showed this to a patient,

00:26:41.070 --> 00:26:42.750
they wouldn't be very satisfied.

00:26:42.750 --> 00:26:47.850
So it's very difficult to make a
case that this is super useful,

00:26:47.850 --> 00:26:50.400
but it seems like this field
has progressed somewhat,

00:26:50.400 --> 00:26:51.970
and I haven't tried this out.

00:26:51.970 --> 00:26:53.930
This is a paper by Max
Welling and company,

00:26:53.930 --> 00:26:55.320
out by a couple of
years, and maybe you guys

00:26:55.320 --> 00:26:56.550
are familiar with this.

00:26:56.550 --> 00:26:59.008
But this ultimately is a little
bit of a different approach

00:26:59.008 --> 00:27:01.230
in the sense that
they take patches,

00:27:01.230 --> 00:27:03.570
the sort of
purple-like patch here,

00:27:03.570 --> 00:27:10.440
and they compare the final
score, or class label,

00:27:10.440 --> 00:27:12.180
relative to what it--

00:27:12.180 --> 00:27:15.090
so taking the intensity
here and replacing it

00:27:15.090 --> 00:27:18.342
by a conditional result
sampling from the periphery.

00:27:18.342 --> 00:27:19.800
And just comparing
those two things

00:27:19.800 --> 00:27:22.210
and seeing whether or not
you either get activation,

00:27:22.210 --> 00:27:24.960
which is the red here.

00:27:24.960 --> 00:27:27.360
This is the way that they
did the conditional sampling,

00:27:27.360 --> 00:27:29.802
and then blue would be
the negative contributors.

00:27:29.802 --> 00:27:31.260
And there, you can
imagine, there's

00:27:31.260 --> 00:27:32.460
a little bit more
distinction here,

00:27:32.460 --> 00:27:34.793
and then something a little
bit more on the medical side

00:27:34.793 --> 00:27:35.910
is this is a brain MRI.

00:27:35.910 --> 00:27:37.980
And so depending
on this patch size,

00:27:37.980 --> 00:27:40.860
you get a different
degree of resolution

00:27:40.860 --> 00:27:44.590
to localizing some areas of
the image that are relevant.

00:27:44.590 --> 00:27:46.710
So this is something
that we're going

00:27:46.710 --> 00:27:50.955
to expect a lot of demands
from the medical field in terms

00:27:50.955 --> 00:27:52.080
of being able to show this.

00:27:52.080 --> 00:27:53.550
And at least our
initial forays weren't

00:27:53.550 --> 00:27:55.675
very satisfying doing this
with what we were doing,

00:27:55.675 --> 00:27:57.930
but maybe these algorithms
have gotten better.

00:27:57.930 --> 00:27:58.430
OK.

00:27:58.430 --> 00:27:59.730
So next thing that matters.

00:27:59.730 --> 00:28:00.230
OK.

00:28:00.230 --> 00:28:01.710
So this is what people do.

00:28:01.710 --> 00:28:06.360
So I did my cardiology
fellowship in MGH,

00:28:06.360 --> 00:28:07.820
and I just traced circles.

00:28:07.820 --> 00:28:08.570
That's what I did.

00:28:08.570 --> 00:28:11.820
I just trace circles, and
I stretched a ruler across,

00:28:11.820 --> 00:28:12.750
and then fed that in.

00:28:12.750 --> 00:28:14.910
At least the program
computed the volumes

00:28:14.910 --> 00:28:17.980
for me, the areas and
volumes, but otherwise, you

00:28:17.980 --> 00:28:20.100
have to do this yourself.

00:28:20.100 --> 00:28:23.880
And so this is like
a task that's done,

00:28:23.880 --> 00:28:26.040
and sometimes you may have to--

00:28:26.040 --> 00:28:28.680
here's an example of
volumes being computed

00:28:28.680 --> 00:28:32.238
by tracing these sorts of things
and much radiology reports just

00:28:32.238 --> 00:28:33.030
involve doing that.

00:28:33.030 --> 00:28:34.800
So this seems like a
very obvious task we

00:28:34.800 --> 00:28:36.870
should be able to improve on.

00:28:36.870 --> 00:28:39.060
So medicine tends
to be not the most

00:28:39.060 --> 00:28:40.675
creative in terms
of trying a bunch

00:28:40.675 --> 00:28:41.800
of different architectures.

00:28:41.800 --> 00:28:44.190
So if you look at the papers,
they all jump on the U-net

00:28:44.190 --> 00:28:47.310
as being the
favorite architecture

00:28:47.310 --> 00:28:48.870
for semantic segmentation.

00:28:48.870 --> 00:28:51.000
So maybe familiar
to people here,

00:28:51.000 --> 00:28:55.760
really, it just captures this
encoding or contracting layer.

00:28:55.760 --> 00:28:58.020
Where you're downsampling,
and then there's

00:28:58.020 --> 00:29:00.600
a symmetric upsampling
that takes place.

00:29:00.600 --> 00:29:03.300
And then ultimately, there's
these skip connections, where

00:29:03.300 --> 00:29:07.290
you take an image, and
then you can catonate it

00:29:07.290 --> 00:29:10.410
with this upsampled layer, and
this helps get a little bit

00:29:10.410 --> 00:29:11.160
more localization.

00:29:11.160 --> 00:29:12.827
So we used this for
our paper, and we'll

00:29:12.827 --> 00:29:15.090
talk about this a little
bit, and it's very popular

00:29:15.090 --> 00:29:16.742
within the medical literature.

00:29:16.742 --> 00:29:18.450
One of the things that
was quite annoying

00:29:18.450 --> 00:29:20.802
is that what you would find
for some of the images,

00:29:20.802 --> 00:29:22.260
you'd find, let's
say, a ventricle.

00:29:22.260 --> 00:29:24.180
You'd find this
nicely segmented area,

00:29:24.180 --> 00:29:26.305
and then you'd find this
little satellite ventricle

00:29:26.305 --> 00:29:28.200
that the image would just pick.

00:29:28.200 --> 00:29:31.350
The problem is that this
pixel-level classification

00:29:31.350 --> 00:29:33.690
tends to be a
problem, and a human

00:29:33.690 --> 00:29:35.010
would never make that mistake.

00:29:35.010 --> 00:29:38.370
But that tends to be something
that sounds like it is common

00:29:38.370 --> 00:29:40.740
in the-- this is a
common tension is

00:29:40.740 --> 00:29:45.750
that this sort of focusing
on relatively limited scales

00:29:45.750 --> 00:29:50.040
ends up being problematic,
when it comes to picking up

00:29:50.040 --> 00:29:51.100
the global architecture.

00:29:51.100 --> 00:29:52.440
And so there's lots
of different solutions

00:29:52.440 --> 00:29:53.800
it looks like in the literature.

00:29:53.800 --> 00:29:55.675
I just highlighted some
of these from a paper

00:29:55.675 --> 00:29:57.785
that was published from
Google a little while ago.

00:29:57.785 --> 00:29:59.160
One of the things
that's captured

00:29:59.160 --> 00:30:01.500
is these ideas of
dilated convolutions,

00:30:01.500 --> 00:30:04.440
and so that you
have convolutions

00:30:04.440 --> 00:30:05.520
built on convolutions.

00:30:05.520 --> 00:30:08.280
And so ultimately, you have
a much bigger receptive field

00:30:08.280 --> 00:30:11.513
for this layer, though
you haven't really

00:30:11.513 --> 00:30:12.930
increased the
number of parameters

00:30:12.930 --> 00:30:13.560
that you have to learn.

00:30:13.560 --> 00:30:14.350
So there is some.

00:30:14.350 --> 00:30:15.475
It seems like there's lots.

00:30:15.475 --> 00:30:17.400
This is not just
a problem for us

00:30:17.400 --> 00:30:19.323
but a problem for many
people in this field.

00:30:19.323 --> 00:30:21.240
So we need to be a little
bit more adventurous

00:30:21.240 --> 00:30:23.198
in terms of trying some
of these other methods.

00:30:23.198 --> 00:30:26.190
We did try a little bit of
that and didn't find a gains,

00:30:26.190 --> 00:30:27.690
but I think,
ultimately, there still

00:30:27.690 --> 00:30:29.270
needs to be a little
bit more work there.

00:30:29.270 --> 00:30:29.610
OK.

00:30:29.610 --> 00:30:31.318
So the last thing I'm
going to talk about

00:30:31.318 --> 00:30:33.840
before getting into my
work is really this idea

00:30:33.840 --> 00:30:35.490
of image registration.

00:30:35.490 --> 00:30:38.400
So I talked about how there are
sometimes some techniques that

00:30:38.400 --> 00:30:42.462
have limitations, either in
terms of spatial resolution

00:30:42.462 --> 00:30:43.420
or temporal resolution.

00:30:43.420 --> 00:30:46.940
So this is a PET scan here,
this sort of reddish glow here,

00:30:46.940 --> 00:30:49.780
and in the background, we
have a CAT scan of the heart.

00:30:49.780 --> 00:30:52.280
And so clearly, this is a
poorly registered image,

00:30:52.280 --> 00:30:55.450
where you have the PET scan kind
of floating out here, when it

00:30:55.450 --> 00:30:56.785
really should be lined up here.

00:30:56.785 --> 00:30:59.160
And so you have something
that's registered better there.

00:30:59.160 --> 00:31:00.868
I also mentioned this
problem but gating.

00:31:00.868 --> 00:31:03.520
So ultimately, if you
have an image taken

00:31:03.520 --> 00:31:05.560
from different
cardiac cycles, you're

00:31:05.560 --> 00:31:08.500
going to have align
them in some way.

00:31:08.500 --> 00:31:10.870
It seems like a very mature
problem in computer vision

00:31:10.870 --> 00:31:11.530
world.

00:31:11.530 --> 00:31:13.640
We haven't done
anything in this space,

00:31:13.640 --> 00:31:16.300
but ultimately, it has
been around for decades.

00:31:16.300 --> 00:31:19.907
If not, I would just at least
touch it, touch upon it.

00:31:19.907 --> 00:31:21.490
So this is sort of
the old school way,

00:31:21.490 --> 00:31:23.110
and then now people
are starting to use

00:31:23.110 --> 00:31:24.610
conditional variational
autoencoders

00:31:24.610 --> 00:31:27.880
to be able to learn
geometric transformations.

00:31:27.880 --> 00:31:32.430
This is the Siemens group out in
Princeton that has this paper.

00:31:32.430 --> 00:31:34.180
Again, nothing I'm
going to focus on, just

00:31:34.180 --> 00:31:36.250
wanted to bring it up
as being an area that

00:31:36.250 --> 00:31:38.300
remains of interest.

00:31:38.300 --> 00:31:39.280
OK.

00:31:39.280 --> 00:31:45.320
So I think we're doing
OK, but you said 4:00.

00:31:45.320 --> 00:31:46.060
PROFESSOR: 3:55

00:31:46.060 --> 00:31:46.600
RAHUL DEO: 3:55.

00:31:46.600 --> 00:31:47.110
OK.

00:31:47.110 --> 00:31:47.920
All right, and interrupt.

00:31:47.920 --> 00:31:48.670
Please, interrupt.

00:31:48.670 --> 00:31:49.720
OK?

00:31:49.720 --> 00:31:52.650
I'm hoping that I'm
not talking too fast.

00:31:52.650 --> 00:31:54.530
OK.

00:31:54.530 --> 00:31:58.160
As David said, this
was not my field,

00:31:58.160 --> 00:32:00.328
but increasingly,
there is some interest

00:32:00.328 --> 00:32:02.120
in terms of getting
involved in it, in part

00:32:02.120 --> 00:32:03.920
because of my frustrations
with clinical medicine.

00:32:03.920 --> 00:32:05.295
So this is one of
my frustrations

00:32:05.295 --> 00:32:06.470
with clinical medicine.

00:32:06.470 --> 00:32:10.380
So cardiology has
not really changed,

00:32:10.380 --> 00:32:13.850
and one of the things
it fails at miserably

00:32:13.850 --> 00:32:17.630
is picking up
early-onset disease.

00:32:17.630 --> 00:32:20.690
So here's the typical
profile, a little facetious.

00:32:20.690 --> 00:32:24.230
So people like me
in our early 40s,

00:32:24.230 --> 00:32:26.365
start to already
have some problems

00:32:26.365 --> 00:32:27.490
with some of these numbers.

00:32:27.490 --> 00:32:29.930
So I like to joke that, since I
came back to the Harvard system

00:32:29.930 --> 00:32:31.847
from California, my blood
pressure has gone up

00:32:31.847 --> 00:32:34.450
10 points which is
true, unfortunately.

00:32:34.450 --> 00:32:38.020
So these changes
already start to happen,

00:32:38.020 --> 00:32:40.310
and nobody does
anything about it.

00:32:40.310 --> 00:32:43.495
So you can go to your doctor,
and you're also saying,

00:32:43.495 --> 00:32:45.120
no, I don't want to
be on any medicine.

00:32:45.120 --> 00:32:47.412
They're like, no, no, you
shouldn't be on any medicine.

00:32:47.412 --> 00:32:52.223
So you kind hem and haw, and a
decade goes by, 15 years go by.

00:32:52.223 --> 00:32:53.890
And then finally,
you're like, OK, well,

00:32:53.890 --> 00:32:55.380
it looks like at
least my coworkers

00:32:55.380 --> 00:32:58.733
are on some medicines, or maybe
I'll be willing to do that.

00:32:58.733 --> 00:33:00.900
And so they've got lots of
stuff you can be treated,

00:33:00.900 --> 00:33:03.382
but it is often very
difficult, and you see

00:33:03.382 --> 00:33:04.590
this at the doctor level too.

00:33:04.590 --> 00:33:05.155
Yes.

00:33:05.155 --> 00:33:06.530
AUDIENCE: For the
optical values,

00:33:06.530 --> 00:33:11.990
how much personal deviation
is there for the values?

00:33:11.990 --> 00:33:17.860
RAHUL DEO: So the optimal
value is fixed and is just

00:33:17.860 --> 00:33:19.780
like a reference value.

00:33:19.780 --> 00:33:21.900
And you can be off--

00:33:21.900 --> 00:33:23.380
so blood pressure, let's say.

00:33:23.380 --> 00:33:25.420
So people consider optimal
to be less than 120

00:33:25.420 --> 00:33:27.600
over less than 80.

00:33:27.600 --> 00:33:29.715
People are in the 200s.

00:33:29.715 --> 00:33:31.590
So you'd be treated in
the 200s, but there'll

00:33:31.590 --> 00:33:34.077
be lots of people in
the 140s and the 150s,

00:33:34.077 --> 00:33:35.910
and there'll be a degree
of kind of nihilism

00:33:35.910 --> 00:33:38.010
about that for some time.

00:33:38.010 --> 00:33:40.560
And my patients would be
like, oh, I got into the fight

00:33:40.560 --> 00:33:42.930
with the parking attendant.

00:33:42.930 --> 00:33:45.160
I just had a really bad phone--

00:33:45.160 --> 00:33:47.250
there's like countless
excuses for why

00:33:47.250 --> 00:33:49.552
it is that one shouldn't
start a medication,

00:33:49.552 --> 00:33:51.010
and this can go on
for a long time.

00:33:51.010 --> 00:33:51.553
Yes.

00:33:51.553 --> 00:33:52.470
AUDIENCE: [INAUDIBLE].

00:33:52.470 --> 00:33:55.676
How can you assess the risk
[INAUDIBLE] for blood pressure?

00:33:55.676 --> 00:33:59.127
Is that, like,
noise [INAUDIBLE]??

00:34:03.933 --> 00:34:04.600
RAHUL DEO: Yeah.

00:34:04.600 --> 00:34:05.130
So OK.

00:34:05.130 --> 00:34:06.720
So that's a great point.

00:34:06.720 --> 00:34:08.100
So yeah.

00:34:08.100 --> 00:34:12.030
So the question is
that many of the things

00:34:12.030 --> 00:34:14.260
that we're seeing
as risk factors have

00:34:14.260 --> 00:34:15.750
inherent variability to them.

00:34:15.750 --> 00:34:19.050
Blood sugar is another great
example of those things.

00:34:19.050 --> 00:34:21.210
If you could have a
single-point estimate that

00:34:21.210 --> 00:34:23.370
arises in the setting of
a single clinic visit,

00:34:23.370 --> 00:34:24.270
how much do you trust that?

00:34:24.270 --> 00:34:26.062
So it's a couple of
things related to that.

00:34:26.062 --> 00:34:28.139
So one of them is
that people could

00:34:28.139 --> 00:34:31.710
be sent home with monitors, and
they can have 24-hour monitors.

00:34:31.710 --> 00:34:34.710
In Europe, that's much
more often done than here.

00:34:34.710 --> 00:34:37.290
And then, the thing is that
often they'll say that,

00:34:37.290 --> 00:34:40.020
and then you go look at
like six consecutive visits,

00:34:40.020 --> 00:34:42.420
and they all have something
elevate, but it's true.

00:34:42.420 --> 00:34:46.260
This is a noisy point
estimate, and people

00:34:46.260 --> 00:34:49.270
have shown that averages
tend to do better.

00:34:49.270 --> 00:34:53.090
But at the same time,
if that's all you have--

00:34:53.090 --> 00:34:54.449
and the bias is interesting.

00:34:54.449 --> 00:34:57.685
Because the bias comes
from some degree of stress,

00:34:57.685 --> 00:34:59.310
but we have lots of
stress in our life.

00:34:59.310 --> 00:35:01.060
I hopefully am not the
most stressful part

00:35:01.060 --> 00:35:04.050
of my patient's life, and so
I think that ultimately there

00:35:04.050 --> 00:35:05.760
are--

00:35:05.760 --> 00:35:07.742
and the problem
with that is it's

00:35:07.742 --> 00:35:09.450
a good reason for
someone to talk you out

00:35:09.450 --> 00:35:10.930
of them starting
them on anything.

00:35:10.930 --> 00:35:13.320
And that's what
ends up happening,

00:35:13.320 --> 00:35:16.270
and so this can be a
really long period of time.

00:35:16.270 --> 00:35:16.770
OK.

00:35:16.770 --> 00:35:17.530
So this is the grim part.

00:35:17.530 --> 00:35:18.030
OK?

00:35:18.030 --> 00:35:19.560
So it turns out
that once symptoms

00:35:19.560 --> 00:35:22.830
develop for something like
heart failure, decline is fast.

00:35:22.830 --> 00:35:26.322
So 50% mortality in five
years, after somebody gets

00:35:26.322 --> 00:35:28.530
hospitalized for their first
heart failure admission,

00:35:28.530 --> 00:35:30.960
and often the symptoms
are just around that time.

00:35:30.960 --> 00:35:32.670
So unfortunately,
these things tend

00:35:32.670 --> 00:35:36.982
to be irreversible changes
that happen in the background,

00:35:36.982 --> 00:35:38.940
and largely, you don't
really have any symptoms

00:35:38.940 --> 00:35:40.030
until late in the game.

00:35:40.030 --> 00:35:42.630
So we have this problem, where
we have this huge stretch.

00:35:42.630 --> 00:35:44.250
We know that there
is risk factors,

00:35:44.250 --> 00:35:46.792
but we have this huge stretch,
where nobody is doing anything

00:35:46.792 --> 00:35:47.470
about them.

00:35:47.470 --> 00:35:49.740
And then we have sort things
going downhill relatively

00:35:49.740 --> 00:35:51.030
quickly after that.

00:35:51.030 --> 00:35:53.310
And unfortunately,
I would make a case

00:35:53.310 --> 00:35:55.050
that probably
responsiveness is probably

00:35:55.050 --> 00:35:56.850
best did this phase over there.

00:35:56.850 --> 00:35:59.160
Expense is really
all over there.

00:35:59.160 --> 00:36:00.600
So we really want to find--

00:36:00.600 --> 00:36:02.850
and this is what I consider
to be missing in medicine.

00:36:02.850 --> 00:36:04.620
I'm going to come back to
this again a little bit later

00:36:04.620 --> 00:36:06.390
on-- but really, we
want to have these--

00:36:06.390 --> 00:36:08.910
if you're going to do something
in this asymptomatic phase,

00:36:08.910 --> 00:36:09.930
it better be cheap.

00:36:09.930 --> 00:36:11.970
You're not going to be
getting MRIs every day

00:36:11.970 --> 00:36:16.820
or every year for people
who have no symptoms.

00:36:16.820 --> 00:36:18.570
The system would
bankrupt if you had that.

00:36:18.570 --> 00:36:20.460
So we need these
low cost metrics

00:36:20.460 --> 00:36:22.570
that can tell us, at
an individual level,

00:36:22.570 --> 00:36:25.260
not just if we had
1,000 people like you,

00:36:25.260 --> 00:36:27.053
somebody would benefit.

00:36:27.053 --> 00:36:28.470
And this is what
my patients would

00:36:28.470 --> 00:36:32.190
say is that they would be
so excited about their EKG

00:36:32.190 --> 00:36:33.690
or their echo being
done every year,

00:36:33.690 --> 00:36:35.130
because they want to know,
how does it look like

00:36:35.130 --> 00:36:36.047
compared to last year?

00:36:36.047 --> 00:36:38.710
They want some comparison
at their level,

00:36:38.710 --> 00:36:41.070
not just some
public health report

00:36:41.070 --> 00:36:45.520
about this being a benefit
to 100 people like you.

00:36:45.520 --> 00:36:48.480
And so it shouldn't
be both low cost,

00:36:48.480 --> 00:36:49.860
should be reflective
at something

00:36:49.860 --> 00:36:51.840
an individual level,
should be relatively

00:36:51.840 --> 00:36:55.200
specific to the disease
process, expressive in some way,

00:36:55.200 --> 00:36:56.670
and should get
better with therapy.

00:36:56.670 --> 00:36:57.870
I think that's one
of the things that's

00:36:57.870 --> 00:36:59.430
pretty important
is if somebody does

00:36:59.430 --> 00:37:01.380
the things you ask
them to do, hopefully,

00:37:01.380 --> 00:37:02.580
that will look better.

00:37:02.580 --> 00:37:04.920
And then that would
be motivating,

00:37:04.920 --> 00:37:06.930
and I think that's how
people get motivated is

00:37:06.930 --> 00:37:08.910
that they get responses.

00:37:08.910 --> 00:37:11.502
So I would make a case
that even simple things

00:37:11.502 --> 00:37:12.960
like an ultrasound--
and I have one

00:37:12.960 --> 00:37:14.310
showed here--
really does capture

00:37:14.310 --> 00:37:15.780
some of these things,
and not all those things,

00:37:15.780 --> 00:37:17.460
but they have some
of those things.

00:37:17.460 --> 00:37:20.100
So you have, for example, that
in the setting of high blood

00:37:20.100 --> 00:37:22.530
pressure, the left ventricular
mass starts to thicken,

00:37:22.530 --> 00:37:25.080
and this is a quantitative,
continuous measure.

00:37:25.080 --> 00:37:28.380
It just thickens over time,
and the heart starts to change.

00:37:28.380 --> 00:37:31.180
The pumping function
can get worse over time.

00:37:31.180 --> 00:37:33.570
The left atrium, which is
this structure over here,

00:37:33.570 --> 00:37:35.730
this thin-walled structure
is amazing in the sense

00:37:35.730 --> 00:37:38.447
that it's almost this barometer
for the pressure in the heart.

00:37:38.447 --> 00:37:39.780
Oh, that's a horrible reference.

00:37:39.780 --> 00:37:41.820
OK, but it tends to
get kind of bigger

00:37:41.820 --> 00:37:44.460
and bigger in a very subtle
way before any symptoms happen.

00:37:44.460 --> 00:37:46.320
So you have this, and
this is just one view.

00:37:46.320 --> 00:37:46.820
Right?

00:37:46.820 --> 00:37:48.240
So this is a simple
view acquired

00:37:48.240 --> 00:37:50.250
from an ultrasound
that captures some

00:37:50.250 --> 00:37:52.420
of these things at
an individual level.

00:37:52.420 --> 00:37:54.270
So this gets to
some of my thoughts

00:37:54.270 --> 00:37:57.590
around where we could imagine
automated interpretation

00:37:57.590 --> 00:37:58.660
benefiting.

00:37:58.660 --> 00:38:03.060
So if you want to think about
where you're less likely.

00:38:03.060 --> 00:38:08.400
So with these very, very
difficult, end-stage,

00:38:08.400 --> 00:38:09.960
or complex decisions,
where you have

00:38:09.960 --> 00:38:12.423
a super skilled
person even collecting

00:38:12.423 --> 00:38:13.590
the data in the first place.

00:38:13.590 --> 00:38:14.840
They've gone through training.

00:38:14.840 --> 00:38:16.320
They're super experienced.

00:38:16.320 --> 00:38:18.210
You have a very expensive
piece of hardware

00:38:18.210 --> 00:38:19.740
used to collect the data.

00:38:19.740 --> 00:38:21.210
You have an expert
interpreting it.

00:38:21.210 --> 00:38:23.310
This is done late in
the disease course.

00:38:23.310 --> 00:38:25.590
You have to make
really hard decisions,

00:38:25.590 --> 00:38:27.120
and you don't want
to mess it up.

00:38:27.120 --> 00:38:29.040
So probably not
good places to try

00:38:29.040 --> 00:38:32.303
to stick in an automated
system in there,

00:38:32.303 --> 00:38:33.720
but what would be
attractive would

00:38:33.720 --> 00:38:37.120
be to try to enable studies that
are not even being done at all.

00:38:37.120 --> 00:38:40.470
So move to the
primary care setting.

00:38:40.470 --> 00:38:41.830
Use low cost handhelds.

00:38:41.830 --> 00:38:43.890
So there's even
now companies that

00:38:43.890 --> 00:38:46.710
are starting to try to automate
acquisition of the data

00:38:46.710 --> 00:38:48.780
by helping people
collect it and guide them

00:38:48.780 --> 00:38:50.700
to collecting the right views.

00:38:50.700 --> 00:38:53.280
Early in the disease course,
no real symptoms here.

00:38:53.280 --> 00:38:55.350
Decision support just
around whether you

00:38:55.350 --> 00:38:58.040
should start some meds or
intensify them, low liability,

00:38:58.040 --> 00:38:58.843
low cost.

00:38:58.843 --> 00:39:00.260
So this is a place
where we wanted

00:39:00.260 --> 00:39:02.093
to focus in terms of
being able to introduce

00:39:02.093 --> 00:39:05.460
some kind of innovations
in this space.

00:39:05.460 --> 00:39:05.960
OK.

00:39:05.960 --> 00:39:07.760
So this comes back
to this slide of I

00:39:07.760 --> 00:39:10.190
talked about where you could
imagine some of these things

00:39:10.190 --> 00:39:12.680
being low hanging fruit, but
maybe those aren't the ones

00:39:12.680 --> 00:39:14.638
that we should be focusing
on we should instead

00:39:14.638 --> 00:39:19.100
be focusing on enabling
more data at low cost,

00:39:19.100 --> 00:39:21.870
getting more out of the
data that we're collecting,

00:39:21.870 --> 00:39:24.120
and helping people even
acquire it in the first place.

00:39:24.120 --> 00:39:26.287
So that's one category of
things, and that's the one

00:39:26.287 --> 00:39:28.105
I just highlighted in
the previous slide.

00:39:28.105 --> 00:39:29.480
You can imagine
something running

00:39:29.480 --> 00:39:31.380
in the background at a
hospital system level

00:39:31.380 --> 00:39:33.380
and just checking to see
whether there's anybody

00:39:33.380 --> 00:39:34.753
who was missed in some ways.

00:39:34.753 --> 00:39:37.170
And then triage I'm going to
talk about in the next slide.

00:39:37.170 --> 00:39:39.140
I'll come back to that,
and then really-- and this

00:39:39.140 --> 00:39:40.280
is, again, one of
the reasons I got

00:39:40.280 --> 00:39:41.947
into this-- we want
to do something that

00:39:41.947 --> 00:39:44.000
elevates practice beyond
just simply repeating

00:39:44.000 --> 00:39:45.540
what we already do.

00:39:45.540 --> 00:39:48.230
And so this idea of
quantitative tracking

00:39:48.230 --> 00:39:50.147
of intermediate states,
subclasses of disease,

00:39:50.147 --> 00:39:52.438
which is actually the real
reason I got into this space

00:39:52.438 --> 00:39:54.410
is because I wanted to
increase scale of data

00:39:54.410 --> 00:39:57.412
to be able to do this, and
this is where you potentially

00:39:57.412 --> 00:39:58.120
would like to go.

00:39:58.120 --> 00:40:00.470
So the ECG example is
an interesting one,

00:40:00.470 --> 00:40:02.810
because automated systems
for ECG interpretation

00:40:02.810 --> 00:40:05.240
have been around
for 40 or 50 years,

00:40:05.240 --> 00:40:11.030
and they really got going
around the early 2000s, when

00:40:11.030 --> 00:40:13.590
people realized--

00:40:13.590 --> 00:40:15.568
there's a pattern
called an ST elevation.

00:40:15.568 --> 00:40:17.360
I'm not sure if you
guys talked about that.

00:40:17.360 --> 00:40:20.225
This is a marker of
complete stoppage

00:40:20.225 --> 00:40:21.350
of blood flow to the heart.

00:40:21.350 --> 00:40:24.080
So muscle starts to die.

00:40:24.080 --> 00:40:28.880
And then the early 2000s, there
was a quality movement that

00:40:28.880 --> 00:40:31.190
said, as soon as
anybody sees that, you

00:40:31.190 --> 00:40:33.200
should get to somebody
doing something

00:40:33.200 --> 00:40:35.700
about it within an
hour and a half or so.

00:40:35.700 --> 00:40:38.180
And so the problem was that in
the old days and the old way

00:40:38.180 --> 00:40:39.020
to do this--

00:40:39.020 --> 00:40:40.395
and even this was
around the time

00:40:40.395 --> 00:40:43.460
I was a resident--
you would have

00:40:43.460 --> 00:40:45.440
to first call the cardiologist.

00:40:45.440 --> 00:40:46.213
Wake him up.

00:40:46.213 --> 00:40:46.880
They would come.

00:40:46.880 --> 00:40:47.963
You'd send them the image.

00:40:47.963 --> 00:40:49.060
They would look at it.

00:40:49.060 --> 00:40:50.180
Then, they would
decide whether or not

00:40:50.180 --> 00:40:51.830
this was the pattern
they were seeing,

00:40:51.830 --> 00:40:54.140
and then they would activate
the lab, the cath lab.

00:40:54.140 --> 00:40:56.870
They would come in, and you
were losing about an hour, hour

00:40:56.870 --> 00:40:58.290
and a half in this process.

00:40:58.290 --> 00:41:01.670
And so instead they decided
that automated systems could

00:41:01.670 --> 00:41:05.270
be used to be able to
enable ambulance personnel

00:41:05.270 --> 00:41:07.550
or emergency room docs,
so non-cardiologists,

00:41:07.550 --> 00:41:09.042
to be able to say,
hey, look, this

00:41:09.042 --> 00:41:10.250
is what we think is going on.

00:41:10.250 --> 00:41:12.890
Let's bring the team in, and
so people would get mobilized.

00:41:12.890 --> 00:41:14.390
People would come
to the hospital.

00:41:14.390 --> 00:41:17.480
Nobody would do anything in
terms of starting the case,

00:41:17.480 --> 00:41:20.598
until somebody confirmed it,
but already, the whole wheels

00:41:20.598 --> 00:41:21.140
were turning.

00:41:21.140 --> 00:41:22.730
And so you have
this triage system,

00:41:22.730 --> 00:41:24.212
where you're making a decision.

00:41:24.212 --> 00:41:25.670
You're not finalizing
the decision,

00:41:25.670 --> 00:41:26.850
but you're speeding things up.

00:41:26.850 --> 00:41:28.040
And so this is an
example where you

00:41:28.040 --> 00:41:29.498
could imagine it's
important to try

00:41:29.498 --> 00:41:31.070
to offload this to something.

00:41:31.070 --> 00:41:33.363
So this is an
example, and there's

00:41:33.363 --> 00:41:34.530
going to be false positives.

00:41:34.530 --> 00:41:36.950
And people will laugh and mock
the emergency room doctors

00:41:36.950 --> 00:41:38.972
and mock the ambulance
drivers and say, ah,

00:41:38.972 --> 00:41:40.430
they don't know
what they're doing.

00:41:40.430 --> 00:41:41.480
They don't have any experience.

00:41:41.480 --> 00:41:43.130
But ultimately,
people were dying,

00:41:43.130 --> 00:41:45.047
because they were waiting
for the cardiologist

00:41:45.047 --> 00:41:47.150
to be available to read the ECG.

00:41:47.150 --> 00:41:50.390
So you've got to think about
those in terms of places

00:41:50.390 --> 00:41:51.930
where there may
be cost for delay.

00:41:51.930 --> 00:41:52.430
OK.

00:41:52.430 --> 00:41:54.420
So coming back to echoes.

00:41:54.420 --> 00:41:54.920
OK.

00:41:54.920 --> 00:41:56.253
So why does an echo get studied?

00:41:56.253 --> 00:42:00.110
Because this is probably not
something that is typical.

00:42:00.110 --> 00:42:04.310
It's a compilation
of videos, and there

00:42:04.310 --> 00:42:06.680
are about 70 different videos
typically in the studies

00:42:06.680 --> 00:42:08.690
that we do at the
centers that we're at.

00:42:08.690 --> 00:42:10.400
And they're taken
over multiple cycles

00:42:10.400 --> 00:42:12.860
and multiple different
views, and often it

00:42:12.860 --> 00:42:15.382
takes somebody pretty skilled
to acquire those views.

00:42:15.382 --> 00:42:17.090
And they take about
45 minutes to an hour

00:42:17.090 --> 00:42:19.820
to gather that data,
multiple different views,

00:42:19.820 --> 00:42:22.010
and the stenographer
is changing the depth

00:42:22.010 --> 00:42:24.050
to zoom in on given structures.

00:42:24.050 --> 00:42:26.090
And so you can understand
that there's already

00:42:26.090 --> 00:42:27.830
somebody who was
already very experienced

00:42:27.830 --> 00:42:30.350
in this process even collecting
the data which is a problem.

00:42:30.350 --> 00:42:32.600
Because you need to take
them out of the picture,

00:42:32.600 --> 00:42:35.490
because they're expensive to
be able to do those things.

00:42:35.490 --> 00:42:38.170
So we were doing at
UCSF 12,000 to 50,000.

00:42:38.170 --> 00:42:41.540
Brigham was probably a little
busier at 30,000 to 35,000.

00:42:41.540 --> 00:42:44.300
Medicare back, in 2011, had
seven million of these perform,

00:42:44.300 --> 00:42:47.630
and there's probably hundreds
of millions of these archives,

00:42:47.630 --> 00:42:50.420
so lots of data.

00:42:50.420 --> 00:42:54.050
So we published
a paper last year

00:42:54.050 --> 00:42:57.440
trying to automate really all of
the main processes around this,

00:42:57.440 --> 00:43:01.130
and part of the reason to do all
is it doesn't help you to have

00:43:01.130 --> 00:43:02.540
one little bit automated.

00:43:02.540 --> 00:43:04.550
Because at the end
of the day, if you

00:43:04.550 --> 00:43:06.140
have to have a
cardiologist doing

00:43:06.140 --> 00:43:07.220
everything else
and a stenographer

00:43:07.220 --> 00:43:09.262
doing everything else,
what have you really saved

00:43:09.262 --> 00:43:10.900
by having one little step?

00:43:10.900 --> 00:43:14.030
So the goal here was to start
from raw study, coming straight

00:43:14.030 --> 00:43:16.410
off the machine, and
try to do everything.

00:43:16.410 --> 00:43:18.110
And so that involves
sorting through all

00:43:18.110 --> 00:43:20.660
these different views, coming
up with empirical quality score

00:43:20.660 --> 00:43:25.100
with it, segmenting all the
five primary views that we use.

00:43:25.100 --> 00:43:27.080
Directly detecting
some diseases,

00:43:27.080 --> 00:43:29.032
and then computing
all the standard mass

00:43:29.032 --> 00:43:31.240
and volume types of measurements
that come from this.

00:43:31.240 --> 00:43:34.070
So we wanted to do it all,
and this was, I think,

00:43:34.070 --> 00:43:37.880
it wasn't strikingly original in
the algorithms that were used.

00:43:37.880 --> 00:43:40.358
But at the same time, it
was very bold for anybody

00:43:40.358 --> 00:43:42.650
in the community to try to
take this on, and of course,

00:43:42.650 --> 00:43:44.983
in general, all the backlash
you could imagine when, you

00:43:44.983 --> 00:43:46.460
try to do something like this.

00:43:46.460 --> 00:43:49.712
I still hear it, but
there's excitement.

00:43:49.712 --> 00:43:51.170
And certainly on
the industry side,

00:43:51.170 --> 00:43:54.110
there's really excitement
in that this is feasible.

00:43:54.110 --> 00:44:01.020
So I was running biology
lab, back in 2016 or so,

00:44:01.020 --> 00:44:02.130
and then decided--

00:44:02.130 --> 00:44:06.720
so my cousin's husband is the
Dean of Engineering at Penn,

00:44:06.720 --> 00:44:09.270
and I emailed him and said, do
you know anyone at Berkeley?

00:44:09.270 --> 00:44:10.112
I live near there.

00:44:10.112 --> 00:44:12.570
I have a very long commute,
and I was like closer to there.

00:44:12.570 --> 00:44:13.653
Is anybody you know there?

00:44:13.653 --> 00:44:14.760
So he's like, yeah.

00:44:14.760 --> 00:44:16.410
I know Ruzena Bajcsy there.

00:44:16.410 --> 00:44:18.840
She used to be a Penn,
and I know Alyosha Efros.

00:44:18.840 --> 00:44:21.970
And so he just emailed them
and said, can you meet?

00:44:21.970 --> 00:44:22.998
[INAUDIBLE]

00:44:22.998 --> 00:44:24.540
And so I met some
of them, and then I

00:44:24.540 --> 00:44:26.260
tried to find some people
who were willing to work.

00:44:26.260 --> 00:44:28.950
So I just spent a day a week
there for about two years,

00:44:28.950 --> 00:44:30.720
just hanging out,
writing, code and try

00:44:30.720 --> 00:44:32.530
to get this project
off the ground.

00:44:32.530 --> 00:44:34.910
So we have a few
different institutions.

00:44:34.910 --> 00:44:37.260
Jeff Zhang was a senior
undergraduate at the time.

00:44:37.260 --> 00:44:40.020
He's at Illinois right
now as a graduate student.

00:44:40.020 --> 00:44:43.170
It's interesting, because it's
hard to get grad student level

00:44:43.170 --> 00:44:45.630
people excited over
stuff that's applications

00:44:45.630 --> 00:44:51.080
of existing algorithms, but
they're happy to advise.

00:44:51.080 --> 00:44:54.070
So I ended up having to write
a lot of the code myself.

00:44:54.070 --> 00:44:55.572
And undergraduates
are, of course,

00:44:55.572 --> 00:44:57.030
excited to do these
kind of things,

00:44:57.030 --> 00:44:59.920
because it's better than
homework, and I can pay.

00:44:59.920 --> 00:45:01.920
But I think, ultimately,
it's interesting to try

00:45:01.920 --> 00:45:05.940
to find that sweet spot and
also find things that ultimately

00:45:05.940 --> 00:45:09.627
could be interesting from an
algorithmic standpoint too.

00:45:09.627 --> 00:45:11.460
So I'm trying to do
more of that these days.

00:45:11.460 --> 00:45:13.230
OK.

00:45:13.230 --> 00:45:15.330
So we aren't the first
to even do something

00:45:15.330 --> 00:45:17.030
around classifying views.

00:45:17.030 --> 00:45:18.780
So somebody already
had publish something,

00:45:18.780 --> 00:45:20.860
but we wanted to be a little
bit more nuanced than that.

00:45:20.860 --> 00:45:22.693
In that we wanted to
be able to distinguish,

00:45:22.693 --> 00:45:25.590
for example, whether this
structure, the left ventricle,

00:45:25.590 --> 00:45:26.220
is cut off.

00:45:26.220 --> 00:45:28.387
Because we don't want to
measure it if it's cut off,

00:45:28.387 --> 00:45:30.803
and we don't want to measure
the atrium if it's completely

00:45:30.803 --> 00:45:31.380
cut off here.

00:45:31.380 --> 00:45:33.390
So we wanted to be able
to have a classifier

00:45:33.390 --> 00:45:35.432
able to distinguish between
some of those things.

00:45:35.432 --> 00:45:37.740
It's not an easy task,
and a lot of these labels

00:45:37.740 --> 00:45:41.400
were me riding the train in
my very long commute from East

00:45:41.400 --> 00:45:44.040
Bay, in California, to UCSF.

00:45:44.040 --> 00:45:47.340
And so I did a lot of labeling,
and I did a lot of segmentation

00:45:47.340 --> 00:45:47.840
too.

00:45:47.840 --> 00:45:49.175
So I could fly a lot.

00:45:49.175 --> 00:45:50.550
And that's the
other thing that's

00:45:50.550 --> 00:45:52.512
kind of interesting is
that you often need--

00:45:52.512 --> 00:45:53.970
even to do the
grunt work-- you may

00:45:53.970 --> 00:45:56.890
need somebody fairly specialized
to do it which is OK, but yeah,

00:45:56.890 --> 00:45:58.723
so that ended up being
me for a lot of this.

00:45:58.723 --> 00:46:01.390
So I traced a lot
of these images,

00:46:01.390 --> 00:46:03.400
and then I got some
other people to help out.

00:46:03.400 --> 00:46:05.900
But you're not going to get a
computer science undergraduate

00:46:05.900 --> 00:46:08.448
to trace art structures
for you, nor are you

00:46:08.448 --> 00:46:10.240
going to get them
excited about doing this.

00:46:10.240 --> 00:46:11.760
So we didn't end up
having that much data,

00:46:11.760 --> 00:46:13.885
and I think we could probably
get better than that.

00:46:13.885 --> 00:46:17.190
But we had the five main views,
and we implemented a modified

00:46:17.190 --> 00:46:18.600
version of unit algorithm.

00:46:18.600 --> 00:46:21.060
We imposed a bit of
a penalty to keep

00:46:21.060 --> 00:46:24.480
this problem of, for example,
a little stray ventricle

00:46:24.480 --> 00:46:25.500
being out there.

00:46:25.500 --> 00:46:27.420
We imposed a penalty
to say, well,

00:46:27.420 --> 00:46:30.022
if that's too far away
from the center then,

00:46:30.022 --> 00:46:31.980
we're going to have the
loss function take that

00:46:31.980 --> 00:46:32.670
into account.

00:46:32.670 --> 00:46:37.130
That helped somewhat, but so
that was our approach to--

00:46:37.130 --> 00:46:38.587
this is a pretty
substantial deal

00:46:38.587 --> 00:46:40.170
to be able to do all
these things that

00:46:40.170 --> 00:46:42.120
normally would be very tedious.

00:46:42.120 --> 00:46:44.310
And as a result, when we
start to analyze things,

00:46:44.310 --> 00:46:47.340
we can segment every single
frame of every single video.

00:46:47.340 --> 00:46:49.900
The typical echo reader will
take two frames and trace them.

00:46:49.900 --> 00:46:50.400
That's it.

00:46:50.400 --> 00:46:51.330
That's all you get.

00:46:51.330 --> 00:46:53.910
So we can do everything over
every single cardiac cycle,

00:46:53.910 --> 00:46:56.290
because there's amazing
variability from beat to beat.

00:46:56.290 --> 00:46:58.560
And so it's silly
to think that that

00:46:58.560 --> 00:47:02.020
should be the gold standard,
but that is the gold standard.

00:47:02.020 --> 00:47:03.837
So we had thousands of echoes.

00:47:03.837 --> 00:47:04.920
So that's the other thing.

00:47:04.920 --> 00:47:07.740
So it turns out that it's
almost impossible to get access

00:47:07.740 --> 00:47:09.930
to echoes, so I wrote a
keystroke encoder that

00:47:09.930 --> 00:47:13.440
sat at the front end and just
mimicked me entering in studies

00:47:13.440 --> 00:47:14.320
and downloading them.

00:47:14.320 --> 00:47:15.862
So that was the only
way I could get.

00:47:15.862 --> 00:47:18.370
So I had about 30,000
studies built up over a year,

00:47:18.370 --> 00:47:21.310
but there's no way
to do bulk download.

00:47:21.310 --> 00:47:23.730
And so again, you've got
to do some grunt work to be

00:47:23.730 --> 00:47:25.570
willing to play this space.

00:47:25.570 --> 00:47:29.495
So we had a fair
number of studies

00:47:29.495 --> 00:47:30.870
we could use in
terms of where we

00:47:30.870 --> 00:47:35.010
had measurements and decent
values in terms of that.

00:47:35.010 --> 00:47:36.570
I think it's
interesting in terms

00:47:36.570 --> 00:47:39.415
of thinking about how good one
can-- how close one can get.

00:47:39.415 --> 00:47:41.040
And one of the things
we found is that,

00:47:41.040 --> 00:47:43.890
when there were big deviations--
these are Bland-Altman plots--

00:47:43.890 --> 00:47:46.170
almost always the
manual ones were wrong.

00:47:46.170 --> 00:47:47.200
AUDIENCE: Why is that?

00:47:47.200 --> 00:47:48.090
RAHUL DEO: Oh, OK.

00:47:48.090 --> 00:47:48.590
OK.

00:47:48.590 --> 00:47:53.340
So Bland-Altman plots, so people
don't like using correlations

00:47:53.340 --> 00:47:54.185
in the medical--

00:47:54.185 --> 00:47:56.310
so Bland and Altman published
a paper in the Lancet

00:47:56.310 --> 00:47:59.190
about 30 years ago
complaining that correlations

00:47:59.190 --> 00:48:01.830
and correlation coefficient are
ultimately not good metrics.

00:48:01.830 --> 00:48:03.940
Because you could have
some substantial bias,

00:48:03.940 --> 00:48:06.480
and really you want to know,
if this is the gold standard,

00:48:06.480 --> 00:48:08.160
you need to get that value.

00:48:08.160 --> 00:48:11.490
So it really is just
looking at differences

00:48:11.490 --> 00:48:15.540
between, let's say, the
reference value and the,

00:48:15.540 --> 00:48:18.360
let's say, automated
value, and then

00:48:18.360 --> 00:48:20.758
plotting that against
the mean of the two.

00:48:20.758 --> 00:48:21.300
So that's it.

00:48:21.300 --> 00:48:23.850
I did it as percentages here,
but ultimately, it's just that.

00:48:23.850 --> 00:48:27.330
It's that you're just
taking the mean of,

00:48:27.330 --> 00:48:29.790
let's, say the left
ventricular volume.

00:48:29.790 --> 00:48:33.450
You have a mean of the automated
versus the manually measured

00:48:33.450 --> 00:48:36.318
one, and then you compare
what the difference is

00:48:36.318 --> 00:48:37.860
of one minus the
other, and so you'll

00:48:37.860 --> 00:48:39.150
be on one side or the other.

00:48:39.150 --> 00:48:41.692
So ideally, you would just be
sitting perfectly on this line,

00:48:41.692 --> 00:48:43.233
and then you're
going to look and see

00:48:43.233 --> 00:48:45.760
whether or not you're clustered
on one side or the other.

00:48:45.760 --> 00:48:48.060
So that's just
the typical thing.

00:48:48.060 --> 00:48:50.280
People try to avoid
correlation coefficients,

00:48:50.280 --> 00:48:52.890
because they kind of consider
them to be not really telling

00:48:52.890 --> 00:48:53.970
you whether or not--

00:48:53.970 --> 00:48:55.970
there really is a gold
standard, and there truly

00:48:55.970 --> 00:48:59.450
is a value here, and you
want to be near that value.

00:48:59.450 --> 00:49:04.670
And so that's the
standard for looking

00:49:04.670 --> 00:49:06.980
at comparison of diagnostics.

00:49:06.980 --> 00:49:09.080
So we had about 8,000 things.

00:49:09.080 --> 00:49:11.417
The reviewers gave us a hard
time for the space up here,

00:49:11.417 --> 00:49:13.250
and there are not that
many studies up here,

00:49:13.250 --> 00:49:14.240
but ultimately, there are some.

00:49:14.240 --> 00:49:16.490
And when we manually looked
at a bunch of them, always

00:49:16.490 --> 00:49:17.990
the manual ones were just wrong.

00:49:17.990 --> 00:49:20.360
Either there is a typo
or something like that,

00:49:20.360 --> 00:49:23.657
so that was reassuring, but
we were sometimes very wrong.

00:49:23.657 --> 00:49:25.490
And you'd find that the
places we'd be wrong

00:49:25.490 --> 00:49:28.640
would be these ridiculously
complex congenital heart

00:49:28.640 --> 00:49:32.390
studies that we had never
been given examples like that

00:49:32.390 --> 00:49:33.360
before.

00:49:33.360 --> 00:49:36.190
So that's a lesson to be learned
is that, sometimes, you're

00:49:36.190 --> 00:49:39.382
going to be really off in
these sorts of approaches,

00:49:39.382 --> 00:49:40.840
and you have to
think a little bit.

00:49:40.840 --> 00:49:41.990
And what we ended
up doing is having

00:49:41.990 --> 00:49:44.448
an interative cycle, where we
would identify those and feed

00:49:44.448 --> 00:49:46.550
them back and of
keep on doing that,

00:49:46.550 --> 00:49:49.750
but that still needs
to be improved upon.

00:49:49.750 --> 00:49:50.300
OK.

00:49:50.300 --> 00:49:55.550
So function, again, there's, a
couple of measures a function.

00:49:55.550 --> 00:49:57.050
There's a company
that has something

00:49:57.050 --> 00:49:58.970
out there in this
space, got FDA approved

00:49:58.970 --> 00:50:00.720
for having an automated
ejection fraction.

00:50:00.720 --> 00:50:02.780
So I think we're better
than their numbers,

00:50:02.780 --> 00:50:04.405
overall, but yeah.

00:50:04.405 --> 00:50:06.530
I think that that's just
one of those things you're

00:50:06.530 --> 00:50:09.090
expected to be able to do.

00:50:09.090 --> 00:50:11.500
And then here's a
problem that we run into.

00:50:11.500 --> 00:50:15.330
So we're comparing to the
status quo which, like I said,

00:50:15.330 --> 00:50:18.220
is one person tracing two
images and comparing them.

00:50:18.220 --> 00:50:19.340
That's it.

00:50:19.340 --> 00:50:24.290
So we're processing potentially
200, 300 different frames

00:50:24.290 --> 00:50:29.210
per study and competing
median, smoothing across.

00:50:29.210 --> 00:50:31.520
We're doing a whole
lot more than that.

00:50:31.520 --> 00:50:34.820
So what do we do about that
in terms of the gold standard?

00:50:34.820 --> 00:50:37.350
And if you just take into
observer variability,

00:50:37.350 --> 00:50:39.080
you're going to
have up to 8% to 9%

00:50:39.080 --> 00:50:41.540
in absolute compared to
60% of the reference.

00:50:41.540 --> 00:50:43.470
So that's horrible.

00:50:43.470 --> 00:50:44.890
So what are you supposed to do?

00:50:44.890 --> 00:50:46.460
And I think so one
thing people do

00:50:46.460 --> 00:50:48.963
is they take multiple readers
and ask them to do that.

00:50:48.963 --> 00:50:50.380
But this is like,
are you're going

00:50:50.380 --> 00:50:51.922
to get a bunch of
cardiologists to do

00:50:51.922 --> 00:50:54.040
like 1,000 studies for you?

00:50:54.040 --> 00:50:57.470
It's very hard to imagine
somebody doing that.

00:50:57.470 --> 00:50:59.210
You could compare it
to another modality.

00:50:59.210 --> 00:51:01.020
So we haven't done this yet,
but you could, for example,

00:51:01.020 --> 00:51:02.840
compare it to MRI and
say whether or not

00:51:02.840 --> 00:51:05.420
you're more consistent
with another modality.

00:51:05.420 --> 00:51:07.100
And then this is
indirect, but you

00:51:07.100 --> 00:51:09.038
can go to like
outcomes in a trial

00:51:09.038 --> 00:51:10.830
and see whether or not
you do a better job.

00:51:10.830 --> 00:51:12.493
So there are things you can do.

00:51:12.493 --> 00:51:13.910
One of the things
we decided to do

00:51:13.910 --> 00:51:16.880
is look for correlations
of structures

00:51:16.880 --> 00:51:22.250
within a study itself
and say, well, the mass--

00:51:22.250 --> 00:51:24.830
so we know that, for
example, thickened hearts

00:51:24.830 --> 00:51:26.583
lead to larger
increases of pressure

00:51:26.583 --> 00:51:27.750
and left atrial enlargement.

00:51:27.750 --> 00:51:29.630
So we can look for correlations
between those things

00:51:29.630 --> 00:51:31.270
and see whether we
do a better job.

00:51:31.270 --> 00:51:33.920
I'd say, for, the most
part we're about on par

00:51:33.920 --> 00:51:35.200
with everything that's there.

00:51:35.200 --> 00:51:36.617
So I don't think
we're any better.

00:51:36.617 --> 00:51:37.575
Sometimes we're better.

00:51:37.575 --> 00:51:38.580
Sometimes we're worse.

00:51:38.580 --> 00:51:40.400
And I think, for the most
part, this was another way

00:51:40.400 --> 00:51:42.740
to try to get at this, because
we were stuck with this.

00:51:42.740 --> 00:51:44.900
How do you work
with a gold standard

00:51:44.900 --> 00:51:46.790
that ultimately I don't
think anybody really

00:51:46.790 --> 00:51:49.130
trusts as a gold standard?

00:51:49.130 --> 00:51:52.970
And this is a problem that
just has to keep on coming up.

00:51:52.970 --> 00:51:54.560
This is just an
example of where you

00:51:54.560 --> 00:51:58.070
could facilitate this idea
of low cost serial imaging

00:51:58.070 --> 00:51:58.860
and point of care.

00:51:58.860 --> 00:52:01.910
So these are patients who
are getting chemotherapy,

00:52:01.910 --> 00:52:05.300
and so so Herceptin-- not
herception, Herceptin,

00:52:05.300 --> 00:52:06.365
it's like inception--

00:52:10.340 --> 00:52:13.160
is an EGFR inhibitor that
causes cardiac toxicity,

00:52:13.160 --> 00:52:15.170
and so people are
getting screening echoes.

00:52:15.170 --> 00:52:17.353
So you could imagine,
if you make it easier

00:52:17.353 --> 00:52:18.770
to acquire and
interpret that, all

00:52:18.770 --> 00:52:20.420
you want to care about is
the function and the size.

00:52:20.420 --> 00:52:21.550
So you can imagine
automating that.

00:52:21.550 --> 00:52:23.420
So we just did this
as proof of concept

00:52:23.420 --> 00:52:26.430
that you could imagine
doing something like this.

00:52:26.430 --> 00:52:29.230
And for the last thing
I want to talk about--

00:52:29.230 --> 00:52:31.010
or sorry, the last
thing in this space--

00:52:31.010 --> 00:52:34.530
is that you could also imagine
directly detecting disease.

00:52:34.530 --> 00:52:37.850
And so you have to say, well,
why is that even worthwhile?

00:52:37.850 --> 00:52:38.389
Yes.

00:52:38.389 --> 00:52:39.389
AUDIENCE: I was curious.

00:52:39.389 --> 00:52:42.049
I guess it's going back
to the idea of if you look

00:52:42.049 --> 00:52:45.375
at blended models
between human groud truth

00:52:45.375 --> 00:52:54.650
and maybe a biological ground
truth, [INAUDIBLE] versus sort

00:52:54.650 --> 00:52:57.500
of what you could get
from an MRI or something--

00:52:57.500 --> 00:53:00.615
or maybe not necessarily an MRI,
but what you were saying based

00:53:00.615 --> 00:53:03.240
on the underlying biology, or if
those two things are generally

00:53:03.240 --> 00:53:03.860
kept separate?

00:53:03.860 --> 00:53:05.720
RAHUL DEO: Yeah.

00:53:05.720 --> 00:53:07.550
These are early days
for a lot of this,

00:53:07.550 --> 00:53:10.040
and I think, anytime you make
anything more complicated,

00:53:10.040 --> 00:53:12.320
then the readers will
give you a hard time,

00:53:12.320 --> 00:53:13.450
but you can imagine that.

00:53:13.450 --> 00:53:15.242
And especially, you
may want to tune things

00:53:15.242 --> 00:53:18.080
to be able to be closer
to something like that.

00:53:18.080 --> 00:53:20.348
So yeah, I think,
unfortunately, people

00:53:20.348 --> 00:53:22.640
are pretty conservative in
terms of how they interpret,

00:53:22.640 --> 00:53:24.890
but it does make some
sense that there's probably

00:53:24.890 --> 00:53:26.060
something that--

00:53:26.060 --> 00:53:29.900
Ideally, you want to be able to
have something that is useful,

00:53:29.900 --> 00:53:33.020
and useful may not be exactly
the same thing as mimicking

00:53:33.020 --> 00:53:34.250
what humans are doing.

00:53:34.250 --> 00:53:35.630
So no, I think it's a good idea.

00:53:35.630 --> 00:53:37.070
And I think that
this is going to be--

00:53:37.070 --> 00:53:39.487
this next wave-- is going to
be thinking a little bit more

00:53:39.487 --> 00:53:41.650
about that in terms
of like how do we

00:53:41.650 --> 00:53:44.150
improve on what's going on
over there, rather than simply

00:53:44.150 --> 00:53:46.940
dragging it back to that?

00:53:46.940 --> 00:53:48.310
OK.

00:53:48.310 --> 00:53:50.028
So there are multiple
rare diseases.

00:53:50.028 --> 00:53:52.070
I use to have a clinic
that would focus on these,

00:53:52.070 --> 00:53:53.653
and they tend to get
missed at centers

00:53:53.653 --> 00:53:54.980
that don't see them that often.

00:53:54.980 --> 00:53:57.140
So one place you
could imagine is

00:53:57.140 --> 00:53:58.940
you can focus on trying
to pick those up,

00:53:58.940 --> 00:54:00.590
and you could imagine, this
could be just surveillance

00:54:00.590 --> 00:54:01.760
running in the background.

00:54:01.760 --> 00:54:06.080
It doesn't have to be kind
of real time identification.

00:54:06.080 --> 00:54:08.510
So there's a few
diseases where it's

00:54:08.510 --> 00:54:10.430
very reasonable to
do these things,

00:54:10.430 --> 00:54:11.490
where it's very obvious.

00:54:11.490 --> 00:54:13.160
So this is a disease called
hypertrophic cardiomyopathy.

00:54:13.160 --> 00:54:14.790
I used to see it in my clinic.

00:54:14.790 --> 00:54:17.660
So abnormally thickened hearts,
leading cause of sudden death

00:54:17.660 --> 00:54:18.560
in young athletes.

00:54:18.560 --> 00:54:24.110
So Reggie Lewis, there's a bunch
of people who've died suddenly

00:54:24.110 --> 00:54:25.970
from this condition.

00:54:25.970 --> 00:54:28.760
Unstable heart rhythm,
sudden death, heart failure,

00:54:28.760 --> 00:54:30.320
it runs in families,
and there are

00:54:30.320 --> 00:54:32.390
things you can do,
if you identified it.

00:54:32.390 --> 00:54:35.180
And so it's actually a fairly
easy task, in the sense

00:54:35.180 --> 00:54:38.130
that it tends to
be quite obvious.

00:54:38.130 --> 00:54:40.460
So we built the classification
model around this,

00:54:40.460 --> 00:54:43.472
and we tried to understand
what it was doing in part.

00:54:43.472 --> 00:54:45.680
And so we tried to do some
of these kind of attention

00:54:45.680 --> 00:54:47.780
or saliency type things, and
they were very unsatisfying,

00:54:47.780 --> 00:54:49.790
in part because I think there's
so many different features

00:54:49.790 --> 00:54:50.895
across the whole image.

00:54:50.895 --> 00:54:52.270
So you're just
getting this blob,

00:54:52.270 --> 00:54:53.870
but I think maybe we just
weren't implementing it

00:54:53.870 --> 00:54:54.370
correctly.

00:54:54.370 --> 00:54:57.740
I'm not really sure, but you
have a left atrium gets bigger.

00:54:57.740 --> 00:54:59.030
The heart gets thicker.

00:54:59.030 --> 00:55:01.790
There's so many changes
across the image.

00:55:01.790 --> 00:55:03.602
It was unsatisfying
in terms of that.

00:55:03.602 --> 00:55:05.060
So we did something
simple and just

00:55:05.060 --> 00:55:06.620
took the output of
the probabilities

00:55:06.620 --> 00:55:08.510
and compared it to
some simple things

00:55:08.510 --> 00:55:10.580
that we actually know
about these things

00:55:10.580 --> 00:55:12.980
and found that there was
some degree of correlation.

00:55:12.980 --> 00:55:16.520
But I would like to make
that a little bit better.

00:55:16.520 --> 00:55:18.620
Cardiac amyloid, a very
popular disease for which

00:55:18.620 --> 00:55:20.128
there are now therapies.

00:55:20.128 --> 00:55:22.670
And so pharma is very interested
in identifying these people,

00:55:22.670 --> 00:55:24.712
and they really get missed
at a pretty high rate.

00:55:24.712 --> 00:55:26.990
So we built another
model for this.

00:55:26.990 --> 00:55:29.420
Usually, we had about
250 or 300 cases

00:55:29.420 --> 00:55:33.652
for each of these things and
maybe a few thousand controls.

00:55:33.652 --> 00:55:35.360
And then this one's
a little interesting.

00:55:35.360 --> 00:55:37.550
This is mitral valve prolapse.

00:55:37.550 --> 00:55:41.830
So this is what a
prolapsing valve looks like.

00:55:41.830 --> 00:55:45.980
If you imagine the plane of the
valve here, it buckles back.

00:55:45.980 --> 00:55:50.425
So it does this,
and that's abnormal,

00:55:50.425 --> 00:55:51.550
and this is a normal valve.

00:55:51.550 --> 00:55:53.520
So you notice, it
doesn't buckle back in.

00:55:53.520 --> 00:55:55.040
So it's a little interesting
in that there's really

00:55:55.040 --> 00:55:57.260
only one part of the cardiac
cycle that would really

00:55:57.260 --> 00:55:59.920
highlight this abnormality,
at least that's the way that--

00:55:59.920 --> 00:56:01.910
so the way that
it's read clinically

00:56:01.910 --> 00:56:04.490
is people wait for this one
part of the cardiac cycle

00:56:04.490 --> 00:56:05.810
where it's buckled back.

00:56:05.810 --> 00:56:07.440
They draw an
imaginary line across,

00:56:07.440 --> 00:56:09.440
and they measure what the
displacement is there,

00:56:09.440 --> 00:56:11.450
and so we built a
reasonable model focusing.

00:56:11.450 --> 00:56:13.013
So we phased these
images and picked

00:56:13.013 --> 00:56:14.930
the part of the cardiac
cycle, those relevant,

00:56:14.930 --> 00:56:16.638
all in an automated
way and built a model

00:56:16.638 --> 00:56:20.990
around that and pretty good, in
terms of being able to do that,

00:56:20.990 --> 00:56:23.700
in terms of being
in detect that.

00:56:23.700 --> 00:56:24.200
Yes.

00:56:24.200 --> 00:56:27.460
AUDIENCE: And so is this model
on images at a certain time?

00:56:27.460 --> 00:56:28.687
Like can you just go back?

00:56:28.687 --> 00:56:30.520
Because obviously, you
weren't doing videos.

00:56:30.520 --> 00:56:31.170
Right?

00:56:31.170 --> 00:56:32.630
RAHUL DEO: Well, so we
would take the whole video.

00:56:32.630 --> 00:56:33.980
We were segmenting it.

00:56:33.980 --> 00:56:36.920
We were phasing it, figuring
out what the part of the--

00:56:36.920 --> 00:56:38.360
when was the end
systole in that,

00:56:38.360 --> 00:56:41.330
and then using those as the--
so using a stack of those

00:56:41.330 --> 00:56:42.377
to be able to classify.

00:56:42.377 --> 00:56:44.210
AUDIENCE: So how do you
know the time point?

00:56:44.210 --> 00:56:45.668
RAHUL DEO: Well,
that's I'm saying.

00:56:45.668 --> 00:56:47.653
So we we're using the
variation in the volumes.

00:56:47.653 --> 00:56:48.710
AUDIENCE: The
segmentation would allow

00:56:48.710 --> 00:56:50.220
you to know the time point.

00:56:50.220 --> 00:56:54.470
RAHUL DEO: Exactly, because so
a typical echo will have an ECG

00:56:54.470 --> 00:56:56.300
to use to gate, but
the handhelds don't.

00:56:56.300 --> 00:56:58.400
So we want to move away
from the things that

00:56:58.400 --> 00:57:01.040
involve the fanciness and
all the bells and whistles.

00:57:01.040 --> 00:57:03.278
We're trying to
use the image alone

00:57:03.278 --> 00:57:04.820
to be able to tell
the cardiac cycle.

00:57:04.820 --> 00:57:06.450
So that's how we did it.

00:57:06.450 --> 00:57:07.870
Yes.

00:57:07.870 --> 00:57:10.280
AUDIENCE: So you
mentioned handhelds.

00:57:10.280 --> 00:57:12.780
With the ultrasounds
[INAUDIBLE],,

00:57:12.780 --> 00:57:14.090
are they different from these?

00:57:14.090 --> 00:57:16.510
RAHUL DEO: They
look pretty similar.

00:57:16.510 --> 00:57:19.340
We got some now, and
they look pretty similar

00:57:19.340 --> 00:57:21.380
in terms of the
quality of the images,

00:57:21.380 --> 00:57:23.810
and you can acquire
the very same view.

00:57:23.810 --> 00:57:27.413
So I think we haven't shown that
we can do it off those, in part

00:57:27.413 --> 00:57:29.330
because there just isn't
enough training data.

00:57:29.330 --> 00:57:32.630
But they look pretty
nice, and I know at UCSF

00:57:32.630 --> 00:57:35.080
and at Brigham, all the
fellows are using it.

00:57:35.080 --> 00:57:38.330
It looks pretty much the same in
terms of the-- the transducers

00:57:38.330 --> 00:57:40.450
are similar, and image
quality is very good.

00:57:40.450 --> 00:57:41.450
Resolution is very good.

00:57:41.450 --> 00:57:43.850
Frame rate probably doesn't
get up as high necessarily,

00:57:43.850 --> 00:57:47.750
but for the most part, I don't
think it's that different.

00:57:47.750 --> 00:57:50.640
So that is the next phase.

00:57:50.640 --> 00:57:51.400
Yes.

00:57:51.400 --> 00:57:52.793
AUDIENCE: Could you comment on--

00:57:52.793 --> 00:57:54.993
so you mentioned how each
of these three examples

00:57:54.993 --> 00:57:56.910
could be used within a
surveillance algorithm.

00:57:56.910 --> 00:57:57.577
RAHUL DEO: Yeah.

00:57:57.577 --> 00:57:59.680
AUDIENCE: Could you
comment on where

00:57:59.680 --> 00:58:02.647
along this true positive,
false positive trade-off

00:58:02.647 --> 00:58:04.480
you would actually be
realistic to use this?

00:58:04.480 --> 00:58:04.880
RAHUL DEO: Yeah.

00:58:04.880 --> 00:58:05.713
That's a good point.

00:58:05.713 --> 00:58:07.880
I think it would vary for
every single one of those,

00:58:07.880 --> 00:58:10.060
and you really want to have
some costs on what the--

00:58:10.060 --> 00:58:14.500
so I would typically err on
the side of higher sensitivity

00:58:14.500 --> 00:58:19.100
and dump it on the
cardiologists to be able to--

00:58:19.100 --> 00:58:23.550
so I would work, but I think
you have to pick some--

00:58:23.550 --> 00:58:25.152
let's say, you're
a product manager.

00:58:25.152 --> 00:58:27.360
AUDIENCE: Just choose one
of these three, and maybe--

00:58:27.360 --> 00:58:28.570
RAHUL DEO: OK.

00:58:28.570 --> 00:58:29.860
Yeah.

00:58:29.860 --> 00:58:32.440
So this is a pretty
rare disease.

00:58:32.440 --> 00:58:36.770
So your priors are pretty low
in terms of these individuals.

00:58:36.770 --> 00:58:39.760
And so I think you
probably would probably

00:58:39.760 --> 00:58:46.330
want to err somewhere
along this area here,

00:58:46.330 --> 00:58:50.110
and so just working
on what the--

00:58:50.110 --> 00:58:53.830
so you probably will still
be a relatively high rate

00:58:53.830 --> 00:58:56.120
of false positives
even that space.

00:58:56.120 --> 00:59:01.810
But I would argue that it would
take the treating cardiologist

00:59:01.810 --> 00:59:04.850
potentially just a few minutes
to look at that study again,

00:59:04.850 --> 00:59:06.850
and if you picked up one
of those patients, that

00:59:06.850 --> 00:59:08.147
would be a big win.

00:59:08.147 --> 00:59:10.480
So I think that the cost
probably wouldn't be that high,

00:59:10.480 --> 00:59:13.290
and you just have
to make the case.

00:59:13.290 --> 00:59:15.840
So therapy for
amyloid, for example,

00:59:15.840 --> 00:59:18.150
this is a nice sharp
up stroke there.

00:59:18.150 --> 00:59:21.552
There's new drugs
out there that are

00:59:21.552 --> 00:59:23.260
sort of begging for
patients, and they're

00:59:23.260 --> 00:59:25.138
having a real hard
time identifying them.

00:59:25.138 --> 00:59:26.680
So you could imagine
again, it's sort

00:59:26.680 --> 00:59:29.830
of a calculus based on
what the benefits would

00:59:29.830 --> 00:59:32.020
be for that identification
and what burden you're

00:59:32.020 --> 00:59:35.460
placing on the individuals to
have to over read something.

00:59:35.460 --> 00:59:37.210
And you could probably
tune that depending

00:59:37.210 --> 00:59:42.070
on what the disease is and
who you're pitching it to.

00:59:42.070 --> 00:59:44.230
But you're right, you're
going to crush people

00:59:44.230 --> 00:59:47.530
if like 1 in 100 ends up
taking a true positive then

00:59:47.530 --> 00:59:50.015
you're not going
to get many fans.

00:59:50.015 --> 00:59:50.515
Yes.

00:59:50.515 --> 00:59:53.930
AUDIENCE: Could you comment
on whether, for example,

00:59:53.930 --> 00:59:56.160
[INAUDIBLE] basis,
the ones that you're

00:59:56.160 --> 00:59:59.670
able to predict very
well at that point

00:59:59.670 --> 01:00:03.470
you just chose what
distinguishes the ones that

01:00:03.470 --> 01:00:04.753
are defined well?

01:00:04.753 --> 01:00:06.170
RAHUL DEO: So
that's a good point,

01:00:06.170 --> 01:00:10.060
and I don't really
know in the sense

01:00:10.060 --> 01:00:11.860
that I haven't
looked that closely.

01:00:11.860 --> 01:00:17.110
But I'm going to guess, they're
very thick and very obvious

01:00:17.110 --> 01:00:18.910
in that sort of sense.

01:00:18.910 --> 01:00:22.295
So we have a ECG model that
may pick this up early.

01:00:22.295 --> 01:00:23.920
What you want is
something to fix it up

01:00:23.920 --> 01:00:26.230
when it's treatable, not
having something that's

01:00:26.230 --> 01:00:27.460
ridiculously exaggerated.

01:00:27.460 --> 01:00:29.800
So you may need multiple
modalities some of which

01:00:29.800 --> 01:00:33.310
are more sensitive than others
that can catch earlier stage

01:00:33.310 --> 01:00:34.827
disease to be able to do that.

01:00:34.827 --> 01:00:36.910
So there are interesting
things about this disease

01:00:36.910 --> 01:00:37.493
in particular.

01:00:37.493 --> 01:00:40.770
So cataracts sometimes
happen before--

01:00:40.770 --> 01:00:43.535
so ideally, the way you do
this is-- and I'm actually

01:00:43.535 --> 01:00:45.160
consulting around
something like this--

01:00:45.160 --> 01:00:49.780
you ideally want a mixture
of electronic health record,

01:00:49.780 --> 01:00:52.870
something from other findings--
mirror findings, eye findings,

01:00:52.870 --> 01:00:54.785
plus maybe something
cardiac plus

01:00:54.785 --> 01:00:56.410
and have something
that ideally catches

01:00:56.410 --> 01:00:58.660
the disease in the ideal
most treated state.

01:00:58.660 --> 01:01:00.130
And maybe echo's
not the best one,

01:01:00.130 --> 01:01:04.070
and I think that we'll come
back to that at the end.

01:01:04.070 --> 01:01:05.350
We have a little bit of time.

01:01:05.350 --> 01:01:05.850
OK.

01:01:08.260 --> 01:01:10.598
So UCSF is filing--

01:01:10.598 --> 01:01:11.140
I don't know.

01:01:11.140 --> 01:01:12.890
I don't think this is
actually patentable,

01:01:12.890 --> 01:01:15.340
but they are filing
for a patent.

01:01:15.340 --> 01:01:18.310
I'm just filling the paperwork
out today in terms of--

01:01:18.310 --> 01:01:19.600
I don't know.

01:01:19.600 --> 01:01:24.730
But my code is all
freely available anyway,

01:01:24.730 --> 01:01:27.078
for academic, non-profit
use, and they're just

01:01:27.078 --> 01:01:28.120
trying to make it better.

01:01:28.120 --> 01:01:31.030
I think, ultimately, my
view as an academic here is

01:01:31.030 --> 01:01:32.770
to try to show what's possible.

01:01:32.770 --> 01:01:35.410
And then, if you want to
get a commercial product,

01:01:35.410 --> 01:01:37.853
then you need people to
weigh in on the industry side

01:01:37.853 --> 01:01:40.270
and make something pretty and
make it usable and all that.

01:01:40.270 --> 01:01:42.010
But I think,
ultimately, I'm trying

01:01:42.010 --> 01:01:44.830
to just show, hey, if we could
do this in a scalable way

01:01:44.830 --> 01:01:46.540
and find out something
new, then you guys

01:01:46.540 --> 01:01:48.430
can catch up and
do something that

01:01:48.430 --> 01:01:51.050
ultimately can be deployed.

01:01:51.050 --> 01:01:53.745
And what's interesting is I have
a collaborator in New Zealand.

01:01:53.745 --> 01:01:55.120
There, they're
are resource poor.

01:01:55.120 --> 01:01:56.948
So they have a huge
backlog of patients.

01:01:56.948 --> 01:01:58.490
They don't have
enough stenographers,

01:01:58.490 --> 01:02:00.198
and they don't have
enough cardiologists.

01:02:00.198 --> 01:02:02.410
So they're trying to
implement this super ultra

01:02:02.410 --> 01:02:06.680
quick five-minute study
and then have automation.

01:02:06.680 --> 01:02:10.857
And so they want our accuracy
to be a little bit better,

01:02:10.857 --> 01:02:12.440
but I think they're
ready to roll out,

01:02:12.440 --> 01:02:15.790
if we're able to get something
that has probably more training

01:02:15.790 --> 01:02:16.290
data.

01:02:16.290 --> 01:02:16.920
Yes.

01:02:16.920 --> 01:02:18.453
Are you from New Zealand?

01:02:18.453 --> 01:02:18.995
AUDIENCE: No.

01:02:18.995 --> 01:02:23.228
I think you started talking
about the trade-off between

01:02:23.228 --> 01:02:24.710
accuracy and--

01:02:24.710 --> 01:02:27.674
so in academia, I get the
sense that they're always

01:02:27.674 --> 01:02:29.173
chasing perfect accuracy.

01:02:29.173 --> 01:02:29.840
RAHUL DEO: Yeah.

01:02:29.840 --> 01:02:31.215
AUDIENCE: But as
you said, you're

01:02:31.215 --> 01:02:35.430
not going to get rid of
cardiologists in the diagnosis.

01:02:35.430 --> 01:02:37.630
So I have a
philosophical question

01:02:37.630 --> 01:02:40.940
of are you chasing
the wrong thing?

01:02:40.940 --> 01:02:45.243
Should we chase
perfect accuracy?

01:02:45.243 --> 01:02:45.910
RAHUL DEO: Yeah.

01:02:45.910 --> 01:02:48.500
So the question is around
what should our goals be?

01:02:51.420 --> 01:02:57.470
So should we be just chasing
after a level of accuracy

01:02:57.470 --> 01:03:00.620
that may be either very,
very difficult to attain?

01:03:00.620 --> 01:03:03.800
And especially, if there's never
a scenario where there'll be

01:03:03.800 --> 01:03:06.680
no clinician involved,
should we instead

01:03:06.680 --> 01:03:08.450
be thinking about
something that gets good

01:03:08.450 --> 01:03:09.590
enough to that next step?

01:03:09.590 --> 01:03:11.215
And I think that's
a really good point.

01:03:15.230 --> 01:03:16.430
And what's interesting is--

01:03:16.430 --> 01:03:18.513
and also it's interesting
from the industry side--

01:03:18.513 --> 01:03:21.260
is the field starts
with the mimicking mode,

01:03:21.260 --> 01:03:24.200
because it's much harder
to change practice.

01:03:24.200 --> 01:03:28.158
It's much easier to just pop
something in and say, hey,

01:03:28.158 --> 01:03:29.950
I know you have to make
these measurements.

01:03:29.950 --> 01:03:32.210
Let me make them for you,
and you could look at them

01:03:32.210 --> 01:03:33.532
and see if you agree.

01:03:33.532 --> 01:03:34.490
So that's what ECGs do.

01:03:34.490 --> 01:03:34.990
Right?

01:03:34.990 --> 01:03:38.100
So nobody these days is
measuring the QR rests width.

01:03:38.100 --> 01:03:38.990
Nobody does that.

01:03:38.990 --> 01:03:39.878
That's just not done.

01:03:39.878 --> 01:03:42.170
If you've got a number that's
absurd, you'll change it.

01:03:42.170 --> 01:03:44.420
But for the most part, you're
like, it's close enough,

01:03:44.420 --> 01:03:46.560
but you almost have
to start with that.

01:03:46.560 --> 01:03:49.590
To do something
that's transformative

01:03:49.590 --> 01:03:51.740
is very hard to do.

01:03:51.740 --> 01:03:53.260
So I think something
that involves--

01:03:53.260 --> 01:03:54.635
and I talked to
David about this.

01:03:54.635 --> 01:03:57.800
It's sort of like the
man-machine interface is

01:03:57.800 --> 01:03:59.780
fascinating to think
about how do we together

01:03:59.780 --> 01:04:01.130
come up with something better?

01:04:01.130 --> 01:04:04.310
But it's just much harder to
get that adopted, because it

01:04:04.310 --> 01:04:07.190
requires buy-in in a way
that's different than just

01:04:07.190 --> 01:04:10.610
you do my work for me, but
more that we come together

01:04:10.610 --> 01:04:12.073
to do something better.

01:04:12.073 --> 01:04:14.240
And I think that's going
to be interesting as to how

01:04:14.240 --> 01:04:16.070
to chip away at that problem.

01:04:20.270 --> 01:04:20.770
OK.

01:04:20.770 --> 01:04:22.490
So a couple of
musings, then I'm going

01:04:22.490 --> 01:04:24.948
to talk a little bit about One
Brave Idea, if we have time,

01:04:24.948 --> 01:04:27.522
or I can stop and take
questions instead,

01:04:27.522 --> 01:04:29.480
because it's a little
bit of a biology venture.

01:04:29.480 --> 01:04:30.020
OK.

01:04:30.020 --> 01:04:33.272
So I do think that we
should really look.

01:04:33.272 --> 01:04:35.480
People give me a hard time
around echo, and I'm like,

01:04:35.480 --> 01:04:37.100
well, ECG's been
around for a long time,

01:04:37.100 --> 01:04:38.308
and there's automation there.

01:04:38.308 --> 01:04:40.100
So let's think about
how it's used there,

01:04:40.100 --> 01:04:41.575
and then see whether or not--

01:04:41.575 --> 01:04:43.200
it's not as outlandish
as people think.

01:04:43.200 --> 01:04:45.117
So I think a lot of these
routine measurements

01:04:45.117 --> 01:04:48.625
are just going to be
done in an automated way.

01:04:48.625 --> 01:04:51.000
Already in our software, you
can put out a little picture

01:04:51.000 --> 01:04:53.060
and overlay the segmentation
on the original image

01:04:53.060 --> 01:04:54.143
and say how good it looks.

01:04:54.143 --> 01:04:54.830
So that's easy.

01:04:54.830 --> 01:04:56.200
So you can do that.

01:04:56.200 --> 01:04:59.540
And then this kind of idea
of point of care automated

01:04:59.540 --> 01:05:01.880
diagnoses can make
some sense around

01:05:01.880 --> 01:05:03.470
some emergency-type situations.

01:05:03.470 --> 01:05:06.380
So maybe you need a
quick check of function.

01:05:06.380 --> 01:05:08.030
Maybe you want to
know if they have

01:05:08.030 --> 01:05:10.447
a lot of fluid around the
heart, and you don't necessarily

01:05:10.447 --> 01:05:11.090
want to wait.

01:05:11.090 --> 01:05:12.465
So those will be
the places where

01:05:12.465 --> 01:05:15.088
there may be some
kind of innovations

01:05:15.088 --> 01:05:16.880
around just getting
something done quickly.

01:05:16.880 --> 01:05:18.380
And then you always
have somebody checking

01:05:18.380 --> 01:05:19.980
in the background,
layer on, a little

01:05:19.980 --> 01:05:21.860
the heart attack
thing I showed you,

01:05:21.860 --> 01:05:23.730
and I think this problem
in echo is there.

01:05:23.730 --> 01:05:26.287
And so if you need
skilled people

01:05:26.287 --> 01:05:28.370
to be able to acquire the
data in the first place,

01:05:28.370 --> 01:05:31.190
you're stuck, because
they can read an echo.

01:05:31.190 --> 01:05:33.720
A really good stenography can
read the whole study for you.

01:05:33.720 --> 01:05:35.750
So if you already have
that person involved

01:05:35.750 --> 01:05:38.090
in the pipeline,
then it's really hard

01:05:38.090 --> 01:05:41.683
to introduce a big advance.

01:05:41.683 --> 01:05:43.850
So you need to figure out
how to take a primary care

01:05:43.850 --> 01:05:46.320
doc off the street, put
a machine in their hand,

01:05:46.320 --> 01:05:48.320
and let them get the image
and then automate all

01:05:48.320 --> 01:05:49.550
the interpretation for them.

01:05:49.550 --> 01:05:52.610
And so until you can task
shift into that space,

01:05:52.610 --> 01:05:55.850
you're stuck with having still
too high a level of skill.

01:05:55.850 --> 01:05:58.170
So there are these companies
that are in the space now,

01:05:58.170 --> 01:06:00.280
and there's a few
that are trying.

01:06:00.280 --> 01:06:03.440
It's easy to imagine, if you
can train a neural network

01:06:03.440 --> 01:06:06.170
to classify a view,
you could get it to--

01:06:06.170 --> 01:06:07.948
this gets to this
idea of registration

01:06:07.948 --> 01:06:10.490
a little bit-- you can recognize
if you're off by 10 degrees,

01:06:10.490 --> 01:06:11.510
or if you need a translation.

01:06:11.510 --> 01:06:13.635
You could just train a
model to be able to do that.

01:06:13.635 --> 01:06:15.830
So I think that's already
happening right now.

01:06:15.830 --> 01:06:19.100
So it's a question as to whether
that will get adopted or not,

01:06:19.100 --> 01:06:20.720
but I think that,
ultimately, if you

01:06:20.720 --> 01:06:24.320
want to get shifting towards
sort of less skilled personnel,

01:06:24.320 --> 01:06:26.460
you need to do
something in that space.

01:06:26.460 --> 01:06:26.960
OK.

01:06:26.960 --> 01:06:28.793
So this is where it
gets a little bit harder

01:06:28.793 --> 01:06:31.940
is to think about how to make
stuff and elevate medicine

01:06:31.940 --> 01:06:34.810
beyond what we're doing.

01:06:34.810 --> 01:06:36.320
And this gets back
to this problem

01:06:36.320 --> 01:06:38.460
I mentioned is, at
the end of the day,

01:06:38.460 --> 01:06:41.990
you can't find
new uses for echo,

01:06:41.990 --> 01:06:43.700
unless the data is
already there for you

01:06:43.700 --> 01:06:45.450
to be able to show
that there's more value

01:06:45.450 --> 01:06:48.110
than there currently is, sort
of this chicken and egg thing.

01:06:48.110 --> 01:06:51.830
So in some sense, what I
hope to introduce in some way

01:06:51.830 --> 01:06:54.180
that we can get much
bigger data sets,

01:06:54.180 --> 01:06:57.310
and they don't have to
be 100 video data sets.

01:06:57.310 --> 01:06:59.160
They can be three
video data sets,

01:06:59.160 --> 01:07:01.005
but we want to be
able to figure out

01:07:01.005 --> 01:07:02.880
how to enable more and
more of these studies.

01:07:02.880 --> 01:07:04.752
So then you can sort
of imagine learning

01:07:04.752 --> 01:07:05.960
many more complicated things.

01:07:05.960 --> 01:07:07.863
You want to track
people over time.

01:07:07.863 --> 01:07:09.530
You want to look at
treatment responses.

01:07:09.530 --> 01:07:11.780
So you've got to look at
where the money is already

01:07:11.780 --> 01:07:13.550
and see who could do this.

01:07:13.550 --> 01:07:15.290
So pharma companies
are interested,

01:07:15.290 --> 01:07:17.930
because they have
these phase II trials.

01:07:17.930 --> 01:07:19.910
They may only have three
months or six months

01:07:19.910 --> 01:07:22.850
to show some benefit
for a drug, and they're

01:07:22.850 --> 01:07:24.770
really interested in
seeing whether there's

01:07:24.770 --> 01:07:26.950
differences after a month,
two months, three months, four

01:07:26.950 --> 01:07:27.450
months.

01:07:27.450 --> 01:07:29.420
So that may be a
place where you get--

01:07:29.420 --> 01:07:31.380
and they're being frugal,
but they have money.

01:07:31.380 --> 01:07:32.797
So you could
imagine, if you could

01:07:32.797 --> 01:07:37.940
introduce this pipeline in there
and just have handheld, simple,

01:07:37.940 --> 01:07:40.850
quick to acquire, far
more frequency, and you

01:07:40.850 --> 01:07:43.620
show a treatment response, and
that's kind of transformative

01:07:43.620 --> 01:07:43.790
then.

01:07:43.790 --> 01:07:44.810
Because then, you
could imagine, that

01:07:44.810 --> 01:07:46.560
can get rolled out in
practice after that.

01:07:46.560 --> 01:07:48.902
So you need somebody to
bankroll this to start with,

01:07:48.902 --> 01:07:51.110
and then you could imagine,
once you have a use case,

01:07:51.110 --> 01:07:53.030
then you could imagine
it getting much more.

01:07:53.030 --> 01:07:54.710
And this idea of
surveillance, you

01:07:54.710 --> 01:07:57.210
could imagine that would be
very doable, that you could just

01:07:57.210 --> 01:07:58.695
have something taking--

01:07:58.695 --> 01:08:01.070
The problem is, you can even
get the data in the archives

01:08:01.070 --> 01:08:02.600
anyway, but let's
say you can get that.

01:08:02.600 --> 01:08:04.820
You could just have this
system looking for amyloid,

01:08:04.820 --> 01:08:06.740
looking for whatever,
and that would be a win

01:08:06.740 --> 01:08:09.200
too is to be able to imagine
doing something like that.

01:08:09.200 --> 01:08:11.720
It's not putting any pressure
on the clinical workflow.

01:08:11.720 --> 01:08:13.107
It's not making
anybody look bad.

01:08:13.107 --> 01:08:15.440
I think, ultimately, it's
trying to just figure out if--

01:08:15.440 --> 01:08:17.609
well, maybe somebody
may be looking bad

01:08:17.609 --> 01:08:19.250
if they miss
something, but yeah.

01:08:19.250 --> 01:08:23.450
I think it is just trying
to identify individuals.

01:08:23.450 --> 01:08:25.910
And so this is an area
I think that's hard,

01:08:25.910 --> 01:08:27.529
and so this kind
of idea, this is

01:08:27.529 --> 01:08:29.779
where I started a little
bit, around this kind of idea

01:08:29.779 --> 01:08:31.880
of this disease
subclassification and risk

01:08:31.880 --> 01:08:32.810
models.

01:08:32.810 --> 01:08:35.939
And so that's like more
sophisticated than anything

01:08:35.939 --> 01:08:36.439
we're doing.

01:08:36.439 --> 01:08:39.040
I think we're pretty crude
at this kind of stuff,

01:08:39.040 --> 01:08:42.260
but one of the
challenges is people just

01:08:42.260 --> 01:08:46.550
aren't interested in new
categories or new risk models,

01:08:46.550 --> 01:08:50.890
if they don't have some way
that they can change practice.

01:08:50.890 --> 01:08:54.319
And that becomes more
difficult, because then you

01:08:54.319 --> 01:08:56.420
need to not only
introduce the model,

01:08:56.420 --> 01:08:58.640
you need to show
how incorporating

01:08:58.640 --> 01:09:01.700
that model in some way is
able to either identify

01:09:01.700 --> 01:09:03.080
people who respond.

01:09:03.080 --> 01:09:04.680
It always comes
down to therapies

01:09:04.680 --> 01:09:05.597
at the end of the day.

01:09:05.597 --> 01:09:08.870
So can you tell me some subclass
of people who will do better

01:09:08.870 --> 01:09:10.670
on this drug, which
means that you

01:09:10.670 --> 01:09:13.399
have to have trial data that
has all those people with all

01:09:13.399 --> 01:09:14.167
that data.

01:09:14.167 --> 01:09:16.250
And unfortunately, because
echoes are so expensive

01:09:16.250 --> 01:09:19.402
and places like the Brigham
charge like $3,000 per echo,

01:09:19.402 --> 01:09:20.819
then you only have
like 100 people

01:09:20.819 --> 01:09:23.111
who have an echo in a trial
or 300 people have an echo.

01:09:23.111 --> 01:09:26.819
You have a 5,000 person trial,
and 5% of them have an echo.

01:09:26.819 --> 01:09:29.700
So you need to change the way
that gets done, because you're

01:09:29.700 --> 01:09:33.270
massively underpowered to be
able to detect anything that's

01:09:33.270 --> 01:09:36.630
sort of a subgroup
within that kind of work.

01:09:36.630 --> 01:09:39.300
So yeah, unfortunately,
the research pace of things

01:09:39.300 --> 01:09:42.510
outpaces the change in
practice in terms of the space,

01:09:42.510 --> 01:09:46.319
until we're able to enable
more data collection.

01:09:46.319 --> 01:09:47.760
So I can stop there.

01:09:47.760 --> 01:09:50.355
I was going to talk about
blood cells in slides.

01:09:50.355 --> 01:09:52.163
PROFESSOR: We can
take some questions.

01:09:52.163 --> 01:09:52.830
RAHUL DEO: Yeah.

01:09:52.830 --> 01:09:53.040
Yeah.

01:09:53.040 --> 01:09:53.250
Yeah.

01:09:53.250 --> 01:09:53.479
OK.

01:09:53.479 --> 01:09:54.354
Why don't we do that.

01:09:57.110 --> 01:09:57.610
Yes.

01:10:00.370 --> 01:10:04.480
AUDIENCE: When CT
reconstruction started,

01:10:04.480 --> 01:10:08.510
I remember seeing some papers
where people said, well,

01:10:08.510 --> 01:10:11.690
we know roughly what to the
anatomy should look like,

01:10:11.690 --> 01:10:14.930
and so we can fill
in missing details.

01:10:14.930 --> 01:10:18.902
In those days, the
slices were run before,

01:10:18.902 --> 01:10:22.073
and so they would hallucinate
what the structure looked like.

01:10:22.073 --> 01:10:22.740
RAHUL DEO: Yeah.

01:10:22.740 --> 01:10:25.730
AUDIENCE: And of course, that
has the benefit of giving you

01:10:25.730 --> 01:10:28.100
a better model, but
it also does risk

01:10:28.100 --> 01:10:30.690
that it's hallucinated data.

01:10:30.690 --> 01:10:34.810
Have you guys tried doing
that with some of the--

01:10:34.810 --> 01:10:35.560
RAHUL DEO: Yeah.

01:10:35.560 --> 01:10:36.500
That's a great point.

01:10:36.500 --> 01:10:37.630
So OK.

01:10:37.630 --> 01:10:40.780
So the question was
so cardiac imaging has

01:10:40.780 --> 01:10:43.920
a very long history, and so
there was a period of time

01:10:43.920 --> 01:10:45.820
where there's these
kind of active modelers

01:10:45.820 --> 01:10:48.370
around morphologies
of the heart.

01:10:48.370 --> 01:10:50.710
And so people had these
models around what

01:10:50.710 --> 01:10:53.480
the heart should look like
from many, many, many studies.

01:10:53.480 --> 01:10:55.480
And they were using that,
back at the time, when

01:10:55.480 --> 01:10:59.560
you had these relatively coarse
multi-slice scanners for a CT,

01:10:59.560 --> 01:11:02.800
they would reconstruct
the 3D image of the heart

01:11:02.800 --> 01:11:06.040
based on some pre-existing
geometric model for what

01:11:06.040 --> 01:11:07.310
the heart should look like.

01:11:07.310 --> 01:11:08.650
And there's, of course,
a benefit to that,

01:11:08.650 --> 01:11:10.317
but some risk in the
sense that somebody

01:11:10.317 --> 01:11:12.555
may be very different in
the space that's missing.

01:11:12.555 --> 01:11:14.680
And so the question is
whether those kind of priors

01:11:14.680 --> 01:11:17.560
can be introduced
in some way, and it

01:11:17.560 --> 01:11:23.290
hasn't been straightforward
as to how to do that.

01:11:23.290 --> 01:11:25.357
Whenever you look at
these ridiculously poor

01:11:25.357 --> 01:11:27.190
segmentations, you're
like, this is idiotic.

01:11:27.190 --> 01:11:29.470
We should be able to
introduce some of that,

01:11:29.470 --> 01:11:33.940
and I've seen people, for
example, put an autoencoder.

01:11:33.940 --> 01:11:35.450
That's not exactly
getting at it,

01:11:35.450 --> 01:11:36.992
but it's actually
getting it somewhat

01:11:36.992 --> 01:11:38.740
with these coarser features.

01:11:38.740 --> 01:11:40.960
But no, I think
in terms of using

01:11:40.960 --> 01:11:43.203
some degree of
geometric priors, I

01:11:43.203 --> 01:11:45.370
think I may have seen some
literature in that space.

01:11:45.370 --> 01:11:46.880
We haven't tried anything there.

01:11:46.880 --> 01:11:49.440
We don't have any data to
do that, unfortunately,

01:11:49.440 --> 01:11:52.090
and I suspect,
yeah, I just don't

01:11:52.090 --> 01:11:53.884
know how difficult that is.

01:11:53.884 --> 01:11:56.104
AUDIENCE: You mentioned
that you don't

01:11:56.104 --> 01:12:01.300
want to see a small additional
atrium off at a distance.

01:12:01.300 --> 01:12:03.113
So that's, in a way,
building in knowledge.

01:12:03.113 --> 01:12:03.780
RAHUL DEO: Yeah.

01:12:03.780 --> 01:12:04.280
No.

01:12:04.280 --> 01:12:06.385
I remember when I was
starting this space.

01:12:06.385 --> 01:12:07.510
I was like this is idiotic.

01:12:07.510 --> 01:12:08.480
Why can't we do this?

01:12:08.480 --> 01:12:10.188
Why don't we have some
way of doing that?

01:12:10.188 --> 01:12:13.270
We couldn't find at that
time any architectures that

01:12:13.270 --> 01:12:16.030
were straightforward
to be able to do that,

01:12:16.030 --> 01:12:20.150
but I'm sure there is
something in that space.

01:12:20.150 --> 01:12:23.200
And we didn't also have the
data for those priors ourselves.

01:12:23.200 --> 01:12:29.400
There's a long history of
these de novo heart modelers

01:12:29.400 --> 01:12:31.715
that exist out there from
Oxford and the New Zealand

01:12:31.715 --> 01:12:33.090
group for that
matter who've been

01:12:33.090 --> 01:12:35.907
doing some of this kind
of multi-scale modeling.

01:12:35.907 --> 01:12:37.740
It will be interesting
to see whether or not

01:12:37.740 --> 01:12:40.440
there is anybody who pushes
forward in that space,

01:12:40.440 --> 01:12:41.650
or is it just more data?

01:12:41.650 --> 01:12:44.250
I think that's
always that tension.

01:12:51.012 --> 01:12:52.950
AUDIENCE: Can I ask
about ultrasounds?

01:12:52.950 --> 01:12:54.300
RAHUL DEO: Yeah.

01:12:54.300 --> 01:12:56.008
AUDIENCE: You didn't
show us ultrasounds.

01:12:56.008 --> 01:12:56.560
Right?

01:12:56.560 --> 01:12:57.220
RAHUL DEO: Yeah, I did.

01:12:57.220 --> 01:12:58.280
AUDIENCE: Oh, you did?

01:12:58.280 --> 01:12:58.420
RAHUL DEO: Yeah.

01:12:58.420 --> 01:12:59.450
The echoes are ultrasounds.

01:12:59.450 --> 01:13:01.830
AUDIENCE: Oh, OK, but that's
really expensive ultrasound.

01:13:01.830 --> 01:13:02.330
Right?

01:13:02.330 --> 01:13:04.193
Like there are
cheaper ultrasounds

01:13:04.193 --> 01:13:06.110
that you could imagine
that you constantly do.

01:13:06.110 --> 01:13:06.610
Right?

01:13:06.610 --> 01:13:07.790
RAHUL DEO: Yeah.

01:13:07.790 --> 01:13:11.210
So there is a company
that just came out

01:13:11.210 --> 01:13:14.210
with the $2,000 handheld
ultrasound, the subscription

01:13:14.210 --> 01:13:15.930
model.

01:13:15.930 --> 01:13:16.430
Yeah.

01:13:16.430 --> 01:13:19.880
So I think that Philips
has a handheld device

01:13:19.880 --> 01:13:24.150
around the $8,000 marker, so
$2,000 is getting quite cheap.

01:13:24.150 --> 01:13:27.650
So that's I think the
space for handheld devices.

01:13:27.650 --> 01:13:29.940
AUDIENCE: We're talking about
resource-poor countries.

01:13:29.940 --> 01:13:30.280
RAHUL DEO: Yeah.

01:13:30.280 --> 01:13:32.405
AUDIENCE: In a developing
country, where maybe they

01:13:32.405 --> 01:13:35.240
have very few doctors per
population kind of thing.

01:13:35.240 --> 01:13:38.130
What kind of imaging
might be useful

01:13:38.130 --> 01:13:41.390
that we could then apply
computer vision algorithms to?

01:13:41.390 --> 01:13:43.940
RAHUL DEO: I think ultrasound
is that sweet spot.

01:13:43.940 --> 01:13:48.250
It has versatility, and
its cost is about where--

01:13:48.250 --> 01:13:50.000
and I'm sure those
companies rented it out

01:13:50.000 --> 01:13:52.590
for much lower cost in
those kinds of places too.

01:13:52.590 --> 01:13:54.840
We're putting together-- or
I put together-- actually,

01:13:54.840 --> 01:13:55.460
it may not have been funded.

01:13:55.460 --> 01:13:56.120
I'm not sure.

01:13:56.120 --> 01:13:59.450
But looking at
sub-Saharan Africa

01:13:59.450 --> 01:14:02.420
and collaborating with
one of the Brigham doctors

01:14:02.420 --> 01:14:05.068
who travels out to
sub-Saharan Africa

01:14:05.068 --> 01:14:07.610
and looking to try to build some
of these automated detection

01:14:07.610 --> 01:14:09.860
type of things in that space.

01:14:09.860 --> 01:14:12.650
So no, I think there is
definite interest in that,

01:14:12.650 --> 01:14:17.450
and then there may be a much
bigger win there then the stuff

01:14:17.450 --> 01:14:18.380
I'm proposing.

01:14:18.380 --> 01:14:20.338
But yeah, no, I think
that's a very good point,

01:14:20.338 --> 01:14:21.218
and that would be--

01:14:21.218 --> 01:14:22.260
it's also, it's portable.

01:14:22.260 --> 01:14:24.260
You could have a
phone-based thing.

01:14:24.260 --> 01:14:29.126
So it's actually very
attractive from that standpoint.

01:14:29.126 --> 01:14:30.043
PROFESSOR: [INAUDIBLE]

01:14:30.043 --> 01:14:30.918
RAHUL DEO: All right.

01:14:30.918 --> 01:14:33.251
I feel like I'm changing the
topic substantially but not

01:14:33.251 --> 01:14:33.751
totally.

01:14:33.751 --> 01:14:34.310
OK.

01:14:34.310 --> 01:14:39.320
So this is that slide I showed,
and I pitched it in a way

01:14:39.320 --> 01:14:41.410
to try to motivate you
to think of ultrasound.

01:14:41.410 --> 01:14:42.930
But I'm not sure
ultrasound really

01:14:42.930 --> 01:14:45.680
achieves all these things, in
the sense I wouldn't call it

01:14:45.680 --> 01:14:48.410
the greatest biological tool
to get at underlying disease

01:14:48.410 --> 01:14:49.850
pathways.

01:14:49.850 --> 01:14:52.070
Some of these things may
be late, like David said,

01:14:52.070 --> 01:14:54.190
or maybe not so reversible.

01:14:54.190 --> 01:14:58.580
So we've been given this One
Brave Idea thing $85 million

01:14:58.580 --> 01:15:02.570
now to make some dent in
a specific disease, so

01:15:02.570 --> 01:15:05.510
coronary artery disease
or coronary heart disease.

01:15:05.510 --> 01:15:07.047
It's that arrogant
tech thing, where

01:15:07.047 --> 01:15:08.630
you just dump a lot
of money somewhere

01:15:08.630 --> 01:15:10.910
and think you're going
to solve all problems.

01:15:10.910 --> 01:15:12.950
And happy to take
it, but I think

01:15:12.950 --> 01:15:14.175
that there are some problems.

01:15:14.175 --> 01:15:15.800
So this is what I
wanted to do, so I've

01:15:15.800 --> 01:15:18.230
wanted to do this for
probably the last five, six

01:15:18.230 --> 01:15:19.880
years, before I
even started here,

01:15:19.880 --> 01:15:23.505
and this has motivated me
in part for quite a while.

01:15:23.505 --> 01:15:24.630
And so here's our problems.

01:15:24.630 --> 01:15:25.000
OK.

01:15:25.000 --> 01:15:27.500
So we're studying heart disease,
so coronary artery disease

01:15:27.500 --> 01:15:30.950
or coronary heart disease is
the arteries in the heart.

01:15:30.950 --> 01:15:32.120
You can't get at those.

01:15:32.120 --> 01:15:33.410
So you can't do any biology.

01:15:33.410 --> 01:15:35.285
You can't do the stuff
the cancer people-- do

01:15:35.285 --> 01:15:36.120
you can biopsy that.

01:15:36.120 --> 01:15:37.575
You can't do anything there.

01:15:37.575 --> 01:15:39.200
So you're stuck with
the thing that you

01:15:39.200 --> 01:15:42.470
want to get at is inaccessible.

01:15:42.470 --> 01:15:45.020
I talked about how a lot of
the imaging is expensive,

01:15:45.020 --> 01:15:48.080
but all those other omic
stuff is really expensive too.

01:15:48.080 --> 01:15:50.980
So that's going to
be not so possible,

01:15:50.980 --> 01:15:54.920
and you're not going to be able
to do serial $1,000 proteomics

01:15:54.920 --> 01:15:55.670
on people either.

01:15:55.670 --> 01:15:57.680
That's not happening
anytime soon.

01:15:57.680 --> 01:16:01.040
And then everything I talked
about, we were woefully

01:16:01.040 --> 01:16:02.690
inadequate in terms
of sample size,

01:16:02.690 --> 01:16:04.730
especially if we
want to characterize

01:16:04.730 --> 01:16:06.833
underlying complex
biological processes.

01:16:06.833 --> 01:16:09.125
So we expect we're going to
need high dimensional data,

01:16:09.125 --> 01:16:10.875
and we're going to
need huge sample sizes.

01:16:10.875 --> 01:16:12.617
There's Vladimir
Vapnik over there.

01:16:12.617 --> 01:16:13.950
And then here's another problem.

01:16:13.950 --> 01:16:14.450
OK?

01:16:14.450 --> 01:16:16.490
So this stuff takes time.

01:16:16.490 --> 01:16:17.720
These diseases take time.

01:16:17.720 --> 01:16:20.085
So if I introduce a
new assay right now,

01:16:20.085 --> 01:16:21.710
how am I going to
show that any of this

01:16:21.710 --> 01:16:22.970
is going to be beneficial?

01:16:22.970 --> 01:16:25.350
Because this disease
develops or 10 to 20 years.

01:16:25.350 --> 01:16:27.517
So I'm not going to talk
about the solution to that,

01:16:27.517 --> 01:16:29.330
well, a little bit.

01:16:29.330 --> 01:16:29.870
OK.

01:16:29.870 --> 01:16:32.630
So one of the issues with
a lot of the data that's

01:16:32.630 --> 01:16:35.010
out there is it's not
particularly expressive.

01:16:35.010 --> 01:16:37.460
It's a lot of that just
the same clinical stuff,

01:16:37.460 --> 01:16:38.690
the same imaging stuff.

01:16:38.690 --> 01:16:42.680
So all these big studies, these
billion dollar big studies,

01:16:42.680 --> 01:16:45.262
ultimately just have
echoes and MRIs and maybe

01:16:45.262 --> 01:16:46.970
a little bit of
genetics, but they really

01:16:46.970 --> 01:16:48.920
don't have stuff
that is this low cost

01:16:48.920 --> 01:16:51.303
expressive biological
stuff that we ideally

01:16:51.303 --> 01:16:52.220
want to be able to do.

01:16:52.220 --> 01:16:55.250
So this is really expensive
and makes $85 million look

01:16:55.250 --> 01:16:57.800
like a joke, and
it's not all that

01:16:57.800 --> 01:16:59.820
rich in terms of complexity.

01:16:59.820 --> 01:17:02.520
So we wanted to do
something different,

01:17:02.520 --> 01:17:05.240
and so this is the crazy thing.

01:17:05.240 --> 01:17:08.340
We're focusing on
circulating cells,

01:17:08.340 --> 01:17:11.270
and so this is a compromise.

01:17:11.270 --> 01:17:12.950
And there's a
reasonably good case

01:17:12.950 --> 01:17:15.250
to be made for
their involvement.

01:17:15.250 --> 01:17:17.270
So there's lots
of data to suggest

01:17:17.270 --> 01:17:19.910
that these are causal mediators
of coronary artery disease

01:17:19.910 --> 01:17:21.270
or coronary heart disease.

01:17:21.270 --> 01:17:24.920
So you can find
them in the plaques.

01:17:24.920 --> 01:17:26.720
So patients who have
autoimmune diseases

01:17:26.720 --> 01:17:29.390
certainly have accelerated
forms after atherosclerosis.

01:17:29.390 --> 01:17:30.175
There are drugs.

01:17:30.175 --> 01:17:31.550
There's a drug
called canakinumab

01:17:31.550 --> 01:17:35.390
that inhibits IL-1 one beta
secretion from macrophages,

01:17:35.390 --> 01:17:38.360
and this has mortality benefit
in coronary artery disease.

01:17:38.360 --> 01:17:40.503
There are mutations in
the white blood cell

01:17:40.503 --> 01:17:42.920
population themselves that are
associated with early heart

01:17:42.920 --> 01:17:43.730
attack.

01:17:43.730 --> 01:17:46.340
So there's a lot there,
and this has been going--

01:17:46.340 --> 01:17:47.840
and there's plenty
of mouse models

01:17:47.840 --> 01:17:49.340
that show that if
you make mutations

01:17:49.340 --> 01:17:51.075
only in the white
blood cell compartment,

01:17:51.075 --> 01:17:53.700
that you will completely change
that the disease course itself.

01:17:53.700 --> 01:17:56.450
So there's a good
amount of data out there

01:17:56.450 --> 01:17:58.940
to suggest that there is an
informative kind of cell type

01:17:58.940 --> 01:17:59.600
there.

01:17:59.600 --> 01:18:01.015
It's accessible.

01:18:01.015 --> 01:18:02.390
There's lots of
predictive models

01:18:02.390 --> 01:18:04.515
already there that could
be done with some of this,

01:18:04.515 --> 01:18:07.010
and they express many of
the genes that are involved.

01:18:07.010 --> 01:18:10.468
And there's a window on many
of these biological processes.

01:18:10.468 --> 01:18:13.010
So we're focusing on computer
vision approaches to this data.

01:18:13.010 --> 01:18:15.050
So we decided, if we
can't do the omic stuff,

01:18:15.050 --> 01:18:16.940
because it costs too
much, we're going

01:18:16.940 --> 01:18:20.240
to take slides and
have tens of thousands

01:18:20.240 --> 01:18:21.850
of cells per individual.

01:18:21.850 --> 01:18:23.600
And then we can introduce
fluorescent dyes

01:18:23.600 --> 01:18:27.350
that can focus on lots
of different organelles.

01:18:27.350 --> 01:18:30.860
And then we can potentially
expand the phenotypic space

01:18:30.860 --> 01:18:32.780
by adding all kinds
of perturbations

01:18:32.780 --> 01:18:35.540
that can be able to
unmask attributes

01:18:35.540 --> 01:18:38.600
of people that may not even be
relatively there at baseline.

01:18:38.600 --> 01:18:41.017
And I think I've been empowered
by the computer vision

01:18:41.017 --> 01:18:43.100
experience with the echo
stuff, and I'm like, hey,

01:18:43.100 --> 01:18:44.310
I can do this.

01:18:44.310 --> 01:18:46.370
I can train these models.

01:18:46.370 --> 01:18:49.790
So we're in a position
now where we can--

01:18:49.790 --> 01:18:52.010
this stuff costs a few
dollars per person.

01:18:52.010 --> 01:18:55.250
It's cheap, and
you can just keep

01:18:55.250 --> 01:18:56.662
on expanding phenotypic space.

01:18:56.662 --> 01:18:57.620
You can bring in drugs.

01:18:57.620 --> 01:18:59.287
You can bring in
whatever you want here,

01:18:59.287 --> 01:19:02.300
and you're still in
that dollars type range.

01:19:02.300 --> 01:19:05.960
So we just piggy-back,
and we just hover around--

01:19:05.960 --> 01:19:07.550
just a couple of
research assistants

01:19:07.550 --> 01:19:09.380
were hovering around clinics.

01:19:09.380 --> 01:19:11.180
And we can do
thousands of patients

01:19:11.180 --> 01:19:13.340
a month, so tens of
thousands of patients a year.

01:19:13.340 --> 01:19:18.410
So we can get into a deep
learning sample size here,

01:19:18.410 --> 01:19:21.710
and so we want
these primary assays

01:19:21.710 --> 01:19:23.570
to be low cost,
reproducible, expressive,

01:19:23.570 --> 01:19:24.830
ideally responsive to therapy.

01:19:24.830 --> 01:19:27.740
So that's this space here,
and there's lots of stuff

01:19:27.740 --> 01:19:28.656
that we have.

01:19:28.656 --> 01:19:31.470
We have all the medical record
data on all these people,

01:19:31.470 --> 01:19:33.810
and we can selectively
do somatic sequencing.

01:19:33.810 --> 01:19:35.130
We can do genome associations.

01:19:35.130 --> 01:19:36.270
We have all ECG data.

01:19:36.270 --> 01:19:38.160
We have selective
positron emission data.

01:19:38.160 --> 01:19:39.960
So it's lots of
additional thought,

01:19:39.960 --> 01:19:42.390
and we want to be able
to walk our cheap assay

01:19:42.390 --> 01:19:45.000
towards those things
are more expensive

01:19:45.000 --> 01:19:47.757
but for which there's
much more historical data.

01:19:47.757 --> 01:19:49.590
So that's what I do
with my life these days,

01:19:49.590 --> 01:19:51.240
and the time problem
has been solved.

01:19:51.240 --> 01:19:54.570
Because we found a collaborary
MGH who has 3 1/2 million

01:19:54.570 --> 01:19:57.870
of these records in terms of
cell counting and cytometer

01:19:57.870 --> 01:19:59.860
data going back for
about three years.

01:19:59.860 --> 01:20:03.390
So we should be able to get
some decent events in that time.

01:20:03.390 --> 01:20:06.072
I need to build a document
classification model for 3 1/2

01:20:06.072 --> 01:20:08.280
million records and decide
whether they have coronary

01:20:08.280 --> 01:20:11.580
heart disease, but sounds
like that's doable.

01:20:11.580 --> 01:20:13.920
We're fearless in this space.

01:20:13.920 --> 01:20:15.960
And then they also
have 13 million images,

01:20:15.960 --> 01:20:18.412
so hundreds of thousands
of people worth of slides.

01:20:18.412 --> 01:20:20.370
So we can at the very
least, get decent weights

01:20:20.370 --> 01:20:22.530
for transfer learning
from some of this data,

01:20:22.530 --> 01:20:25.730
and we're doing this for
acute heart attack patients.

01:20:25.730 --> 01:20:29.140
So yeah, so this is what
I'm doing, ultimately,

01:20:29.140 --> 01:20:32.760
and so it's this bridge between
existing imaging, existing

01:20:32.760 --> 01:20:36.660
conventional medical
data, and this low cost,

01:20:36.660 --> 01:20:39.030
expressive, serial-type
of stuff that ultimately

01:20:39.030 --> 01:20:42.090
hoping to expand phenotypic
space and keep the cost down.

01:20:42.090 --> 01:20:44.670
I think all my lessons from
working with expensive imaging

01:20:44.670 --> 01:20:47.300
data has motivated me to build
something around this space.

01:20:47.300 --> 01:20:50.890
So this is my it's
my baby right now.

01:20:50.890 --> 01:20:53.550
And so lots of things for
people to be involved in,

01:20:53.550 --> 01:20:58.030
if they want to, and these are
some of the funding sources.

01:20:58.030 --> 01:20:58.530
All right.

01:20:58.530 --> 01:20:59.340
Thank you.

01:20:59.340 --> 01:21:02.690
[APPLAUSE]