WEBVTT

00:00:04.500 --> 00:00:08.640
So far, we have used what is
known as internal validation

00:00:08.640 --> 00:00:10.600
to test our model.

00:00:10.600 --> 00:00:14.300
This means that we took the
data from one set of patients

00:00:14.300 --> 00:00:17.990
and split them into a training
set and a testing set.

00:00:17.990 --> 00:00:20.530
While this confirms
that our model is

00:00:20.530 --> 00:00:23.220
good at making predictions
for patients in the Framingham

00:00:23.220 --> 00:00:26.120
Heart Study population,
it's unclear

00:00:26.120 --> 00:00:29.820
if the model generalizes
to other populations.

00:00:29.820 --> 00:00:31.770
The Framingham
cohort of patients

00:00:31.770 --> 00:00:34.600
were white, middle class adults.

00:00:34.600 --> 00:00:36.350
To be sure that
the model extends

00:00:36.350 --> 00:00:38.670
to other types of
patients, we need

00:00:38.670 --> 00:00:41.100
to test on other populations.

00:00:41.100 --> 00:00:43.260
This is known as
external validation.

00:00:45.990 --> 00:00:48.820
There have been many studies
to test the Framingham model

00:00:48.820 --> 00:00:53.990
from the influential 1998
paper on diverse courts.

00:00:53.990 --> 00:00:56.510
This table shows a
sample of studies

00:00:56.510 --> 00:00:58.590
that tested the
model on populations

00:00:58.590 --> 00:01:00.730
with different races.

00:01:00.730 --> 00:01:03.570
The researchers for each
study collected the same risk

00:01:03.570 --> 00:01:06.390
factors used in
the original study,

00:01:06.390 --> 00:01:10.360
predicted CHD using the
Framingham Heart Study model,

00:01:10.360 --> 00:01:12.440
and then analyzed how
accurate the model

00:01:12.440 --> 00:01:15.940
was for that population.

00:01:15.940 --> 00:01:19.700
For some populations, the
Framingham model was accurate.

00:01:19.700 --> 00:01:24.180
For the ARIC study that tested
the model with black men,

00:01:24.180 --> 00:01:28.050
this figure shows a bar graph of
how the Framingham predictions

00:01:28.050 --> 00:01:30.600
compare with the actual results.

00:01:30.600 --> 00:01:32.759
The gray bars are
the predictions.

00:01:32.759 --> 00:01:35.830
And the black bars are
the actual outcomes.

00:01:35.830 --> 00:01:39.840
The patients are sorted on
the x-axis by predicted risk

00:01:39.840 --> 00:01:42.550
and on the y-axis by the
percentage of patients

00:01:42.550 --> 00:01:47.060
in each group who
actually developed CHD.

00:01:47.060 --> 00:01:50.020
For the most part, the
predictions are accurate.

00:01:50.020 --> 00:01:53.660
There's one group for which the
model under-predicted the risk

00:01:53.660 --> 00:01:58.270
and one group for which the
model over-predicted the risk.

00:01:58.270 --> 00:02:01.240
But for other populations,
the Framingham model

00:02:01.240 --> 00:02:02.850
was not as accurate.

00:02:02.850 --> 00:02:06.540
For the HHS study with
Japanese-American men,

00:02:06.540 --> 00:02:08.440
the Framingham
model systematically

00:02:08.440 --> 00:02:10.090
over-predicts a risk of CHD.

00:02:12.720 --> 00:02:15.650
The model can be recalibrated
for this population

00:02:15.650 --> 00:02:18.060
by scaling down the predictions.

00:02:18.060 --> 00:02:20.510
This changes the
predicted risk but not

00:02:20.510 --> 00:02:22.380
the order of the predictions.

00:02:22.380 --> 00:02:25.579
The high risk patients still
have higher predictions

00:02:25.579 --> 00:02:28.150
than the lower risk patients.

00:02:28.150 --> 00:02:31.290
This allows the model to have
more accurate risk estimates

00:02:31.290 --> 00:02:36.130
for populations not included in
the original group of patients.

00:02:36.130 --> 00:02:38.880
For models that will be used
on different populations

00:02:38.880 --> 00:02:41.390
than the one used
to create the model,

00:02:41.390 --> 00:02:44.570
external validation is critical.