WEBVTT 00:00:00.000 --> 00:00:00.620 align:middle line:90% PROFESSOR: OK. 00:00:00.620 --> 00:00:05.670 align:middle line:84% So last time, we spent the hour and a half 00:00:05.670 --> 00:00:13.680 align:middle line:84% talking about classification methods and their use 00:00:13.680 --> 00:00:20.880 align:middle line:84% of genomic technologies in direct and close 00:00:20.880 --> 00:00:23.220 align:middle line:84% to direct, which we called indirect, applications 00:00:23.220 --> 00:00:24.870 align:middle line:90% of clinical medicine. 00:00:24.870 --> 00:00:27.750 align:middle line:84% And we talked about three different types 00:00:27.750 --> 00:00:34.530 align:middle line:84% of class methods again, class exploration, class prediction, 00:00:34.530 --> 00:00:37.260 align:middle line:90% and class discovery. 00:00:37.260 --> 00:00:39.780 align:middle line:84% And I think we finished with a discussion 00:00:39.780 --> 00:00:43.710 align:middle line:84% about class prediction, which is a basic approach that's 00:00:43.710 --> 00:00:46.110 align:middle line:90% used to-- 00:00:46.110 --> 00:00:51.150 align:middle line:84% that starts with microarray data from a certain number 00:00:51.150 --> 00:00:54.000 align:middle line:84% of patients, builds a certain model, 00:00:54.000 --> 00:00:56.790 align:middle line:84% and then tries to predict something. 00:00:56.790 --> 00:00:59.160 align:middle line:84% Everybody who was here kind of remember that, 00:00:59.160 --> 00:01:00.210 align:middle line:90% vaguely at least? 00:01:00.210 --> 00:01:02.910 align:middle line:84% And the basic steps of class prediction 00:01:02.910 --> 00:01:06.720 align:middle line:84% were, first, to choose a gene set, 00:01:06.720 --> 00:01:08.220 align:middle line:84% so choose a number of genes that's 00:01:08.220 --> 00:01:12.570 align:middle line:84% representative of the data and that 00:01:12.570 --> 00:01:17.100 align:middle line:84% divides the data into classes for which you're targeting 00:01:17.100 --> 00:01:21.510 align:middle line:84% your predictions to construct a function, a prediction 00:01:21.510 --> 00:01:25.800 align:middle line:84% function, that will be a mathematical function 00:01:25.800 --> 00:01:30.720 align:middle line:84% that you'll take a new sample with its expression data, 00:01:30.720 --> 00:01:33.150 align:middle line:84% and plug into the function, and get out an answer, 00:01:33.150 --> 00:01:36.450 align:middle line:84% and then determine a rule to try and then 00:01:36.450 --> 00:01:38.850 align:middle line:84% use the rule to classify one way or the other, 00:01:38.850 --> 00:01:42.330 align:middle line:90% and then, lastly, validation. 00:01:42.330 --> 00:01:46.240 align:middle line:84% So I thought what we would do today is go through an example. 00:01:46.240 --> 00:01:48.070 align:middle line:90% So we went through all this. 00:01:48.070 --> 00:01:51.840 align:middle line:84% So I'm not sure we need this anymore. 00:01:51.840 --> 00:01:55.990 align:middle line:84% I thought what we would do today is go through an example. 00:01:55.990 --> 00:02:01.890 align:middle line:84% And the example is related to breast cancer and the paper 00:02:01.890 --> 00:02:06.352 align:middle line:84% from Nature 2002 that we talked about. 00:02:06.352 --> 00:02:08.310 align:middle line:84% So the issue of New England Journal of Medicine 00:02:08.310 --> 00:02:10.320 align:middle line:84% that was published today had a number 00:02:10.320 --> 00:02:14.160 align:middle line:84% of articles related to use of these classification 00:02:14.160 --> 00:02:21.410 align:middle line:84% methods in AML, in acute myelogenous leukemia. 00:02:21.410 --> 00:02:23.943 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:02:23.943 --> 00:02:25.860 align:middle line:84% PROFESSOR: Well, there's two different papers. 00:02:25.860 --> 00:02:32.788 align:middle line:84% There's an editorial and there's a perspective written. 00:02:32.788 --> 00:02:34.830 align:middle line:84% Maybe we'll look at the perspective for a second. 00:02:34.830 --> 00:02:36.840 align:middle line:84% And the perspective is called "Microarrays 00:02:36.840 --> 00:02:39.000 align:middle line:90% in clinical investigations." 00:02:39.000 --> 00:02:44.330 align:middle line:90% 00:02:44.330 --> 00:02:57.640 align:middle line:84% And so the perspective is pretty bold in its views here. 00:02:57.640 --> 00:03:01.090 align:middle line:84% And essentially, the last paragraph, the 00:03:01.090 --> 00:03:02.600 align:middle line:90% take home message-- 00:03:02.600 --> 00:03:05.950 align:middle line:84% so the take home message is that our usual thinking 00:03:05.950 --> 00:03:09.010 align:middle line:84% about biomarker discovery in clinical trials 00:03:09.010 --> 00:03:11.230 align:middle line:90% is about to change dramatically. 00:03:11.230 --> 00:03:13.210 align:middle line:84% In the future, clinical investigations 00:03:13.210 --> 00:03:17.650 align:middle line:84% will consist of small trials with a high density of data, 00:03:17.650 --> 00:03:20.650 align:middle line:84% precise patient stratification according 00:03:20.650 --> 00:03:23.770 align:middle line:84% to the expression profile, and highly 00:03:23.770 --> 00:03:26.590 align:middle line:84% tailored analysis of microarray data. 00:03:26.590 --> 00:03:33.700 align:middle line:84% So they are looking at the articles in this issue 00:03:33.700 --> 00:03:36.850 align:middle line:90% of gene expression and AML. 00:03:36.850 --> 00:03:41.990 align:middle line:84% And they focus on identifying the relevant-- 00:03:41.990 --> 00:03:45.820 align:middle line:84% so looking through 7,000 genes or I 00:03:45.820 --> 00:03:50.140 align:middle line:84% think one study used 133 arrays with 13,000, another 00:03:50.140 --> 00:03:52.060 align:middle line:90% with 25,000-- 00:03:52.060 --> 00:03:55.030 align:middle line:84% looking at large number of genes and their expression 00:03:55.030 --> 00:03:58.240 align:middle line:84% in patients and identifying smaller numbers that 00:03:58.240 --> 00:04:00.760 align:middle line:90% are very relevant to disease. 00:04:00.760 --> 00:04:05.920 align:middle line:84% And that's what's meant by the term biomarkers here. 00:04:05.920 --> 00:04:08.020 align:middle line:84% And they're making a prediction that this 00:04:08.020 --> 00:04:13.120 align:middle line:84% is going to change the way clinical trials are 00:04:13.120 --> 00:04:15.730 align:middle line:90% being done dramatically. 00:04:15.730 --> 00:04:19.037 align:middle line:84% This seems very unlikely to me that this will-- 00:04:19.037 --> 00:04:20.829 align:middle line:84% AUDIENCE: I was really curious because when 00:04:20.829 --> 00:04:24.220 align:middle line:84% you reading the articles, at least glancingly, 00:04:24.220 --> 00:04:28.240 align:middle line:84% they seem very bread and butter, just classification 00:04:28.240 --> 00:04:30.050 align:middle line:90% and some prediction. 00:04:30.050 --> 00:04:33.442 align:middle line:84% And I was just curious how they [INAUDIBLE] 00:04:33.442 --> 00:04:34.900 align:middle line:84% the New England Journal of Medicine 00:04:34.900 --> 00:04:38.970 align:middle line:84% is very reticent to take anything with [INAUDIBLE].. 00:04:38.970 --> 00:04:44.680 align:middle line:84% And so that sort of struck me really. 00:04:44.680 --> 00:04:47.320 align:middle line:84% PROFESSOR: Well, it's a learning process for everybody. 00:04:47.320 --> 00:04:50.170 align:middle line:84% There are things to learn from the breast cancer 00:04:50.170 --> 00:04:51.010 align:middle line:90% work that was done. 00:04:51.010 --> 00:04:52.750 align:middle line:84% And we'll go over the Nature paper. 00:04:52.750 --> 00:04:54.670 align:middle line:84% And that also was published in the New England 00:04:54.670 --> 00:04:57.400 align:middle line:90% Journal of Medicine as well. 00:04:57.400 --> 00:04:59.600 align:middle line:90% Let me give you-- 00:04:59.600 --> 00:05:00.100 align:middle line:90% let's see. 00:05:00.100 --> 00:05:02.650 align:middle line:90% I had this one for you. 00:05:02.650 --> 00:05:07.980 align:middle line:84% And this one is important because when you have 00:05:07.980 --> 00:05:08.920 align:middle line:90% [INAUDIBLE]. 00:05:08.920 --> 00:05:22.440 align:middle line:90% 00:05:22.440 --> 00:05:30.180 align:middle line:84% So let's-- I didn't have time to review the AML stuff 00:05:30.180 --> 00:05:35.490 align:middle line:84% that's accompanying the editorials and perspective 00:05:35.490 --> 00:05:36.570 align:middle line:90% in today's issue. 00:05:36.570 --> 00:05:38.790 align:middle line:84% I think it would be nice to go through that 00:05:38.790 --> 00:05:42.330 align:middle line:84% and look at the data directly instead 00:05:42.330 --> 00:05:45.270 align:middle line:90% of the claims in the editorial. 00:05:45.270 --> 00:05:51.070 align:middle line:84% But let's do that for the breast cancer work that was done. 00:05:51.070 --> 00:05:55.560 align:middle line:90% So this is the paper. 00:05:55.560 --> 00:05:57.820 align:middle line:84% You might want to have it just so you have it here, 00:05:57.820 --> 00:05:59.580 align:middle line:90% because we're going to-- 00:05:59.580 --> 00:06:04.520 align:middle line:84% I thought rather than talking in the abstract 00:06:04.520 --> 00:06:06.620 align:middle line:84% as we did last time, to really go 00:06:06.620 --> 00:06:10.610 align:middle line:84% through concretely the use of the prediction method 00:06:10.610 --> 00:06:13.790 align:middle line:84% here, which is used quite similarly 00:06:13.790 --> 00:06:15.770 align:middle line:90% in one of these papers here. 00:06:15.770 --> 00:06:18.690 align:middle line:90% 00:06:18.690 --> 00:06:21.590 align:middle line:84% So let's go through this Nature paper 00:06:21.590 --> 00:06:24.740 align:middle line:84% on gene expression profiling predicting clinical outcome 00:06:24.740 --> 00:06:28.050 align:middle line:90% of breast cancer. 00:06:28.050 --> 00:06:35.630 align:middle line:84% So in this paper, essentially what these authors did 00:06:35.630 --> 00:06:37.430 align:middle line:84% in the abstract, you can see that they 00:06:37.430 --> 00:06:42.170 align:middle line:84% used microarray analysis on primary breast tumors of 117 00:06:42.170 --> 00:06:48.650 align:middle line:84% young patients and applied supervised classification 00:06:48.650 --> 00:06:51.110 align:middle line:84% to identify a gene expression signature strongly 00:06:51.110 --> 00:06:55.910 align:middle line:84% predictive of a short interval to distant metastasis. 00:06:55.910 --> 00:06:57.990 align:middle line:90% So again this is-- 00:06:57.990 --> 00:07:00.470 align:middle line:84% again, I often get tripped up by viewing things 00:07:00.470 --> 00:07:04.430 align:middle line:84% in terms of the specific tool that was used. 00:07:04.430 --> 00:07:08.215 align:middle line:84% And really what they're doing is doing prediction here. 00:07:08.215 --> 00:07:09.590 align:middle line:84% Some of our different pigeonholes 00:07:09.590 --> 00:07:13.700 align:middle line:84% that I try to put things in, class, exploration, prediction, 00:07:13.700 --> 00:07:14.870 align:middle line:90% and discovery. 00:07:14.870 --> 00:07:16.790 align:middle line:90% This is prediction. 00:07:16.790 --> 00:07:19.290 align:middle line:84% Supervised classification is one aspect of it. 00:07:19.290 --> 00:07:22.310 align:middle line:84% But there are other parts of this to doing prediction. 00:07:22.310 --> 00:07:23.750 align:middle line:84% But what they were interested in, 00:07:23.750 --> 00:07:26.720 align:middle line:84% essentially, is taking patients and making 00:07:26.720 --> 00:07:32.930 align:middle line:84% good or bad predictions of prognosis of outcome 00:07:32.930 --> 00:07:36.960 align:middle line:90% based on their gene expression. 00:07:36.960 --> 00:07:38.420 align:middle line:90% So they did a variety of things. 00:07:38.420 --> 00:07:41.560 align:middle line:84% But most of this paper is focused-- 00:07:41.560 --> 00:07:46.460 align:middle line:84% we look at, say, the second paragraph on the first page 00:07:46.460 --> 00:07:47.510 align:middle line:90% after the abstract. 00:07:47.510 --> 00:07:50.870 align:middle line:84% An unsupervised hierarchical clustering algorithm 00:07:50.870 --> 00:07:54.140 align:middle line:84% allowed us to cluster the 98 tumors-- 00:07:54.140 --> 00:07:56.870 align:middle line:84% now we're down to 98 from 117, I'll 00:07:56.870 --> 00:07:59.750 align:middle line:84% come back to that-- on the basis of their similarities measured 00:07:59.750 --> 00:08:03.830 align:middle line:84% over approximately 5,000 significant genes. 00:08:03.830 --> 00:08:05.540 align:middle line:90% And that's figure 1A. 00:08:05.540 --> 00:08:10.730 align:middle line:90% 00:08:10.730 --> 00:08:14.180 align:middle line:84% So here they have things sideways. 00:08:14.180 --> 00:08:18.710 align:middle line:84% They have the specimens as the rows 00:08:18.710 --> 00:08:22.940 align:middle line:84% and the genes as the columns in that top diagram. 00:08:22.940 --> 00:08:26.670 align:middle line:90% 00:08:26.670 --> 00:08:35.539 align:middle line:84% And so this is hierarchical clustering of this data set 00:08:35.539 --> 00:08:40.220 align:middle line:90% that consists of a big table. 00:08:40.220 --> 00:08:45.070 align:middle line:90% 00:08:45.070 --> 00:09:02.880 align:middle line:84% And the table again has 98 patient samples and 5,000 genes 00:09:02.880 --> 00:09:08.250 align:middle line:90% and expression levels that-- 00:09:08.250 --> 00:09:10.410 align:middle line:90% I might be short on the paper. 00:09:10.410 --> 00:09:13.150 align:middle line:90% 00:09:13.150 --> 00:09:15.370 align:middle line:84% Why don't you guys look on that for now, yeah. 00:09:15.370 --> 00:09:19.070 align:middle line:84% Yeah, this paper is up on the site. 00:09:19.070 --> 00:09:24.550 align:middle line:84% OK, so this is the data set that they constructed. 00:09:24.550 --> 00:09:29.830 align:middle line:84% AUDIENCE: I just have a question about figure 1A in particular. 00:09:29.830 --> 00:09:35.560 align:middle line:84% What's the utility of something like this in the paper? 00:09:35.560 --> 00:09:38.620 align:middle line:84% I mean they're obviously they're not looking at this-- 00:09:38.620 --> 00:09:39.803 align:middle line:90% like if I were to-- 00:09:39.803 --> 00:09:42.220 align:middle line:84% when they get their data and they see something like this, 00:09:42.220 --> 00:09:45.290 align:middle line:84% they feed into a computer that does all this analysis. 00:09:45.290 --> 00:09:47.710 align:middle line:84% So what's the purpose of showing us 00:09:47.710 --> 00:09:52.510 align:middle line:84% this rather than just kind of cutting to the chase of what 00:09:52.510 --> 00:09:53.950 align:middle line:90% their actual results are. 00:09:53.950 --> 00:09:58.450 align:middle line:84% Should this be standard for when you're 00:09:58.450 --> 00:10:01.090 align:middle line:84% doing these types of studies or whatever? 00:10:01.090 --> 00:10:03.850 align:middle line:84% PROFESSOR: I think they're pretty valuable diagrams 00:10:03.850 --> 00:10:07.000 align:middle line:90% because they do allow you-- 00:10:07.000 --> 00:10:10.180 align:middle line:84% this is the black and white, so there's the color version-- 00:10:10.180 --> 00:10:14.650 align:middle line:84% they do allow you to see that there 00:10:14.650 --> 00:10:19.540 align:middle line:84% are different patterns of expressions of genes 00:10:19.540 --> 00:10:20.960 align:middle line:90% and different subgroups. 00:10:20.960 --> 00:10:23.500 align:middle line:90% 00:10:23.500 --> 00:10:26.800 align:middle line:84% AUDIENCE: I mean, I guess maybe these lower ones are blown up, 00:10:26.800 --> 00:10:30.580 align:middle line:84% so where [INAUDIBLE] it may become more obvious. 00:10:30.580 --> 00:10:35.950 align:middle line:84% But I'm thinking just in terms of if I saw an SDS-PAGE 00:10:35.950 --> 00:10:39.490 align:middle line:84% gel in the paper, it's just like, right there you 00:10:39.490 --> 00:10:40.625 align:middle line:90% can see the results. 00:10:40.625 --> 00:10:42.000 align:middle line:84% If I look at something like this, 00:10:42.000 --> 00:10:48.370 align:middle line:84% it just looks like a bunch of pixels [INAUDIBLE].. 00:10:48.370 --> 00:10:50.090 align:middle line:90% I just I guess. 00:10:50.090 --> 00:10:52.585 align:middle line:84% PROFESSOR: So you're asking about the motivation? 00:10:52.585 --> 00:10:53.960 align:middle line:84% AUDIENCE: Well, yeah, I mean so-- 00:10:53.960 --> 00:10:55.060 align:middle line:90% [INTERPOSING VOICES] 00:10:55.060 --> 00:10:57.860 align:middle line:84% AUDIENCE: Yeah, so like the sub figures can be useful. 00:10:57.860 --> 00:11:00.790 align:middle line:84% I guess and maybe I can see how you'd say, 00:11:00.790 --> 00:11:03.185 align:middle line:84% OK, this is the reason we expanded on it. 00:11:03.185 --> 00:11:04.810 align:middle line:84% PROFESSOR: But even in the big figures, 00:11:04.810 --> 00:11:08.290 align:middle line:84% you can often see, and maybe I have a picture of this on color 00:11:08.290 --> 00:11:10.780 align:middle line:84% that I can put up, I'm not sure if I do. 00:11:10.780 --> 00:11:12.220 align:middle line:90% But I should. 00:11:12.220 --> 00:11:16.120 align:middle line:84% But you can see that there are different groups, even 00:11:16.120 --> 00:11:17.860 align:middle line:90% within color. 00:11:17.860 --> 00:11:21.460 align:middle line:84% [BEEPING] Sorry, I have some patients. 00:11:21.460 --> 00:11:24.010 align:middle line:84% AUDIENCE: I guess sort of as a newbie 00:11:24.010 --> 00:11:29.320 align:middle line:84% here, looking at my [INAUDIBLE] traditionally, 00:11:29.320 --> 00:11:32.510 align:middle line:84% you see something like that-- when I'm seeing something 00:11:32.510 --> 00:11:35.560 align:middle line:84% like this, I just go, well, gee, that looks pretty. 00:11:35.560 --> 00:11:39.800 align:middle line:84% And I have no idea what it means or whatever. 00:11:39.800 --> 00:11:43.420 align:middle line:84% And I think that would be kind of intimidating for people 00:11:43.420 --> 00:11:47.140 align:middle line:84% to want to pick up a paper like that. 00:11:47.140 --> 00:11:51.295 align:middle line:84% And they don't quite get the point of seeing something like. 00:11:51.295 --> 00:11:52.420 align:middle line:90% - Well, there's two issues. 00:11:52.420 --> 00:11:53.620 align:middle line:90% That's an independent issue. 00:11:53.620 --> 00:11:58.640 align:middle line:84% I think there's an issue that I find 00:11:58.640 --> 00:12:01.490 align:middle line:84% talking about these things is, the same thing 00:12:01.490 --> 00:12:05.180 align:middle line:84% as looking at a scattergram, why should you involve a-- 00:12:05.180 --> 00:12:07.850 align:middle line:84% look at a scattergram rather than just the correlation 00:12:07.850 --> 00:12:11.220 align:middle line:90% coefficient described in here. 00:12:11.220 --> 00:12:15.560 align:middle line:84% Just give me a look right path to see, to actually 00:12:15.560 --> 00:12:17.420 align:middle line:84% see, by visual inspection that it 00:12:17.420 --> 00:12:19.590 align:middle line:90% looks like this class is there. 00:12:19.590 --> 00:12:22.490 align:middle line:84% And so when I look at this thing, 00:12:22.490 --> 00:12:26.330 align:middle line:84% I can actually convince myself but there's a lot of crap. 00:12:26.330 --> 00:12:30.020 align:middle line:84% But there is also some big chunks of red. 00:12:30.020 --> 00:12:32.420 align:middle line:84% AUDIENCE: So definitely the color picture helps a lot. 00:12:32.420 --> 00:12:34.772 align:middle line:90% 00:12:34.772 --> 00:12:36.980 align:middle line:84% PROFESSOR: The other part of these pictures, which is 00:12:36.980 --> 00:12:40.140 align:middle line:90% helpful, are the dendrograms. 00:12:40.140 --> 00:12:44.090 align:middle line:84% OK, so again we talked about the use of hierarchical 00:12:44.090 --> 00:12:46.610 align:middle line:84% clustering as a classification method 00:12:46.610 --> 00:12:50.120 align:middle line:84% and that it does not really produce classes. 00:12:50.120 --> 00:12:52.710 align:middle line:84% OK, again, everything is related to everything else. 00:12:52.710 --> 00:12:55.310 align:middle line:90% You don't have disjoint classes. 00:12:55.310 --> 00:12:57.500 align:middle line:84% It's just that the degree of relationship 00:12:57.500 --> 00:13:02.510 align:middle line:84% is represented by the distance, literally 00:13:02.510 --> 00:13:08.120 align:middle line:84% the length of this line, right here in the diagram, 00:13:08.120 --> 00:13:12.950 align:middle line:84% represents the relationship between these two over here 00:13:12.950 --> 00:13:17.070 align:middle line:90% and say these two over here. 00:13:17.070 --> 00:13:21.530 align:middle line:84% And so again everything every cluster is split. 00:13:21.530 --> 00:13:22.740 align:middle line:90% This is binary. 00:13:22.740 --> 00:13:27.470 align:middle line:90% So everything is split into two. 00:13:27.470 --> 00:13:33.650 align:middle line:84% And so the adjacency is actually another issue too, about these, 00:13:33.650 --> 00:13:39.260 align:middle line:84% is that you can spin any one of these clusters at any node 00:13:39.260 --> 00:13:42.990 align:middle line:84% without changing the organization here. 00:13:42.990 --> 00:13:47.240 align:middle line:84% So sometimes you will see diagrams-- 00:13:47.240 --> 00:13:50.480 align:middle line:84% why don't I show you one since we're 00:13:50.480 --> 00:13:52.090 align:middle line:90% getting to nitty gritty here. 00:13:52.090 --> 00:13:54.680 align:middle line:84% AUDIENCE: When you have a paper and you see that figure, 00:13:54.680 --> 00:13:57.920 align:middle line:84% just apart from of the obvious, like here's a really big red 00:13:57.920 --> 00:14:01.730 align:middle line:90% patch, [INAUDIBLE] that's-- 00:14:01.730 --> 00:14:06.110 align:middle line:84% you're also, if someone is familiar with this, 00:14:06.110 --> 00:14:08.570 align:middle line:84% you're also gathering other information. 00:14:08.570 --> 00:14:11.250 align:middle line:84% By not having data in hand, you can see, 00:14:11.250 --> 00:14:15.920 align:middle line:84% you can look at that figure and say, ah, pattern, but here's 00:14:15.920 --> 00:14:17.810 align:middle line:84% another part of the figure that also gave me 00:14:17.810 --> 00:14:19.680 align:middle line:90% some more information. 00:14:19.680 --> 00:14:22.010 align:middle line:84% And me as the newbie, I'm just not 00:14:22.010 --> 00:14:24.197 align:middle line:84% seeing it because I'm not as familiar with that. 00:14:24.197 --> 00:14:26.030 align:middle line:84% PROFESSOR: And I don't have a password here. 00:14:26.030 --> 00:14:28.034 align:middle line:90% I can't do this. 00:14:28.034 --> 00:14:30.890 align:middle line:84% AUDIENCE: Is that a fair assessment? 00:14:30.890 --> 00:14:34.010 align:middle line:84% PROFESSOR: That's a fair assessment, OK. 00:14:34.010 --> 00:14:39.020 align:middle line:84% What I was going to show you is that there's been, in my area, 00:14:39.020 --> 00:14:40.827 align:middle line:84% last week there was a publication-- 00:14:40.827 --> 00:14:43.160 align:middle line:84% it's the second one this group has done-- where they put 00:14:43.160 --> 00:14:45.080 align:middle line:90% one of these diagrams up here. 00:14:45.080 --> 00:14:51.920 align:middle line:84% And they're showing that three patients with one condition, 00:14:51.920 --> 00:14:57.260 align:middle line:84% they all cluster together and separate from another seven. 00:14:57.260 --> 00:15:01.670 align:middle line:84% But they don't put the dendrogram on that. 00:15:01.670 --> 00:15:04.550 align:middle line:84% All they show is adjacency of rows. 00:15:04.550 --> 00:15:07.970 align:middle line:84% And they don't show exactly how they're organized or connected. 00:15:07.970 --> 00:15:10.073 align:middle line:84% So you see three that are here and then 00:15:10.073 --> 00:15:11.240 align:middle line:90% another seven that are here. 00:15:11.240 --> 00:15:13.100 align:middle line:84% But it's possible that this one over here 00:15:13.100 --> 00:15:18.830 align:middle line:84% is actually part of a group of four over here. 00:15:18.830 --> 00:15:21.500 align:middle line:84% And so that's another value to these diagrams, 00:15:21.500 --> 00:15:24.050 align:middle line:84% is when people claim classes to look and see. 00:15:24.050 --> 00:15:29.240 align:middle line:84% When you look at this one, it's not overwhelming, actually. 00:15:29.240 --> 00:15:30.860 align:middle line:84% And this is where they drew the line. 00:15:30.860 --> 00:15:33.230 align:middle line:90% And they took this as one class. 00:15:33.230 --> 00:15:34.880 align:middle line:90% But this is really-- 00:15:34.880 --> 00:15:39.560 align:middle line:84% I mean every one of these is just subdivided here, 00:15:39.560 --> 00:15:44.190 align:middle line:84% instead of one fairly large class. 00:15:44.190 --> 00:15:46.880 align:middle line:84% So there is helpful information for seeing those diagrams. 00:15:46.880 --> 00:15:50.450 align:middle line:84% AUDIENCE: So the leftmost is the two classes in [INAUDIBLE] 00:15:50.450 --> 00:15:52.397 align:middle line:90% 117-- 00:15:52.397 --> 00:15:53.480 align:middle line:90% PROFESSOR: That's correct. 00:15:53.480 --> 00:15:55.190 align:middle line:90% They said they are two classes. 00:15:55.190 --> 00:15:58.650 align:middle line:90% So this is one branch here. 00:15:58.650 --> 00:16:03.330 align:middle line:84% And this is the other branch here. 00:16:03.330 --> 00:16:04.680 align:middle line:90% OK. 00:16:04.680 --> 00:16:09.180 align:middle line:84% OK, yeah, well, let's get into the nitty gritty, 00:16:09.180 --> 00:16:12.270 align:middle line:84% at least so we have one example that we've understood 00:16:12.270 --> 00:16:15.030 align:middle line:90% fairly well what's happening. 00:16:15.030 --> 00:16:21.210 align:middle line:90% OK, so this is an example. 00:16:21.210 --> 00:16:22.940 align:middle line:84% So this is just hierarchical clustering. 00:16:22.940 --> 00:16:26.390 align:middle line:84% What would you call this if I made you pigeonhole it 00:16:26.390 --> 00:16:30.590 align:middle line:84% into class exploration, class prediction, or class discovery? 00:16:30.590 --> 00:16:32.960 align:middle line:84% These are 98 patients with breast cancer. 00:16:32.960 --> 00:16:35.480 align:middle line:90% 00:16:35.480 --> 00:16:37.970 align:middle line:84% These are both lymph node positive and lymph 00:16:37.970 --> 00:16:38.930 align:middle line:90% node negative. 00:16:38.930 --> 00:16:42.680 align:middle line:84% They're pretty much all of their samples except for a group 00:16:42.680 --> 00:16:46.640 align:middle line:84% that they've left off for other purposes. 00:16:46.640 --> 00:16:49.520 align:middle line:84% This would be an example of class exploration. 00:16:49.520 --> 00:16:52.530 align:middle line:84% So they're starting by looking at their data. 00:16:52.530 --> 00:16:54.290 align:middle line:84% And they're just saying, well, does it 00:16:54.290 --> 00:16:59.210 align:middle line:84% look like the expression profiles differ among patients 00:16:59.210 --> 00:17:04.579 align:middle line:84% and allow us to find different subgroups, different profiles? 00:17:04.579 --> 00:17:08.250 align:middle line:90% 00:17:08.250 --> 00:17:10.130 align:middle line:84% And we talked about one step that was 00:17:10.130 --> 00:17:11.930 align:middle line:90% important to do before this. 00:17:11.930 --> 00:17:14.060 align:middle line:84% And that was the stuff about signatures 00:17:14.060 --> 00:17:18.079 align:middle line:84% and their reproducibility and their distinctness. 00:17:18.079 --> 00:17:20.900 align:middle line:84% They don't provide any data here in that regard. 00:17:20.900 --> 00:17:24.650 align:middle line:84% They say that they did every tumor sample twice, 00:17:24.650 --> 00:17:27.867 align:middle line:84% that they did two independent array profiles on each one. 00:17:27.867 --> 00:17:29.450 align:middle line:84% But they don't say anything about what 00:17:29.450 --> 00:17:31.250 align:middle line:90% they did with that data. 00:17:31.250 --> 00:17:33.450 align:middle line:84% They don't provide any correlation coefficients, 00:17:33.450 --> 00:17:36.290 align:middle line:84% for example, to say that when you do the same experiment 00:17:36.290 --> 00:17:39.755 align:middle line:84% twice on a given tissue, you get out the same pattern of genes. 00:17:39.755 --> 00:17:44.520 align:middle line:90% 00:17:44.520 --> 00:17:48.540 align:middle line:84% But accepting that, this first step is exploration. 00:17:48.540 --> 00:17:57.960 align:middle line:84% And here there is an attempt to correlate these groups 00:17:57.960 --> 00:18:03.130 align:middle line:84% with different clinical features. 00:18:03.130 --> 00:18:06.150 align:middle line:84% So first, they do hierarchical clustering 00:18:06.150 --> 00:18:08.520 align:middle line:90% of the entire data set. 00:18:08.520 --> 00:18:12.000 align:middle line:84% And they look and see broad patterns. 00:18:12.000 --> 00:18:14.910 align:middle line:84% Maybe there is a group over here that's 00:18:14.910 --> 00:18:17.950 align:middle line:84% more red for these genes and greens for these. 00:18:17.950 --> 00:18:20.380 align:middle line:84% And so maybe there is some structure here. 00:18:20.380 --> 00:18:25.710 align:middle line:84% So the next step they do is to, for each one of these patients, 00:18:25.710 --> 00:18:31.110 align:middle line:84% they look at binary outcomes, and say if this patient here 00:18:31.110 --> 00:18:35.820 align:middle line:84% had a mutation in the BRCA1 gene, 00:18:35.820 --> 00:18:38.220 align:middle line:90% they'll color it in black. 00:18:38.220 --> 00:18:41.820 align:middle line:84% And this patient here did not have a mutation. 00:18:41.820 --> 00:18:44.190 align:middle line:84% So for each one of these 98 patients, 00:18:44.190 --> 00:18:46.600 align:middle line:84% they're now looking at one, two, three, four, five, 00:18:46.600 --> 00:18:52.920 align:middle line:84% six different phenotypic markers and coloring it in this way 00:18:52.920 --> 00:18:55.060 align:middle line:84% so that we can then look and say, 00:18:55.060 --> 00:18:56.610 align:middle line:84% is there any relationship between 00:18:56.610 --> 00:19:01.500 align:middle line:84% these phenotypic variables and these two classes 00:19:01.500 --> 00:19:03.989 align:middle line:84% which were defined by the dendrogram? 00:19:03.989 --> 00:19:08.120 align:middle line:90% 00:19:08.120 --> 00:19:10.978 align:middle line:84% Again to pigeonhole it into one of the three class methods, 00:19:10.978 --> 00:19:12.020 align:middle line:90% what are they doing here? 00:19:12.020 --> 00:19:14.930 align:middle line:90% 00:19:14.930 --> 00:19:15.680 align:middle line:90% What's that? 00:19:15.680 --> 00:19:19.030 align:middle line:90% 00:19:19.030 --> 00:19:21.640 align:middle line:84% Class exploration, class discovery-- 00:19:21.640 --> 00:19:23.870 align:middle line:84% they're doing class discovery here. 00:19:23.870 --> 00:19:26.230 align:middle line:90% So they looked at their data. 00:19:26.230 --> 00:19:27.400 align:middle line:90% They explored it. 00:19:27.400 --> 00:19:29.560 align:middle line:84% They wondered that there might be classes. 00:19:29.560 --> 00:19:31.540 align:middle line:84% And they're now asking is there something 00:19:31.540 --> 00:19:35.080 align:middle line:84% significant phenotypically, something 00:19:35.080 --> 00:19:39.070 align:middle line:84% meaningful clinically, that relates to these two classes? 00:19:39.070 --> 00:19:40.720 align:middle line:84% Or is it just an artifact of doing 00:19:40.720 --> 00:19:46.450 align:middle line:84% clustering, which will always reduce things into classes? 00:19:46.450 --> 00:19:48.610 align:middle line:84% Depending on where you cut this tree, 00:19:48.610 --> 00:19:52.900 align:middle line:84% you can have two classes here, or this one here 00:19:52.900 --> 00:19:55.330 align:middle line:84% that looks like there's a large class, which goes 00:19:55.330 --> 00:19:58.090 align:middle line:90% from here down to here. 00:19:58.090 --> 00:20:01.720 align:middle line:84% You can make up classes at any different level 00:20:01.720 --> 00:20:03.520 align:middle line:84% here, because fundamentally there really 00:20:03.520 --> 00:20:07.390 align:middle line:84% are no classes with a dendrogram. 00:20:07.390 --> 00:20:09.310 align:middle line:84% But they're now saying, well, what 00:20:09.310 --> 00:20:14.315 align:middle line:84% phenotypic variables might correlate with this split right 00:20:14.315 --> 00:20:14.815 align:middle line:90% here? 00:20:14.815 --> 00:20:18.460 align:middle line:90% 00:20:18.460 --> 00:20:20.200 align:middle line:90% This is still discovery. 00:20:20.200 --> 00:20:22.750 align:middle line:84% Yeah, I would call this class discovery, 00:20:22.750 --> 00:20:25.060 align:middle line:90% that they found two classes. 00:20:25.060 --> 00:20:31.690 align:middle line:84% And now that to establish them as truly classes that 00:20:31.690 --> 00:20:33.970 align:middle line:84% have some meaning, some clinical meaning, 00:20:33.970 --> 00:20:37.000 align:middle line:84% they want to say what clinical variables might 00:20:37.000 --> 00:20:41.170 align:middle line:90% this relate to here? 00:20:41.170 --> 00:20:43.630 align:middle line:84% And so looking at the things here, 00:20:43.630 --> 00:20:46.800 align:middle line:84% there are few things which are fairly striking, I guess. 00:20:46.800 --> 00:20:51.660 align:middle line:84% The presence of estrogen receptors on the tumor cells 00:20:51.660 --> 00:20:53.520 align:middle line:90% here is pretty good. 00:20:53.520 --> 00:20:56.670 align:middle line:84% Most of these patients are ER positive. 00:20:56.670 --> 00:21:00.272 align:middle line:84% Most of these patients here are-- 00:21:00.272 --> 00:21:01.980 align:middle line:84% actually, I can't remember which is black 00:21:01.980 --> 00:21:03.660 align:middle line:84% and which is white, which is positive 00:21:03.660 --> 00:21:06.600 align:middle line:84% and which is negative for their classes here. 00:21:06.600 --> 00:21:10.020 align:middle line:90% 00:21:10.020 --> 00:21:13.590 align:middle line:84% Metastases-- so the presence of-- 00:21:13.590 --> 00:21:16.275 align:middle line:84% and I think this is distance metastases at five years-- 00:21:16.275 --> 00:21:19.140 align:middle line:90% 00:21:19.140 --> 00:21:21.060 align:middle line:90% not really too much difference. 00:21:21.060 --> 00:21:24.580 align:middle line:90% Maybe I think white is present. 00:21:24.580 --> 00:21:25.770 align:middle line:90% So maybe more of these. 00:21:25.770 --> 00:21:27.000 align:middle line:90% What's that? 00:21:27.000 --> 00:21:28.540 align:middle line:90% White is positive. 00:21:28.540 --> 00:21:31.200 align:middle line:84% So there may be more consistently, certainly 00:21:31.200 --> 00:21:33.300 align:middle line:84% a larger percentage of this group, 00:21:33.300 --> 00:21:36.975 align:middle line:84% had distant metastases at five years. 00:21:36.975 --> 00:21:38.910 align:middle line:90% 00:21:38.910 --> 00:21:46.170 align:middle line:84% So once they do this and believe that they do have classes here, 00:21:46.170 --> 00:21:49.080 align:middle line:84% they then move to the next step and say we're 00:21:49.080 --> 00:21:51.810 align:middle line:84% going to build a predictive model. 00:21:51.810 --> 00:21:55.080 align:middle line:84% And this is the one that they focus on, 00:21:55.080 --> 00:21:58.800 align:middle line:84% is whether there's metastases or not. 00:21:58.800 --> 00:22:00.390 align:middle line:84% So they're saying in this data, there 00:22:00.390 --> 00:22:05.970 align:middle line:84% seems to be some way to classify the data that 00:22:05.970 --> 00:22:12.720 align:middle line:84% has some distinctions in terms of predicting outcome. 00:22:12.720 --> 00:22:15.330 align:middle line:84% And we're going to take it a step further here 00:22:15.330 --> 00:22:18.180 align:middle line:84% and actually go through the trouble of building a model 00:22:18.180 --> 00:22:19.050 align:middle line:90% to predict this. 00:22:19.050 --> 00:22:26.467 align:middle line:90% 00:22:26.467 --> 00:22:28.300 align:middle line:84% AUDIENCE: So they want to use the metastases 00:22:28.300 --> 00:22:34.410 align:middle line:84% as the principal predictor for or [INAUDIBLE].. 00:22:34.410 --> 00:22:38.080 align:middle line:84% Why, maybe this is just another ignorant question. 00:22:38.080 --> 00:22:40.190 align:middle line:84% PROFESSOR: No, there's no ignorant questions here. 00:22:40.190 --> 00:22:44.110 align:middle line:84% AUDIENCE: Why did they choose that as like estrogen receptor? 00:22:44.110 --> 00:22:46.240 align:middle line:84% PROFESSOR: Well, yeah, because people already 00:22:46.240 --> 00:22:50.350 align:middle line:84% know about the estrogen receptor and BRCA1 mutations. 00:22:50.350 --> 00:22:50.860 align:middle line:90% OK. 00:22:50.860 --> 00:22:54.405 align:middle line:90% So and metastasis is the-- 00:22:54.405 --> 00:22:56.030 align:middle line:84% it's fundamentally the important thing. 00:22:56.030 --> 00:22:58.870 align:middle line:84% It's the important clinical endpoint. 00:22:58.870 --> 00:23:00.610 align:middle line:84% If we can make better predictions 00:23:00.610 --> 00:23:03.580 align:middle line:84% based on gene expression profiles of tumor 00:23:03.580 --> 00:23:08.000 align:middle line:84% when we do an initial biopsy on the breast cancer, 00:23:08.000 --> 00:23:11.050 align:middle line:84% can we use that to then decide how to treat people? 00:23:11.050 --> 00:23:12.700 align:middle line:84% People have a good prognosis don't 00:23:12.700 --> 00:23:16.680 align:middle line:84% need the same treatment as people with a bad prognosis, 00:23:16.680 --> 00:23:17.285 align:middle line:90% maybe. 00:23:17.285 --> 00:23:19.660 align:middle line:84% It doesn't actually-- there's no treatment intervention-- 00:23:19.660 --> 00:23:20.830 align:middle line:90% not a treatment study here. 00:23:20.830 --> 00:23:24.700 align:middle line:84% It's not saying that we can intervene and alter prognosis 00:23:24.700 --> 00:23:25.900 align:middle line:90% through treatment. 00:23:25.900 --> 00:23:31.150 align:middle line:84% It's strictly predictive of prognosis at this point. 00:23:31.150 --> 00:23:32.740 align:middle line:90% OK? 00:23:32.740 --> 00:23:35.770 align:middle line:84% OK, so let's go back to the paper. 00:23:35.770 --> 00:23:38.570 align:middle line:90% 00:23:38.570 --> 00:23:45.490 align:middle line:84% And so the next thing they do after this class exploration, 00:23:45.490 --> 00:23:50.770 align:middle line:84% this sort of partial class discovery, 00:23:50.770 --> 00:23:54.670 align:middle line:84% I mean they don't carry it further and say that we really 00:23:54.670 --> 00:23:58.120 align:middle line:84% have discovered a new class here, 00:23:58.120 --> 00:23:59.800 align:middle line:90% because they really haven't. 00:23:59.800 --> 00:24:01.870 align:middle line:84% I mean what the prognosis is really 00:24:01.870 --> 00:24:05.330 align:middle line:84% correlating with is the estrogen receptor status here, 00:24:05.330 --> 00:24:07.310 align:middle line:90% which is a well-known fact. 00:24:07.310 --> 00:24:12.010 align:middle line:84% So patients who are estrogen receptor positive 00:24:12.010 --> 00:24:14.630 align:middle line:84% have a better outcome, I think, in breast cancer. 00:24:14.630 --> 00:24:18.100 align:middle line:84% So I'm not sure about the black being positive or negative. 00:24:18.100 --> 00:24:19.468 align:middle line:90% The black is negative, OK. 00:24:19.468 --> 00:24:20.260 align:middle line:90% Yeah, that's right. 00:24:20.260 --> 00:24:22.510 align:middle line:84% So patients who are estrogen receptor positive 00:24:22.510 --> 00:24:25.090 align:middle line:90% have a better outcome. 00:24:25.090 --> 00:24:26.860 align:middle line:84% The estrogen receptor negative patients, 00:24:26.860 --> 00:24:31.810 align:middle line:84% by and large, have a bad outcome. 00:24:31.810 --> 00:24:35.270 align:middle line:90% 00:24:35.270 --> 00:24:37.900 align:middle line:90% So there's actually nothing new. 00:24:37.900 --> 00:24:41.410 align:middle line:84% There's no important class discovery 00:24:41.410 --> 00:24:43.855 align:middle line:90% that took place in this paper. 00:24:43.855 --> 00:24:46.670 align:middle line:90% 00:24:46.670 --> 00:24:52.250 align:middle line:84% So what they next do is change the data set. 00:24:52.250 --> 00:24:54.740 align:middle line:90% Let's shut this off. 00:24:54.740 --> 00:24:56.840 align:middle line:90% It's not shut down. 00:24:56.840 --> 00:25:01.310 align:middle line:84% So the next step that they do in this paper, 00:25:01.310 --> 00:25:10.080 align:middle line:90% and that's on this page 532. 00:25:10.080 --> 00:25:15.010 align:middle line:84% Bottom of the first page, sorry, bottom of page 530-- 00:25:15.010 --> 00:25:20.110 align:middle line:84% is they focus now on a subgroup of these samples on the 78 00:25:20.110 --> 00:25:24.020 align:middle line:84% patients with sporadic lymph node negative disease. 00:25:24.020 --> 00:25:26.800 align:middle line:84% So these are all patients who, at the time of the diagnosis 00:25:26.800 --> 00:25:29.170 align:middle line:84% of their breast cancer, had lymph node biopsies which 00:25:29.170 --> 00:25:32.950 align:middle line:84% were negative, and did not show any spread to the lymph nodes. 00:25:32.950 --> 00:25:35.050 align:middle line:84% And so they then focused the rest of the paper 00:25:35.050 --> 00:25:37.270 align:middle line:84% on this select group of patients, 00:25:37.270 --> 00:25:41.650 align:middle line:84% lymph node negative, and also sporadic, 00:25:41.650 --> 00:25:45.250 align:middle line:84% which means none of these had mutations, the BRCA1 or even 00:25:45.250 --> 00:25:47.050 align:middle line:90% BRCA2 mutations. 00:25:47.050 --> 00:25:50.020 align:middle line:84% They were not genetic breast cancers, 00:25:50.020 --> 00:25:52.030 align:middle line:90% patients who are genetic. 00:25:52.030 --> 00:25:56.080 align:middle line:84% And so in that 78 group, 44 patients were free of disease 00:25:56.080 --> 00:26:00.960 align:middle line:84% after five years, and 34 patients developed metastasis. 00:26:00.960 --> 00:26:04.650 align:middle line:90% 00:26:04.650 --> 00:26:06.660 align:middle line:90% So we're down to 78 patients. 00:26:06.660 --> 00:26:10.760 align:middle line:90% 00:26:10.760 --> 00:26:18.560 align:middle line:84% And 44 had good prognosis, or turned out 00:26:18.560 --> 00:26:20.255 align:middle line:90% to have good outcome. 00:26:20.255 --> 00:26:22.880 align:middle line:90% 00:26:22.880 --> 00:26:24.770 align:middle line:90% And 34 had poor outcomes. 00:26:24.770 --> 00:26:32.060 align:middle line:90% 00:26:32.060 --> 00:26:35.540 align:middle line:84% So what they then say on the next page, 00:26:35.540 --> 00:26:39.860 align:middle line:84% 532, is to identify reliably good and poor 00:26:39.860 --> 00:26:44.480 align:middle line:84% prognostic tumors, we used a powerful three-step supervised 00:26:44.480 --> 00:26:48.830 align:middle line:84% classification method similar to those used previously. 00:26:48.830 --> 00:26:51.710 align:middle line:84% In brief, approximately 5,000 genes 00:26:51.710 --> 00:26:56.090 align:middle line:84% were selected from the 25,000 genes on the microarray. 00:26:56.090 --> 00:27:01.910 align:middle line:84% OK, so let's get into how they build the model here. 00:27:01.910 --> 00:27:04.560 align:middle line:84% So they're going to use this data. 00:27:04.560 --> 00:27:10.130 align:middle line:84% So they're going to use 78 instead of 98. 00:27:10.130 --> 00:27:14.090 align:middle line:90% And they do have 5,000 genes. 00:27:14.090 --> 00:27:20.570 align:middle line:84% So the microarray has actually measured 25,000 genes. 00:27:20.570 --> 00:27:25.940 align:middle line:84% So they had this data set to start with. 00:27:25.940 --> 00:27:30.320 align:middle line:84% And the first thing they did is use basic, but still fairly 00:27:30.320 --> 00:27:33.380 align:middle line:84% ad hoc, procedures to get rid of genes that 00:27:33.380 --> 00:27:37.550 align:middle line:84% had very little meaning, so genes that just didn't vary, 00:27:37.550 --> 00:27:39.860 align:middle line:84% a gene that had a low expression level 00:27:39.860 --> 00:27:42.980 align:middle line:90% for every single one of these. 00:27:42.980 --> 00:27:50.030 align:middle line:84% And they removed such genes and ended up with 5,000 00:27:50.030 --> 00:27:53.060 align:middle line:90% after that point. 00:27:53.060 --> 00:27:56.030 align:middle line:84% AUDIENCE: So was 5,000 predetermined, or was that-- 00:27:56.030 --> 00:27:58.940 align:middle line:84% PROFESSOR: They-- let's see how they ended up with five. 00:27:58.940 --> 00:28:01.940 align:middle line:84% They said significantly regulated and more than three 00:28:01.940 --> 00:28:04.550 align:middle line:90% tumors out of 78. 00:28:04.550 --> 00:28:08.660 align:middle line:84% So this is an area which remains completely ad hoc 00:28:08.660 --> 00:28:11.900 align:middle line:84% and different in virtually every paper. 00:28:11.900 --> 00:28:16.550 align:middle line:84% We call it the initial filtering of meaningless genes. 00:28:16.550 --> 00:28:18.950 align:middle line:90% And they decided that-- 00:28:18.950 --> 00:28:23.390 align:middle line:84% I believe that they did T-tests for significance 00:28:23.390 --> 00:28:28.040 align:middle line:84% and required there was some significance in at least three 00:28:28.040 --> 00:28:33.200 align:middle line:84% out of 78 at some particular significant level, 00:28:33.200 --> 00:28:35.330 align:middle line:90% and ended up with 5,000 genes. 00:28:35.330 --> 00:28:36.810 align:middle line:84% There's many ways people do this. 00:28:36.810 --> 00:28:41.120 align:middle line:84% Some people will say I'll look at the standard deviation 00:28:41.120 --> 00:28:44.070 align:middle line:90% of these gene vectors. 00:28:44.070 --> 00:28:47.810 align:middle line:84% And if the standard deviation is less than some cutoff, 00:28:47.810 --> 00:28:52.310 align:middle line:84% I'll assume those genes are not varying enough in this data set 00:28:52.310 --> 00:28:53.195 align:middle line:90% and get rid of those. 00:28:53.195 --> 00:28:57.190 align:middle line:90% 00:28:57.190 --> 00:29:01.940 align:middle line:84% So now they want to build their predictor. 00:29:01.940 --> 00:29:04.820 align:middle line:84% And so we talked about those three steps. 00:29:04.820 --> 00:29:09.910 align:middle line:84% And the first step is choosing a discriminative gene set. 00:29:09.910 --> 00:29:14.530 align:middle line:90% 00:29:14.530 --> 00:29:17.110 align:middle line:84% So they're going to whittle the 5,000 genes down 00:29:17.110 --> 00:29:19.060 align:middle line:84% into small numbers, which they're 00:29:19.060 --> 00:29:24.430 align:middle line:84% going to make put in their black box to make their model. 00:29:24.430 --> 00:29:28.660 align:middle line:84% And they do this by correlation with ideal outcome. 00:29:28.660 --> 00:29:31.100 align:middle line:84% I think we talked about that method before. 00:29:31.100 --> 00:29:32.620 align:middle line:84% So first they'll arrange the data 00:29:32.620 --> 00:29:37.360 align:middle line:84% set so that all of the good prognosis are up here. 00:29:37.360 --> 00:29:40.020 align:middle line:90% 00:29:40.020 --> 00:29:46.960 align:middle line:84% OK, and then T+ 45 through 78 are the poor prognosis. 00:29:46.960 --> 00:29:51.010 align:middle line:84% And then they'll make up an ideal vector, i, 00:29:51.010 --> 00:30:00.370 align:middle line:84% and put let's say 1's here and then 0's here. 00:30:00.370 --> 00:30:03.550 align:middle line:84% And then for each of these 5,000 genes, 00:30:03.550 --> 00:30:05.770 align:middle line:84% they'll calculate a correlation coefficient 00:30:05.770 --> 00:30:07.880 align:middle line:90% to this ideal vector. 00:30:07.880 --> 00:30:15.820 align:middle line:84% So r for gene one correlated with ideal vector i 00:30:15.820 --> 00:30:18.520 align:middle line:90% was some number, 0.6. 00:30:18.520 --> 00:30:22.570 align:middle line:84% And r for gene two correlated with vector i 00:30:22.570 --> 00:30:27.370 align:middle line:84% is some other number, negative 0.45, let's say. 00:30:27.370 --> 00:30:39.168 align:middle line:90% And do this for the 5,000 genes. 00:30:39.168 --> 00:30:39.835 align:middle line:90% Everyone follow? 00:30:39.835 --> 00:30:46.140 align:middle line:90% 00:30:46.140 --> 00:30:51.090 align:middle line:84% So the first thing they do is they get this down to 231 genes 00:30:51.090 --> 00:30:56.500 align:middle line:84% by using a cutoff of 0.3 negative or positive. 00:30:56.500 --> 00:31:08.370 align:middle line:84% So anything that was between an r between 0.3 and negative 0.3 00:31:08.370 --> 00:31:12.000 align:middle line:90% is gotten rid of here. 00:31:12.000 --> 00:31:14.220 align:middle line:84% So they get rid of this one here. 00:31:14.220 --> 00:31:16.080 align:middle line:90% This they would keep. 00:31:16.080 --> 00:31:19.410 align:middle line:84% This one they would keep, and so forth. 00:31:19.410 --> 00:31:26.160 align:middle line:84% And so that left them with 231 genes, 00:31:26.160 --> 00:31:32.350 align:middle line:84% which we'll renumber and call these genes now. 00:31:32.350 --> 00:31:33.400 align:middle line:90% Yes-- 00:31:33.400 --> 00:31:35.920 align:middle line:84% AUDIENCE: Question related to the coefficient 00:31:35.920 --> 00:31:40.270 align:middle line:84% for [INAUDIBLE] normalizing their variables in [INAUDIBLE] 00:31:40.270 --> 00:31:45.080 align:middle line:84% gene expression so as to be in the same order of the outcome 00:31:45.080 --> 00:31:47.760 align:middle line:90% variables, presumably binary. 00:31:47.760 --> 00:31:49.822 align:middle line:84% What kind of transformation goes on? 00:31:49.822 --> 00:31:51.530 align:middle line:84% PROFESSOR: Well, there's two issues here. 00:31:51.530 --> 00:31:53.750 align:middle line:90% One is normalizing the data set. 00:31:53.750 --> 00:31:57.430 align:middle line:90% So they used cDNA arrays. 00:31:57.430 --> 00:32:00.760 align:middle line:84% So their normalization was built in because they 00:32:00.760 --> 00:32:02.510 align:middle line:90% used a reference. 00:32:02.510 --> 00:32:04.750 align:middle line:84% So a cDNA array is an experiment you have 00:32:04.750 --> 00:32:07.390 align:middle line:90% two samples applied each time. 00:32:07.390 --> 00:32:11.320 align:middle line:84% And their reference consisted of a little bit of RNA 00:32:11.320 --> 00:32:16.570 align:middle line:84% from each one of, I think, the larger set. 00:32:16.570 --> 00:32:21.430 align:middle line:84% I don't if it's the 78 or the 98 or 117 samples. 00:32:21.430 --> 00:32:24.010 align:middle line:90% So that was the normalization. 00:32:24.010 --> 00:32:27.100 align:middle line:84% To do this, they just organize it that way. 00:32:27.100 --> 00:32:30.370 align:middle line:84% Just move these up here and move these down here. 00:32:30.370 --> 00:32:32.470 align:middle line:84% AUDIENCE: I guess my question was, 00:32:32.470 --> 00:32:36.250 align:middle line:84% I'm just trying to figure out the [INAUDIBLE] do you actually 00:32:36.250 --> 00:32:38.580 align:middle line:90% quantify for-- 00:32:38.580 --> 00:32:42.020 align:middle line:84% in the correlation coefficient, you're looking at direction. 00:32:42.020 --> 00:32:46.030 align:middle line:84% But does the amount of expression or difference 00:32:46.030 --> 00:32:49.780 align:middle line:84% in expression between positive and negative side 00:32:49.780 --> 00:32:53.120 align:middle line:84% matter In terms of your correlation? 00:32:53.120 --> 00:32:55.250 align:middle line:84% PROFESSOR: No, that won't matter. 00:32:55.250 --> 00:32:58.250 align:middle line:84% Again, you could-- again, Pearson correlation 00:32:58.250 --> 00:33:00.290 align:middle line:84% coefficients, at least, are invariant 00:33:00.290 --> 00:33:02.610 align:middle line:90% to linear transformations. 00:33:02.610 --> 00:33:05.900 align:middle line:84% So you could take every piece here and multiply by 15 00:33:05.900 --> 00:33:09.140 align:middle line:84% and add 7, and it won't change its correlation coefficient. 00:33:09.140 --> 00:33:14.250 align:middle line:90% 00:33:14.250 --> 00:33:19.660 align:middle line:84% OK, so now they were left with 231 genes. 00:33:19.660 --> 00:33:24.610 align:middle line:84% And all of these genes have some threshold of high correlation, 00:33:24.610 --> 00:33:31.450 align:middle line:84% either positive or negative to this ideal outcome here. 00:33:31.450 --> 00:33:35.620 align:middle line:84% So gene one had a certain set of expression values 00:33:35.620 --> 00:33:36.410 align:middle line:90% in this group. 00:33:36.410 --> 00:33:39.910 align:middle line:84% And it was different than in this group, 00:33:39.910 --> 00:33:42.520 align:middle line:84% because it correlates, has a high correlation 00:33:42.520 --> 00:33:44.080 align:middle line:90% coefficient to that factor. 00:33:44.080 --> 00:33:47.550 align:middle line:90% 00:33:47.550 --> 00:33:48.660 align:middle line:90% Everyone follow? 00:33:48.660 --> 00:33:51.000 align:middle line:90% OK. 00:33:51.000 --> 00:33:52.890 align:middle line:90% OK, so now they do-- 00:33:52.890 --> 00:33:55.020 align:middle line:84% some people would stop here and say this 00:33:55.020 --> 00:33:58.133 align:middle line:90% is my discriminative gene set. 00:33:58.133 --> 00:33:59.550 align:middle line:84% And the next thing I'm going to do 00:33:59.550 --> 00:34:02.265 align:middle line:90% is build my prediction function. 00:34:02.265 --> 00:34:05.812 align:middle line:90% 00:34:05.812 --> 00:34:07.270 align:middle line:84% And then I'm going to make my rule. 00:34:07.270 --> 00:34:10.750 align:middle line:90% 00:34:10.750 --> 00:34:12.070 align:middle line:90% And then I'm going to validate. 00:34:12.070 --> 00:34:16.710 align:middle line:90% 00:34:16.710 --> 00:34:19.500 align:middle line:84% But they decide to further optimize 00:34:19.500 --> 00:34:22.590 align:middle line:90% this discriminative gene set. 00:34:22.590 --> 00:34:28.090 align:middle line:90% So this is a little complex. 00:34:28.090 --> 00:34:30.900 align:middle line:90% But what they do-- 00:34:30.900 --> 00:34:31.679 align:middle line:90% not that complex. 00:34:31.679 --> 00:34:36.179 align:middle line:84% So they take the 231 genes that they have here. 00:34:36.179 --> 00:34:39.239 align:middle line:84% And they rank order them by largest magnitude 00:34:39.239 --> 00:34:41.530 align:middle line:84% of their correlation coefficients. 00:34:41.530 --> 00:34:47.070 align:middle line:84% So gene 16 had a correlation coefficient of 0.9. 00:34:47.070 --> 00:34:54.840 align:middle line:84% And G 12 had a correlation coefficient of negative 0.85. 00:34:54.840 --> 00:34:58.800 align:middle line:84% And they rank these going down to G 231, which just made it 00:34:58.800 --> 00:35:05.320 align:middle line:84% with a correlation coefficient of 0.31, into this group. 00:35:05.320 --> 00:35:07.060 align:middle line:84% So now they're even looking deeper 00:35:07.060 --> 00:35:08.200 align:middle line:90% into this data structure. 00:35:08.200 --> 00:35:12.430 align:middle line:84% And they're looking for the very best genes that 00:35:12.430 --> 00:35:13.750 align:middle line:90% correlate with this outcome. 00:35:13.750 --> 00:35:19.320 align:middle line:90% 00:35:19.320 --> 00:35:21.270 align:middle line:84% An example of such a gene might be 00:35:21.270 --> 00:35:24.960 align:middle line:84% gene 16, which had expression levels that 00:35:24.960 --> 00:35:30.090 align:middle line:84% were 1,000 for these samples, and was 5 00:35:30.090 --> 00:35:32.010 align:middle line:90% for all of these samples. 00:35:32.010 --> 00:35:35.685 align:middle line:84% It was a perfect classifier, that one gene. 00:35:35.685 --> 00:35:38.570 align:middle line:90% 00:35:38.570 --> 00:35:41.840 align:middle line:84% So now they're going to really build 00:35:41.840 --> 00:35:44.120 align:middle line:84% a discriminative gene set through 00:35:44.120 --> 00:35:45.410 align:middle line:90% the following procedure. 00:35:45.410 --> 00:35:49.820 align:middle line:84% They're going to take the top five from this list 00:35:49.820 --> 00:35:57.410 align:middle line:84% and call that our discriminative gene set 00:35:57.410 --> 00:36:00.030 align:middle line:90% consisting of only five genes. 00:36:00.030 --> 00:36:02.900 align:middle line:84% OK, then they're going to go through this, 00:36:02.900 --> 00:36:06.170 align:middle line:84% build a prediction function, make a rule, 00:36:06.170 --> 00:36:12.530 align:middle line:84% and validate using leave one out cross validation here. 00:36:12.530 --> 00:36:15.810 align:middle line:90% 00:36:15.810 --> 00:36:17.730 align:middle line:84% Then they will add five more genes 00:36:17.730 --> 00:36:19.380 align:middle line:90% to this discriminative gene set. 00:36:19.380 --> 00:36:22.470 align:middle line:84% So they'll go down five more in this group 00:36:22.470 --> 00:36:28.800 align:middle line:84% and now have a discriminative gene set of 10 genes. 00:36:28.800 --> 00:36:31.470 align:middle line:84% They'll take that, build a prediction function, 00:36:31.470 --> 00:36:33.555 align:middle line:90% make a rule, and validate again. 00:36:33.555 --> 00:36:36.840 align:middle line:90% 00:36:36.840 --> 00:36:38.730 align:middle line:84% Now that they've done a second validation, 00:36:38.730 --> 00:36:42.930 align:middle line:84% they have a second accuracy number to look at. 00:36:42.930 --> 00:36:45.700 align:middle line:84% And they'll say did my accuracy get better or not? 00:36:45.700 --> 00:36:48.480 align:middle line:84% So the first time they did this with the top five genes, 00:36:48.480 --> 00:36:51.978 align:middle line:84% they found an accuracy in terms of prediction of, 00:36:51.978 --> 00:36:53.020 align:middle line:90% I don't know what it was. 00:36:53.020 --> 00:36:54.145 align:middle line:90% But we'll make up a number. 00:36:54.145 --> 00:36:59.152 align:middle line:84% Let's say 60% predictions were correct 00:36:59.152 --> 00:37:00.360 align:middle line:90% after they built their model. 00:37:00.360 --> 00:37:03.310 align:middle line:90% 00:37:03.310 --> 00:37:06.640 align:middle line:84% So then they did this with another of the larger gene set, 00:37:06.640 --> 00:37:09.360 align:middle line:84% including the next five best genes. 00:37:09.360 --> 00:37:11.580 align:middle line:84% And they found they had a prediction function 00:37:11.580 --> 00:37:13.600 align:middle line:90% of 70, that got them up to 70%. 00:37:13.600 --> 00:37:16.150 align:middle line:90% 00:37:16.150 --> 00:37:19.480 align:middle line:84% And they kept on going until their accuracy was 00:37:19.480 --> 00:37:22.270 align:middle line:84% the best it could possibly be and wasn't getting 00:37:22.270 --> 00:37:24.940 align:middle line:84% any better with the addition of more genes 00:37:24.940 --> 00:37:28.380 align:middle line:90% from the bottom of this list. 00:37:28.380 --> 00:37:30.840 align:middle line:90% There they stopped at 70 genes. 00:37:30.840 --> 00:37:33.955 align:middle line:90% 00:37:33.955 --> 00:37:35.830 align:middle line:84% So they constructed their discriminative gene 00:37:35.830 --> 00:37:44.190 align:middle line:84% set right here as a set of 70 genes 00:37:44.190 --> 00:37:50.790 align:middle line:84% based on this optimization of trying first the top five, 00:37:50.790 --> 00:37:53.550 align:middle line:84% building the model, testing the model, 00:37:53.550 --> 00:37:56.910 align:middle line:84% and looking at its accuracy, and improving the accuracy 00:37:56.910 --> 00:37:59.940 align:middle line:84% until it was as best as it could be. 00:37:59.940 --> 00:38:01.980 align:middle line:90% Yes-- 00:38:01.980 --> 00:38:04.938 align:middle line:90% AUDIENCE: Why five and not one? 00:38:04.938 --> 00:38:07.230 align:middle line:84% PROFESSOR: One would have taken them five times as long 00:38:07.230 --> 00:38:09.270 align:middle line:90% to get there. 00:38:09.270 --> 00:38:11.450 align:middle line:84% AUDIENCE: That's true but [INAUDIBLE] one 00:38:11.450 --> 00:38:15.030 align:middle line:90% is more relevant. 00:38:15.030 --> 00:38:17.670 align:middle line:84% PROFESSOR: Well, that's actually the point that people are 00:38:17.670 --> 00:38:21.610 align:middle line:90% trying to make in this field. 00:38:21.610 --> 00:38:24.300 align:middle line:84% That one is not relevant, in that 00:38:24.300 --> 00:38:27.090 align:middle line:84% the biomarkers that we're now looking for 00:38:27.090 --> 00:38:30.460 align:middle line:90% are groups of genes. 00:38:30.460 --> 00:38:33.360 align:middle line:84% AUDIENCE: I mean in the sense of implementing, I don't mean-- 00:38:33.360 --> 00:38:35.820 align:middle line:84% PROFESSOR: This can be implemented clinically. 00:38:35.820 --> 00:38:42.870 align:middle line:84% Once you set up to make an inkjet-synthesized or even 00:38:42.870 --> 00:38:46.960 align:middle line:84% robotically spotted array, you can do it for a few dollars. 00:38:46.960 --> 00:38:50.340 align:middle line:84% You can spot arrays for a few dollars, basically. 00:38:50.340 --> 00:38:53.400 align:middle line:90% This is not actually expensive. 00:38:53.400 --> 00:38:54.900 align:middle line:84% AUDIENCE: So basically they're going 00:38:54.900 --> 00:39:03.120 align:middle line:84% to find the minimal amount of these genes, 00:39:03.120 --> 00:39:08.210 align:middle line:84% and we give them the maximum amount of predictive value 00:39:08.210 --> 00:39:11.072 align:middle line:90% [INAUDIBLE]. 00:39:11.072 --> 00:39:12.780 align:middle line:84% PROFESSOR: Correct, a predictive accuracy 00:39:12.780 --> 00:39:16.020 align:middle line:84% in terms of going for this one prediction of good 00:39:16.020 --> 00:39:17.490 align:middle line:90% versus bad outcome. 00:39:17.490 --> 00:39:19.560 align:middle line:90% AUDIENCE: Is there any value-- 00:39:19.560 --> 00:39:25.620 align:middle line:84% so at each step before they got to the extra revision 00:39:25.620 --> 00:39:28.890 align:middle line:84% that you mentioned, [INAUDIBLE] close 00:39:28.890 --> 00:39:30.540 align:middle line:90% where most people would stop. 00:39:30.540 --> 00:39:36.270 align:middle line:84% Is there any value in there going outside of their 231 back 00:39:36.270 --> 00:39:38.070 align:middle line:90% into the 5,000. 00:39:38.070 --> 00:39:40.230 align:middle line:90% And just randomly-- 00:39:40.230 --> 00:39:43.170 align:middle line:84% PROFESSOR: I don't know how far up they went. 00:39:43.170 --> 00:39:46.660 align:middle line:90% I don't if they went past 231. 00:39:46.660 --> 00:39:51.000 align:middle line:84% I doubt it, that they then looked back at other genes. 00:39:51.000 --> 00:39:54.780 align:middle line:84% AUDIENCE: [INAUDIBLE] and they never [INAUDIBLE] at 231 genes, 00:39:54.780 --> 00:39:56.310 align:middle line:84% then they said OK, well, from there 00:39:56.310 --> 00:39:59.760 align:middle line:84% we've got certain candidates, which I think is quite fair. 00:39:59.760 --> 00:40:03.270 align:middle line:84% Once they've got their 70, is there 00:40:03.270 --> 00:40:06.330 align:middle line:84% any value in going back out to the larger data 00:40:06.330 --> 00:40:12.465 align:middle line:84% and seeing if we can call a few more things that maybe-- 00:40:12.465 --> 00:40:18.420 align:middle line:84% PROFESSOR: Maybe, but the main critical point of this approach 00:40:18.420 --> 00:40:22.740 align:middle line:84% is that this is overfitting, this is very, very serious 00:40:22.740 --> 00:40:27.210 align:middle line:90% overfitting of data. 00:40:27.210 --> 00:40:30.180 align:middle line:84% I don't think this is what you want to do. 00:40:30.180 --> 00:40:31.620 align:middle line:90% But people are still doing it. 00:40:31.620 --> 00:40:34.830 align:middle line:84% And I mean, it's fairly reasonable 00:40:34.830 --> 00:40:41.640 align:middle line:84% to make this gene set here and stop there. 00:40:41.640 --> 00:40:46.470 align:middle line:84% But to do a procedure that you then optimize this set, 00:40:46.470 --> 00:40:47.880 align:middle line:90% you stop with the 231. 00:40:47.880 --> 00:40:53.400 align:middle line:84% To then take that and to do repetitive cross validation 00:40:53.400 --> 00:40:59.400 align:middle line:84% on the very data that you're using to build the model, 00:40:59.400 --> 00:41:03.900 align:middle line:84% and to do it to select the genes, even in that way, 00:41:03.900 --> 00:41:09.510 align:middle line:84% is very serious overfitting of a data set. 00:41:09.510 --> 00:41:12.540 align:middle line:84% And I put up that fictitious graph yesterday 00:41:12.540 --> 00:41:15.420 align:middle line:84% about the points and overfitting it 00:41:15.420 --> 00:41:17.940 align:middle line:90% with the best possible curve. 00:41:17.940 --> 00:41:19.620 align:middle line:84% But it may not be the right one to make 00:41:19.620 --> 00:41:25.900 align:middle line:84% a prediction of a new point here because of how overfitting it 00:41:25.900 --> 00:41:26.400 align:middle line:90% is. 00:41:26.400 --> 00:41:29.830 align:middle line:84% So it works perfectly for the data set. 00:41:29.830 --> 00:41:32.040 align:middle line:84% And I think I brought up some of the pitfalls 00:41:32.040 --> 00:41:37.830 align:middle line:84% in terms of validation that's been done with microarray data. 00:41:37.830 --> 00:41:42.870 align:middle line:84% I think I passed out a copy of that paper mostly. 00:41:42.870 --> 00:41:46.770 align:middle line:84% AUDIENCE: Can't be overcome by cutting the data set in half 00:41:46.770 --> 00:41:49.950 align:middle line:84% and doing exactly what you just described, 00:41:49.950 --> 00:41:51.640 align:middle line:84% the method that you just described, 00:41:51.640 --> 00:41:53.300 align:middle line:90% and then taking that-- 00:41:53.300 --> 00:41:59.160 align:middle line:84% [INAUDIBLE] and then taking that other half a [INAUDIBLE] 00:41:59.160 --> 00:42:00.365 align:middle line:90% because you did describe-- 00:42:00.365 --> 00:42:01.740 align:middle line:84% PROFESSOR: You can build-- right, 00:42:01.740 --> 00:42:03.070 align:middle line:90% that's the right way to do it. 00:42:03.070 --> 00:42:08.690 align:middle line:84% So that's the training set and then the validation set. 00:42:08.690 --> 00:42:11.130 align:middle line:84% I mean even the real right way to do 00:42:11.130 --> 00:42:13.560 align:middle line:84% it is to make up any model you want 00:42:13.560 --> 00:42:17.520 align:middle line:84% and then prospectively test it on the next 100 patients. 00:42:17.520 --> 00:42:20.220 align:middle line:84% AUDIENCE: [INAUDIBLE] is that typically, the real performance 00:42:20.220 --> 00:42:26.700 align:middle line:84% is particularly in 2004, you have so few patients 00:42:26.700 --> 00:42:30.240 align:middle line:84% that you loathe to get away from many of your test set, 00:42:30.240 --> 00:42:32.160 align:middle line:84% because you might not have enough signal 00:42:32.160 --> 00:42:35.760 align:middle line:84% in your training set that you could [INAUDIBLE] for people. 00:42:35.760 --> 00:42:38.070 align:middle line:90% Otherwise, [INAUDIBLE]. 00:42:38.070 --> 00:42:41.850 align:middle line:90% 00:42:41.850 --> 00:42:43.860 align:middle line:84% That's why it's so amazing that [INAUDIBLE] 00:42:43.860 --> 00:42:48.372 align:middle line:84% can [? write ?] a paper besides a third of patients 00:42:48.372 --> 00:42:49.296 align:middle line:90% [INAUDIBLE]. 00:42:49.296 --> 00:42:55.598 align:middle line:90% 00:42:55.598 --> 00:42:57.640 align:middle line:84% PROFESSOR: Yeah, when these chips first came out, 00:42:57.640 --> 00:43:02.790 align:middle line:90% they were $2,000 a piece. 00:43:02.790 --> 00:43:05.640 align:middle line:84% Yeah, they're very-- they're even more than that, but-- 00:43:05.640 --> 00:43:09.000 align:middle line:90% AUDIENCE: Even more [INAUDIBLE] 00:43:09.000 --> 00:43:11.790 align:middle line:84% PROFESSOR: So these were very costly experiments 00:43:11.790 --> 00:43:14.700 align:middle line:90% to do back then. 00:43:14.700 --> 00:43:19.020 align:middle line:84% So this is their discriminative gene set. 00:43:19.020 --> 00:43:22.270 align:middle line:84% OK, now they have to choose a predictive function. 00:43:22.270 --> 00:43:23.860 align:middle line:84% So the predictive function, again, 00:43:23.860 --> 00:43:32.100 align:middle line:84% is if one was to give this model a new clinical specimen, 00:43:32.100 --> 00:43:36.700 align:middle line:84% how do we then make a prediction on the new clinical specimen? 00:43:36.700 --> 00:43:41.780 align:middle line:84% And so they used a fairly simple predictive function here, 00:43:41.780 --> 00:43:48.740 align:middle line:84% which is correlation coefficient to this ideal outcome. 00:43:48.740 --> 00:43:53.350 align:middle line:84% So if I gave you a patient tissue sample. 00:43:53.350 --> 00:43:55.390 align:middle line:84% And you did one of these microarray experiments 00:43:55.390 --> 00:44:06.160 align:middle line:84% and measured 25,000 different genes on it, 00:44:06.160 --> 00:44:08.860 align:middle line:84% then either you would use that in this model 00:44:08.860 --> 00:44:11.467 align:middle line:84% by pulling out the numbers from the 70 genes 00:44:11.467 --> 00:44:14.050 align:middle line:84% that were relevant-- that were part of the discriminative gene 00:44:14.050 --> 00:44:16.240 align:middle line:90% set. 00:44:16.240 --> 00:44:27.470 align:middle line:84% And then taking that number of 70 genes and-- 00:44:27.470 --> 00:44:35.280 align:middle line:84% let's see, sorry, I'm goofing up here. 00:44:35.280 --> 00:44:41.916 align:middle line:90% So-- improved-- 00:44:41.916 --> 00:44:54.610 align:middle line:90% 00:44:54.610 --> 00:44:58.600 align:middle line:84% The classifier predicted correctly the outcome. 00:44:58.600 --> 00:45:10.290 align:middle line:90% 00:45:10.290 --> 00:45:13.890 align:middle line:84% I'm sorry, I thought I understood 00:45:13.890 --> 00:45:15.660 align:middle line:84% their predictive function in this paper. 00:45:15.660 --> 00:45:18.560 align:middle line:90% 00:45:18.560 --> 00:45:19.560 align:middle line:90% Let's take a peek at it. 00:45:19.560 --> 00:45:27.630 align:middle line:90% 00:45:27.630 --> 00:45:29.745 align:middle line:84% The predictive function is a threshold rule. 00:45:29.745 --> 00:45:33.600 align:middle line:90% 00:45:33.600 --> 00:45:35.475 align:middle line:84% And that's evident like on figure 2. 00:45:35.475 --> 00:45:57.580 align:middle line:90% 00:45:57.580 --> 00:46:00.670 align:middle line:84% I believe they took an average number. 00:46:00.670 --> 00:46:04.250 align:middle line:90% So they took the average. 00:46:04.250 --> 00:46:12.620 align:middle line:84% So bad profile-- they took these profiles here and just average. 00:46:12.620 --> 00:46:16.360 align:middle line:84% So there's 34 in this bad group, bad prognosis group. 00:46:16.360 --> 00:46:22.630 align:middle line:84% And they took the 34 numbers here and averaged them together 00:46:22.630 --> 00:46:26.530 align:middle line:90% to get one number here. 00:46:26.530 --> 00:46:29.000 align:middle line:84% And then they did that for their other genes. 00:46:29.000 --> 00:46:32.570 align:middle line:84% So we're down to 70 genes right now. 00:46:32.570 --> 00:46:39.550 align:middle line:84% And so they have this ideal bad profile. 00:46:39.550 --> 00:46:45.538 align:middle line:90% 00:46:45.538 --> 00:46:47.330 align:middle line:84% And then pretty sure this is what they did. 00:46:47.330 --> 00:46:49.970 align:middle line:84% And they did the same thing for the good here. 00:46:49.970 --> 00:46:57.040 align:middle line:84% And they have their good profile, 00:46:57.040 --> 00:47:03.190 align:middle line:84% which is just an arithmetic mean of the expression levels. 00:47:03.190 --> 00:47:09.490 align:middle line:84% And now when the model is presented with a new row 00:47:09.490 --> 00:47:15.190 align:middle line:84% vector there of expression data for 70 genes for patients, 00:47:15.190 --> 00:47:21.070 align:middle line:84% and we'll call that new patient patient n, 00:47:21.070 --> 00:47:25.180 align:middle line:84% then they will calculate the correlation coefficient 00:47:25.180 --> 00:47:33.040 align:middle line:84% of patient n to maybe just the good vector. 00:47:33.040 --> 00:47:34.150 align:middle line:90% I guess you can't do it. 00:47:34.150 --> 00:47:37.698 align:middle line:84% Well, so I'm sorry I'm forgetting the details here 00:47:37.698 --> 00:47:38.240 align:middle line:90% a little bit. 00:47:38.240 --> 00:47:40.750 align:middle line:84% And I'm having trouble finding them right off here. 00:47:40.750 --> 00:47:43.840 align:middle line:84% But I'm pretty sure this is what they did. 00:47:43.840 --> 00:47:46.540 align:middle line:84% But so then which one did they correlate it with? 00:47:46.540 --> 00:47:47.560 align:middle line:90% The good one? 00:47:47.560 --> 00:47:50.080 align:middle line:90% 00:47:50.080 --> 00:47:50.843 align:middle line:90% Uh-huh. 00:47:50.843 --> 00:47:52.510 align:middle line:84% OK, yeah, which brings up another point, 00:47:52.510 --> 00:47:57.700 align:middle line:84% why not do it to the bad profile and correlate to that 00:47:57.700 --> 00:47:59.557 align:middle line:90% as a means of your algorithm? 00:47:59.557 --> 00:48:01.390 align:middle line:84% That might give you a very different answer. 00:48:01.390 --> 00:48:04.522 align:middle line:90% I'm not quite sure. 00:48:04.522 --> 00:48:05.980 align:middle line:84% But in any event-- so, OK, it seems 00:48:05.980 --> 00:48:07.702 align:middle line:84% like there is this choice here that they 00:48:07.702 --> 00:48:09.160 align:middle line:84% look at its correlation coefficient 00:48:09.160 --> 00:48:12.740 align:middle line:84% to the average good profile and got a number. 00:48:12.740 --> 00:48:17.248 align:middle line:84% So this will give you r, we'll give you a number, 0.4. 00:48:17.248 --> 00:48:18.790 align:middle line:84% And that's their prediction function. 00:48:18.790 --> 00:48:22.540 align:middle line:90% 00:48:22.540 --> 00:48:25.780 align:middle line:84% And then the last step in this model building is to make up 00:48:25.780 --> 00:48:31.180 align:middle line:84% a rule and say if there's-- and they used a threshold rule 00:48:31.180 --> 00:48:33.230 align:middle line:90% and a non-ambiguous classifier. 00:48:33.230 --> 00:48:35.590 align:middle line:84% So if it's above the threshold, it's in one group. 00:48:35.590 --> 00:48:39.070 align:middle line:84% If it's below the threshold, it's in another group. 00:48:39.070 --> 00:48:42.490 align:middle line:84% And they actually explored a couple 00:48:42.490 --> 00:48:46.210 align:middle line:84% of different possibilities for the threshold. 00:48:46.210 --> 00:48:52.360 align:middle line:84% And we talked about that Tuesday that this rule and varying 00:48:52.360 --> 00:48:54.610 align:middle line:84% the threshold is the classic trade 00:48:54.610 --> 00:48:59.170 align:middle line:84% off of sensitivity versus specificity here of a test. 00:48:59.170 --> 00:49:03.520 align:middle line:84% And they talk about that in this paper and show-- 00:49:03.520 --> 00:49:08.200 align:middle line:84% so on this diagram here, so how to divide 00:49:08.200 --> 00:49:10.810 align:middle line:84% the blacks and the whites to best advantage 00:49:10.810 --> 00:49:15.370 align:middle line:84% is the question here, the good versus the bad prognosis. 00:49:15.370 --> 00:49:18.490 align:middle line:84% And do we cut it here and get all of the whites 00:49:18.490 --> 00:49:20.980 align:middle line:84% with just a couple of blacks or do we cut a little bit more 00:49:20.980 --> 00:49:22.390 align:middle line:90% and get more whites? 00:49:22.390 --> 00:49:25.720 align:middle line:84% But we're trading off sensitivity and specificity 00:49:25.720 --> 00:49:26.960 align:middle line:90% here. 00:49:26.960 --> 00:49:30.805 align:middle line:84% And that's the and that's the rule that they use. 00:49:30.805 --> 00:49:34.870 align:middle line:90% 00:49:34.870 --> 00:49:39.560 align:middle line:84% So that's the model that they use. 00:49:39.560 --> 00:49:44.800 align:middle line:84% And then they actually do a couple-- 00:49:44.800 --> 00:49:51.550 align:middle line:84% in this paper, they do a couple of things for validation of it, 00:49:51.550 --> 00:49:54.710 align:middle line:90% for this step right here. 00:49:54.710 --> 00:49:59.770 align:middle line:84% And they do pick out a new test, a new validation 00:49:59.770 --> 00:50:01.930 align:middle line:84% set of 19 samples that they did not 00:50:01.930 --> 00:50:05.355 align:middle line:90% use to build the model here. 00:50:05.355 --> 00:50:05.980 align:middle line:90% And it's small. 00:50:05.980 --> 00:50:11.350 align:middle line:84% They don't say exactly how they constructed that group of 19. 00:50:11.350 --> 00:50:16.840 align:middle line:84% But they do use a separate group of 19 to then test this on. 00:50:16.840 --> 00:50:22.540 align:middle line:84% And when they do that-- so that's I guess page 534, 00:50:22.540 --> 00:50:24.550 align:middle line:90% the third paragraph-- 00:50:24.550 --> 00:50:26.570 align:middle line:84% to validate the prognosis classifier 00:50:26.570 --> 00:50:28.780 align:middle line:84% and additional independent set of primary tumors 00:50:28.780 --> 00:50:31.030 align:middle line:84% from 19 young lymph node negative patients 00:50:31.030 --> 00:50:33.150 align:middle line:90% were selected. 00:50:33.150 --> 00:50:35.970 align:middle line:84% There were seven patients who had the good outcome, 12 00:50:35.970 --> 00:50:38.130 align:middle line:90% who had the bad. 00:50:38.130 --> 00:50:44.196 align:middle line:84% And it resulted in two out of 19 incorrect classifications. 00:50:44.196 --> 00:50:47.410 align:middle line:90% 00:50:47.410 --> 00:50:52.210 align:middle line:84% I think both-- so that's how they got an accuracy number out 00:50:52.210 --> 00:50:57.080 align:middle line:84% of that, two of 19 as the accuracy. 00:50:57.080 --> 00:50:59.080 align:middle line:84% One of the important issues that you 00:50:59.080 --> 00:51:02.410 align:middle line:84% get into when you do something like that is the accuracy 00:51:02.410 --> 00:51:07.370 align:middle line:84% may not be the same for the good versus the bad patients. 00:51:07.370 --> 00:51:10.150 align:middle line:84% So you might have a classifier that 00:51:10.150 --> 00:51:12.250 align:middle line:84% predicts all of the good outcomes 00:51:12.250 --> 00:51:17.260 align:middle line:84% correctly, but is terrible at predicting the bad ones. 00:51:17.260 --> 00:51:19.750 align:middle line:84% Half the bad ones it says are good outcomes. 00:51:19.750 --> 00:51:23.570 align:middle line:84% And that's not very helpful to you. 00:51:23.570 --> 00:51:26.770 align:middle line:84% And so I think the data is in here. 00:51:26.770 --> 00:51:29.470 align:middle line:84% And I think it's mentioned in the pitfall article 00:51:29.470 --> 00:51:31.330 align:middle line:90% that I handed out. 00:51:31.330 --> 00:51:35.020 align:middle line:84% That teases that out a little bit more 00:51:35.020 --> 00:51:39.640 align:middle line:84% as to the importance of saying whether the predictor is 00:51:39.640 --> 00:51:43.600 align:middle line:84% working equally well on the good and the bad cases 00:51:43.600 --> 00:51:44.980 align:middle line:90% that are presented to it. 00:51:44.980 --> 00:51:49.200 align:middle line:90% 00:51:49.200 --> 00:51:53.840 align:middle line:84% So I think that's essentially this paper. 00:51:53.840 --> 00:51:56.300 align:middle line:84% There was a follow up for this, which 00:51:56.300 --> 00:51:58.310 align:middle line:90% was in the New England Journal. 00:51:58.310 --> 00:52:05.510 align:middle line:84% It's on the website, which is even more focused on prediction 00:52:05.510 --> 00:52:07.940 align:middle line:84% as survival, gene expression signature as a predictor 00:52:07.940 --> 00:52:10.820 align:middle line:90% of survival in breast cancer. 00:52:10.820 --> 00:52:13.550 align:middle line:84% They used a larger data set here. 00:52:13.550 --> 00:52:16.880 align:middle line:84% But they use the same model here. 00:52:16.880 --> 00:52:20.330 align:middle line:84% They did validation on the data set. 00:52:20.330 --> 00:52:24.420 align:middle line:84% But they didn't do the full validation procedure. 00:52:24.420 --> 00:52:27.200 align:middle line:84% So we talked about that, how a lot of validation 00:52:27.200 --> 00:52:29.870 align:middle line:90% is done through leave one out. 00:52:29.870 --> 00:52:33.560 align:middle line:84% So leave one sample out, build the model, 00:52:33.560 --> 00:52:35.570 align:middle line:90% and then test that sample. 00:52:35.570 --> 00:52:39.530 align:middle line:84% And then do it for each of the samples 78 times or 200 times, 00:52:39.530 --> 00:52:40.230 align:middle line:90% whatever. 00:52:40.230 --> 00:52:42.800 align:middle line:84% And see how accurate your predictions are each time. 00:52:42.800 --> 00:52:47.030 align:middle line:84% But it's important when you do that to go back and rebuild 00:52:47.030 --> 00:52:51.200 align:middle line:84% the choice of the discriminative gene set each time. 00:52:51.200 --> 00:52:56.180 align:middle line:84% And this actually is in the New England Journal of Medicine. 00:52:56.180 --> 00:52:58.130 align:middle line:90% They do actually report both. 00:52:58.130 --> 00:53:01.130 align:middle line:84% They're aware of the need to do this. 00:53:01.130 --> 00:53:07.490 align:middle line:84% And the numbers I think were 24% versus 41% or something, 27%. 00:53:07.490 --> 00:53:10.910 align:middle line:90% So when they do the error-- 00:53:10.910 --> 00:53:16.040 align:middle line:84% when they do the validation based on not doing this step, 00:53:16.040 --> 00:53:18.200 align:middle line:84% here we're choosing the gene set, 00:53:18.200 --> 00:53:23.100 align:middle line:84% there was a 27% error rate there. 00:53:23.100 --> 00:53:25.460 align:middle line:84% But when they did it repeating this gene step, 00:53:25.460 --> 00:53:30.530 align:middle line:84% there was a 41% error rate, which 00:53:30.530 --> 00:53:33.590 align:middle line:84% is getting close to chance, flipping a coin. 00:53:33.590 --> 00:53:36.440 align:middle line:90% 00:53:36.440 --> 00:53:40.310 align:middle line:84% But in any event, so really on the basis of this, 00:53:40.310 --> 00:53:43.003 align:middle line:84% the Netherlands Cancer Institute that 00:53:43.003 --> 00:53:44.420 align:middle line:84% did this said that they were going 00:53:44.420 --> 00:53:48.980 align:middle line:84% to start using this 70 gene set to make clinical decisions. 00:53:48.980 --> 00:53:51.230 align:middle line:84% They're going to take new patients with breast cancer. 00:53:51.230 --> 00:53:52.820 align:middle line:90% Do a microarray experiment. 00:53:52.820 --> 00:53:54.320 align:middle line:90% Measure the 70 genes. 00:53:54.320 --> 00:53:57.260 align:middle line:84% Use this model, and make a prediction 00:53:57.260 --> 00:54:00.770 align:middle line:84% about good or bad outcome, and tailor their treatment 00:54:00.770 --> 00:54:02.360 align:middle line:90% according to that. 00:54:02.360 --> 00:54:03.460 align:middle line:90% Yes-- 00:54:03.460 --> 00:54:05.630 align:middle line:84% AUDIENCE: I actually had a question related 00:54:05.630 --> 00:54:08.340 align:middle line:90% to this and more [INAUDIBLE]. 00:54:08.340 --> 00:54:11.570 align:middle line:84% So the lab puts out a study like this, 00:54:11.570 --> 00:54:13.850 align:middle line:84% says here's 70 genes that we think 00:54:13.850 --> 00:54:21.850 align:middle line:84% are good indicator of [INAUDIBLE] [? By the logic, ?] 00:54:21.850 --> 00:54:24.940 align:middle line:84% I want to validate these results, right? 00:54:24.940 --> 00:54:31.480 align:middle line:84% Now in gathering and collecting this data, 00:54:31.480 --> 00:54:33.040 align:middle line:84% like this ridiculous study, they're 00:54:33.040 --> 00:54:35.780 align:middle line:84% looking at patients over five years. 00:54:35.780 --> 00:54:38.530 align:middle line:84% You do a microarray analysis on a patient 00:54:38.530 --> 00:54:44.320 align:middle line:84% and they pop up positive or whatever for this indicator, 00:54:44.320 --> 00:54:47.440 align:middle line:84% now you've got people making clinical decisions on this. 00:54:47.440 --> 00:54:51.180 align:middle line:84% And it seems like the validation method is 00:54:51.180 --> 00:54:54.940 align:middle line:90% an [INAUDIBLE] withheld data. 00:54:54.940 --> 00:55:01.690 align:middle line:84% They need to have a bad prognosis or a bad outcome 00:55:01.690 --> 00:55:07.120 align:middle line:84% in order to say that these candidates-- for the patient-- 00:55:07.120 --> 00:55:10.810 align:middle line:84% if these 70 genes actually do have a negative effect, 00:55:10.810 --> 00:55:14.810 align:middle line:84% or indicate [INAUDIBLE] that effect has to be there. 00:55:14.810 --> 00:55:18.990 align:middle line:84% So how does that how does a clinician 00:55:18.990 --> 00:55:23.910 align:middle line:84% or a clinical researcher, who kind of sees patients and does 00:55:23.910 --> 00:55:25.890 align:middle line:90% this, how do they balance that? 00:55:25.890 --> 00:55:32.302 align:middle line:84% I mean what would your suggestion or advice be? 00:55:32.302 --> 00:55:33.870 align:middle line:84% Does that makes any sense at all? 00:55:33.870 --> 00:55:36.510 align:middle line:84% PROFESSOR: Well you're talking about how do you do research 00:55:36.510 --> 00:55:37.957 align:middle line:90% on patients this area? 00:55:37.957 --> 00:55:38.790 align:middle line:90% [INTERPOSING VOICES] 00:55:38.790 --> 00:55:39.870 align:middle line:84% AUDIENCE: [INAUDIBLE] how you [INAUDIBLE] patients 00:55:39.870 --> 00:55:40.890 align:middle line:90% clinically? 00:55:40.890 --> 00:55:45.050 align:middle line:84% AUDIENCE: Well, [INAUDIBLE] so we have this result, 00:55:45.050 --> 00:55:46.758 align:middle line:90% people need to validate it. 00:55:46.758 --> 00:55:48.300 align:middle line:84% In the process of validating, they're 00:55:48.300 --> 00:55:54.300 align:middle line:84% going to find people who have this particular pattern, right? 00:55:54.300 --> 00:55:58.110 align:middle line:84% And they need to follow those people over a certain time 00:55:58.110 --> 00:55:59.040 align:middle line:90% course. 00:55:59.040 --> 00:56:06.120 align:middle line:84% And in order to validate their own [INAUDIBLE] 00:56:06.120 --> 00:56:10.140 align:middle line:84% essentially, if none of the people who have these 70 00:56:10.140 --> 00:56:13.710 align:middle line:84% genes, or expression profiles or whatever, 00:56:13.710 --> 00:56:17.277 align:middle line:84% develop breast cancer, then the model's kind of [INAUDIBLE] 00:56:17.277 --> 00:56:18.360 align:middle line:90% back to the drawing board. 00:56:18.360 --> 00:56:21.520 align:middle line:84% So essentially, and I hate to say it this way-- 00:56:21.520 --> 00:56:23.700 align:middle line:84% PROFESSOR: No, say what you're thinking. 00:56:23.700 --> 00:56:25.888 align:middle line:84% AUDIENCE: They need these people, in order 00:56:25.888 --> 00:56:27.930 align:middle line:84% to validate the model, they need them to progress 00:56:27.930 --> 00:56:30.420 align:middle line:90% right to disease state. 00:56:30.420 --> 00:56:33.150 align:middle line:84% And yet at the same time, they're, 00:56:33.150 --> 00:56:37.800 align:middle line:84% as clinicians, so putting aside the researcher part, 00:56:37.800 --> 00:56:40.510 align:middle line:84% as clinicians, they need to treat these patients. 00:56:40.510 --> 00:56:43.560 align:middle line:84% And if there's a strong indication from prior studies 00:56:43.560 --> 00:56:46.870 align:middle line:84% that the chances are you're going to get breast cancer 00:56:46.870 --> 00:56:48.120 align:middle line:90% and it's going to metastasize. 00:56:48.120 --> 00:56:50.930 align:middle line:84% And it's going to get really bad for you. 00:56:50.930 --> 00:56:54.120 align:middle line:84% They can't just not treat these people aggressively. 00:56:54.120 --> 00:56:56.550 align:middle line:84% Or that treatment is going to affect that outcome. 00:56:56.550 --> 00:56:58.410 align:middle line:84% PROFESSOR: Well, people-- yeah, yeah, 00:56:58.410 --> 00:57:02.430 align:middle line:84% so I don't think anybody's doing-- 00:57:02.430 --> 00:57:05.170 align:middle line:84% AUDIENCE: You think about the Tuskegee-like experiments. 00:57:05.170 --> 00:57:09.670 align:middle line:84% PROFESSOR: Yes, so nobody's denying treatment 00:57:09.670 --> 00:57:13.367 align:middle line:84% or withholding best known treatments to patients. 00:57:13.367 --> 00:57:14.950 align:middle line:84% AUDIENCE: So how do you calculate that 00:57:14.950 --> 00:57:18.470 align:middle line:84% into the validation of the model? 00:57:18.470 --> 00:57:20.710 align:middle line:90% That if you get-- you can't-- 00:57:20.710 --> 00:57:22.150 align:middle line:84% because you can't say, well, they 00:57:22.150 --> 00:57:24.350 align:middle line:84% didn't have metastasis because of my treatment. 00:57:24.350 --> 00:57:25.870 align:middle line:84% PROFESSOR: There are a few issues. 00:57:25.870 --> 00:57:28.660 align:middle line:84% And you actually-- so in this New England Journal of Medicine 00:57:28.660 --> 00:57:30.850 align:middle line:90% paper-- 00:57:30.850 --> 00:57:32.500 align:middle line:84% here, I did not pass that one out. 00:57:32.500 --> 00:57:33.820 align:middle line:90% It's on the website. 00:57:33.820 --> 00:57:36.250 align:middle line:84% What they did is they used this model in a larger 00:57:36.250 --> 00:57:38.620 align:middle line:84% group of patients and they then compared 00:57:38.620 --> 00:57:43.310 align:middle line:84% predictions of this model to best clinical predictors. 00:57:43.310 --> 00:57:47.830 align:middle line:84% So for example, there's other predictive scales. 00:57:47.830 --> 00:57:51.400 align:middle line:84% And they used one called the St. Gallen scale, which predicts 00:57:51.400 --> 00:57:52.930 align:middle line:90% outcome in breast cancer. 00:57:52.930 --> 00:57:56.900 align:middle line:84% And they said how does our model compare to the St. Gallen 00:57:56.900 --> 00:57:57.400 align:middle line:90% scale. 00:57:57.400 --> 00:57:59.025 align:middle line:84% If we take all of these patients and we 00:57:59.025 --> 00:58:03.700 align:middle line:84% see what their St. Gallen number was and how they did. 00:58:03.700 --> 00:58:07.960 align:middle line:84% And we take our predictions and how they did, which is better? 00:58:07.960 --> 00:58:11.380 align:middle line:84% And they concluded that their method was better 00:58:11.380 --> 00:58:15.070 align:middle line:84% than the St. Gallen method of making predictions. 00:58:15.070 --> 00:58:18.100 align:middle line:84% And that was really one of the principal-- the principal 00:58:18.100 --> 00:58:20.680 align:middle line:84% justification for saying we have a better method 00:58:20.680 --> 00:58:23.020 align:middle line:84% to make predictions about how patients are going to do. 00:58:23.020 --> 00:58:25.630 align:middle line:84% And we're going to start using this 00:58:25.630 --> 00:58:28.060 align:middle line:84% in the Netherlands at the Cancer Institute 00:58:28.060 --> 00:58:31.840 align:middle line:84% to make our decisions, rather than St. Gallen. 00:58:31.840 --> 00:58:36.370 align:middle line:84% And what you point out is actually a big flaw in research 00:58:36.370 --> 00:58:40.510 align:middle line:84% that does this, because patients had their St. Gallen 00:58:40.510 --> 00:58:42.940 align:middle line:84% scale determined when they were diagnosed. 00:58:42.940 --> 00:58:45.940 align:middle line:84% And depending on what their clinicians thought, 00:58:45.940 --> 00:58:50.300 align:middle line:84% treated them accordingly, in order to achieve best outcome. 00:58:50.300 --> 00:58:52.120 align:middle line:84% So if they were in a poor outcome group 00:58:52.120 --> 00:58:55.570 align:middle line:84% to begin with, based on St. Gallen criteria, 00:58:55.570 --> 00:58:57.370 align:middle line:84% they got treated more aggressively 00:58:57.370 --> 00:59:00.370 align:middle line:84% to try and make their outcome better. 00:59:00.370 --> 00:59:02.260 align:middle line:84% So actually, the goal of the clinician 00:59:02.260 --> 00:59:06.220 align:middle line:84% is for the St. Gallen scale not to be 00:59:06.220 --> 00:59:10.060 align:middle line:90% right on this group of patients. 00:59:10.060 --> 00:59:13.180 align:middle line:84% The interventions that took place on these patients, 00:59:13.180 --> 00:59:16.540 align:middle line:84% because it was not a perspective controlled 00:59:16.540 --> 00:59:19.690 align:middle line:84% study, the interventions that took place were specifically 00:59:19.690 --> 00:59:24.310 align:middle line:84% designed to mitigate the poor prognosis that the St. Gallen 00:59:24.310 --> 00:59:27.500 align:middle line:84% criteria indicated on these patients. 00:59:27.500 --> 00:59:31.420 align:middle line:84% And so it may not be a surprise that the St. Gallen predictor 00:59:31.420 --> 00:59:35.620 align:middle line:84% was not all that accurate later on, because people were 00:59:35.620 --> 00:59:40.690 align:middle line:84% intervening actively with the results of that predictor 00:59:40.690 --> 00:59:42.190 align:middle line:90% to change outcome. 00:59:42.190 --> 00:59:45.850 align:middle line:84% And that wasn't happening with gene expression profiles. 00:59:45.850 --> 00:59:47.560 align:middle line:84% Nobody looked at the gene expression 00:59:47.560 --> 00:59:50.140 align:middle line:84% profile of these patients 10 years ago 00:59:50.140 --> 00:59:51.700 align:middle line:84% and then decided how they were going 00:59:51.700 --> 00:59:54.520 align:middle line:84% to treat them over the next five years 00:59:54.520 --> 00:59:56.060 align:middle line:90% to make their disease better. 00:59:56.060 --> 00:59:58.900 align:middle line:84% AUDIENCE: So is it just that because for right 00:59:58.900 --> 01:00:03.700 align:middle line:84% now the turnaround on the analysis of the gene 01:00:03.700 --> 01:00:10.660 align:middle line:84% expression profiles is long enough that the St. 01:00:10.660 --> 01:00:15.100 align:middle line:84% Gallen approach of aggressive treatment or whatever, 01:00:15.100 --> 01:00:17.530 align:middle line:84% that turnaround is long enough that you'll 01:00:17.530 --> 01:00:19.900 align:middle line:84% have your answer and your validation 01:00:19.900 --> 01:00:22.660 align:middle line:84% before clinical decision start getting made, 01:00:22.660 --> 01:00:28.268 align:middle line:84% repeat experiments are done to cross validate-- 01:00:28.268 --> 01:00:29.560 align:middle line:90% PROFESSOR: They'll never be a-- 01:00:29.560 --> 01:00:33.490 align:middle line:84% I mean to get a new drug approved in this country 01:00:33.490 --> 01:00:38.920 align:middle line:84% for labeling requires that you take a group of patients 01:00:38.920 --> 01:00:42.130 align:middle line:84% with a disease, and you treat some of them with the drug 01:00:42.130 --> 01:00:45.850 align:middle line:84% and some of them with a placebo, and you see which works. 01:00:45.850 --> 01:00:49.960 align:middle line:84% To ever really know whether prediction models are going 01:00:49.960 --> 01:00:54.880 align:middle line:84% to work accurately, one needs to do the same thing, 01:00:54.880 --> 01:00:59.470 align:middle line:84% to take patients prospectively to apply prediction models 01:00:59.470 --> 01:01:02.110 align:middle line:84% to them ahead of time, and to see 01:01:02.110 --> 01:01:06.670 align:middle line:84% what happens with them, without an intervention that's 01:01:06.670 --> 01:01:10.960 align:middle line:84% based on the results of the prediction. 01:01:10.960 --> 01:01:15.910 align:middle line:84% That's why I find a paper like this perspective here kind 01:01:15.910 --> 01:01:17.650 align:middle line:90% of scary. 01:01:17.650 --> 01:01:19.420 align:middle line:84% The microarrays in clinical investigations 01:01:19.420 --> 01:01:21.580 align:middle line:84% from today's journal, where the authors 01:01:21.580 --> 01:01:24.940 align:middle line:84% envision that in the future clinical investigations will 01:01:24.940 --> 01:01:28.990 align:middle line:84% consist of small trials with a high density of data, 01:01:28.990 --> 01:01:31.180 align:middle line:84% precise patient stratification according 01:01:31.180 --> 01:01:33.250 align:middle line:84% to expression profile, and highly 01:01:33.250 --> 01:01:36.290 align:middle line:84% tailored analysis of microarray data, 01:01:36.290 --> 01:01:39.520 align:middle line:84% otherwise known as massive overfitting 01:01:39.520 --> 01:01:46.490 align:middle line:84% of tons of expression data in a small number of patients here. 01:01:46.490 --> 01:01:48.400 align:middle line:84% And that's never going to tell you 01:01:48.400 --> 01:01:51.868 align:middle line:84% whether your predictions are correct. 01:01:51.868 --> 01:01:53.410 align:middle line:84% And if you're going to make decisions 01:01:53.410 --> 01:01:57.460 align:middle line:84% about treatment or predictions based on that, 01:01:57.460 --> 01:01:59.650 align:middle line:90% you won't be accurate. 01:01:59.650 --> 01:02:03.790 align:middle line:84% I don't believe that there'll be sufficient accuracy from small. 01:02:03.790 --> 01:02:06.130 align:middle line:84% I don't think that trials need to get smaller. 01:02:06.130 --> 01:02:08.440 align:middle line:84% And trials can be made smaller because we now 01:02:08.440 --> 01:02:10.990 align:middle line:90% have microarray data. 01:02:10.990 --> 01:02:13.570 align:middle line:84% In fact, trials need to be probably larger, 01:02:13.570 --> 01:02:17.995 align:middle line:84% because you have that much more undetermined data set. 01:02:17.995 --> 01:02:21.220 align:middle line:90% 01:02:21.220 --> 01:02:30.532 align:middle line:84% AUDIENCE: If you're [INAUDIBLE] the reproducibility [INAUDIBLE] 01:02:30.532 --> 01:02:32.740 align:middle line:84% PROFESSOR: Well, so the Netherlands Cancer Institute, 01:02:32.740 --> 01:02:36.820 align:middle line:84% they patented their set of genes as a predictor 01:02:36.820 --> 01:02:38.420 align:middle line:90% of breast cancer. 01:02:38.420 --> 01:02:43.400 align:middle line:84% So, hoping-- right, well, hoping that they'll have the patent. 01:02:43.400 --> 01:02:45.760 align:middle line:84% And if they show that their stuff works, 01:02:45.760 --> 01:02:48.160 align:middle line:84% then people will come to them and they'll 01:02:48.160 --> 01:02:54.570 align:middle line:84% start wholesaling their type of arrays and their approach. 01:02:54.570 --> 01:02:55.480 align:middle line:90% Do you know that-- 01:02:55.480 --> 01:02:57.130 align:middle line:90% I haven't seen anything, Zack. 01:02:57.130 --> 01:02:59.410 align:middle line:84% I mean there was an article that they 01:02:59.410 --> 01:03:02.460 align:middle line:84% were going to start doing this in the summer of 2003. 01:03:02.460 --> 01:03:03.547 align:middle line:90% And it was in Nature. 01:03:03.547 --> 01:03:04.880 align:middle line:90% We're going to start doing this. 01:03:04.880 --> 01:03:06.680 align:middle line:90% And silence. 01:03:06.680 --> 01:03:07.930 align:middle line:90% Did they come to their senses? 01:03:07.930 --> 01:03:09.190 align:middle line:84% AUDIENCE: That's what I'm thinking must have happened. 01:03:09.190 --> 01:03:11.145 align:middle line:84% I think that probably somebody who actually 01:03:11.145 --> 01:03:12.520 align:middle line:84% knew this [INAUDIBLE] say, do you 01:03:12.520 --> 01:03:15.250 align:middle line:84% want to screw our patients over by doing this? 01:03:15.250 --> 01:03:16.630 align:middle line:90% I don't know that for a fact. 01:03:16.630 --> 01:03:17.338 align:middle line:90% PROFESSOR: Right. 01:03:17.338 --> 01:03:19.900 align:middle line:90% 01:03:19.900 --> 01:03:25.270 align:middle line:84% AUDIENCE: Yeah, what's an article [INAUDIBLE] 01:03:25.270 --> 01:03:27.160 align:middle line:90% AUDIENCE: Oh, yeah. 01:03:27.160 --> 01:03:31.220 align:middle line:84% AUDIENCE: I can't remember how many marketing [INAUDIBLE] 01:03:31.220 --> 01:03:34.163 align:middle line:90% [INTERPOSING VOICES] 01:03:34.163 --> 01:03:35.580 align:middle line:84% AUDIENCE: There's two issues here. 01:03:35.580 --> 01:03:41.150 align:middle line:84% One is whether FDA approves these gene chips in general. 01:03:41.150 --> 01:03:43.460 align:middle line:84% We'll get back to that in a second. 01:03:43.460 --> 01:03:47.870 align:middle line:84% Narrowly, whether you use a [INAUDIBLE] formula 01:03:47.870 --> 01:03:52.560 align:middle line:84% to prescribe these kind of [INAUDIBLE] on any platform. 01:03:52.560 --> 01:03:56.900 align:middle line:84% And I think as a responsible clinician, most of us 01:03:56.900 --> 01:03:59.630 align:middle line:84% would not right now necessarily stratify our patients 01:03:59.630 --> 01:04:01.730 align:middle line:90% based on the prognosis alone. 01:04:01.730 --> 01:04:02.600 align:middle line:90% You agree with that? 01:04:02.600 --> 01:04:03.225 align:middle line:90% PROFESSOR: Yes. 01:04:03.225 --> 01:04:05.540 align:middle line:90% 01:04:05.540 --> 01:04:07.940 align:middle line:84% AUDIENCE: It's been done kind of for cancer. 01:04:07.940 --> 01:04:09.697 align:middle line:84% And some oncology trials, it's kind of 01:04:09.697 --> 01:04:11.030 align:middle line:90% scary to me that have been done. 01:04:11.030 --> 01:04:13.790 align:middle line:90% 01:04:13.790 --> 01:04:16.120 align:middle line:90% [INAUDIBLE] been conducted. 01:04:16.120 --> 01:04:18.820 align:middle line:84% The way cancer is done in this country 01:04:18.820 --> 01:04:23.260 align:middle line:84% is within impactive centers and cooperative oncology groups 01:04:23.260 --> 01:04:26.530 align:middle line:84% where large numbers of individuals 01:04:26.530 --> 01:04:28.990 align:middle line:90% are treated in protocols. 01:04:28.990 --> 01:04:31.150 align:middle line:84% And those are now being stratified 01:04:31.150 --> 01:04:34.630 align:middle line:90% by [INAUDIBLE] for some studies. 01:04:34.630 --> 01:04:38.140 align:middle line:84% I can't say that I fully agree with that. 01:04:38.140 --> 01:04:41.530 align:middle line:84% [INAUDIBLE] actually using the measurement [? technology. ?] 01:04:41.530 --> 01:04:46.060 align:middle line:84% Roche Diagnostics made the headlines about two months ago 01:04:46.060 --> 01:04:49.990 align:middle line:84% when they tried to get a chip, not for expression, 01:04:49.990 --> 01:04:55.750 align:middle line:84% but for genotyping mutations in the P450 of the proteins 01:04:55.750 --> 01:04:59.590 align:middle line:84% that clear idea of toxic chemicals out 01:04:59.590 --> 01:05:01.930 align:middle line:90% of our body in the liver. 01:05:01.930 --> 01:05:04.720 align:middle line:84% They tried [INAUDIBLE] it's very important for [? promo ?] 01:05:04.720 --> 01:05:07.930 align:middle line:84% genomics because it'll tell you how fast growth appeared. 01:05:07.930 --> 01:05:14.680 align:middle line:84% Roche tried to get it simply through a waiver that 01:05:14.680 --> 01:05:18.020 align:middle line:84% essentially said this is just a method of measurement. 01:05:18.020 --> 01:05:21.200 align:middle line:84% We don't really have to get specific FDA approval. 01:05:21.200 --> 01:05:23.230 align:middle line:84% The FDA said no, no, no, hold it. 01:05:23.230 --> 01:05:25.700 align:middle line:84% Let's go through the formal approval process. 01:05:25.700 --> 01:05:27.520 align:middle line:84% So what Cecily is talking about is 01:05:27.520 --> 01:05:31.420 align:middle line:84% that it announced that another [INAUDIBLE] 01:05:31.420 --> 01:05:34.010 align:middle line:84% will see if they can repeat it the same way. 01:05:34.010 --> 01:05:37.070 align:middle line:90% And this is-- 01:05:37.070 --> 01:05:38.320 align:middle line:90% It's actually an act of valor. 01:05:38.320 --> 01:05:40.615 align:middle line:84% As some of you are involved [? in HSP ?] should 01:05:40.615 --> 01:05:43.540 align:middle line:84% know that HSP is actually very much involved in help 01:05:43.540 --> 01:05:46.930 align:middle line:84% the FDA figure out exactly how to look 01:05:46.930 --> 01:05:50.710 align:middle line:84% at this, because [INAUDIBLE] the FDA has only now developed 01:05:50.710 --> 01:05:54.400 align:middle line:84% the minimum standards of data submission 01:05:54.400 --> 01:05:56.770 align:middle line:84% from the pharmaceutical companies 01:05:56.770 --> 01:05:58.180 align:middle line:90% on the microarray data. 01:05:58.180 --> 01:06:00.550 align:middle line:84% They have no guidelines whatsoever 01:06:00.550 --> 01:06:04.460 align:middle line:84% on the standard identification patients. 01:06:04.460 --> 01:06:07.210 align:middle line:84% So I think there's going to be several hops, 01:06:07.210 --> 01:06:13.000 align:middle line:84% skips, and jumps to get these kind of measurements translated 01:06:13.000 --> 01:06:23.380 align:middle line:84% from research descriptions to clinical measures. 01:06:23.380 --> 01:06:26.740 align:middle line:84% It's still going to be used, by the way, to give you a-- 01:06:26.740 --> 01:06:29.350 align:middle line:84% it still can be used in a Medical Center 01:06:29.350 --> 01:06:33.460 align:middle line:84% without FDA approval as long as they're not sold. 01:06:33.460 --> 01:06:37.090 align:middle line:84% So for instance [INAUDIBLE] the genetics and genomics 01:06:37.090 --> 01:06:43.810 align:middle line:84% has contracted with Affymetrics for a sequencing 01:06:43.810 --> 01:06:46.900 align:middle line:84% chip for genes involved in hearing loss 01:06:46.900 --> 01:06:49.300 align:middle line:90% and cardiomyopathy. 01:06:49.300 --> 01:06:53.860 align:middle line:84% But [INAUDIBLE] is not selling this for general use. 01:06:53.860 --> 01:07:00.060 align:middle line:84% It's been used internally by its clinicians. 01:07:00.060 --> 01:07:04.080 align:middle line:84% So it's a very tricky process that that's 01:07:04.080 --> 01:07:06.900 align:middle line:90% been where regulation-- 01:07:06.900 --> 01:07:09.480 align:middle line:84% it's genomics in fact [INAUDIBLE] 01:07:09.480 --> 01:07:13.774 align:middle line:84% will be giving a lecture about that [? later on. ?] 01:07:13.774 --> 01:07:15.607 align:middle line:84% PROFESSOR: Is Margulies giving that lecture? 01:07:15.607 --> 01:07:16.190 align:middle line:90% AUDIENCE: Yes. 01:07:16.190 --> 01:07:19.090 align:middle line:90% 01:07:19.090 --> 01:07:23.070 align:middle line:84% PROFESSOR: OK, that's all for today.