WEBVTT 00:00:00.000 --> 00:00:03.390 align:middle line:84% PETER PARK: For today, I'll just talk a little bit more 00:00:03.390 --> 00:00:08.189 align:middle line:84% generally at the beginning about a few observations 00:00:08.189 --> 00:00:10.530 align:middle line:90% that I've had. 00:00:10.530 --> 00:00:14.640 align:middle line:84% Perhaps a little bit about reliability microarray studies. 00:00:14.640 --> 00:00:17.370 align:middle line:84% I'll talk about classification problem in general. 00:00:17.370 --> 00:00:19.200 align:middle line:84% And then, I'll talk more about phenotypes. 00:00:19.200 --> 00:00:25.020 align:middle line:84% And then review some literature that are well cited. 00:00:25.020 --> 00:00:28.530 align:middle line:84% So I bet you've had some exposure to that, right? 00:00:28.530 --> 00:00:31.500 align:middle line:84% Did anyone-- maybe Marco, or Zack, 00:00:31.500 --> 00:00:35.040 align:middle line:90% talk about lymphoma studies? 00:00:35.040 --> 00:00:38.425 align:middle line:84% Do you remember any other papers that you've covered? 00:00:38.425 --> 00:00:38.925 align:middle line:90% Yeah? 00:00:38.925 --> 00:00:40.144 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:00:40.144 --> 00:00:44.850 align:middle line:90% 00:00:44.850 --> 00:00:48.150 align:middle line:84% PETER PARK: Anything else you remember? 00:00:48.150 --> 00:00:50.190 align:middle line:90% All right, so-- 00:00:50.190 --> 00:00:52.267 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:00:52.267 --> 00:00:53.850 align:middle line:84% PETER PARK: Yeah, right, so maybe this 00:00:53.850 --> 00:00:55.020 align:middle line:90% stuff that I have is new. 00:00:55.020 --> 00:00:57.240 align:middle line:90% OK, good. 00:00:57.240 --> 00:00:59.760 align:middle line:90% So you probably know all this. 00:00:59.760 --> 00:01:04.200 align:middle line:84% It's actually-- so the cDNA arrays are cheaper. 00:01:04.200 --> 00:01:10.680 align:middle line:84% So they are used a lot of biology labs. 00:01:10.680 --> 00:01:14.400 align:middle line:84% The ones that I work with, for example, in the new building, 00:01:14.400 --> 00:01:15.690 align:middle line:90% they have their own libraries. 00:01:15.690 --> 00:01:18.630 align:middle line:90% They print their own. 00:01:18.630 --> 00:01:21.060 align:middle line:90% Affy arrays are-- 00:01:21.060 --> 00:01:23.730 align:middle line:84% I think the consensus that the Affy arrays are probably 00:01:23.730 --> 00:01:25.110 align:middle line:90% the most accurate at this point. 00:01:25.110 --> 00:01:30.450 align:middle line:84% But it's still $8 to $1,000 per array. 00:01:30.450 --> 00:01:31.860 align:middle line:84% A lot of groups are experimenting 00:01:31.860 --> 00:01:34.650 align:middle line:84% with these printed oligonucleotide arrays. 00:01:34.650 --> 00:01:39.780 align:middle line:84% So Agilent has-- so there are just whole variety now-- 00:01:39.780 --> 00:01:46.588 align:middle line:84% Agilent has one type of platform where they actually 00:01:46.588 --> 00:01:48.630 align:middle line:84% give you the chips, and you have to basically buy 00:01:48.630 --> 00:01:50.940 align:middle line:90% their whole setup. 00:01:50.940 --> 00:01:53.400 align:middle line:90% And it's cheaper. 00:01:53.400 --> 00:01:56.420 align:middle line:84% People seem to like it, but who knows what's 00:01:56.420 --> 00:01:57.670 align:middle line:90% going to happen in the future. 00:01:57.670 --> 00:02:00.810 align:middle line:84% I mean, it's all unclear at this point 00:02:00.810 --> 00:02:03.270 align:middle line:84% exactly what platform is the best 00:02:03.270 --> 00:02:06.000 align:middle line:90% information per dollar spent. 00:02:06.000 --> 00:02:08.220 align:middle line:84% But there are other companies that will actually 00:02:08.220 --> 00:02:10.830 align:middle line:90% make these oligonucleotides. 00:02:10.830 --> 00:02:13.630 align:middle line:84% And then you just buy those and then you print it yourself. 00:02:13.630 --> 00:02:18.660 align:middle line:84% So you just have those spray printers and then print those. 00:02:18.660 --> 00:02:22.500 align:middle line:84% I think those will run at this point, maybe 00:02:22.500 --> 00:02:24.150 align:middle line:90% a third of the Affy cost. 00:02:24.150 --> 00:02:27.100 align:middle line:84% So that's a good option for the future. 00:02:27.100 --> 00:02:30.150 align:middle line:84% But it's not clear how good those are. 00:02:30.150 --> 00:02:37.950 align:middle line:84% So it's cheaper at this point, but not as good. 00:02:37.950 --> 00:02:41.020 align:middle line:84% We are actually trying to print that at the [? Harvard ?] 00:02:41.020 --> 00:02:44.110 align:middle line:90% Partner Center ourselves. 00:02:44.110 --> 00:02:45.598 align:middle line:84% And so if we can get that to work-- 00:02:45.598 --> 00:02:47.640 align:middle line:84% we're doing it for human and mouse at this point. 00:02:47.640 --> 00:02:52.950 align:middle line:84% If we can get that to work, that would be very 00:02:52.950 --> 00:02:56.310 align:middle line:90% good for our investigators. 00:02:56.310 --> 00:03:01.830 align:middle line:84% SAGE Is a very accurate way of doing expression data analysis. 00:03:01.830 --> 00:03:05.370 align:middle line:84% It basically involves sequencing every little tag. 00:03:05.370 --> 00:03:07.530 align:middle line:84% So advantage is that you get a very-- even 00:03:07.530 --> 00:03:10.110 align:middle line:84% for a very small copy number of transcripts, 00:03:10.110 --> 00:03:11.880 align:middle line:90% you can get them accurately. 00:03:11.880 --> 00:03:15.360 align:middle line:84% The downside is that it's expensive. 00:03:15.360 --> 00:03:17.310 align:middle line:84% But there are a bunch of groups around here 00:03:17.310 --> 00:03:20.400 align:middle line:90% that have large libraries. 00:03:20.400 --> 00:03:26.230 align:middle line:84% So I'm personally interested in the second point, 00:03:26.230 --> 00:03:28.800 align:middle line:84% which is, nowadays, called systems biology-- 00:03:28.800 --> 00:03:32.430 align:middle line:84% so that is integrating a lot of high throughput 00:03:32.430 --> 00:03:35.730 align:middle line:84% technologies in genomics and proteomics, 00:03:35.730 --> 00:03:36.850 align:middle line:90% and putting that together. 00:03:36.850 --> 00:03:39.480 align:middle line:84% So microarray is where everything started. 00:03:39.480 --> 00:03:41.070 align:middle line:84% Now that all these other technologies 00:03:41.070 --> 00:03:42.660 align:middle line:84% are getting more mature, we're all 00:03:42.660 --> 00:03:45.840 align:middle line:84% interested in combining all of those. 00:03:45.840 --> 00:03:48.900 align:middle line:84% And there are a lot of public and commercial tools, 00:03:48.900 --> 00:03:51.100 align:middle line:84% tools, and databases that you should be aware of. 00:03:51.100 --> 00:03:56.730 align:middle line:84% So I think the problem is that even the best biologists don't 00:03:56.730 --> 00:03:57.720 align:middle line:90% know what's out there. 00:03:57.720 --> 00:04:01.110 align:middle line:84% So the challenge for me and a lot 00:04:01.110 --> 00:04:04.703 align:middle line:84% of other people in a similar situation is that you have to, 00:04:04.703 --> 00:04:06.870 align:middle line:84% instead of assuming that they know what's out there, 00:04:06.870 --> 00:04:08.790 align:middle line:84% you have to tell them what to do. 00:04:08.790 --> 00:04:11.730 align:middle line:84% Tell the biologists that, you should try such and such, 00:04:11.730 --> 00:04:14.920 align:middle line:84% because they're just unaware of what's going on 00:04:14.920 --> 00:04:18.130 align:middle line:90% in mathematics, generally. 00:04:18.130 --> 00:04:21.570 align:middle line:84% So in terms of data integration, initially 00:04:21.570 --> 00:04:23.940 align:middle line:84% people did a lot of expression data studies. 00:04:23.940 --> 00:04:28.200 align:middle line:84% There have been some nice programs 00:04:28.200 --> 00:04:30.540 align:middle line:90% that linked literature. 00:04:30.540 --> 00:04:33.900 align:middle line:84% For example, we, at the center, just got a commercial license 00:04:33.900 --> 00:04:36.300 align:middle line:90% to something called Ingenuity. 00:04:36.300 --> 00:04:38.190 align:middle line:84% You basically put your expression data 00:04:38.190 --> 00:04:41.580 align:middle line:84% into the software and it'll generate nice networks. 00:04:41.580 --> 00:04:45.480 align:middle line:84% And you can-- well, there are many ways of being this data. 00:04:45.480 --> 00:04:48.510 align:middle line:84% But you can, for example, click on links between genes, 00:04:48.510 --> 00:04:52.790 align:middle line:84% and it will bring up all literature 00:04:52.790 --> 00:04:54.753 align:middle line:90% that site those genes. 00:04:54.753 --> 00:04:56.920 align:middle line:84% I mean, there are lots of programs too that do that, 00:04:56.920 --> 00:04:58.920 align:middle line:84% but this particular program does it really well. 00:04:58.920 --> 00:05:02.060 align:middle line:90% 00:05:02.060 --> 00:05:04.040 align:middle line:84% So something like that is enormously helpful, 00:05:04.040 --> 00:05:06.322 align:middle line:84% if you know that such things exist 00:05:06.322 --> 00:05:07.530 align:middle line:90% and if you have access to it. 00:05:07.530 --> 00:05:09.432 align:middle line:84% There are some free ones out there, 00:05:09.432 --> 00:05:10.640 align:middle line:90% but they're just not as good. 00:05:10.640 --> 00:05:14.330 align:middle line:84% I mean, this software, they have tons of people-- 00:05:14.330 --> 00:05:17.690 align:middle line:84% not tons, a lot of people, just sitting there, curating 00:05:17.690 --> 00:05:19.440 align:middle line:90% literature by hand. 00:05:19.440 --> 00:05:21.830 align:middle line:84% And so they've accumulated this huge database. 00:05:21.830 --> 00:05:23.720 align:middle line:90% But I think that's-- 00:05:23.720 --> 00:05:26.630 align:middle line:84% eventually, this is, I think, how a lot of us 00:05:26.630 --> 00:05:27.710 align:middle line:90% are going to do research. 00:05:27.710 --> 00:05:31.010 align:middle line:84% So these tools will get smarter and smarter, 00:05:31.010 --> 00:05:35.060 align:middle line:84% so you don't have to go through PubMed, yourself. 00:05:35.060 --> 00:05:37.050 align:middle line:90% There are other types of data. 00:05:37.050 --> 00:05:40.860 align:middle line:84% I don't if you've actually covered any of these in class, 00:05:40.860 --> 00:05:44.240 align:middle line:84% but there's a lot of protein DNA binding data. 00:05:44.240 --> 00:05:48.140 align:middle line:84% So the most famous data set is from the Whitehead, 00:05:48.140 --> 00:05:54.230 align:middle line:84% from Rick Young's lab, where for yeast, they basically 00:05:54.230 --> 00:05:57.350 align:middle line:84% took all the transcription factors, about a little more 00:05:57.350 --> 00:06:03.650 align:middle line:84% than 100 for yeast, and then they found where in the genome 00:06:03.650 --> 00:06:06.930 align:middle line:84% these transcription factors actually bind. 00:06:06.930 --> 00:06:08.310 align:middle line:90% So in vivo. 00:06:08.310 --> 00:06:10.355 align:middle line:90% So they can't locate-- 00:06:10.355 --> 00:06:12.230 align:middle line:84% you don't have enough resolutions to pinpoint 00:06:12.230 --> 00:06:18.320 align:middle line:84% exactly what base pair they are binding, 00:06:18.320 --> 00:06:21.560 align:middle line:84% but you know which genes are actually 00:06:21.560 --> 00:06:25.820 align:middle line:84% using these transcription factors for their regulation. 00:06:25.820 --> 00:06:28.430 align:middle line:90% So that data is enormous useful. 00:06:28.430 --> 00:06:31.490 align:middle line:84% I think they're trying to do that for human genome, 00:06:31.490 --> 00:06:34.400 align:middle line:84% but I don't know how that's going. 00:06:34.400 --> 00:06:39.620 align:middle line:84% Protein microarrays-- so that has not 00:06:39.620 --> 00:06:42.590 align:middle line:84% been as successful as people initially, thought it would be. 00:06:42.590 --> 00:06:44.240 align:middle line:84% There are just a lot of difficulties 00:06:44.240 --> 00:06:46.730 align:middle line:90% in just manufacturing. 00:06:46.730 --> 00:06:52.250 align:middle line:84% But for example, at the center, Harvard Partner Center, 00:06:52.250 --> 00:06:56.000 align:middle line:84% there are some contracts with some companies that 00:06:56.000 --> 00:07:04.520 align:middle line:84% make these protein arrays with about 50 or 60 molecules 00:07:04.520 --> 00:07:07.760 align:middle line:84% sitting there, then I guess these are 50 or 60 00:07:07.760 --> 00:07:12.660 align:middle line:84% really common antibody detection systems. 00:07:12.660 --> 00:07:17.533 align:middle line:84% So I think that could be very helpful. 00:07:17.533 --> 00:07:19.700 align:middle line:84% But I think the technology is still not quite there. 00:07:19.700 --> 00:07:22.100 align:middle line:84% At least not for the price that's 00:07:22.100 --> 00:07:24.860 align:middle line:84% reasonable for most investigators. 00:07:24.860 --> 00:07:28.790 align:middle line:84% Mass spec-- that's an enormous area that has taken off 00:07:28.790 --> 00:07:32.180 align:middle line:84% in the past, I'd say, at least in terms of informatics, 00:07:32.180 --> 00:07:34.940 align:middle line:90% maybe just past a year or two. 00:07:34.940 --> 00:07:38.780 align:middle line:84% There are enormous amounts of data 00:07:38.780 --> 00:07:40.530 align:middle line:84% and a lot of informatics challenges. 00:07:40.530 --> 00:07:45.950 align:middle line:84% And so another set of data that could be 00:07:45.950 --> 00:07:48.660 align:middle line:90% integrated into your studies. 00:07:48.660 --> 00:07:51.410 align:middle line:84% So when people talk about systems biology, 00:07:51.410 --> 00:07:53.060 align:middle line:90% people mean different things. 00:07:53.060 --> 00:07:55.070 align:middle line:84% But generally, they refer to the fact 00:07:55.070 --> 00:07:58.730 align:middle line:84% that you're combining all these different types to generate 00:07:58.730 --> 00:08:00.380 align:middle line:90% some coherent picture. 00:08:00.380 --> 00:08:02.625 align:middle line:90% And it's actually really hard. 00:08:02.625 --> 00:08:04.250 align:middle line:84% And I think some of the reasons will be 00:08:04.250 --> 00:08:07.560 align:middle line:90% clear in the next few slides. 00:08:07.560 --> 00:08:11.070 align:middle line:90% OK, so let me just skip that. 00:08:11.070 --> 00:08:14.150 align:middle line:84% So just to give you a funny story 00:08:14.150 --> 00:08:19.890 align:middle line:84% on the necessity for a good analysis with all this data. 00:08:19.890 --> 00:08:26.900 align:middle line:84% There was a well-cited paper in Nature Genetics 00:08:26.900 --> 00:08:30.170 align:middle line:84% in 2001 called Transcriptional Regulation and Function 00:08:30.170 --> 00:08:32.640 align:middle line:90% During the Human Cell Cycle. 00:08:32.640 --> 00:08:35.690 align:middle line:84% So I think it actually came from the group that 00:08:35.690 --> 00:08:38.600 align:middle line:84% did the yeast cell cycle a couple of years back, 00:08:38.600 --> 00:08:40.110 align:middle line:90% prior to that. 00:08:40.110 --> 00:08:41.929 align:middle line:84% And in that paper, they claimed that there 00:08:41.929 --> 00:08:43.909 align:middle line:84% are large number of mammalian genes, 00:08:43.909 --> 00:08:47.150 align:middle line:84% about 700, to have a cell cycle specific patterns. 00:08:47.150 --> 00:08:54.890 align:middle line:84% So I think this was done on Affy data, actually. 00:08:54.890 --> 00:08:57.350 align:middle line:84% Anyway, the next year, there's a paper 00:08:57.350 --> 00:09:00.800 align:middle line:84% in PNAS called "Analysis of cell cycle 00:09:00.800 --> 00:09:03.350 align:middle line:84% specific gene expression in human cells 00:09:03.350 --> 00:09:06.140 align:middle line:84% as determined by microarrays and double diamidine block 00:09:06.140 --> 00:09:06.860 align:middle line:90% synchronization. 00:09:06.860 --> 00:09:08.870 align:middle line:84% So from the title, you can't really 00:09:08.870 --> 00:09:12.890 align:middle line:90% tell what the subject is. 00:09:12.890 --> 00:09:15.980 align:middle line:84% But basically what they did in the 2002 paper 00:09:15.980 --> 00:09:20.820 align:middle line:84% is go back and reanalyze the data from 2001 paper. 00:09:20.820 --> 00:09:24.740 align:middle line:84% And so, if you look at the text here, 00:09:24.740 --> 00:09:26.660 align:middle line:84% the original microarray data presented 00:09:26.660 --> 00:09:30.590 align:middle line:84% to support the existence of cyclic gene 00:09:30.590 --> 00:09:33.890 align:middle line:84% expression in human cells is now re-examined with a statistical 00:09:33.890 --> 00:09:38.043 align:middle line:84% approach, we find that there is internal evidence implying 00:09:38.043 --> 00:09:39.710 align:middle line:84% that the original microarray data do not 00:09:39.710 --> 00:09:42.900 align:middle line:84% support the proposed patterns of gene expression. 00:09:42.900 --> 00:09:44.750 align:middle line:90% So one of these-- 00:09:44.750 --> 00:09:47.270 align:middle line:84% I think it was the first author that's a statistician. 00:09:47.270 --> 00:09:54.180 align:middle line:84% So they claimed that to study the cell cycle, 00:09:54.180 --> 00:09:56.030 align:middle line:84% you need to synchronize at the beginning, 00:09:56.030 --> 00:09:59.080 align:middle line:90% using one of many methods. 00:09:59.080 --> 00:10:01.160 align:middle line:84% And they look at this data and say, 00:10:01.160 --> 00:10:04.780 align:middle line:84% well after randomizing data, we still see 00:10:04.780 --> 00:10:08.010 align:middle line:84% a bunch of genes that have cyclic behavior. 00:10:08.010 --> 00:10:10.750 align:middle line:84% In fact, if you study it carefully, 00:10:10.750 --> 00:10:12.250 align:middle line:90% cells are not even synchronized. 00:10:12.250 --> 00:10:16.870 align:middle line:84% So they claim that the first experiment was just junk. 00:10:16.870 --> 00:10:20.500 align:middle line:84% And the 700 genes that they observed 00:10:20.500 --> 00:10:23.890 align:middle line:84% was something that just could have come by chance. 00:10:23.890 --> 00:10:26.372 align:middle line:90% 00:10:26.372 --> 00:10:28.330 align:middle line:84% Because I mean, because you have so many genes, 00:10:28.330 --> 00:10:33.040 align:middle line:84% you're going to see some genes that go in cycles. 00:10:33.040 --> 00:10:34.390 align:middle line:90% So it came out in PNAS. 00:10:34.390 --> 00:10:40.300 align:middle line:84% And so since then, at least, there were just exchanges 00:10:40.300 --> 00:10:41.170 align:middle line:90% back and forth. 00:10:41.170 --> 00:10:44.980 align:middle line:84% So the first group has a website that details 00:10:44.980 --> 00:10:48.220 align:middle line:84% this is what we did, this is what the PNAS paper did, 00:10:48.220 --> 00:10:49.720 align:middle line:90% and this is why they're wrong. 00:10:49.720 --> 00:10:51.100 align:middle line:90% It's just a whole website. 00:10:51.100 --> 00:10:52.990 align:middle line:84% And I think the second guys has also 00:10:52.990 --> 00:10:56.030 align:middle line:90% had a rebuttal to that website. 00:10:56.030 --> 00:11:04.450 align:middle line:84% So I mean, this is an important thing to settle, right? 00:11:04.450 --> 00:11:07.600 align:middle line:84% Because it came out in Nature Genetics, a lot of people 00:11:07.600 --> 00:11:09.700 align:middle line:84% are using this data, but the other group says, 00:11:09.700 --> 00:11:11.510 align:middle line:90% this is all nonsense. 00:11:11.510 --> 00:11:15.040 align:middle line:84% And I think-- I mean, so I read a little bit of that. 00:11:15.040 --> 00:11:18.070 align:middle line:90% I think what happened was-- 00:11:18.070 --> 00:11:20.410 align:middle line:84% I mean, as usual the truth is somewhere in between. 00:11:20.410 --> 00:11:23.440 align:middle line:84% I mean, the first group was so interested in showing 00:11:23.440 --> 00:11:26.350 align:middle line:84% that there are lots of genes that are cycling, 00:11:26.350 --> 00:11:32.440 align:middle line:84% that they just used methods that will be advantageous to them. 00:11:32.440 --> 00:11:35.200 align:middle line:84% And the second paper, they were so 00:11:35.200 --> 00:11:37.550 align:middle line:84% interested in showing that the data was wrong, 00:11:37.550 --> 00:11:39.580 align:middle line:84% that they defined things in such a way 00:11:39.580 --> 00:11:46.298 align:middle line:84% that there would be no cycling genes. 00:11:46.298 --> 00:11:48.340 align:middle line:84% AUDIENCE: [INAUDIBLE] if you found the same genes 00:11:48.340 --> 00:11:51.177 align:middle line:90% in another [INAUDIBLE]. 00:11:51.177 --> 00:11:52.510 align:middle line:90% PETER PARK: Maybe, I don't know. 00:11:52.510 --> 00:11:55.060 align:middle line:90% Yeah. 00:11:55.060 --> 00:11:58.060 align:middle line:84% I mean, I bet, depending on how you do it. 00:11:58.060 --> 00:11:59.800 align:middle line:84% I would think that they would find 00:11:59.800 --> 00:12:07.120 align:middle line:84% some overlap with the first one, but not 700. 00:12:07.120 --> 00:12:09.010 align:middle line:84% So I guess the lesson is that you 00:12:09.010 --> 00:12:14.200 align:middle line:84% should make sure you do things well before you publish. 00:12:14.200 --> 00:12:17.720 align:middle line:84% At least in such a way that no one can attack you like this. 00:12:17.720 --> 00:12:19.070 align:middle line:90% And this is just one example. 00:12:19.070 --> 00:12:23.350 align:middle line:84% There are a lot of problems with early studies. 00:12:23.350 --> 00:12:27.640 align:middle line:84% And I'll actually tell you about another problem 00:12:27.640 --> 00:12:29.005 align:middle line:90% that has been very common. 00:12:29.005 --> 00:12:31.940 align:middle line:90% 00:12:31.940 --> 00:12:34.720 align:middle line:84% So one thing that I've been interested 00:12:34.720 --> 00:12:39.460 align:middle line:84% in with a student, who actually published a paper on this 00:12:39.460 --> 00:12:43.210 align:middle line:84% before anyone else, so there are substantial differences 00:12:43.210 --> 00:12:45.580 align:middle line:84% in different technologies for microarray platforms. 00:12:45.580 --> 00:12:50.920 align:middle line:84% So cDNAs-- they're very different from Affy arrays. 00:12:50.920 --> 00:12:57.910 align:middle line:84% So one of the common questions is, what platform should I use? 00:12:57.910 --> 00:12:59.890 align:middle line:84% If something's cheaper, does it work as well? 00:12:59.890 --> 00:13:02.710 align:middle line:90% Do they agree? 00:13:02.710 --> 00:13:04.210 align:middle line:90% So which platform is the best? 00:13:04.210 --> 00:13:07.120 align:middle line:84% And so there are a bunch of papers on this, some of which 00:13:07.120 --> 00:13:10.270 align:middle line:84% are cited in the article, but I mean, 00:13:10.270 --> 00:13:12.730 align:middle line:90% that's just a very small subset. 00:13:12.730 --> 00:13:16.390 align:middle line:84% And most of them are conflicting, 00:13:16.390 --> 00:13:21.070 align:middle line:84% except that they agree that they do not agree very well. 00:13:21.070 --> 00:13:25.360 align:middle line:84% So they're conflicting in terms of which one is better, 00:13:25.360 --> 00:13:29.110 align:middle line:90% how accurate are they. 00:13:29.110 --> 00:13:33.130 align:middle line:84% But I think overall, the overlap is surprisingly small. 00:13:33.130 --> 00:13:36.040 align:middle line:84% So if you do your experiment with cDNA arrays, 00:13:36.040 --> 00:13:38.110 align:middle line:84% find genes that are different with [INAUDIBLE].. 00:13:38.110 --> 00:13:40.630 align:middle line:84% So you repeat the experiment with Affy arrays, 00:13:40.630 --> 00:13:43.490 align:middle line:90% you get a very different result. 00:13:43.490 --> 00:13:45.490 align:middle line:90% So that's a big problem, right? 00:13:45.490 --> 00:13:47.200 align:middle line:84% I mean, for one thing, you're not sure 00:13:47.200 --> 00:13:51.710 align:middle line:84% if the results that you get are correct, or not. 00:13:51.710 --> 00:13:54.790 align:middle line:84% And another thing is that eventually, we'd 00:13:54.790 --> 00:13:58.360 align:middle line:84% like to combine data from different people. 00:13:58.360 --> 00:14:02.320 align:middle line:84% So if someone else, in another school, 00:14:02.320 --> 00:14:07.090 align:middle line:84% has done very similar experiment, 00:14:07.090 --> 00:14:10.360 align:middle line:84% and you want to see how your experiment compares with that, 00:14:10.360 --> 00:14:12.850 align:middle line:84% then you don't want to have to repeat their experiment. 00:14:12.850 --> 00:14:15.610 align:middle line:84% You'd like to just do your experiment on whatever platform 00:14:15.610 --> 00:14:17.200 align:middle line:84% you have and compare it against what 00:14:17.200 --> 00:14:20.630 align:middle line:84% someone else has done, which may not be on the same platform. 00:14:20.630 --> 00:14:22.570 align:middle line:90% And so this is a big issue. 00:14:22.570 --> 00:14:26.530 align:middle line:84% And it's far away from being settled-- 00:14:26.530 --> 00:14:29.690 align:middle line:84% just there are so many issues and problems, 00:14:29.690 --> 00:14:35.230 align:middle line:84% so that I think it'll be a while before these things get 00:14:35.230 --> 00:14:36.410 align:middle line:90% sorted out. 00:14:36.410 --> 00:14:41.170 align:middle line:84% So as I mentioned, Affy arrays appear 00:14:41.170 --> 00:14:43.480 align:middle line:84% to have the most reproducible experiment. 00:14:43.480 --> 00:14:45.760 align:middle line:84% I think most people that have tried 00:14:45.760 --> 00:14:48.370 align:middle line:84% multiple privatizations will say that they 00:14:48.370 --> 00:14:52.134 align:middle line:84% agree with correlation of coin 98, 99. 00:14:52.134 --> 00:14:54.640 align:middle line:90% 00:14:54.640 --> 00:14:59.190 align:middle line:84% But one thing I've done with Steve Greenberg, whom 00:14:59.190 --> 00:15:02.430 align:middle line:84% you've met, I think, couple weeks ago, 00:15:02.430 --> 00:15:05.520 align:middle line:84% is that we've studied how reproducible 00:15:05.520 --> 00:15:09.150 align:middle line:84% there are between two generations of Affy arrays. 00:15:09.150 --> 00:15:15.160 align:middle line:84% So there is a very popular set of arrays-- 00:15:15.160 --> 00:15:16.890 align:middle line:90% the first one was U95-- 00:15:16.890 --> 00:15:22.020 align:middle line:84% that's 95 refers to the version of gene database 00:15:22.020 --> 00:15:27.990 align:middle line:84% that they use to collect the information about the probes. 00:15:27.990 --> 00:15:34.080 align:middle line:84% So that, I think, people used for a couple of years-- 00:15:34.080 --> 00:15:37.380 align:middle line:84% lots of chips, I mean, many thousands of chips. 00:15:37.380 --> 00:15:40.005 align:middle line:84% And then, you U133 is the newer one. 00:15:40.005 --> 00:15:42.630 align:middle line:90% 00:15:42.630 --> 00:15:45.630 align:middle line:84% So U95 actually came in five chips, 00:15:45.630 --> 00:15:47.430 align:middle line:90% and U133 came in two chips. 00:15:47.430 --> 00:15:50.740 align:middle line:84% So they changed a bunch of things 00:15:50.740 --> 00:15:52.920 align:middle line:84% so they were able to fit into two chips. 00:15:52.920 --> 00:16:00.210 align:middle line:84% Now, there is just a single chip with the whole human genes. 00:16:00.210 --> 00:16:05.970 align:middle line:84% So from five arrays, three or four years ago, now you 00:16:05.970 --> 00:16:08.610 align:middle line:90% have a single array. 00:16:08.610 --> 00:16:12.210 align:middle line:84% And so a lot of experiments have been done. 00:16:12.210 --> 00:16:14.310 align:middle line:90% These are human arrays. 00:16:14.310 --> 00:16:16.740 align:middle line:84% Just to give you a sense of how many arrays are out there, 00:16:16.740 --> 00:16:20.310 align:middle line:84% when I was talking to someone from Aventis pharmaceutical, 00:16:20.310 --> 00:16:25.200 align:middle line:84% they have a proprietary database with-- 00:16:25.200 --> 00:16:27.720 align:middle line:84% I think they said about I can't remember, 00:16:27.720 --> 00:16:31.680 align:middle line:84% it was either 20,000 or 30,000 chips. 00:16:31.680 --> 00:16:32.670 align:middle line:90% AUDIENCE: Of their own? 00:16:32.670 --> 00:16:34.003 align:middle line:90% PETER PARK: Or their own, right. 00:16:34.003 --> 00:16:35.430 align:middle line:90% So their internal database. 00:16:35.430 --> 00:16:39.180 align:middle line:84% So 20,000, let's say, time's $1,000, that's $20 million, 00:16:39.180 --> 00:16:40.080 align:middle line:90% right? 00:16:40.080 --> 00:16:41.530 align:middle line:90% That's a lot of money. 00:16:41.530 --> 00:16:47.460 align:middle line:84% So anyway, there's a lot of data using these. 00:16:47.460 --> 00:16:51.300 align:middle line:84% Actually, I don't if these arrays are not the one-- 00:16:51.300 --> 00:16:53.370 align:middle line:84% they didn't have 20,000 of these arrays. 00:16:53.370 --> 00:16:54.900 align:middle line:84% But they have a lot of Affy arrays. 00:16:54.900 --> 00:16:57.340 align:middle line:90% These are 20,000 Affy arrays. 00:16:57.340 --> 00:16:59.730 align:middle line:90% And that's just one company. 00:16:59.730 --> 00:17:02.610 align:middle line:84% Anyway, so what Steve and I wanted to do 00:17:02.610 --> 00:17:08.640 align:middle line:84% is look at how the results would vary depending 00:17:08.640 --> 00:17:10.560 align:middle line:90% on which platform you use. 00:17:10.560 --> 00:17:15.060 align:middle line:84% And he actually had done the experiment on U95, 00:17:15.060 --> 00:17:19.800 align:middle line:84% and then he had a new grant, he had more samples, 00:17:19.800 --> 00:17:21.359 align:middle line:84% so he want to do uniformity again, 00:17:21.359 --> 00:17:23.790 align:middle line:90% and so he did it on U133. 00:17:23.790 --> 00:17:26.220 align:middle line:84% And it's not because he had so much money 00:17:26.220 --> 00:17:28.569 align:middle line:90% that he just wanted to try this. 00:17:28.569 --> 00:17:33.060 align:middle line:84% So what he did was he hybridized 14 muscle life samples 00:17:33.060 --> 00:17:38.280 align:middle line:84% from with patients with inflammatory myopathies, which 00:17:38.280 --> 00:17:42.660 align:middle line:84% is his specialty, then they were hybridized to both chips. 00:17:42.660 --> 00:17:44.410 align:middle line:90% So I looked at this data-- 00:17:44.410 --> 00:17:47.460 align:middle line:84% we submitted the paper just recently. 00:17:47.460 --> 00:17:49.145 align:middle line:84% But what we looked at was, what happens 00:17:49.145 --> 00:17:51.520 align:middle line:84% when we grouped the arrays using hierarchical clustering, 00:17:51.520 --> 00:17:54.660 align:middle line:84% which you know by now, and when we find differentially 00:17:54.660 --> 00:17:58.290 align:middle line:90% expressed genes in each case. 00:17:58.290 --> 00:18:02.190 align:middle line:84% So you can't read the labels from there, 00:18:02.190 --> 00:18:05.970 align:middle line:84% I bet, if you look at the top left one-- 00:18:05.970 --> 00:18:09.730 align:middle line:84% so we basically pulled all the data together and said, 00:18:09.730 --> 00:18:12.360 align:middle line:90% OK, just can I cluster them? 00:18:12.360 --> 00:18:16.740 align:middle line:84% So what you should see is, same sample, 00:18:16.740 --> 00:18:20.610 align:middle line:84% hybridized to 95 and 133 should be next to each other, 00:18:20.610 --> 00:18:25.050 align:middle line:84% and that's going to be the case for all your samples. 00:18:25.050 --> 00:18:29.082 align:middle line:84% In this clustering, everything on the left is U133, 00:18:29.082 --> 00:18:30.540 align:middle line:84% and everything on the right is U95. 00:18:30.540 --> 00:18:35.010 align:middle line:90% 00:18:35.010 --> 00:18:37.430 align:middle line:90% Yeah. 00:18:37.430 --> 00:18:41.780 align:middle line:84% So if you cluster, it's not the sample characteristics, 00:18:41.780 --> 00:18:44.778 align:middle line:84% but the array type that is the distinguishing feature. 00:18:44.778 --> 00:18:46.236 align:middle line:84% AUDIENCE: What did he [INAUDIBLE]?? 00:18:46.236 --> 00:18:49.680 align:middle line:90% 00:18:49.680 --> 00:18:52.570 align:middle line:84% PETER PARK: Well, in this case, we used Pearson correlations. 00:18:52.570 --> 00:18:58.140 align:middle line:84% So any linear normalization would not affect the result. 00:18:58.140 --> 00:19:01.780 align:middle line:84% But no matter what you try, it's going to turn out this way. 00:19:01.780 --> 00:19:06.030 align:middle line:84% And so this is disturbing, right? 00:19:06.030 --> 00:19:08.200 align:middle line:84% So there are some ways to fix this. 00:19:08.200 --> 00:19:14.280 align:middle line:84% So in this particular case, we normalized the genes 00:19:14.280 --> 00:19:17.370 align:middle line:84% in a particular way, and then we were able to actually get 00:19:17.370 --> 00:19:20.560 align:middle line:90% this to line up correctly. 00:19:20.560 --> 00:19:25.270 align:middle line:84% But what we did in this case, in general, doesn't work. 00:19:25.270 --> 00:19:27.580 align:middle line:90% So we could only do it here. 00:19:27.580 --> 00:19:29.800 align:middle line:90% But in general, it doesn't work. 00:19:29.800 --> 00:19:33.060 align:middle line:84% So remember, this is one platform, just 00:19:33.060 --> 00:19:34.530 align:middle line:90% different generations. 00:19:34.530 --> 00:19:36.495 align:middle line:84% So if you were to compare different platforms, 00:19:36.495 --> 00:19:37.995 align:middle line:84% I mean, things are just much harder. 00:19:37.995 --> 00:19:40.750 align:middle line:90% 00:19:40.750 --> 00:19:46.080 align:middle line:90% And so, that's just an idea of-- 00:19:46.080 --> 00:19:48.580 align:middle line:84% that should give you some idea as to how different they are. 00:19:48.580 --> 00:19:54.230 align:middle line:84% Another way that we looked at was, from each platform, 00:19:54.230 --> 00:19:56.680 align:middle line:90% we looked at-- 00:19:56.680 --> 00:20:00.380 align:middle line:84% so the samples were naturally divided into two groups. 00:20:00.380 --> 00:20:02.890 align:middle line:84% So we looked at, for each platform, what genes 00:20:02.890 --> 00:20:04.150 align:middle line:90% are differentially expressed. 00:20:04.150 --> 00:20:11.710 align:middle line:84% So from the U95 samples, we got a long list, from 133, 00:20:11.710 --> 00:20:13.620 align:middle line:90% we got a long list. 00:20:13.620 --> 00:20:16.300 align:middle line:90% How much overlap is there? 00:20:16.300 --> 00:20:17.920 align:middle line:90% That was the question. 00:20:17.920 --> 00:20:19.720 align:middle line:90% Does that make sense? 00:20:19.720 --> 00:20:23.830 align:middle line:84% So if you look at the dotted line, if you look at-- 00:20:23.830 --> 00:20:26.560 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:20:26.560 --> 00:20:29.770 align:middle line:90% PETER PARK: Right, so we had 14. 00:20:29.770 --> 00:20:33.280 align:middle line:84% And there were basically five of one type, 00:20:33.280 --> 00:20:37.850 align:middle line:84% and the rest could be considered as another type. 00:20:37.850 --> 00:20:41.050 align:middle line:84% So using those five and nine chips, 00:20:41.050 --> 00:20:43.820 align:middle line:84% we looked at the different questions. 00:20:43.820 --> 00:20:45.860 align:middle line:84% So we did that separately for each one. 00:20:45.860 --> 00:20:48.940 align:middle line:84% So I mean, they should overlap, right? 00:20:48.940 --> 00:20:51.820 align:middle line:90% Exactly same samples. 00:20:51.820 --> 00:20:53.470 align:middle line:84% We did a lot of analysis to make sure 00:20:53.470 --> 00:20:58.750 align:middle line:84% that there wasn't a lot of degradation of RNA, and such. 00:20:58.750 --> 00:21:00.130 align:middle line:90% But if you look at this-- 00:21:00.130 --> 00:21:02.500 align:middle line:84% so forget the solid line for now-- 00:21:02.500 --> 00:21:06.940 align:middle line:84% if you look at the dotted dashed line, out of top 100, 00:21:06.940 --> 00:21:10.940 align:middle line:84% there are about 20%, less than 20% that overlap. 00:21:10.940 --> 00:21:14.200 align:middle line:84% So about 20 genes will show up in both, 00:21:14.200 --> 00:21:18.140 align:middle line:90% but 80 genes are not in common. 00:21:18.140 --> 00:21:22.240 align:middle line:84% So that's somewhat disturbing, right? 00:21:22.240 --> 00:21:25.450 align:middle line:90% I mean, if you look at top 50-- 00:21:25.450 --> 00:21:28.690 align:middle line:84% and usually, when investigators do these experiments, 00:21:28.690 --> 00:21:34.600 align:middle line:84% they can't really look at more than top 20, 30, 50, 00:21:34.600 --> 00:21:36.590 align:middle line:84% it's about 15% that are in common. 00:21:36.590 --> 00:21:39.620 align:middle line:84% So 15%, that's like seven or eight genes out of 50. 00:21:39.620 --> 00:21:42.490 align:middle line:90% So that's somewhat disturbing. 00:21:42.490 --> 00:21:45.790 align:middle line:84% So for the paper, we actually found 00:21:45.790 --> 00:21:49.490 align:middle line:84% some way of fixing this problem in the general case. 00:21:49.490 --> 00:21:55.300 align:middle line:84% So we could raise the percentages a lot. 00:21:55.300 --> 00:22:00.820 align:middle line:84% But most people who will not go through this trouble of what 00:22:00.820 --> 00:22:06.070 align:middle line:84% we did should be aware that their list is not 00:22:06.070 --> 00:22:09.040 align:middle line:90% all that robust. 00:22:09.040 --> 00:22:11.200 align:middle line:84% And in some sense, this is not surprising. 00:22:11.200 --> 00:22:13.270 align:middle line:84% Although the degree to which that they disagree 00:22:13.270 --> 00:22:14.950 align:middle line:90% is surprising. 00:22:14.950 --> 00:22:17.080 align:middle line:84% Generally, you have several hundred, 00:22:17.080 --> 00:22:19.863 align:middle line:84% or maybe even thousands of genes, 00:22:19.863 --> 00:22:21.280 align:middle line:84% that are differentially expressed. 00:22:21.280 --> 00:22:23.260 align:middle line:84% And there are different genes expressed 00:22:23.260 --> 00:22:24.550 align:middle line:90% at a very similar level. 00:22:24.550 --> 00:22:27.640 align:middle line:84% So any perturbation that you make to the data, 00:22:27.640 --> 00:22:30.820 align:middle line:84% things are going to just shuffle, 00:22:30.820 --> 00:22:33.650 align:middle line:84% and they're generally not very stable. 00:22:33.650 --> 00:22:36.710 align:middle line:84% So it's not surprising that they don't agree very much. 00:22:36.710 --> 00:22:39.760 align:middle line:84% But in this case, because it was so tightly controlled, I mean, 00:22:39.760 --> 00:22:44.710 align:middle line:84% because they get the same RNA labeled at the same time, 00:22:44.710 --> 00:22:48.580 align:middle line:84% they were hybridized in the same facility, 00:22:48.580 --> 00:22:50.320 align:middle line:84% we expected this to be a little higher. 00:22:50.320 --> 00:22:52.960 align:middle line:90% But it's not. 00:22:52.960 --> 00:22:56.080 align:middle line:90% So this is a challenge, I think. 00:22:56.080 --> 00:23:01.120 align:middle line:84% And it should be a caution for people doing experiments. 00:23:01.120 --> 00:23:06.940 align:middle line:84% One thing that-- so one thing that people 00:23:06.940 --> 00:23:10.510 align:middle line:84% should pay more attention to, therefore, is not so much 00:23:10.510 --> 00:23:13.240 align:middle line:84% what genes are expressed, but what kind of genes 00:23:13.240 --> 00:23:14.000 align:middle line:90% are expressed. 00:23:14.000 --> 00:23:16.780 align:middle line:84% So people are now thinking of more robust way 00:23:16.780 --> 00:23:18.550 align:middle line:90% of looking at the list. 00:23:18.550 --> 00:23:23.230 align:middle line:84% I mean, for example, what kind of goal category genes 00:23:23.230 --> 00:23:26.356 align:middle line:84% are represented at the top, that kind of thing. 00:23:26.356 --> 00:23:30.797 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:23:30.797 --> 00:23:32.630 align:middle line:84% PETER PARK: Yeah, so that's a good question. 00:23:32.630 --> 00:23:34.010 align:middle line:90% So we looked at that in detail. 00:23:34.010 --> 00:23:37.240 align:middle line:84% So it turns out that if the probes-- 00:23:37.240 --> 00:23:39.890 align:middle line:84% so you all know how Affy arrays work. 00:23:39.890 --> 00:23:41.710 align:middle line:84% So if the probes are exactly the same, 00:23:41.710 --> 00:23:43.520 align:middle line:84% the sequences are exactly the same. 00:23:43.520 --> 00:23:45.490 align:middle line:90% They are very reproducible. 00:23:45.490 --> 00:23:48.830 align:middle line:90% So they give very good results. 00:23:48.830 --> 00:23:52.660 align:middle line:84% But if they are different, I mean, 00:23:52.660 --> 00:23:56.260 align:middle line:84% I don't have pictures here to show you, but no relationship. 00:23:56.260 --> 00:24:02.110 align:middle line:84% Even though they are supposed to detect the same transcript. 00:24:02.110 --> 00:24:03.760 align:middle line:84% So basically, I mean, the lesson is 00:24:03.760 --> 00:24:07.090 align:middle line:84% that slight changes in the sequence information 00:24:07.090 --> 00:24:11.672 align:middle line:84% results in huge changes in your expression. 00:24:11.672 --> 00:24:12.780 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:24:12.780 --> 00:24:17.280 align:middle line:90% 00:24:17.280 --> 00:24:19.000 align:middle line:84% PETER PARK: No, the length is the same. 00:24:19.000 --> 00:24:21.050 align:middle line:84% The length that-- those are 25 mers. 00:24:21.050 --> 00:24:24.430 align:middle line:84% And the reason that they're-- so there is a debate as to what 00:24:24.430 --> 00:24:25.940 align:middle line:90% the optimal length is. 00:24:25.940 --> 00:24:27.340 align:middle line:90% And so people have-- 00:24:27.340 --> 00:24:30.683 align:middle line:84% so the commercial suppliers that I listed earlier, 00:24:30.683 --> 00:24:31.850 align:middle line:90% they have different lengths. 00:24:31.850 --> 00:24:36.010 align:middle line:84% So there are some 30 mers, 50 mers, a lot of people 00:24:36.010 --> 00:24:41.980 align:middle line:84% think that the longer nucleotides, oligonucleotides 00:24:41.980 --> 00:24:48.300 align:middle line:84% usually have 70 mers, or that size, are the best. 00:24:48.300 --> 00:24:50.790 align:middle line:84% But that's still a bit of an open question. 00:24:50.790 --> 00:24:54.420 align:middle line:84% But Affy would probably like to have longer probes, 00:24:54.420 --> 00:24:57.040 align:middle line:84% but they're limited by technology. 00:24:57.040 --> 00:24:58.470 align:middle line:90% So they are built by-- 00:24:58.470 --> 00:25:00.210 align:middle line:84% it's the same technology that you 00:25:00.210 --> 00:25:01.950 align:middle line:84% use to build semiconductor chips, 00:25:01.950 --> 00:25:08.420 align:middle line:84% and so you can't tile so many nucleotides. 00:25:08.420 --> 00:25:10.390 align:middle line:90% So what they did-- 00:25:10.390 --> 00:25:13.710 align:middle line:84% so the reason that Affy got better, is that this-- 00:25:13.710 --> 00:25:15.390 align:middle line:84% it's so hard to figure out exactly 00:25:15.390 --> 00:25:18.840 align:middle line:84% which sequences would give you the best result. So basically, 00:25:18.840 --> 00:25:20.430 align:middle line:84% what Affy has done over the years is 00:25:20.430 --> 00:25:23.160 align:middle line:84% that, throughout the generations, they figured out, 00:25:23.160 --> 00:25:27.780 align:middle line:84% just through trial and error, which ones give bad results. 00:25:27.780 --> 00:25:30.450 align:middle line:84% And so next time the product comes out, 00:25:30.450 --> 00:25:33.870 align:middle line:84% they get rid of all the bad ones and just try new ones. 00:25:33.870 --> 00:25:36.690 align:middle line:84% So by that trial and error process, in my view, 00:25:36.690 --> 00:25:38.620 align:middle line:90% they've gotten pretty good. 00:25:38.620 --> 00:25:43.020 align:middle line:84% So if you look at two or three years ago, 00:25:43.020 --> 00:25:48.390 align:middle line:84% if you look at the old chip, just how much variation there 00:25:48.390 --> 00:25:51.360 align:middle line:84% is among the probes that are supposed 00:25:51.360 --> 00:25:53.970 align:middle line:84% to detect the same gene, you'd see just things all 00:25:53.970 --> 00:25:55.470 align:middle line:90% over the place. 00:25:55.470 --> 00:25:58.320 align:middle line:84% It's just maybe half the probes may give you 00:25:58.320 --> 00:26:00.210 align:middle line:90% completely different numbers. 00:26:00.210 --> 00:26:03.240 align:middle line:90% But now, it is much better. 00:26:03.240 --> 00:26:04.930 align:middle line:90% Things have certainly improved. 00:26:04.930 --> 00:26:08.190 align:middle line:84% And I think Affy, the platform itself, 00:26:08.190 --> 00:26:09.780 align:middle line:84% has some advantages, because they 00:26:09.780 --> 00:26:12.420 align:middle line:90% are using some multiple probes. 00:26:12.420 --> 00:26:15.930 align:middle line:84% Other platforms, they don't have the luxury of doing that. 00:26:15.930 --> 00:26:18.270 align:middle line:84% So they have a single, longer probe, 00:26:18.270 --> 00:26:21.490 align:middle line:84% but they have no idea whether it's correct or not. 00:26:21.490 --> 00:26:25.770 align:middle line:84% And I've looked at, actually, commercial arrays, not Affy, 00:26:25.770 --> 00:26:27.990 align:middle line:84% and compared that against Affy, and you just 00:26:27.990 --> 00:26:31.440 align:middle line:90% get disturbing results. 00:26:31.440 --> 00:26:37.170 align:middle line:84% So you probably have heard a lot of good things 00:26:37.170 --> 00:26:42.120 align:middle line:84% about microarrays, and that's probably true overall. 00:26:42.120 --> 00:26:47.550 align:middle line:84% But there are some issues that you need to be careful about. 00:26:47.550 --> 00:26:49.570 align:middle line:84% So yeah, I was a little surprised about this. 00:26:49.570 --> 00:26:52.710 align:middle line:84% And I guess the recommendation is 00:26:52.710 --> 00:26:55.185 align:middle line:84% that if you have different platforms, at least, 00:26:55.185 --> 00:26:57.000 align:middle line:84% unless you're going to go through, 00:26:57.000 --> 00:27:01.080 align:middle line:84% unless you are able to go through what we went through, 00:27:01.080 --> 00:27:02.940 align:middle line:90% not to combine the arrays. 00:27:02.940 --> 00:27:06.090 align:middle line:84% But then, that's not a very satisfactory solution. 00:27:06.090 --> 00:27:10.320 align:middle line:84% I mean, if you have a database with all these steps 00:27:10.320 --> 00:27:13.740 align:middle line:84% that everyone else has done, what use is it 00:27:13.740 --> 00:27:15.900 align:middle line:90% if you can't use them. 00:27:15.900 --> 00:27:22.110 align:middle line:84% Other than just each data set in its own state. 00:27:22.110 --> 00:27:26.190 align:middle line:84% So OK, I don't think we need to talk 00:27:26.190 --> 00:27:28.410 align:middle line:90% about how many chips we need. 00:27:28.410 --> 00:27:32.950 align:middle line:84% I don't if anyone is planning to do array experiments. 00:27:32.950 --> 00:27:35.715 align:middle line:90% Your array experiments-- oh, OK. 00:27:35.715 --> 00:27:38.880 align:middle line:90% 00:27:38.880 --> 00:27:44.880 align:middle line:84% So generally, I think for most platforms that have been around 00:27:44.880 --> 00:27:47.970 align:middle line:84% for a while, most people would agree 00:27:47.970 --> 00:27:51.090 align:middle line:84% that biological variability dominates technical measurement 00:27:51.090 --> 00:27:51.610 align:middle line:90% errors. 00:27:51.610 --> 00:27:54.420 align:middle line:84% So for Affy, you don't want to do many replicates 00:27:54.420 --> 00:27:56.580 align:middle line:90% of the same tissue or sample. 00:27:56.580 --> 00:28:01.080 align:middle line:90% 00:28:01.080 --> 00:28:04.410 align:middle line:84% In terms of exactly how many you need, how many chips you need, 00:28:04.410 --> 00:28:07.290 align:middle line:90% that's a hard question. 00:28:07.290 --> 00:28:10.620 align:middle line:84% Only because you don't know how much variability 00:28:10.620 --> 00:28:13.180 align:middle line:90% that you have in your samples. 00:28:13.180 --> 00:28:15.540 align:middle line:84% So if someone can tell me, I have this much variability 00:28:15.540 --> 00:28:19.363 align:middle line:84% in my samples, I want this false positive rate, 00:28:19.363 --> 00:28:21.030 align:middle line:84% this false negative rate, I can give you 00:28:21.030 --> 00:28:23.130 align:middle line:90% numbers based on some model. 00:28:23.130 --> 00:28:25.270 align:middle line:84% But usually people can't give me those numbers. 00:28:25.270 --> 00:28:28.590 align:middle line:84% So there's no point in talking about this. 00:28:28.590 --> 00:28:32.310 align:middle line:84% Some, generally speaking, if you have data in cancer data, 00:28:32.310 --> 00:28:34.020 align:middle line:84% where you have a lot of variability, 00:28:34.020 --> 00:28:35.580 align:middle line:90% I think people would say-- 00:28:35.580 --> 00:28:40.260 align:middle line:84% some studies have said, having samples like 10 to 15 00:28:40.260 --> 00:28:43.620 align:middle line:84% per group for a tumor comparison might give you something 00:28:43.620 --> 00:28:47.070 align:middle line:90% like 75% false positive rate. 00:28:47.070 --> 00:28:52.840 align:middle line:84% And for detecting genes that are three-fold variable. 00:28:52.840 --> 00:28:55.650 align:middle line:84% So yeah, there can be some estimates 00:28:55.650 --> 00:28:58.050 align:middle line:90% like that for given parameters. 00:28:58.050 --> 00:28:59.880 align:middle line:90% But in general, it's hard. 00:28:59.880 --> 00:29:02.910 align:middle line:90% So yeah, let's leave it at that. 00:29:02.910 --> 00:29:05.970 align:middle line:90% 00:29:05.970 --> 00:29:08.100 align:middle line:84% I think I'm going to skip most of these. 00:29:08.100 --> 00:29:09.690 align:middle line:84% I'm sure you've seen these before. 00:29:09.690 --> 00:29:13.250 align:middle line:90% 00:29:13.250 --> 00:29:18.620 align:middle line:84% Let's talk about p-values, multiple testing. 00:29:18.620 --> 00:29:20.180 align:middle line:90% OK, so maybe not. 00:29:20.180 --> 00:29:26.040 align:middle line:90% So just to give you one example. 00:29:26.040 --> 00:29:31.760 align:middle line:84% So suppose you flip a coin 10 times and get all heads. 00:29:31.760 --> 00:29:37.520 align:middle line:84% This is like a statistics 101 problem question. 00:29:37.520 --> 00:29:41.390 align:middle line:84% What's the probability that you get all heads? 00:29:41.390 --> 00:29:49.300 align:middle line:84% And it turns out it's about 0.001-- 00:29:49.300 --> 00:29:54.690 align:middle line:84% it's like 1 of our 2 to the 10th. 00:29:54.690 --> 00:29:57.810 align:middle line:90% So yeah, it's about 0.1%. 00:29:57.810 --> 00:30:00.180 align:middle line:84% So if the question is, is it biased? 00:30:00.180 --> 00:30:02.640 align:middle line:90% You'd say, probably, right? 00:30:02.640 --> 00:30:04.540 align:middle line:90% Pretty unlikely. 00:30:04.540 --> 00:30:07.410 align:middle line:84% But what if there are 10,000 people flipping coins, 00:30:07.410 --> 00:30:08.775 align:middle line:90% and one person gets 10 heads. 00:30:08.775 --> 00:30:11.670 align:middle line:90% 00:30:11.670 --> 00:30:14.790 align:middle line:84% So should you be surprised that someone got 10 heads? 00:30:14.790 --> 00:30:16.550 align:middle line:90% Probably not, right? 00:30:16.550 --> 00:30:21.540 align:middle line:90% So you need to adjust for this. 00:30:21.540 --> 00:30:25.980 align:middle line:84% This is the same problem that I mentioned earlier 00:30:25.980 --> 00:30:27.630 align:middle line:90% with the Nature Genetics paper. 00:30:27.630 --> 00:30:33.660 align:middle line:84% You need to adjust for the fact that there are so many genes. 00:30:33.660 --> 00:30:37.410 align:middle line:84% In some sense, people find it somewhat difficult 00:30:37.410 --> 00:30:40.320 align:middle line:84% to understand, because I'm interested in one gene, 00:30:40.320 --> 00:30:45.120 align:middle line:84% but if my probability depends on what kind of chip 00:30:45.120 --> 00:30:48.750 align:middle line:84% I use-- for example, if I use a chip with 10,000 genes, 00:30:48.750 --> 00:30:53.910 align:middle line:84% versus a chip with 30,000 genes, my probability 00:30:53.910 --> 00:30:57.280 align:middle line:84% will be different for the gene that I'm interested in. 00:30:57.280 --> 00:30:59.650 align:middle line:84% So people find it sometimes hard to understand. 00:30:59.650 --> 00:31:02.550 align:middle line:84% But some correction has to be made, I think. 00:31:02.550 --> 00:31:06.330 align:middle line:84% So it's often referred to as multiple testing adjustments. 00:31:06.330 --> 00:31:10.450 align:middle line:84% I think there is some consensus now on how to do this. 00:31:10.450 --> 00:31:13.770 align:middle line:84% So maybe we won't talk about this here. 00:31:13.770 --> 00:31:19.590 align:middle line:84% But people generally-- and I agree with this, as well, 00:31:19.590 --> 00:31:21.330 align:middle line:84% there's something called false discovery, 00:31:21.330 --> 00:31:23.040 align:middle line:90% that you may have heard of. 00:31:23.040 --> 00:31:25.140 align:middle line:84% I think that's a very reasonable approach. 00:31:25.140 --> 00:31:27.550 align:middle line:84% Exactly how to calculate that, it's not so simple. 00:31:27.550 --> 00:31:31.810 align:middle line:90% But it can be done. 00:31:31.810 --> 00:31:35.670 align:middle line:84% All right, so let's talk a little more about the subject 00:31:35.670 --> 00:31:39.250 align:middle line:84% that I'm interested in, in terms of application. 00:31:39.250 --> 00:31:45.180 align:middle line:90% 00:31:45.180 --> 00:31:52.560 align:middle line:84% So eventually, for people that are interested in applications, 00:31:52.560 --> 00:31:57.990 align:middle line:84% clinical application of microarray, or any other type 00:31:57.990 --> 00:32:03.060 align:middle line:84% of high throughput data, is that you 00:32:03.060 --> 00:32:06.480 align:middle line:84% like to use this in your clinical setting. 00:32:06.480 --> 00:32:09.070 align:middle line:84% That is treatment related things. 00:32:09.070 --> 00:32:14.700 align:middle line:84% So the challenge is to stop doing 00:32:14.700 --> 00:32:18.450 align:middle line:84% these simple methodological problems at some point 00:32:18.450 --> 00:32:20.120 align:middle line:90% and move on to real problems. 00:32:20.120 --> 00:32:20.620 align:middle line:90% 00:32:20.620 --> 00:32:24.330 align:middle line:84% And that means incorporating large amount 00:32:24.330 --> 00:32:26.143 align:middle line:90% of phenotypic data. 00:32:26.143 --> 00:32:27.810 align:middle line:84% So in a clinical setting, typically, you 00:32:27.810 --> 00:32:34.200 align:middle line:84% have lots of other types of data listed here. 00:32:34.200 --> 00:32:36.150 align:middle line:84% And we like to find relationships 00:32:36.150 --> 00:32:39.090 align:middle line:84% between genomic data and phenotype data-- 00:32:39.090 --> 00:32:40.890 align:middle line:90% phenotypic data. 00:32:40.890 --> 00:32:43.320 align:middle line:84% So questions could be, what genes 00:32:43.320 --> 00:32:46.530 align:middle line:84% are variable or correlated with particular phenotype? 00:32:46.530 --> 00:32:49.840 align:middle line:84% What should we use as predictors? 00:32:49.840 --> 00:32:54.430 align:middle line:84% So I'm just introducing the framework at a basic level. 00:32:54.430 --> 00:32:56.595 align:middle line:84% And then we'll look at some papers. 00:32:56.595 --> 00:33:00.440 align:middle line:90% 00:33:00.440 --> 00:33:03.940 align:middle line:84% So the most simple case that people 00:33:03.940 --> 00:33:08.410 align:middle line:84% have spent a lot of effort on is the binary case. 00:33:08.410 --> 00:33:14.080 align:middle line:84% So the phenotypic data, in this case, is just a label-- 00:33:14.080 --> 00:33:18.310 align:middle line:84% zero, one, yes, or no, disease, versus normal. 00:33:18.310 --> 00:33:21.640 align:middle line:84% And there are basically every method out there 00:33:21.640 --> 00:33:23.800 align:middle line:84% that could be applied to this, has been applied, 00:33:23.800 --> 00:33:26.620 align:middle line:84% and there's tons of papers on this-- how to do this better. 00:33:26.620 --> 00:33:29.680 align:middle line:84% How do you pick out genes that are related to the label? 00:33:29.680 --> 00:33:33.093 align:middle line:84% And how do you do the prediction? 00:33:33.093 --> 00:33:34.510 align:middle line:84% Have guys talked about prediction? 00:33:34.510 --> 00:33:38.350 align:middle line:84% Like how to-- do you know leave-one-out cross 00:33:38.350 --> 00:33:40.370 align:middle line:90% validation is? 00:33:40.370 --> 00:33:42.670 align:middle line:90% Right, so good. 00:33:42.670 --> 00:33:47.780 align:middle line:84% And then, people did multiple subclasses, not just two, 00:33:47.780 --> 00:33:51.190 align:middle line:90% but multiple cases. 00:33:51.190 --> 00:33:54.010 align:middle line:84% And then, in this case, it's non-ordered-- 00:33:54.010 --> 00:33:55.600 align:middle line:90% so you have different subtypes. 00:33:55.600 --> 00:33:58.180 align:middle line:84% You could have an ordered subclasses-- 00:33:58.180 --> 00:34:01.660 align:middle line:84% that is, if you have a rating for severity of disease, 00:34:01.660 --> 00:34:09.580 align:middle line:84% maybe you have one through 5 as your phenotype. 00:34:09.580 --> 00:34:14.800 align:middle line:84% Continuous-- for some reason, this hasn't really-- 00:34:14.800 --> 00:34:18.790 align:middle line:84% the last two haven't been done as well as they should 00:34:18.790 --> 00:34:21.730 align:middle line:90% have been, or they should be. 00:34:21.730 --> 00:34:25.130 align:middle line:90% Yeah, as well as they should be. 00:34:25.130 --> 00:34:30.190 align:middle line:84% But it's certainly in an area where a lot of progress 00:34:30.190 --> 00:34:31.010 align:middle line:90% needs to be made. 00:34:31.010 --> 00:34:35.020 align:middle line:84% So for example, I did one collaboration 00:34:35.020 --> 00:34:38.960 align:middle line:84% where the phenotype was some invasive ability of cells. 00:34:38.960 --> 00:34:44.860 align:middle line:84% So what genes predict the ability of these cancer cells? 00:34:44.860 --> 00:34:49.120 align:middle line:84% And so how to apply through all this finding genes, and then 00:34:49.120 --> 00:34:52.030 align:middle line:90% finding predictions-- 00:34:52.030 --> 00:34:54.370 align:middle line:84% prediction algorithms, and then making predictions, 00:34:54.370 --> 00:35:00.100 align:middle line:84% how to do that optimally, it's not completely settled. 00:35:00.100 --> 00:35:02.710 align:middle line:84% And finally, the sensor data type 00:35:02.710 --> 00:35:05.660 align:middle line:84% is something that I'm very interested in. 00:35:05.660 --> 00:35:08.560 align:middle line:84% And I'll talk a little more about that later. 00:35:08.560 --> 00:35:11.110 align:middle line:84% So it turns out that a lot of these phenotypes 00:35:11.110 --> 00:35:13.850 align:middle line:84% can be reduced to the binary type. 00:35:13.850 --> 00:35:15.970 align:middle line:84% So if you have multiple subclasses, 00:35:15.970 --> 00:35:18.190 align:middle line:90% and combine them into two. 00:35:18.190 --> 00:35:22.270 align:middle line:84% If you have continuous data you can say, well, low invasive 00:35:22.270 --> 00:35:25.090 align:middle line:84% versus high, you have severity of disease, 00:35:25.090 --> 00:35:26.170 align:middle line:90% you can divide into two. 00:35:26.170 --> 00:35:30.220 align:middle line:84% So you can always turn into a simpler problem. 00:35:30.220 --> 00:35:32.920 align:middle line:84% And that's what people have done in many cases. 00:35:32.920 --> 00:35:34.600 align:middle line:84% But you lose a lot of information. 00:35:34.600 --> 00:35:37.870 align:middle line:84% So you'd like to preserve that and do that-- 00:35:37.870 --> 00:35:39.610 align:middle line:90% carry the same procedure out. 00:35:39.610 --> 00:35:42.550 align:middle line:90% 00:35:42.550 --> 00:35:46.570 align:middle line:84% So I mean, you probably have seen this 00:35:46.570 --> 00:35:49.510 align:middle line:84% before, there are lots of questions that could be asked, 00:35:49.510 --> 00:35:52.650 align:middle line:90% and have been asked. 00:35:52.650 --> 00:35:56.310 align:middle line:84% So just to give you a general framework, 00:35:56.310 --> 00:36:03.360 align:middle line:84% so because there are so many genes in data, 00:36:03.360 --> 00:36:06.720 align:middle line:84% usually there are some type of dimensionality reduction. 00:36:06.720 --> 00:36:08.310 align:middle line:84% Even when you have a very good method, 00:36:08.310 --> 00:36:12.030 align:middle line:84% you still need to reduce to the data set size. 00:36:12.030 --> 00:36:14.760 align:middle line:84% Sometimes unless, you have a nice computer, 00:36:14.760 --> 00:36:16.290 align:middle line:84% things may not even fit on your data 00:36:16.290 --> 00:36:18.160 align:middle line:90% unless you do some of reduction. 00:36:18.160 --> 00:36:22.150 align:middle line:84% So this problem is called feature selection 00:36:22.150 --> 00:36:24.100 align:middle line:90% in computer science. 00:36:24.100 --> 00:36:26.580 align:middle line:84% So there are lots of different ways of filtering genes 00:36:26.580 --> 00:36:29.070 align:middle line:84% you know threshold value from expression, 00:36:29.070 --> 00:36:30.660 align:middle line:90% variation filtering, and so on. 00:36:30.660 --> 00:36:32.220 align:middle line:84% There are lots of tools for doing 00:36:32.220 --> 00:36:34.080 align:middle line:90% the dimensionality reduction. 00:36:34.080 --> 00:36:39.990 align:middle line:84% So typically, either you reduce dimension 00:36:39.990 --> 00:36:45.510 align:middle line:84% by just cutting down the number of genes, 00:36:45.510 --> 00:36:50.200 align:middle line:84% or you could actually find some of minor combinations of genes. 00:36:50.200 --> 00:36:53.200 align:middle line:84% And then, use those as the reduced dimension. 00:36:53.200 --> 00:36:56.970 align:middle line:84% So if you do like principal components, or a single value 00:36:56.970 --> 00:36:59.670 align:middle line:84% composition, you reduce the dimension, 00:36:59.670 --> 00:37:03.630 align:middle line:84% but the dimensions that you have in the end are not genes. 00:37:03.630 --> 00:37:05.130 align:middle line:90% Those are combinations of genes. 00:37:05.130 --> 00:37:07.890 align:middle line:84% So there are different ways of doing this. 00:37:07.890 --> 00:37:12.970 align:middle line:84% And with disadvantages and advantages for each one. 00:37:12.970 --> 00:37:15.930 align:middle line:84% So I won't go through this, but t-test you probably 00:37:15.930 --> 00:37:17.490 align:middle line:90% have heard of. 00:37:17.490 --> 00:37:21.450 align:middle line:84% So let me just spend maybe two or three slides 00:37:21.450 --> 00:37:25.170 align:middle line:84% on just one thing that a lot of people have noticed. 00:37:25.170 --> 00:37:29.220 align:middle line:84% This problem has been fixed in most cases, 00:37:29.220 --> 00:37:31.560 align:middle line:90% say, starting a year or two ago. 00:37:31.560 --> 00:37:36.900 align:middle line:84% But if you go back to earlier papers, 00:37:36.900 --> 00:37:42.130 align:middle line:84% they've done a lot of suboptimal things, or in this case, 00:37:42.130 --> 00:37:44.655 align:middle line:84% just some wrong thing in their study. 00:37:44.655 --> 00:37:47.520 align:middle line:90% 00:37:47.520 --> 00:37:51.180 align:middle line:84% So there are a lot of papers on classifying disease categories. 00:37:51.180 --> 00:37:53.100 align:middle line:84% And this has been the primary, or one 00:37:53.100 --> 00:37:56.520 align:middle line:84% of the primary applications, of Affy technology, 00:37:56.520 --> 00:37:59.670 align:middle line:90% or cDNA technology. 00:37:59.670 --> 00:38:02.160 align:middle line:84% Of course, the performance depends 00:38:02.160 --> 00:38:03.600 align:middle line:90% on the methodology used. 00:38:03.600 --> 00:38:07.770 align:middle line:84% A lot of times you see these machine learning algorithms, 00:38:07.770 --> 00:38:09.780 align:middle line:90% like singular-- 00:38:09.780 --> 00:38:12.030 align:middle line:84% not singular, support vector machines. 00:38:12.030 --> 00:38:14.430 align:middle line:84% Neural networks, they appear to give 00:38:14.430 --> 00:38:15.780 align:middle line:90% pretty good results, often. 00:38:15.780 --> 00:38:19.900 align:middle line:90% 00:38:19.900 --> 00:38:23.050 align:middle line:84% And because you probably know, if there aren't enough samples, 00:38:23.050 --> 00:38:26.170 align:middle line:84% you want to use set training and testing sets 00:38:26.170 --> 00:38:28.720 align:middle line:90% to assess your accuracy. 00:38:28.720 --> 00:38:32.290 align:middle line:84% If not, use leave-one-out cross-validation. 00:38:32.290 --> 00:38:34.540 align:middle line:84% It turns out that a lot of papers in good journals 00:38:34.540 --> 00:38:38.390 align:middle line:84% have made mistakes from prediction accuracy. 00:38:38.390 --> 00:38:41.800 align:middle line:84% And what they state are overestimates 00:38:41.800 --> 00:38:43.070 align:middle line:90% of the real accuracy. 00:38:43.070 --> 00:38:49.613 align:middle line:84% So this wasn't discussed by other people, right? 00:38:49.613 --> 00:38:51.655 align:middle line:84% AUDIENCE: Talked about a paper that [INAUDIBLE].. 00:38:51.655 --> 00:38:55.360 align:middle line:90% PETER PARK: Oh, he did? 00:38:55.360 --> 00:38:57.640 align:middle line:90% In breast cancer? 00:38:57.640 --> 00:39:00.850 align:middle line:84% Right, so let me very briefly then-- so 00:39:00.850 --> 00:39:04.210 align:middle line:84% let me just talk about this one simulation study. 00:39:04.210 --> 00:39:05.245 align:middle line:90% Did he talk about this? 00:39:05.245 --> 00:39:08.110 align:middle line:90% 00:39:08.110 --> 00:39:12.910 align:middle line:84% OK, all right, so this is Rick's-- 00:39:12.910 --> 00:39:15.670 align:middle line:84% Rex Simon at NIH, he's a statistician 00:39:15.670 --> 00:39:17.110 align:middle line:90% who's done a lot of good work. 00:39:17.110 --> 00:39:19.810 align:middle line:90% He published this paper-- 00:39:19.810 --> 00:39:21.625 align:middle line:90% actually I think the date is-- 00:39:21.625 --> 00:39:23.800 align:middle line:90% it may have been 2002. 00:39:23.800 --> 00:39:27.220 align:middle line:84% Anyway, Journal of National Cancer Institute. 00:39:27.220 --> 00:39:28.840 align:middle line:84% So he did a simulation study just 00:39:28.840 --> 00:39:31.880 align:middle line:84% to show you what kind of mistakes could be made. 00:39:31.880 --> 00:39:37.150 align:middle line:90% So he generated simulated data-- 00:39:37.150 --> 00:39:40.990 align:middle line:84% 20 expression profiles, 10 randomly assigned 00:39:40.990 --> 00:39:44.240 align:middle line:84% to one class, the rest of the other. 00:39:44.240 --> 00:39:47.560 align:middle line:84% So in this case, because the data are all random, 00:39:47.560 --> 00:39:49.780 align:middle line:84% there's no true underlying differences. 00:39:49.780 --> 00:39:52.480 align:middle line:84% And that means whatever prediction you should make, 00:39:52.480 --> 00:39:55.990 align:middle line:84% the error should be about half, right? 00:39:55.990 --> 00:40:02.800 align:middle line:84% And he averaged this procedure over 2,000 data sets. 00:40:02.800 --> 00:40:07.890 align:middle line:90% So resubstitution here means-- 00:40:07.890 --> 00:40:10.500 align:middle line:90% OK, so you have 20 samples. 00:40:10.500 --> 00:40:13.290 align:middle line:84% So you built your model, prediction model, 00:40:13.290 --> 00:40:17.190 align:middle line:84% based on all 20, so you have your model. 00:40:17.190 --> 00:40:22.260 align:middle line:84% And then, you test the model on each one. 00:40:22.260 --> 00:40:29.190 align:middle line:84% So if you do that, 98.2% of the time, you make no error. 00:40:29.190 --> 00:40:32.970 align:middle line:84% You always get the labels correct. 00:40:32.970 --> 00:40:35.250 align:middle line:84% Cancer or not, or whatever labels 00:40:35.250 --> 00:40:37.770 align:middle line:84% you might be in a particular study, that's the label. 00:40:37.770 --> 00:40:41.160 align:middle line:90% 00:40:41.160 --> 00:40:43.710 align:middle line:84% And that's clearly a wrong thing to do, right? 00:40:43.710 --> 00:40:46.540 align:middle line:84% You can't build your model based on your data, 00:40:46.540 --> 00:40:49.830 align:middle line:90% and then predict the data. 00:40:49.830 --> 00:40:53.010 align:middle line:84% But what a lot of people have done in good journals 00:40:53.010 --> 00:40:58.440 align:middle line:84% is to remove the test sample after gene selection. 00:40:58.440 --> 00:41:03.660 align:middle line:84% So I have my 20 samples, and then I filter the genes, 00:41:03.660 --> 00:41:08.160 align:middle line:84% and then I pick the genes that are informative of the classes. 00:41:08.160 --> 00:41:12.330 align:middle line:84% And then, to predict the model, I'll leave one out, 00:41:12.330 --> 00:41:16.200 align:middle line:84% or to predict the label, I leave one out, build my model, 00:41:16.200 --> 00:41:19.470 align:middle line:84% but based on the genes that I've already selected. 00:41:19.470 --> 00:41:22.440 align:middle line:84% And then predict the one that's left out. 00:41:22.440 --> 00:41:25.260 align:middle line:84% Then you leave another one out, build a new model 00:41:25.260 --> 00:41:27.990 align:middle line:90% to the prediction, and so on. 00:41:27.990 --> 00:41:32.670 align:middle line:84% So you think, well, that should be OK, right? 00:41:32.670 --> 00:41:38.175 align:middle line:84% But it's OK if the gene set that you get is the same every time 00:41:38.175 --> 00:41:39.000 align:middle line:90% you-- 00:41:39.000 --> 00:41:40.980 align:middle line:84% if they were the same every time you 00:41:40.980 --> 00:41:43.350 align:middle line:84% delete the sample and recalculate. 00:41:43.350 --> 00:41:48.700 align:middle line:84% But I mean, as we saw earlier, the gene not very stable. 00:41:48.700 --> 00:41:52.600 align:middle line:84% So if you do that, there's a lot of bias. 00:41:52.600 --> 00:41:55.500 align:middle line:84% So if you do that in this simulated data set, 00:41:55.500 --> 00:41:58.320 align:middle line:84% 90% of the time, on a random data set, 00:41:58.320 --> 00:41:59.760 align:middle line:90% you have 90% accuracy. 00:41:59.760 --> 00:42:02.280 align:middle line:90% 00:42:02.280 --> 00:42:07.050 align:middle line:84% So generally, the difference is not this big. 00:42:07.050 --> 00:42:10.350 align:middle line:84% But it could be substantial, depending on, say, 00:42:10.350 --> 00:42:12.390 align:middle line:90% the size of your data set. 00:42:12.390 --> 00:42:14.535 align:middle line:84% Now, if you do it correctly, that is, 00:42:14.535 --> 00:42:18.525 align:middle line:84% you remove the data set before selection, before selecting 00:42:18.525 --> 00:42:20.250 align:middle line:84% the genes, you reselect the genes 00:42:20.250 --> 00:42:25.140 align:middle line:84% and you do the prediction, and so on, then you get median 00:42:25.140 --> 00:42:27.600 align:middle line:90% at 11 misclassification. 00:42:27.600 --> 00:42:29.970 align:middle line:84% So you should get 10, so you get something close 00:42:29.970 --> 00:42:31.620 align:middle line:90% to something close to the same. 00:42:31.620 --> 00:42:34.300 align:middle line:84% Of course, no one really does it exactly the way 00:42:34.300 --> 00:42:35.050 align:middle line:90% it should be done. 00:42:35.050 --> 00:42:38.725 align:middle line:84% So the way it should be done is, delete one sample, 00:42:38.725 --> 00:42:40.350 align:middle line:84% and then start everything from scratch. 00:42:40.350 --> 00:42:44.220 align:middle line:84% Like you renormalize, you do everything. 00:42:44.220 --> 00:42:45.780 align:middle line:90% But no one really does that. 00:42:45.780 --> 00:42:50.100 align:middle line:84% So they just actually normalize using all the data, 00:42:50.100 --> 00:42:54.510 align:middle line:84% and then, nowadays, you move on, and then 00:42:54.510 --> 00:42:56.770 align:middle line:84% you move one to the gene selection, and so on. 00:42:56.770 --> 00:42:59.970 align:middle line:84% So I think people probably suspected 00:42:59.970 --> 00:43:05.790 align:middle line:84% that there was a problem with this, but not as big 00:43:05.790 --> 00:43:08.640 align:middle line:90% as it is in certain cases. 00:43:08.640 --> 00:43:11.200 align:middle line:90% 00:43:11.200 --> 00:43:17.678 align:middle line:84% So nowadays, like if you see a paper in good journals, 00:43:17.678 --> 00:43:19.470 align:middle line:84% it will have the supplementary information, 00:43:19.470 --> 00:43:21.900 align:middle line:84% they'll talk about what the bias is. 00:43:21.900 --> 00:43:26.230 align:middle line:84% But they often report the better rate in the paper. 00:43:26.230 --> 00:43:29.700 align:middle line:90% So not good. 00:43:29.700 --> 00:43:35.040 align:middle line:84% OK, so let's talk about survival times now. 00:43:35.040 --> 00:43:39.670 align:middle line:84% So how many of you are familiar with my censor data? 00:43:39.670 --> 00:43:46.050 align:middle line:84% OK, so I won't really talk about the basics too much then. 00:43:46.050 --> 00:43:50.900 align:middle line:84% I mean, the censoring rate often is in the order of-- 00:43:50.900 --> 00:43:52.230 align:middle line:90% it could be like 50%. 00:43:52.230 --> 00:43:53.220 align:middle line:90% That's not unusual. 00:43:53.220 --> 00:43:56.110 align:middle line:84% So you want to make sure that you take care of your censoring 00:43:56.110 --> 00:43:58.440 align:middle line:90% correctly. 00:43:58.440 --> 00:44:01.530 align:middle line:84% And generally, we talk about the right censoring-- 00:44:01.530 --> 00:44:03.600 align:middle line:84% you have the patient, a study's terminated, 00:44:03.600 --> 00:44:06.100 align:middle line:84% [INAUDIBLE] patients die, your patient drops out of a study. 00:44:06.100 --> 00:44:09.420 align:middle line:90% 00:44:09.420 --> 00:44:13.950 align:middle line:84% OK, so as n the other phenotypes, 00:44:13.950 --> 00:44:20.490 align:middle line:84% the simplest thing you can do is to use standard univariate 00:44:20.490 --> 00:44:21.060 align:middle line:90% approach. 00:44:21.060 --> 00:44:24.690 align:middle line:84% That is, I look at my genes one at a times, 00:44:24.690 --> 00:44:27.480 align:middle line:84% see as this gene relates to survival. 00:44:27.480 --> 00:44:31.110 align:middle line:84% So there are many ways of doing that because, I mean, 00:44:31.110 --> 00:44:33.730 align:middle line:84% this problem has been around for decades. 00:44:33.730 --> 00:44:36.690 align:middle line:84% And so there are good methods for doing this. 00:44:36.690 --> 00:44:42.390 align:middle line:84% For example, log ranked tests will give you 00:44:42.390 --> 00:44:46.470 align:middle line:84% some p-value as to how that gene is correlated 00:44:46.470 --> 00:44:49.150 align:middle line:90% with your survival data. 00:44:49.150 --> 00:44:51.630 align:middle line:84% And so a lot of people will do that, 00:44:51.630 --> 00:44:53.078 align:middle line:84% some sort of univariate approach, 00:44:53.078 --> 00:44:54.870 align:middle line:84% considering [INAUDIBLE] time, and then they 00:44:54.870 --> 00:44:58.420 align:middle line:84% put together at the end, some type of voting method. 00:44:58.420 --> 00:45:00.480 align:middle line:84% So to do a prediction of the patient, 00:45:00.480 --> 00:45:04.810 align:middle line:84% you say, well, what does my gene say about this patient. 00:45:04.810 --> 00:45:07.050 align:middle line:90% What about gene two? 00:45:07.050 --> 00:45:08.640 align:middle line:90% What about 100? 00:45:08.640 --> 00:45:09.780 align:middle line:90% They add up. 00:45:09.780 --> 00:45:13.620 align:middle line:84% Sometimes weighted differently, depending 00:45:13.620 --> 00:45:17.400 align:middle line:84% on how good the gene is in predicting-- 00:45:17.400 --> 00:45:22.360 align:middle line:84% or in its correlation to the phenotype. 00:45:22.360 --> 00:45:28.060 align:middle line:84% So as I'll talk a little more later, 00:45:28.060 --> 00:45:29.740 align:middle line:90% this is often not optimal. 00:45:29.740 --> 00:45:33.010 align:middle line:84% So I mean, it's how people do it because it's easy, 00:45:33.010 --> 00:45:33.885 align:middle line:90% but it's not optimal. 00:45:33.885 --> 00:45:36.580 align:middle line:90% 00:45:36.580 --> 00:45:39.340 align:middle line:90% So I'll skip that. 00:45:39.340 --> 00:45:40.770 align:middle line:90% So let's just do some-- 00:45:40.770 --> 00:45:44.460 align:middle line:90% 00:45:44.460 --> 00:45:48.310 align:middle line:90% take a look at some examples. 00:45:48.310 --> 00:45:51.900 align:middle line:84% So eventually, you like to do your study 00:45:51.900 --> 00:45:55.170 align:middle line:84% and then correlate that with your phenotype. 00:45:55.170 --> 00:45:57.720 align:middle line:84% And then actually go on to show that whatever-- 00:45:57.720 --> 00:46:00.690 align:middle line:84% you don't want to just show that it's related to the phenotype. 00:46:00.690 --> 00:46:03.780 align:middle line:84% You want to show that it's better 00:46:03.780 --> 00:46:06.150 align:middle line:90% than what people use currently. 00:46:06.150 --> 00:46:09.990 align:middle line:84% So this wasn't done until, let's say, 2001, 2002. 00:46:09.990 --> 00:46:12.840 align:middle line:84% So people just said, oh, my expression profiles 00:46:12.840 --> 00:46:15.690 align:middle line:90% are related to my phenotype. 00:46:15.690 --> 00:46:17.650 align:middle line:90% But it's actually better. 00:46:17.650 --> 00:46:19.230 align:middle line:90% So it's actually-- 00:46:19.230 --> 00:46:21.730 align:middle line:84% I guess this is an interesting question for debate. 00:46:21.730 --> 00:46:24.780 align:middle line:84% I mean, people have published all these papers 00:46:24.780 --> 00:46:26.670 align:middle line:90% for two or three years. 00:46:26.670 --> 00:46:30.180 align:middle line:84% But I actually don't know of any hospital that 00:46:30.180 --> 00:46:36.930 align:middle line:84% will do the testing and then do prediction on the patient. 00:46:36.930 --> 00:46:40.170 align:middle line:84% I mean, there was some report of a plan 00:46:40.170 --> 00:46:49.000 align:middle line:84% to do this in Netherlands, but this hasn't really come about. 00:46:49.000 --> 00:46:50.970 align:middle line:84% So I mean, I actually had a good friend 00:46:50.970 --> 00:46:53.370 align:middle line:84% in the pathology department who's actually 00:46:53.370 --> 00:46:56.760 align:middle line:84% done a lot of arrays himself, and so now 00:46:56.760 --> 00:47:02.070 align:middle line:84% that he finished his PhD, he's in a pathology fellowship, 00:47:02.070 --> 00:47:06.870 align:middle line:84% he said, OK, maybe I'll try to get this thing to the clinic. 00:47:06.870 --> 00:47:10.260 align:middle line:84% And it turns out, there's just a lot of problems. 00:47:10.260 --> 00:47:13.410 align:middle line:84% For one thing, a lot of these studies 00:47:13.410 --> 00:47:19.140 align:middle line:84% have not shown that, compared to the cost involved, 00:47:19.140 --> 00:47:20.940 align:middle line:90% that it's actually a good deal. 00:47:20.940 --> 00:47:23.190 align:middle line:84% So apparently, there are very simple tests 00:47:23.190 --> 00:47:26.575 align:middle line:84% for a lot of, say, cancer-related diagnoses. 00:47:26.575 --> 00:47:28.950 align:middle line:84% There are very simple tests that are very cheap, and very 00:47:28.950 --> 00:47:30.990 align:middle line:90% easy to do. 00:47:30.990 --> 00:47:32.710 align:middle line:90% Take very little time. 00:47:32.710 --> 00:47:35.740 align:middle line:84% So he said, it's not clear in the clinic 00:47:35.740 --> 00:47:41.490 align:middle line:84% whether those tests are actually less accurate than this. 00:47:41.490 --> 00:47:43.690 align:middle line:84% It's much simpler, people already do this. 00:47:43.690 --> 00:47:49.290 align:middle line:84% And so there isn't a great incentive to switch to this. 00:47:49.290 --> 00:47:53.610 align:middle line:84% And probably the bigger reason, though, is money. 00:47:53.610 --> 00:48:01.440 align:middle line:84% So insurance companies these days are paid per patient-- 00:48:01.440 --> 00:48:04.650 align:middle line:84% so I forget the terminology for this. 00:48:04.650 --> 00:48:09.360 align:middle line:84% But if the hospital wants to do an extra procedure 00:48:09.360 --> 00:48:12.685 align:middle line:84% like this test, basically they don't 00:48:12.685 --> 00:48:14.310 align:middle line:84% get reimbursed by the insurance company 00:48:14.310 --> 00:48:16.380 align:middle line:84% unless it can be shown to the insurance company 00:48:16.380 --> 00:48:19.920 align:middle line:84% that this is a necessary procedure to be 00:48:19.920 --> 00:48:22.240 align:middle line:90% done on all these patients. 00:48:22.240 --> 00:48:24.030 align:middle line:90% And that's a difficult thing. 00:48:24.030 --> 00:48:25.480 align:middle line:90% So my friend looked into this. 00:48:25.480 --> 00:48:28.140 align:middle line:84% And it's not easy-- a hospital, or insurance companies 00:48:28.140 --> 00:48:30.930 align:middle line:84% are not willing to shell out money 00:48:30.930 --> 00:48:36.330 align:middle line:84% unless it's proven so in a reasonably good way. 00:48:36.330 --> 00:48:38.790 align:middle line:84% And of course, if I were to, for example, 00:48:38.790 --> 00:48:43.020 align:middle line:84% show them the earlier results on how [? useful ?] these are, 00:48:43.020 --> 00:48:46.060 align:middle line:84% no one's going to, say, pay $1,000 per patient. 00:48:46.060 --> 00:48:52.380 align:middle line:84% So I don't know at what point this is going to really 00:48:52.380 --> 00:48:53.870 align:middle line:90% be done in the clinic. 00:48:53.870 --> 00:48:56.550 align:middle line:90% 00:48:56.550 --> 00:49:03.810 align:middle line:84% But I think as more people publish better papers, 00:49:03.810 --> 00:49:08.280 align:middle line:84% and do a more comprehensive cost benefit analysis, 00:49:08.280 --> 00:49:10.140 align:middle line:90% maybe it will happen. 00:49:10.140 --> 00:49:12.508 align:middle line:84% I think initially, people are very excited that this is 00:49:12.508 --> 00:49:13.800 align:middle line:90% what you're going to do, right? 00:49:13.800 --> 00:49:16.680 align:middle line:84% You walk in the clinic, they do this, and give you 00:49:16.680 --> 00:49:19.230 align:middle line:90% all these diagnoses. 00:49:19.230 --> 00:49:24.340 align:middle line:84% But who knows when this is going to happen. 00:49:24.340 --> 00:49:28.770 align:middle line:84% So yeah, those are some practical issues that 00:49:28.770 --> 00:49:32.610 align:middle line:90% are not so easy to resolve. 00:49:32.610 --> 00:49:36.930 align:middle line:84% In any case, this is a paper that came out in Nature. 00:49:36.930 --> 00:49:41.020 align:middle line:84% This is, I think, a group from the Netherlands, 00:49:41.020 --> 00:49:44.550 align:middle line:90% where I think they are more-- 00:49:44.550 --> 00:49:47.190 align:middle line:84% at least from a couple articles that I read, 00:49:47.190 --> 00:49:52.680 align:middle line:84% they are more closer to actually doing this in the clinic. 00:49:52.680 --> 00:49:56.820 align:middle line:84% And one article actually cited the results 00:49:56.820 --> 00:49:59.470 align:middle line:84% of this paper as a basis for doing this in clinic. 00:49:59.470 --> 00:50:02.670 align:middle line:84% So one problem is that I don't think 00:50:02.670 --> 00:50:04.680 align:middle line:84% what was done in this paper is that great. 00:50:04.680 --> 00:50:07.450 align:middle line:90% So it's a bit of an issue. 00:50:07.450 --> 00:50:11.220 align:middle line:84% But at least in this paper, they claim that this gene expression 00:50:11.220 --> 00:50:13.710 align:middle line:84% profile will outperform all currently 00:50:13.710 --> 00:50:15.720 align:middle line:84% used clinical parameters in predicting disease 00:50:15.720 --> 00:50:17.650 align:middle line:90% outcome in breast cancer. 00:50:17.650 --> 00:50:22.425 align:middle line:84% So is anyone familiar with this paper? 00:50:22.425 --> 00:50:28.670 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:50:28.670 --> 00:50:31.170 align:middle line:84% PETER PARK: All right, so did he talk about the methodology, 00:50:31.170 --> 00:50:32.530 align:middle line:90% as well? 00:50:32.530 --> 00:50:33.405 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:50:33.405 --> 00:50:34.320 align:middle line:90% PETER PARK: Oh, OK. 00:50:34.320 --> 00:50:36.960 align:middle line:90% I must have forgotten. 00:50:36.960 --> 00:50:39.060 align:middle line:84% Did he talk about this briefly, as well? 00:50:39.060 --> 00:50:40.715 align:middle line:90% It's also a breast cancer-- 00:50:40.715 --> 00:50:42.840 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:50:42.840 --> 00:50:44.970 align:middle line:84% PETER PARK: OK, but so you talked about this paper, 00:50:44.970 --> 00:50:45.470 align:middle line:90% but not-- 00:50:45.470 --> 00:50:52.110 align:middle line:90% 00:50:52.110 --> 00:50:54.820 align:middle line:90% OK, I apologize. 00:50:54.820 --> 00:50:59.610 align:middle line:84% I can't remember from last year what papers were covered. 00:50:59.610 --> 00:51:03.150 align:middle line:84% And I knew that the lymphoma papers were 00:51:03.150 --> 00:51:05.730 align:middle line:90% covered by Zack, or somebody. 00:51:05.730 --> 00:51:10.620 align:middle line:84% But then I forgot that Steve might have covered this. 00:51:10.620 --> 00:51:16.620 align:middle line:84% Anyway, so actually, there isn't that much time 00:51:16.620 --> 00:51:20.880 align:middle line:84% so it's good that he covered this. 00:51:20.880 --> 00:51:24.120 align:middle line:84% So basically, the approach is you 00:51:24.120 --> 00:51:28.530 align:middle line:84% build some sort of classifier, you pick out the genes 00:51:28.530 --> 00:51:29.820 align:middle line:90% through some classifier. 00:51:29.820 --> 00:51:34.560 align:middle line:84% And then, so the way that these papers prove 00:51:34.560 --> 00:51:40.260 align:middle line:84% that things are good, or the expression profiles are better, 00:51:40.260 --> 00:51:45.780 align:middle line:84% is that you do some of Kaplan-Meier plot for each one, 00:51:45.780 --> 00:51:48.690 align:middle line:84% and then you do a log rank test, or some tests like that, 00:51:48.690 --> 00:51:52.980 align:middle line:84% to show that there's a big difference. 00:51:52.980 --> 00:51:59.490 align:middle line:84% So they do this for a variety of stratified groups. 00:51:59.490 --> 00:52:04.830 align:middle line:84% So you might take patients that were assigned 00:52:04.830 --> 00:52:11.610 align:middle line:84% one category of disease by your typical prognosis, 00:52:11.610 --> 00:52:14.190 align:middle line:84% and then you show that, within that group, 00:52:14.190 --> 00:52:15.880 align:middle line:90% there's enough variability. 00:52:15.880 --> 00:52:19.920 align:middle line:84% So somehow, there is some information 00:52:19.920 --> 00:52:22.275 align:middle line:84% in the expression data that's not captured by this. 00:52:22.275 --> 00:52:28.670 align:middle line:90% 00:52:28.670 --> 00:52:34.200 align:middle line:84% And the next step that are done increasingly now 00:52:34.200 --> 00:52:36.540 align:middle line:84% is after you get this classifier, 00:52:36.540 --> 00:52:39.780 align:middle line:84% you put that together with all the other data. 00:52:39.780 --> 00:52:44.520 align:middle line:90% So in this case-- 00:52:44.520 --> 00:52:50.040 align:middle line:84% in this case, they have, basically 00:52:50.040 --> 00:52:53.220 align:middle line:84% out of all the expression data that they have, 00:52:53.220 --> 00:52:54.960 align:middle line:84% they just come up with one signature. 00:52:54.960 --> 00:52:59.950 align:middle line:84% So yes, it looks good, versus no, it doesn't. 00:52:59.950 --> 00:53:03.060 align:middle line:84% And so that becomes just a one variable 00:53:03.060 --> 00:53:07.540 align:middle line:84% in your multivariate model that you have. 00:53:07.540 --> 00:53:11.790 align:middle line:84% So typically, most studies, without expression data, 00:53:11.790 --> 00:53:18.120 align:middle line:84% will just have a multivariate model, Cox model, sometimes. 00:53:18.120 --> 00:53:24.540 align:middle line:84% So Cox model is just a multiple linear regression, sort of done 00:53:24.540 --> 00:53:28.590 align:middle line:90% differently for censor data. 00:53:28.590 --> 00:53:31.410 align:middle line:84% So for Cox model, you fit everything. 00:53:31.410 --> 00:53:34.620 align:middle line:84% And then if you say, if you see that the p-value is 00:53:34.620 --> 00:53:37.830 align:middle line:84% small for that, then you say, well 00:53:37.830 --> 00:53:42.780 align:middle line:84% it's a new parameter, or new variable 00:53:42.780 --> 00:53:45.340 align:middle line:84% that you should take into account in your study. 00:53:45.340 --> 00:53:47.370 align:middle line:84% So I think this is a fairly reasonable approach. 00:53:47.370 --> 00:53:50.130 align:middle line:90% 00:53:50.130 --> 00:53:51.690 align:middle line:90% There is another study-- 00:53:51.690 --> 00:53:54.090 align:middle line:84% you may have seen this, as well-- 00:53:54.090 --> 00:53:58.230 align:middle line:84% where they do something they do something better, I think. 00:53:58.230 --> 00:54:07.500 align:middle line:84% Which is, after you get all your expression data, 00:54:07.500 --> 00:54:11.430 align:middle line:84% they actually manually go through the top genes, 00:54:11.430 --> 00:54:14.530 align:middle line:84% and then try to come up with new variables. 00:54:14.530 --> 00:54:18.240 align:middle line:84% So I think this good signature-- which is batch signature-- 00:54:18.240 --> 00:54:19.440 align:middle line:90% is too crude. 00:54:19.440 --> 00:54:25.620 align:middle line:84% It doesn't really give you as much information from the data. 00:54:25.620 --> 00:54:29.220 align:middle line:84% I think you could get, I think, a lot more information 00:54:29.220 --> 00:54:30.310 align:middle line:90% from your expression data. 00:54:30.310 --> 00:54:33.420 align:middle line:84% So in this example, they actually 00:54:33.420 --> 00:54:36.330 align:middle line:84% have not yes or no score, but they actually 00:54:36.330 --> 00:54:37.950 align:middle line:90% have a prediction score. 00:54:37.950 --> 00:54:41.320 align:middle line:90% So that's a continuous variable. 00:54:41.320 --> 00:54:44.920 align:middle line:84% And then, they actually classify their genes, 00:54:44.920 --> 00:54:46.950 align:middle line:84% depending on their expression profiles, 00:54:46.950 --> 00:54:52.750 align:middle line:84% and their known and annotations into these groups. 00:54:52.750 --> 00:54:54.900 align:middle line:84% So in this case they have five groups, 00:54:54.900 --> 00:54:56.760 align:middle line:84% so proliferation signature value-- 00:54:56.760 --> 00:55:02.665 align:middle line:90% this is a set of genes in P6-- 00:55:02.665 --> 00:55:04.290 align:middle line:84% that's just one gene that just happened 00:55:04.290 --> 00:55:07.670 align:middle line:84% to be different from all the other profiles. 00:55:07.670 --> 00:55:09.610 align:middle line:84% And then they have these signature values. 00:55:09.610 --> 00:55:13.770 align:middle line:84% So I think this is a fairly reasonable approach. 00:55:13.770 --> 00:55:15.930 align:middle line:84% And I think it gives you a lot more information. 00:55:15.930 --> 00:55:19.200 align:middle line:84% Plus these coefficients, I think, 00:55:19.200 --> 00:55:24.900 align:middle line:84% give you a more robust view of how 00:55:24.900 --> 00:55:27.330 align:middle line:84% that expression, that group of genes, 00:55:27.330 --> 00:55:33.660 align:middle line:84% is impacting your survival in this case. 00:55:33.660 --> 00:55:35.280 align:middle line:90% So you get a prediction score. 00:55:35.280 --> 00:55:37.740 align:middle line:84% And then, I guess from here, you could do-- 00:55:37.740 --> 00:55:41.760 align:middle line:84% you could put other things, other variables 00:55:41.760 --> 00:55:45.360 align:middle line:84% in this setting, in the regression setting. 00:55:45.360 --> 00:55:50.095 align:middle line:84% Actually, there's a paper that came out this past week-- 00:55:50.095 --> 00:55:50.595 align:middle line:90% this week. 00:55:50.595 --> 00:55:54.180 align:middle line:90% 00:55:54.180 --> 00:55:57.930 align:middle line:90% And-- you're smiling? 00:55:57.930 --> 00:55:59.070 align:middle line:90% AUDIENCE: I know this one. 00:55:59.070 --> 00:56:00.510 align:middle line:84% PETER PARK: Oh, you know this one. 00:56:00.510 --> 00:56:01.290 align:middle line:90% OK. 00:56:01.290 --> 00:56:06.300 align:middle line:84% So I mean, I feel like at some point, 00:56:06.300 --> 00:56:09.670 align:middle line:84% someone can write a software where you add the data set, 00:56:09.670 --> 00:56:12.780 align:middle line:84% and it'll just do all the things that are done. 00:56:12.780 --> 00:56:19.500 align:middle line:84% Because it's, I think, fairly similar in all the papers. 00:56:19.500 --> 00:56:25.170 align:middle line:84% Except the algorithms-- and basically, 00:56:25.170 --> 00:56:28.320 align:middle line:84% the choice of algorithms used is like who 00:56:28.320 --> 00:56:31.590 align:middle line:84% the investigator knows, who knows how to do these things. 00:56:31.590 --> 00:56:35.940 align:middle line:84% So in this case, there's Tef Srini 00:56:35.940 --> 00:56:40.080 align:middle line:84% is a statistician at Stanford, and so this paper uses 00:56:40.080 --> 00:56:42.240 align:middle line:84% the methods that he developed, which I 00:56:42.240 --> 00:56:44.740 align:middle line:90% think is a fairly good method. 00:56:44.740 --> 00:56:52.190 align:middle line:84% So just to give you a rough idea of what they do here 00:56:52.190 --> 00:56:56.330 align:middle line:84% is, so they make predictions in the end-- 00:56:56.330 --> 00:57:00.200 align:middle line:84% so they select genes by the method 00:57:00.200 --> 00:57:03.620 align:middle line:84% that Tef Srini, which is actually-- it's 00:57:03.620 --> 00:57:05.000 align:middle line:90% a little complicated to explain. 00:57:05.000 --> 00:57:11.480 align:middle line:84% But it basically does a search through all your expression 00:57:11.480 --> 00:57:15.410 align:middle line:84% of genes, but you can also actually 00:57:15.410 --> 00:57:17.240 align:middle line:90% look for combinations of genes. 00:57:17.240 --> 00:57:23.370 align:middle line:84% So if you remember, basically all the typical methods 00:57:23.370 --> 00:57:26.040 align:middle line:84% will look for a gene that's related to the phenotype. 00:57:26.040 --> 00:57:30.260 align:middle line:84% So if there is any interaction between genes, 00:57:30.260 --> 00:57:33.920 align:middle line:90% you basically lose them all. 00:57:33.920 --> 00:57:36.260 align:middle line:84% So there is some fancy method that he 00:57:36.260 --> 00:57:39.530 align:middle line:84% developed where you basically search through all your genes-- 00:57:39.530 --> 00:57:45.320 align:middle line:84% it's like regression, stepwise regression type, 00:57:45.320 --> 00:57:47.652 align:middle line:84% where you basically search through your genes, 00:57:47.652 --> 00:57:49.860 align:middle line:84% you keep the one that gives you the best information. 00:57:49.860 --> 00:57:52.820 align:middle line:84% And then, you try next set, and then you 00:57:52.820 --> 00:57:54.780 align:middle line:84% try combinations, and that kind of thing. 00:57:54.780 --> 00:57:58.260 align:middle line:84% So they identify a bunch of genes. 00:57:58.260 --> 00:58:00.570 align:middle line:90% And then they do a prediction. 00:58:00.570 --> 00:58:05.060 align:middle line:84% So they had 116 adults, you have one group-- 00:58:05.060 --> 00:58:09.830 align:middle line:84% a training set with 59, you do a prediction on 57. 00:58:09.830 --> 00:58:12.080 align:middle line:84% The way they do the prediction is 00:58:12.080 --> 00:58:16.310 align:middle line:84% an algorithm that was published in TNAS a year or two ago. 00:58:16.310 --> 00:58:22.820 align:middle line:84% Basically, you take a new sample, 00:58:22.820 --> 00:58:26.960 align:middle line:84% and then see which of the clusters 00:58:26.960 --> 00:58:28.610 align:middle line:90% the profile is closest to. 00:58:28.610 --> 00:58:31.470 align:middle line:84% But they do this in a slightly clever way. 00:58:31.470 --> 00:58:35.060 align:middle line:84% So instead of just looking at correlations, 00:58:35.060 --> 00:58:38.540 align:middle line:84% they do something a little bit fancier. 00:58:38.540 --> 00:58:43.640 align:middle line:84% But I mean, the underlying idea is the same. 00:58:43.640 --> 00:58:47.810 align:middle line:84% And then, in the end, they do the multivariate analysis gene 00:58:47.810 --> 00:58:49.580 align:middle line:84% expression predictor-- is strongly 00:58:49.580 --> 00:58:54.170 align:middle line:90% independent prognostic factor. 00:58:54.170 --> 00:58:58.100 align:middle line:84% So this is a figure in the paper. 00:58:58.100 --> 00:59:05.640 align:middle line:84% I guess at this point, we'll skip this. 00:59:05.640 --> 00:59:09.530 align:middle line:84% So for the next actually about five, 10 minutes, 00:59:09.530 --> 00:59:14.270 align:middle line:84% let me talk about something that I did. 00:59:14.270 --> 00:59:20.420 align:middle line:84% So most studies so far are used survival curves 00:59:20.420 --> 00:59:21.920 align:middle line:84% as a way of verifying the results. 00:59:21.920 --> 00:59:27.380 align:middle line:84% So you basically do the clustering, unsupervised 00:59:27.380 --> 00:59:29.880 align:middle line:84% clustering, and then you define groups and say, 00:59:29.880 --> 00:59:31.740 align:middle line:90% are they really different? 00:59:31.740 --> 00:59:37.610 align:middle line:84% So if they're really different as verified by these curves, 00:59:37.610 --> 00:59:41.000 align:middle line:90% then you say, oh, that's great. 00:59:41.000 --> 00:59:47.330 align:middle line:84% But in some sense, this is an indirect way of doing things. 00:59:47.330 --> 00:59:49.490 align:middle line:84% I mean, if maybe there is a different way 00:59:49.490 --> 00:59:51.950 align:middle line:84% of clustering that gives you a better result, who knows. 00:59:51.950 --> 00:59:54.740 align:middle line:84% This is just checking to see if your clustering was 00:59:54.740 --> 00:59:57.570 align:middle line:90% done correctly. 00:59:57.570 --> 01:00:03.815 align:middle line:84% So in another way, another problem with this is that-- 01:00:03.815 --> 01:00:09.800 align:middle line:90% 01:00:09.800 --> 01:00:14.540 align:middle line:84% yeah, well, I mean, so that's, I think, one way of doing things. 01:00:14.540 --> 01:00:17.630 align:middle line:84% Another way, actually, to do the censoring is 01:00:17.630 --> 01:00:20.030 align:middle line:84% as was done in a lot of the other papers, 01:00:20.030 --> 01:00:22.250 align:middle line:84% is to turn survival times into binary indicator, 01:00:22.250 --> 01:00:23.925 align:middle line:90% as I mentioned. 01:00:23.925 --> 01:00:25.550 align:middle line:84% So something that we were interested in 01:00:25.550 --> 01:00:28.190 align:middle line:84% was whether there is some combination. 01:00:28.190 --> 01:00:30.590 align:middle line:84% So maybe it's not really gene A that's 01:00:30.590 --> 01:00:34.340 align:middle line:84% predictive, but it's gene A plus half of gene B, 01:00:34.340 --> 01:00:36.770 align:middle line:90% plus 2 times gene C-- 01:00:36.770 --> 01:00:42.290 align:middle line:84% maybe that's really the most predictive of survival time. 01:00:42.290 --> 01:00:47.210 align:middle line:84% So otherwise, you don't know exactly how 01:00:47.210 --> 01:00:49.160 align:middle line:90% to combine your genes. 01:00:49.160 --> 01:00:55.387 align:middle line:84% If you do some of [? boding ?] method, for example, 01:00:55.387 --> 01:00:57.470 align:middle line:84% you are combining information from all your genes, 01:00:57.470 --> 01:00:59.220 align:middle line:84% but you're not doing it in an optimal way. 01:00:59.220 --> 01:01:02.300 align:middle line:84% So we actually thought about this problem, 01:01:02.300 --> 01:01:06.500 align:middle line:84% and then came up with what I think is a good solution. 01:01:06.500 --> 01:01:11.030 align:middle line:84% Of course, the difficulty is that it's too complicated, 01:01:11.030 --> 01:01:15.410 align:middle line:84% and so it's hard to write a software that someone can use 01:01:15.410 --> 01:01:20.310 align:middle line:84% to just press a button and get a good result. 01:01:20.310 --> 01:01:25.770 align:middle line:84% And we can't just do analysis for other people. 01:01:25.770 --> 01:01:33.440 align:middle line:84% So as in many bioinformatics algorithms, it's kind of there, 01:01:33.440 --> 01:01:36.260 align:middle line:84% but we haven't really done as much 01:01:36.260 --> 01:01:38.640 align:middle line:90% as we could have done with it. 01:01:38.640 --> 01:01:42.110 align:middle line:84% So just to give you a brief overview of this-- 01:01:42.110 --> 01:01:45.620 align:middle line:84% so the basic problem is that you have too many variables. 01:01:45.620 --> 01:01:49.790 align:middle line:84% So we use some method, it's called partial least squares. 01:01:49.790 --> 01:01:53.150 align:middle line:84% It's actually becoming very popular now. 01:01:53.150 --> 01:01:56.690 align:middle line:84% And it turns out that it's a compromise between just doing 01:01:56.690 --> 01:01:59.480 align:middle line:84% least squares, like your regression, versus PCA. 01:01:59.480 --> 01:02:03.500 align:middle line:84% So principle component analysis is 01:02:03.500 --> 01:02:07.490 align:middle line:84% good in that you try to maximize information contained 01:02:07.490 --> 01:02:11.240 align:middle line:84% in your few components, but it doesn't really have anything 01:02:11.240 --> 01:02:13.250 align:middle line:90% to do with the phenotype. 01:02:13.250 --> 01:02:16.340 align:middle line:84% So if you pick variables based on PCA, 01:02:16.340 --> 01:02:19.400 align:middle line:84% you do it to measure reduction in some optimal way. 01:02:19.400 --> 01:02:21.680 align:middle line:84% But it may not be related to your phenotype. 01:02:21.680 --> 01:02:25.700 align:middle line:84% If you do regression, you are picking 01:02:25.700 --> 01:02:29.000 align:middle line:84% something that's really tightly correlated with your phenotype. 01:02:29.000 --> 01:02:31.850 align:middle line:84% But you're not really doing any dimension reduction. 01:02:31.850 --> 01:02:33.567 align:middle line:84% Or you can't fit it in an optimal way. 01:02:33.567 --> 01:02:35.900 align:middle line:84% So there's something called partial least squares that's 01:02:35.900 --> 01:02:39.680 align:middle line:84% a compromise between the two, and it appears to work really 01:02:39.680 --> 01:02:42.035 align:middle line:90% well in most cases. 01:02:42.035 --> 01:02:44.537 align:middle line:90% 01:02:44.537 --> 01:02:46.620 align:middle line:84% And it turns out that someone had worked this out. 01:02:46.620 --> 01:02:49.880 align:middle line:84% So they figured out how to do partial least squares 01:02:49.880 --> 01:02:54.680 align:middle line:84% for different types of phenotypes continuous 01:02:54.680 --> 01:02:59.280 align:middle line:90% and binary phenotype. 01:02:59.280 --> 01:03:02.120 align:middle line:84% So it gets a little complicated, as soon 01:03:02.120 --> 01:03:07.310 align:middle line:84% as you have a phenotype that's not binary, or as soon 01:03:07.310 --> 01:03:10.320 align:middle line:84% as it's not continuous but it has been worked out. 01:03:10.320 --> 01:03:14.000 align:middle line:84% But when you have censoring, well, again, 01:03:14.000 --> 01:03:15.500 align:middle line:84% if you have a small number of genes, 01:03:15.500 --> 01:03:18.080 align:middle line:84% you could use something like Cox model 01:03:18.080 --> 01:03:22.040 align:middle line:84% to correlate gene expression to censoring. 01:03:22.040 --> 01:03:24.470 align:middle line:84% But when you have too many genes, you can't do that. 01:03:24.470 --> 01:03:25.740 align:middle line:90% Your standard methods fail. 01:03:25.740 --> 01:03:27.590 align:middle line:84% So the question was, can we do this? 01:03:27.590 --> 01:03:30.050 align:middle line:84% Can we apply a method that's well 01:03:30.050 --> 01:03:34.190 align:middle line:84% known to work for regular, just other simpler phenotype, 01:03:34.190 --> 01:03:35.570 align:middle line:90% but can we work-- 01:03:35.570 --> 01:03:37.640 align:middle line:84% can we get this to work for censor data? 01:03:37.640 --> 01:03:43.980 align:middle line:84% So it turns out that we came up with a fairly good solution. 01:03:43.980 --> 01:03:47.190 align:middle line:84% It turns out that this was worked out many years ago, 01:03:47.190 --> 01:03:49.267 align:middle line:84% we discovered, but there was no need-- 01:03:49.267 --> 01:03:50.850 align:middle line:84% they didn't have high throughput data. 01:03:50.850 --> 01:03:52.968 align:middle line:84% So there was no need to use this. 01:03:52.968 --> 01:03:55.010 align:middle line:84% It was done by a statistician, and it's very hard 01:03:55.010 --> 01:03:55.593 align:middle line:90% to understand. 01:03:55.593 --> 01:03:58.190 align:middle line:84% I mean, I can barely understand it myself. 01:03:58.190 --> 01:04:02.540 align:middle line:84% So in some extent, it was reinventing the wheel. 01:04:02.540 --> 01:04:05.460 align:middle line:84% But we do it a little differently. 01:04:05.460 --> 01:04:08.340 align:middle line:84% And then we were able to apply this. 01:04:08.340 --> 01:04:10.820 align:middle line:84% So it turns out that we were actually 01:04:10.820 --> 01:04:16.530 align:middle line:84% able to get some really good results. 01:04:16.530 --> 01:04:19.880 align:middle line:84% For example, if I use the results of that algorithm 01:04:19.880 --> 01:04:22.440 align:middle line:84% to divide the patients into different groups, 01:04:22.440 --> 01:04:25.400 align:middle line:84% then it's more significant than using some other method 01:04:25.400 --> 01:04:26.670 align:middle line:90% of doing things. 01:04:26.670 --> 01:04:31.160 align:middle line:84% So there are methods like this that could be-- 01:04:31.160 --> 01:04:32.720 align:middle line:90% I think that will be available. 01:04:32.720 --> 01:04:35.930 align:middle line:84% And unfortunately, it does require 01:04:35.930 --> 01:04:39.860 align:middle line:84% some mathematical training, but for people here 01:04:39.860 --> 01:04:43.550 align:middle line:84% that are interested in this and can invest the time and energy, 01:04:43.550 --> 01:04:49.760 align:middle line:84% I mean, I think knowing that the better algorithms will give you 01:04:49.760 --> 01:04:55.430 align:middle line:84% a lot of insights into what's going on underneath your data. 01:04:55.430 --> 01:05:04.613 align:middle line:90% AUDIENCE: [INAUDIBLE] 01:05:04.613 --> 01:05:06.405 align:middle line:84% PETER PARK: Well, that's the problem right? 01:05:06.405 --> 01:05:09.030 align:middle line:84% AUDIENCE: You know, because just so many things will always 01:05:09.030 --> 01:05:10.740 align:middle line:90% be a problem. 01:05:10.740 --> 01:05:13.830 align:middle line:84% Even if they randomly [INAUDIBLE] 01:05:13.830 --> 01:05:15.435 align:middle line:84% had nothing to do with the survival, 01:05:15.435 --> 01:05:20.040 align:middle line:84% they can always find [INAUDIBLE].. 01:05:20.040 --> 01:05:23.490 align:middle line:84% PETER PARK: But not as well, right? 01:05:23.490 --> 01:05:26.220 align:middle line:84% Yeah, but it's actually not that likely, 01:05:26.220 --> 01:05:28.510 align:middle line:84% if you actually do the calculation. 01:05:28.510 --> 01:05:29.010 align:middle line:90% But-- 01:05:29.010 --> 01:05:30.660 align:middle line:90% AUDIENCE: [INAUDIBLE] 01:05:30.660 --> 01:05:36.300 align:middle line:84% PETER PARK: Yeah, you could find some combinations that will-- 01:05:36.300 --> 01:05:39.780 align:middle line:84% and it's true for anything, right? 01:05:39.780 --> 01:05:44.543 align:middle line:84% But even for the binary case, if you have 10 in each group, 01:05:44.543 --> 01:05:45.960 align:middle line:84% what's the probability that you're 01:05:45.960 --> 01:05:50.580 align:middle line:84% going to find a random predictor that will be lined-- 01:05:50.580 --> 01:05:52.110 align:middle line:84% if it's continuous data-- that will 01:05:52.110 --> 01:05:55.510 align:middle line:84% be lined up exactly as in the order of the patients? 01:05:55.510 --> 01:05:57.180 align:middle line:90% It's not that likely. 01:05:57.180 --> 01:05:59.160 align:middle line:90% So if it's not binary, right? 01:05:59.160 --> 01:06:01.380 align:middle line:90% It's much less likely. 01:06:01.380 --> 01:06:03.480 align:middle line:84% AUDIENCE: So you'd think the binary [INAUDIBLE].. 01:06:03.480 --> 01:06:04.860 align:middle line:90% PETER PARK: Yeah, yeah, exactly. 01:06:04.860 --> 01:06:08.040 align:middle line:84% Yeah, so that's the problem, right? 01:06:08.040 --> 01:06:13.290 align:middle line:84% Even for these Kaplan-Meier curves that people show, 01:06:13.290 --> 01:06:18.960 align:middle line:84% they need to adjust for multiple testing in some way, right? 01:06:18.960 --> 01:06:21.390 align:middle line:84% So you can certainly find genes that 01:06:21.390 --> 01:06:23.580 align:middle line:84% will give you a significant p-value, 01:06:23.580 --> 01:06:26.700 align:middle line:90% even though the data are random. 01:06:26.700 --> 01:06:29.850 align:middle line:84% So some people do permutation tests 01:06:29.850 --> 01:06:33.700 align:middle line:90% to adjust for the p-value. 01:06:33.700 --> 01:06:41.050 align:middle line:84% So conclusions-- I don't need to go with that here. 01:06:41.050 --> 01:06:47.460 align:middle line:84% So let's see-- just as a final comment. 01:06:47.460 --> 01:06:51.840 align:middle line:90% 01:06:51.840 --> 01:06:54.120 align:middle line:84% To summarize briefly, I would say 01:06:54.120 --> 01:06:58.240 align:middle line:84% that some of the recent papers have done it pretty well. 01:06:58.240 --> 01:06:59.220 align:middle line:90% I think, analysis. 01:06:59.220 --> 01:07:05.580 align:middle line:84% I think that involves not just doing the typical cluster, 01:07:05.580 --> 01:07:09.570 align:middle line:84% and then finding the p-value for the Kaplan-Meier curves, 01:07:09.570 --> 01:07:14.250 align:middle line:84% but actually getting a good score of some sort, 01:07:14.250 --> 01:07:18.480 align:middle line:84% and then incorporating that into the multivariate model. 01:07:18.480 --> 01:07:21.210 align:middle line:84% And then, I mean, that part is actually not trivial. 01:07:21.210 --> 01:07:27.300 align:middle line:84% But if you're familiar with this statistical package, something 01:07:27.300 --> 01:07:30.540 align:middle line:84% like this, it is actually not hard. 01:07:30.540 --> 01:07:33.960 align:middle line:84% So you should be able to, I think, 01:07:33.960 --> 01:07:35.880 align:middle line:84% if you understand these papers really well, 01:07:35.880 --> 01:07:38.422 align:middle line:84% there are a lot of little steps that you have to worry about, 01:07:38.422 --> 01:07:41.700 align:middle line:84% you should probably be able to do everything on your own, 01:07:41.700 --> 01:07:43.860 align:middle line:90% I think. 01:07:43.860 --> 01:07:47.350 align:middle line:90% So without too much difficulty. 01:07:47.350 --> 01:07:53.760 align:middle line:84% So finally, the software that I use all the time is R-- 01:07:53.760 --> 01:07:56.820 align:middle line:84% are you guys familiar with the software? 01:07:56.820 --> 01:08:02.430 align:middle line:84% What statistics software, if any, do you guys know? 01:08:02.430 --> 01:08:04.200 align:middle line:90% MATLAB? 01:08:04.200 --> 01:08:06.400 align:middle line:90% MATLAB, a lot of people know. 01:08:06.400 --> 01:08:10.470 align:middle line:84% So R is-- so if you go to a statistics department, 01:08:10.470 --> 01:08:13.710 align:middle line:84% they'll use one of two softwares. 01:08:13.710 --> 01:08:17.850 align:middle line:90% One is at SAS, people use that. 01:08:17.850 --> 01:08:19.229 align:middle line:90% And the other one is S+. 01:08:19.229 --> 01:08:23.680 align:middle line:90% So SAS has been around forever. 01:08:23.680 --> 01:08:27.340 align:middle line:84% It's like Fortran-- it's like it's been around forever. 01:08:27.340 --> 01:08:29.160 align:middle line:84% There's a lot of people that have 01:08:29.160 --> 01:08:30.689 align:middle line:90% written good software for it. 01:08:30.689 --> 01:08:31.920 align:middle line:90% You know that it works. 01:08:31.920 --> 01:08:33.450 align:middle line:90% But it's very clumsy. 01:08:33.450 --> 01:08:36.779 align:middle line:84% A lot of things that you want to do, you can't do. 01:08:36.779 --> 01:08:43.500 align:middle line:84% They have a better Windows-driven package now 01:08:43.500 --> 01:08:44.550 align:middle line:90% for SAS. 01:08:44.550 --> 01:08:47.130 align:middle line:84% But even just a few years ago, you 01:08:47.130 --> 01:08:49.830 align:middle line:84% will draw plot like using asterisks 01:08:49.830 --> 01:08:57.180 align:middle line:84% on a text kind of page, so draw little characters 01:08:57.180 --> 01:08:59.850 align:middle line:90% on your screen as a plot. 01:08:59.850 --> 01:09:03.960 align:middle line:84% But I think the younger generation that are more 01:09:03.960 --> 01:09:09.240 align:middle line:84% computer savvy are more likely to use S+ unless they have 01:09:09.240 --> 01:09:11.880 align:middle line:84% to use SAS in their statistics courses. 01:09:11.880 --> 01:09:17.490 align:middle line:84% So S+ is more like MATLAB, so it's fairly powerful, 01:09:17.490 --> 01:09:19.660 align:middle line:84% and it gives you a lot of freedom. 01:09:19.660 --> 01:09:24.760 align:middle line:84% And R-- so there's so the reason it's called S+ is there is 01:09:24.760 --> 01:09:27.798 align:middle line:84% a statistical language called S-- 01:09:27.798 --> 01:09:31.410 align:middle line:84% I don't know why it's called S. Why C? 01:09:31.410 --> 01:09:34.109 align:middle line:90% Programming language called C. 01:09:34.109 --> 01:09:38.250 align:middle line:84% And then some-- and this was developed at Bell Labs. 01:09:38.250 --> 01:09:42.479 align:middle line:84% And some company took the code, and then made that 01:09:42.479 --> 01:09:45.450 align:middle line:84% as a product, and they called it S+. 01:09:45.450 --> 01:09:47.580 align:middle line:84% And then now there's a bunch of people, 01:09:47.580 --> 01:09:51.420 align:middle line:84% a lot of whom worked on-- a lot of whom are familiar with S 01:09:51.420 --> 01:09:53.189 align:middle line:90% that developed R. 01:09:53.189 --> 01:09:57.330 align:middle line:84% So actually the guy that wrote one 01:09:57.330 --> 01:10:02.130 align:middle line:84% of the two co-authors of the original R is at Dana-Farber, 01:10:02.130 --> 01:10:06.590 align:middle line:84% and he's been doing a lot of work in microarray analysis, 01:10:06.590 --> 01:10:07.140 align:middle line:90% as well. 01:10:07.140 --> 01:10:12.740 align:middle line:84% But so the software is free, and it's like Linux. 01:10:12.740 --> 01:10:18.020 align:middle line:84% People contribute-- it's well tested. 01:10:18.020 --> 01:10:20.690 align:middle line:90% And very powerful. 01:10:20.690 --> 01:10:22.830 align:middle line:84% And that's what I use all the time. 01:10:22.830 --> 01:10:26.500 align:middle line:84% There are some issues like memory management, 01:10:26.500 --> 01:10:30.050 align:middle line:90% and certain data types. 01:10:30.050 --> 01:10:32.190 align:middle line:84% But I think things are improving. 01:10:32.190 --> 01:10:34.010 align:middle line:90% And so I'm quite happy with it. 01:10:34.010 --> 01:10:36.200 align:middle line:84% And everyone that I've recommended to it 01:10:36.200 --> 01:10:38.040 align:middle line:90% are pretty happy with it. 01:10:38.040 --> 01:10:41.250 align:middle line:84% It has a feel of MATLAB, like you don't declare variables. 01:10:41.250 --> 01:10:42.920 align:middle line:90% You just kind of use them. 01:10:42.920 --> 01:10:44.840 align:middle line:84% Matrix manipulations are very similar. 01:10:44.840 --> 01:10:47.510 align:middle line:84% Everything is done in vector form. 01:10:47.510 --> 01:10:53.450 align:middle line:84% I showed this to one guy in one of the labs 01:10:53.450 --> 01:10:55.430 align:middle line:84% at the children's hospital-- he's a fellow. 01:10:55.430 --> 01:10:58.340 align:middle line:84% And then he was trying to do some microarray data 01:10:58.340 --> 01:11:02.480 align:middle line:84% analysis in Excel, like doing some sort of permutation. 01:11:02.480 --> 01:11:04.160 align:middle line:84% And I showed him how to do this, and he 01:11:04.160 --> 01:11:06.470 align:middle line:84% said, this one line, I mean, it took me 01:11:06.470 --> 01:11:08.450 align:middle line:84% three hours to do this in Excel, this is 01:11:08.450 --> 01:11:11.030 align:middle line:90% one line in R. He's very happy. 01:11:11.030 --> 01:11:15.550 align:middle line:90% AUDIENCE: [INAUDIBLE] 01:11:15.550 --> 01:11:17.220 align:middle line:84% PETER PARK: MATLAB has more statistics, 01:11:17.220 --> 01:11:20.970 align:middle line:84% but I think R has more sophisticated statistics. 01:11:20.970 --> 01:11:25.350 align:middle line:84% So MATLAB is more engineering oriented. 01:11:25.350 --> 01:11:26.520 align:middle line:90% MATLAB has better graphics. 01:11:26.520 --> 01:11:29.100 align:middle line:90% 01:11:29.100 --> 01:11:33.210 align:middle line:84% It's probably better for solving for algorithms-- 01:11:33.210 --> 01:11:35.370 align:middle line:84% like if you want to solve a large matrix, 01:11:35.370 --> 01:11:39.720 align:middle line:90% it's probably better in MATLAB. 01:11:39.720 --> 01:11:42.900 align:middle line:84% Just because a lot more people are working with MATLAB, 01:11:42.900 --> 01:11:46.960 align:middle line:84% and it's a big company, they spent a lot of money on it. 01:11:46.960 --> 01:11:49.950 align:middle line:84% But for statistics, which means, I think, 01:11:49.950 --> 01:11:53.070 align:middle line:90% R is more sophisticated. 01:11:53.070 --> 01:11:54.300 align:middle line:90% You have a lot more options. 01:11:54.300 --> 01:11:57.120 align:middle line:90% 01:11:57.120 --> 01:12:00.390 align:middle line:84% Yeah, so I mean, I used to use MATLAB all the time. 01:12:00.390 --> 01:12:04.740 align:middle line:84% But now, I'm happy with R. But if you know one, 01:12:04.740 --> 01:12:07.560 align:middle line:90% it's easy to pick up the other. 01:12:07.560 --> 01:12:09.090 align:middle line:84% I mean, you always get confused as 01:12:09.090 --> 01:12:12.450 align:middle line:84% to how do I comment something out, or that sort of thing. 01:12:12.450 --> 01:12:15.092 align:middle line:84% But they're basically the same flavor-- the same reason 01:12:15.092 --> 01:12:16.050 align:middle line:90% that MATLAB is popular. 01:12:16.050 --> 01:12:18.420 align:middle line:90% 01:12:18.420 --> 01:12:24.130 align:middle line:84% Let's see, steep learning curve, but worthwhile in the long run. 01:12:24.130 --> 01:12:26.970 align:middle line:84% And there is something called Bioconductor-- 01:12:26.970 --> 01:12:28.780 align:middle line:84% I don't if anyone has heard of this. 01:12:28.780 --> 01:12:34.740 align:middle line:84% So this was an effort that Robert Gentleman, who's 01:12:34.740 --> 01:12:38.220 align:middle line:84% at the Dana-Farber Institute, who was one 01:12:38.220 --> 01:12:40.980 align:middle line:90% of the founders of R, started. 01:12:40.980 --> 01:12:45.030 align:middle line:84% So I was of more of a part of it-- 01:12:45.030 --> 01:12:47.100 align:middle line:84% more part of this project before, 01:12:47.100 --> 01:12:52.920 align:middle line:84% and he suggested at the very beginning for the title, 01:12:52.920 --> 01:12:55.200 align:middle line:84% or for the name of the project, MAD Men-- 01:12:55.200 --> 01:12:58.650 align:middle line:90% like microarray data management. 01:12:58.650 --> 01:13:02.310 align:middle line:84% But in the end, it was decided that NIH would not 01:13:02.310 --> 01:13:05.040 align:middle line:84% want to fund something called MAD Men 01:13:05.040 --> 01:13:08.980 align:middle line:84% So he came up with Bioconductor, which is a much better name. 01:13:08.980 --> 01:13:11.040 align:middle line:84% So if you go to bioconductor.org, 01:13:11.040 --> 01:13:13.660 align:middle line:84% there are packages that you can download. 01:13:13.660 --> 01:13:16.740 align:middle line:90% So what's happening now is-- 01:13:16.740 --> 01:13:18.690 align:middle line:84% I think this package has caught on, 01:13:18.690 --> 01:13:23.430 align:middle line:84% so that if someone writes or comes up with a good algorithm, 01:13:23.430 --> 01:13:26.370 align:middle line:84% they'll write a routine and deposit it 01:13:26.370 --> 01:13:28.680 align:middle line:84% somewhere in this website so that you 01:13:28.680 --> 01:13:30.030 align:middle line:90% can download it and run it. 01:13:30.030 --> 01:13:32.550 align:middle line:90% 01:13:32.550 --> 01:13:35.380 align:middle line:84% And that's been done for a lot of good algorithms out there. 01:13:35.380 --> 01:13:38.160 align:middle line:84% So if you want to normalize your cDNA data using 01:13:38.160 --> 01:13:41.040 align:middle line:90% some fancy method, it's there. 01:13:41.040 --> 01:13:43.620 align:middle line:90% Download it. 01:13:43.620 --> 01:13:44.610 align:middle line:90% It's pretty easy. 01:13:44.610 --> 01:13:47.910 align:middle line:84% And there's a Windows version of this that's pretty easy. 01:13:47.910 --> 01:13:50.610 align:middle line:84% And R comes in Linux and Windows, 01:13:50.610 --> 01:13:55.260 align:middle line:90% but it gets compiled for-- 01:13:55.260 --> 01:13:57.780 align:middle line:84% gets ported to Windows frequently, 01:13:57.780 --> 01:14:01.170 align:middle line:84% and I've had no problems with it. 01:14:01.170 --> 01:14:04.447 align:middle line:84% I think someone was trying to port it to Apple, 01:14:04.447 --> 01:14:06.030 align:middle line:84% but I don't know how successful it is. 01:14:06.030 --> 01:14:08.430 align:middle line:90% I wouldn't recommend it. 01:14:08.430 --> 01:14:09.857 align:middle line:90% But I'm pretty happy with it. 01:14:09.857 --> 01:14:11.190 align:middle line:90% This is what I use all the time. 01:14:11.190 --> 01:14:17.770 align:middle line:84% This is what a lot of people use who do microarray analysis. 01:14:17.770 --> 01:14:23.170 align:middle line:84% OK, so I see that I have ended three minutes early. 01:14:23.170 --> 01:14:27.420 align:middle line:90% AUDIENCE: [INAUDIBLE] 01:14:27.420 --> 01:14:28.890 align:middle line:84% PETER PARK: Yeah, so the problem-- 01:14:28.890 --> 01:14:30.060 align:middle line:90% yeah, there is. 01:14:30.060 --> 01:14:33.450 align:middle line:84% The problem is that I think R is just a bad name, because you 01:14:33.450 --> 01:14:36.780 align:middle line:90% can't really search for R. 01:14:36.780 --> 01:14:37.350 align:middle line:90% 01:14:37.350 --> 01:14:40.680 align:middle line:84% You can go to a Bioconductor R, and there's a link back to R, 01:14:40.680 --> 01:14:41.610 align:middle line:90% or you can do-- 01:14:41.610 --> 01:14:47.100 align:middle line:84% you can if you do a statistical package R. R used to be-- 01:14:47.100 --> 01:14:50.700 align:middle line:84% I mean, it used to be hard to run these things. 01:14:50.700 --> 01:14:54.120 align:middle line:84% But Robert and his friends have really made this easy, 01:14:54.120 --> 01:14:56.280 align:middle line:84% so that you basically click a button 01:14:56.280 --> 01:14:58.710 align:middle line:84% and it'll install on your Windows. 01:14:58.710 --> 01:15:02.880 align:middle line:90% So all right.