WEBVTT 00:00:00.090 --> 00:00:02.430 The following content is provided under a Creative 00:00:02.430 --> 00:00:03.820 Commons license. 00:00:03.820 --> 00:00:06.030 Your support will help MIT Open CourseWare 00:00:06.030 --> 00:00:10.120 continue to offer high-quality educational resources for free. 00:00:10.120 --> 00:00:12.660 To make a donation or to view additional materials 00:00:12.660 --> 00:00:16.620 from hundreds of MIT courses, visit MIT OpenCourseWare 00:00:16.620 --> 00:00:20.420 at ocw.mit.edu 00:00:20.420 --> 00:00:22.170 WILLIAM BONVILLIAN: Rasheed, why don't you 00:00:22.170 --> 00:00:29.080 give us that point that you were making to me about DARPA. 00:00:29.080 --> 00:00:33.670 RASHEED: Yes, so DARPA relies very heavily on this program 00:00:33.670 --> 00:00:34.540 manager structure. 00:00:34.540 --> 00:00:36.220 So they give these program managers a lot of power 00:00:36.220 --> 00:00:38.320 and a lot of leeway to do whatever they want. 00:00:38.320 --> 00:00:41.950 But they have-- they have to have a very distinct vision 00:00:41.950 --> 00:00:43.840 from the outset, like really when they 00:00:43.840 --> 00:00:45.540 start to be able to do things. 00:00:45.540 --> 00:00:49.658 And so I was concerned that these program managers were 00:00:49.658 --> 00:00:51.700 going to basically be coming from the same places 00:00:51.700 --> 00:00:54.605 over and over again, and so have very similar visions. 00:00:54.605 --> 00:00:56.230 And you're going to be able to miss out 00:00:56.230 --> 00:00:57.760 on a lot of different talent that 00:00:57.760 --> 00:01:00.040 exists outside, or a lot of different new projects, 00:01:00.040 --> 00:01:03.220 or new fields, maybe, just like you're talking 00:01:03.220 --> 00:01:06.490 about-- these new collaboration between biology 00:01:06.490 --> 00:01:10.000 and kind of humans and computer science, things like that. 00:01:10.000 --> 00:01:11.860 But that was really my main concern, 00:01:11.860 --> 00:01:17.570 was program managers weren't being chosen as-- 00:01:17.570 --> 00:01:20.290 equitably isn't the word, but that the selection 00:01:20.290 --> 00:01:22.767 process of these program managers was actually-- 00:01:22.767 --> 00:01:23.600 might be too narrow. 00:01:26.170 --> 00:01:29.280 WILLIAM BONVILLIAN: But they do get visionaries. 00:01:29.280 --> 00:01:30.488 STUDENT: Sometimes. 00:01:30.488 --> 00:01:32.280 I mean, I think we mentioned that they only 00:01:32.280 --> 00:01:35.338 got white males [INAUDIBLE]. 00:01:39.040 --> 00:01:40.540 WILLIAM BONVILLIAN: OK, so I'm going 00:01:40.540 --> 00:01:42.940 to move us to Tammy Carleton. 00:01:42.940 --> 00:01:47.500 And Tammy's big perception about DARPA 00:01:47.500 --> 00:01:53.890 is this vision perception, that that's a critical factor that 00:01:53.890 --> 00:01:55.780 makes DARPA different. 00:01:55.780 --> 00:01:57.940 And there is a process that DARPA 00:01:57.940 --> 00:02:02.920 has developed for this radical technological visioning. 00:02:02.920 --> 00:02:06.880 And essentially, she argues that there's six stages that they 00:02:06.880 --> 00:02:08.139 use to kind of get there. 00:02:08.139 --> 00:02:11.050 First, recruitment of great talent-- 00:02:11.050 --> 00:02:13.600 and the talent, they hear about talent 00:02:13.600 --> 00:02:16.870 from this DARPA network, which is now quite huge. 00:02:16.870 --> 00:02:19.720 Because all of the people that get DARPA contracts and awards, 00:02:19.720 --> 00:02:22.180 they stay in this kind of DARPA family. 00:02:22.180 --> 00:02:24.460 And typically, researchers love to work for DARPA. 00:02:24.460 --> 00:02:31.480 Because working on DARPA-hard problems is really intriguing. 00:02:31.480 --> 00:02:34.780 So that's a community that tends to know each other, 00:02:34.780 --> 00:02:36.370 and be in communication, as well as 00:02:36.370 --> 00:02:40.920 the community of former DARPA program managers and officials. 00:02:40.920 --> 00:02:44.800 So a lot of the talent comes through that network, 00:02:44.800 --> 00:02:49.090 which kind of knows what's going to work. 00:02:49.090 --> 00:02:52.120 So the recruitment and the vision get united in a way, 00:02:52.120 --> 00:02:54.160 in that recruitment process. 00:02:54.160 --> 00:02:57.402 Then comes the stage of the vision formulation. 00:02:57.402 --> 00:02:59.110 The program manager has got to figure out 00:02:59.110 --> 00:03:02.080 what they're going to do with their four or five years. 00:03:02.080 --> 00:03:06.920 And they've got to formulate a radical vision, 00:03:06.920 --> 00:03:08.800 a right/left vision, what's going to come out 00:03:08.800 --> 00:03:11.217 of that pipeline, and what's the fundamental research that 00:03:11.217 --> 00:03:13.450 going to get me there. 00:03:13.450 --> 00:03:16.570 And it's driving for that breakthrough that's 00:03:16.570 --> 00:03:19.330 the organizing proposition. 00:03:19.330 --> 00:03:21.010 Then, they've got to go through defining 00:03:21.010 --> 00:03:23.260 a program and its launch. 00:03:23.260 --> 00:03:26.500 Often, they will have a workshop, 00:03:26.500 --> 00:03:28.090 an invitation-only workshop, where 00:03:28.090 --> 00:03:34.270 they'll assemble 25, 30, 40 of the kind of best thinkers 00:03:34.270 --> 00:03:38.320 on a project area, and meet for a day or two days. 00:03:38.320 --> 00:03:44.620 I've watched one of those, which got held up at MIT as part 00:03:44.620 --> 00:03:47.980 of the beginnings of creating this Biological Technologies 00:03:47.980 --> 00:03:49.000 office. 00:03:49.000 --> 00:03:52.210 And they brought in just an amazing collection, 00:03:52.210 --> 00:03:54.940 from MIT and from elsewhere, of thinkers 00:03:54.940 --> 00:03:57.040 about what some of the early projects 00:03:57.040 --> 00:04:00.630 might be to achieve some of these capabilities. 00:04:00.630 --> 00:04:03.520 So the Deputy Director of DARPA came up, 00:04:03.520 --> 00:04:05.202 and kind of helped run this workshop 00:04:05.202 --> 00:04:06.160 and think this through. 00:04:06.160 --> 00:04:07.870 The office wasn't set up for a while 00:04:07.870 --> 00:04:10.540 until the next DARPA director, but it really 00:04:10.540 --> 00:04:13.720 helped contribute to that portfolio. 00:04:13.720 --> 00:04:18.190 And it was a really serious effort. 00:04:18.190 --> 00:04:23.320 So the program manager develops a portfolio approach 00:04:23.320 --> 00:04:25.090 of standing up a series of groups 00:04:25.090 --> 00:04:28.090 that are going to be working on this, and inter-relating them. 00:04:28.090 --> 00:04:29.940 And the program manager is also going 00:04:29.940 --> 00:04:33.635 to be responsible for making this technology transfer. 00:04:33.635 --> 00:04:35.260 So in other words, it's not enough just 00:04:35.260 --> 00:04:36.190 to do something cool. 00:04:36.190 --> 00:04:38.800 You've got to move it into the implementation stage. 00:04:38.800 --> 00:04:40.520 And how is that going to happen? 00:04:40.520 --> 00:04:43.300 So the program manager has got to undertake 00:04:43.300 --> 00:04:46.810 all of these various steps. 00:04:46.810 --> 00:04:51.220 So the criteria for being DARPA hard 00:04:51.220 --> 00:04:54.460 is that the project has to be technologically very 00:04:54.460 --> 00:04:55.160 challenging. 00:04:55.160 --> 00:04:58.120 It's got to extend beyond the current limits of technology 00:04:58.120 --> 00:04:59.050 and knowledge. 00:04:59.050 --> 00:05:01.900 Otherwise, they don't really want to do it. 00:05:01.900 --> 00:05:04.280 It's got to be actionable, which means it can be made. 00:05:04.280 --> 00:05:06.670 It can be built. It can be produced. 00:05:06.670 --> 00:05:09.580 So if it can't be, why do it? 00:05:09.580 --> 00:05:11.080 You've got to be able to demonstrate 00:05:11.080 --> 00:05:12.940 that it's actionable. 00:05:12.940 --> 00:05:14.860 Typically multidisciplinary-- that it's 00:05:14.860 --> 00:05:17.800 going to draw on a variety of areas of expertise. 00:05:17.800 --> 00:05:19.000 It's got to be far reaching. 00:05:19.000 --> 00:05:20.650 It's got to change things. 00:05:20.650 --> 00:05:24.640 It's got to be very ambitious at a big scale. 00:05:24.640 --> 00:05:27.280 "Don't do little things" is kind of a DARPA sub-rule. 00:05:29.830 --> 00:05:32.980 And various techniques to flesh out the vision 00:05:32.980 --> 00:05:35.050 include those expert-only workshops, 00:05:35.050 --> 00:05:38.170 and a proof of concept effort. 00:05:38.170 --> 00:05:40.450 Sometimes, if it's some idea that looks promising, 00:05:40.450 --> 00:05:42.880 they'll do what DARPA calls a "seedling," just put 00:05:42.880 --> 00:05:45.190 a little money on it, and let somebody play 00:05:45.190 --> 00:05:47.680 with it, a real thinker play with it, 00:05:47.680 --> 00:05:50.380 and see what it might materialize into. 00:05:50.380 --> 00:05:53.080 So they've got that seedling capability. 00:05:53.080 --> 00:05:57.138 There is no training system for program managers. 00:05:57.138 --> 00:05:58.930 They don't go through any kind of training. 00:05:58.930 --> 00:06:00.880 They just arrive. 00:06:00.880 --> 00:06:05.133 And they kind of learn from other program managers 00:06:05.133 --> 00:06:06.300 what they're supposed to do. 00:06:06.300 --> 00:06:10.390 So there's a very strong kind of DARPA culture 00:06:10.390 --> 00:06:13.600 about how it operates, that gets communicated by word of mouth 00:06:13.600 --> 00:06:16.870 and kind of learning by doing. 00:06:16.870 --> 00:06:23.260 And it's informal exchange, but there is a very strong kind 00:06:23.260 --> 00:06:26.170 of operating rules set. 00:06:26.170 --> 00:06:28.060 DARPA program managers really have 00:06:28.060 --> 00:06:29.900 to be quite entrepreneurial. 00:06:29.900 --> 00:06:32.470 And Martine, you were raising this earlier. 00:06:32.470 --> 00:06:34.720 They've got to stand up a whole new territory. 00:06:34.720 --> 00:06:36.550 And to do that, they've got to be 00:06:36.550 --> 00:06:41.440 highly entrepreneurial about how to bring that about. 00:06:41.440 --> 00:06:45.030 And the decision to stand up a vision program-- 00:06:45.030 --> 00:06:48.100 this is not set up through peer review. 00:06:48.100 --> 00:06:50.860 There isn't consensus decision-making at DARPA. 00:06:50.860 --> 00:06:54.850 That program manager has got to have a really strong vision, 00:06:54.850 --> 00:06:58.060 and it's got to get really organized. 00:06:58.060 --> 00:07:01.600 And then it's got to withstand the office director 00:07:01.600 --> 00:07:06.310 and then the DARPA director really tearing through it, 00:07:06.310 --> 00:07:08.110 and really analyzing it. 00:07:08.110 --> 00:07:10.390 So once you get that, though, the approvals 00:07:10.390 --> 00:07:13.000 can be very fast, once the decision is 00:07:13.000 --> 00:07:15.160 made to stand something up. 00:07:15.160 --> 00:07:17.590 And that the program manager really is-- 00:07:17.590 --> 00:07:21.080 to go back to that term that you liked, Steph-- 00:07:21.080 --> 00:07:23.380 a vision champion. 00:07:23.380 --> 00:07:26.600 They've got to champion this vision and make it happen. 00:07:26.600 --> 00:07:29.200 That's what the program manager has got to do. 00:07:29.200 --> 00:07:34.780 So DARPA starts with division. 00:07:34.780 --> 00:07:40.820 That is not how industry operates. 00:07:40.820 --> 00:07:44.968 That's not how other government agencies operate. 00:07:49.570 --> 00:07:54.880 Industry works largely on what's called a "stage gate system." 00:07:54.880 --> 00:07:57.730 So industry is going to want to do typically 00:07:57.730 --> 00:07:59.410 more incremental kind of advances, 00:07:59.410 --> 00:08:02.320 because the advances can snap onto markets that it's already 00:08:02.320 --> 00:08:05.560 got, and economic opportunities that it already sees, 00:08:05.560 --> 00:08:08.710 and technologies that it's already starting to explore. 00:08:08.710 --> 00:08:10.300 So it's typically going to want to do 00:08:10.300 --> 00:08:12.850 more incremental advances. 00:08:12.850 --> 00:08:18.060 And it's only going to do those incremental advances if they 00:08:18.060 --> 00:08:21.070 can optimize them economically. 00:08:21.070 --> 00:08:23.650 So an industry R&D project has got 00:08:23.650 --> 00:08:29.400 to go through a series of stage gates, which essentially 00:08:29.400 --> 00:08:34.890 is a way of tearing down a large menu of stuff 00:08:34.890 --> 00:08:39.270 and getting to the most economically optimal. 00:08:39.270 --> 00:08:42.580 Industry stage gate process is not a technology visioning 00:08:42.580 --> 00:08:43.080 system. 00:08:45.940 --> 00:08:48.582 It's not deciding on a vision and doing 00:08:48.582 --> 00:08:50.290 whatever is necessary to get there, which 00:08:50.290 --> 00:08:52.000 is the way DARPA operates. 00:08:52.000 --> 00:08:55.240 It's a much more limited set of procedural stage gates 00:08:55.240 --> 00:08:57.670 that a technology idea has got to go through. 00:08:57.670 --> 00:09:01.570 So DARPA just working on a very different kind 00:09:01.570 --> 00:09:04.390 of organizational model from the way in which things are done, 00:09:04.390 --> 00:09:13.030 either at other agencies or within industry. 00:09:13.030 --> 00:09:15.100 And starting with this vision up front 00:09:15.100 --> 00:09:18.850 is a really interesting organizational idea, 00:09:18.850 --> 00:09:22.270 that you all should kind of keep in mind when you're 00:09:22.270 --> 00:09:24.820 operating out there, and setting up your own companies, 00:09:24.820 --> 00:09:27.490 and doing your own startups. 00:09:27.490 --> 00:09:30.450 So Max, you've got Tammy Carleton's piece. 00:09:30.450 --> 00:09:31.240 MAX: Right. 00:09:31.240 --> 00:09:36.622 The key step within Carleton's five steps 00:09:36.622 --> 00:09:38.080 that I thought was most interesting 00:09:38.080 --> 00:09:40.060 was the fifth one, technology transfer, which 00:09:40.060 --> 00:09:44.200 is supposed to bridge the R&D-- 00:09:44.200 --> 00:09:47.680 the valley of death, as it were, that 00:09:47.680 --> 00:09:52.280 enables a different scientific idea to be commercialized, 00:09:52.280 --> 00:09:57.220 and to actually affect people all over the world, eventually. 00:09:57.220 --> 00:09:59.980 While reading, I kind of realized 00:09:59.980 --> 00:10:03.020 that DARPA seems like they have a pretty decent organization. 00:10:03.020 --> 00:10:05.260 They get things done relatively efficiently. 00:10:05.260 --> 00:10:07.220 They have a lot of money. 00:10:07.220 --> 00:10:08.970 Why is it they have-- 00:10:08.970 --> 00:10:11.230 and I actually just decided to start googling, 00:10:11.230 --> 00:10:13.750 to try to figure out if DARPA had done anything related 00:10:13.750 --> 00:10:14.260 to fusion. 00:10:14.260 --> 00:10:16.090 And I couldn't find anything. 00:10:16.090 --> 00:10:20.492 So I'm curious, why not? 00:10:20.492 --> 00:10:21.950 WILLIAM BONVILLIAN: Well, actually, 00:10:21.950 --> 00:10:24.730 DARPA deferred on energy technologies 00:10:24.730 --> 00:10:28.030 to ARPA-E, when ARPA-E got stood up, 00:10:28.030 --> 00:10:32.680 because DARPA views ARPA-E as a compatriot. 00:10:32.680 --> 00:10:35.860 And ARPA-E interestingly, has done some extremely interesting 00:10:35.860 --> 00:10:38.992 work on fusion. 00:10:38.992 --> 00:10:41.590 STUDENT: I know Lockheed [INAUDIBLE] fusion-- 00:10:41.590 --> 00:10:42.673 WILLIAM BONVILLIAN: Right. 00:10:42.673 --> 00:10:43.630 STUDENT: [INAUDIBLE] 00:10:43.630 --> 00:10:47.243 MAX: Yeah, but Lockheed Martin's-- 00:10:47.243 --> 00:10:49.410 WILLIAM BONVILLIAN: Well, they have the Skunk Works. 00:10:49.410 --> 00:10:50.050 MAX: They do. 00:10:50.050 --> 00:10:52.675 WILLIAM BONVILLIAN: And DARPA's done an enormous amount of work 00:10:52.675 --> 00:10:56.013 with the Skunk Works over time, including stealth. 00:10:56.013 --> 00:10:58.180 MAX: Granted, yes-- they definitely have a facility, 00:10:58.180 --> 00:11:00.880 and they definitely have the project managers. 00:11:00.880 --> 00:11:02.620 I just feel that it's a bit of an area 00:11:02.620 --> 00:11:04.150 outside of their expertise. 00:11:04.150 --> 00:11:06.657 They're going to accidentally make it fly and stuff. 00:11:06.657 --> 00:11:07.740 WILLIAM BONVILLIAN: Right. 00:11:07.740 --> 00:11:09.790 And look-- another issue for DARPA is-- remember, 00:11:09.790 --> 00:11:11.582 these technologies have got to get stood up 00:11:11.582 --> 00:11:14.213 in the lifetime of the project. 00:11:14.213 --> 00:11:15.130 MAX: Yeah, five years. 00:11:15.130 --> 00:11:16.370 WILLIAM BONVILLIAN: Like a five year project. 00:11:16.370 --> 00:11:18.040 So if you can't do it within five years, 00:11:18.040 --> 00:11:20.410 it's very hard to put it on the DARPA agenda. 00:11:20.410 --> 00:11:22.832 MAX: Well, you mentioned that some of the projects 00:11:22.832 --> 00:11:24.790 would get handed off to other project managers. 00:11:24.790 --> 00:11:25.120 WILLIAM BONVILLIAN: Yes. 00:11:25.120 --> 00:11:27.630 MAX: So that's how long a manager lasts, right? 00:11:27.630 --> 00:11:28.630 WILLIAM BONVILLIAN: Yes. 00:11:28.630 --> 00:11:30.790 MAX: But it's not always how long a project lasts. 00:11:30.790 --> 00:11:31.790 WILLIAM BONVILLIAN: Yes. 00:11:31.790 --> 00:11:34.690 Sometimes, there can be intergenerational transfers. 00:11:34.690 --> 00:11:38.620 And then Licklider developed a different kind of system. 00:11:38.620 --> 00:11:44.080 He created a community that kept continuing 00:11:44.080 --> 00:11:45.880 with the same kind of ethos, working 00:11:45.880 --> 00:11:47.770 on that same set of agenda items, 00:11:47.770 --> 00:11:50.200 working them down one after another, 00:11:50.200 --> 00:11:53.205 and making progress year after year on each of them. 00:11:53.205 --> 00:11:54.580 So that's a different kind of way 00:11:54.580 --> 00:11:56.080 that DARPA is going to be organized. 00:11:56.080 --> 00:11:59.530 But typically, once the program manager 00:11:59.530 --> 00:12:03.250 leaves who had the vision, their job 00:12:03.250 --> 00:12:05.790 is to get that implemented in their, in effect, 00:12:05.790 --> 00:12:07.240 DARPA lifetime. 00:12:07.240 --> 00:12:10.150 And if they haven't, then it's hard for DARPA 00:12:10.150 --> 00:12:13.810 to continue that, which we could view 00:12:13.810 --> 00:12:16.525 as a weakness of the program. 00:12:16.525 --> 00:12:18.650 On the other hand, you could view it as a strength. 00:12:18.650 --> 00:12:20.817 MAX: Yeah, because then they make sure that things-- 00:12:20.817 --> 00:12:22.600 that if they are working, then they keep-- 00:12:22.600 --> 00:12:24.250 they may proceed with it, or they pass it off 00:12:24.250 --> 00:12:25.090 to someone else. 00:12:25.090 --> 00:12:26.090 WILLIAM BONVILLIAN: Yes. 00:12:30.570 --> 00:12:33.990 MAX: So speaking of Licklider, actually, while I was reading, 00:12:33.990 --> 00:12:37.980 I noticed that Licklider had mentioned that he heavily 00:12:37.980 --> 00:12:40.480 relied on standardized tests when picking candidates, 00:12:40.480 --> 00:12:43.440 so like that recruitment of great talent. 00:12:43.440 --> 00:12:45.300 Given that literally everyone in this room 00:12:45.300 --> 00:12:48.270 has experience with standardized testing, and at least 00:12:48.270 --> 00:12:51.840 I can't speak for everyone, but I don't like it very much. 00:12:51.840 --> 00:12:52.550 I wonder-- 00:12:52.550 --> 00:12:53.842 STUDENT: [INAUDIBLE] two tests. 00:12:53.842 --> 00:12:54.690 MAX: Which two? 00:12:54.690 --> 00:12:55.350 It was ACTs. 00:12:55.350 --> 00:12:56.490 STUDENT: One was like a psych test, 00:12:56.490 --> 00:12:57.626 and the other one was like a [INAUDIBLE].. 00:12:57.626 --> 00:12:58.320 STUDENT 1: GRE. 00:12:58.320 --> 00:12:59.742 We did the GRE, it said. 00:12:59.742 --> 00:13:01.200 STUDENT: Well, there's another one, 00:13:01.200 --> 00:13:02.742 but it seemed like a psychology test. 00:13:02.742 --> 00:13:04.013 I'm not sure. 00:13:04.013 --> 00:13:05.680 STUDENT 1: So I think that she mentioned 00:13:05.680 --> 00:13:08.800 that he was the only one who asked for test scores at all. 00:13:08.800 --> 00:13:12.997 And one was the GRE, which is kind of weird. 00:13:12.997 --> 00:13:15.580 WILLIAM BONVILLIAN: Remember, he was a psychologist coming out 00:13:15.580 --> 00:13:16.930 of the testing world. 00:13:16.930 --> 00:13:21.060 So that's probably part of the reason. 00:13:21.060 --> 00:13:25.270 MAX: I'll chalk it up to [INAUDIBLE],, because there's 00:13:25.270 --> 00:13:27.187 only so much a standardized test can tell you. 00:13:27.187 --> 00:13:29.353 They won't really tell you how you work with people. 00:13:29.353 --> 00:13:30.970 They won't tell you anything about how 00:13:30.970 --> 00:13:33.580 you work on anything that's not related to, I don't know, 00:13:33.580 --> 00:13:36.205 math or literature. 00:13:36.205 --> 00:13:38.830 WILLIAM BONVILLIAN: He did come up with a pretty talented team, 00:13:38.830 --> 00:13:39.140 though. 00:13:39.140 --> 00:13:40.098 MAX: Yeah, that's true. 00:13:40.098 --> 00:13:42.870 STUDENT: I mean, you get pretty far on psychology tests, 00:13:42.870 --> 00:13:44.460 especially when it comes to intertia. 00:13:44.460 --> 00:13:46.835 Because you have to figure out, for really good visionary 00:13:46.835 --> 00:13:49.560 leaders, what they'll figure out is 00:13:49.560 --> 00:13:52.040 they're intellectuals, introverts, extroverts, 00:13:52.040 --> 00:13:55.800 and they have a supporting side to their psyche. 00:13:55.800 --> 00:13:58.560 And they're kind of like the avatar of team groups. 00:13:58.560 --> 00:14:02.190 So it's pretty good for if you want to find out good leaders. 00:14:02.190 --> 00:14:04.964 And I know it's been pretty effective in manageent theory 00:14:04.964 --> 00:14:06.862 when you use psych tests. 00:14:06.862 --> 00:14:08.320 It's also really good to figure out 00:14:08.320 --> 00:14:10.535 who is going to be a really good scientist, 00:14:10.535 --> 00:14:12.160 because it's usually highly introverted 00:14:12.160 --> 00:14:16.120 and comes up with new insights that doesn't follow the crowd. 00:14:16.120 --> 00:14:17.830 So I can see why that's important. 00:14:17.830 --> 00:14:20.840 The GRE might be, but also, if your score is low, 00:14:20.840 --> 00:14:22.967 but if you've come out with a good body of work, 00:14:22.967 --> 00:14:24.550 then they can probably assume that you 00:14:24.550 --> 00:14:26.383 are pretty smart, even if you didn't do well 00:14:26.383 --> 00:14:27.331 on standardized tests. 00:14:29.902 --> 00:14:31.860 STUDENT 2: Because I'm looking up her LinkedIn. 00:14:31.860 --> 00:14:35.080 And it says that she has-- her background is in communication. 00:14:35.080 --> 00:14:37.080 And then she got a Master's in Public Relations. 00:14:37.080 --> 00:14:39.625 And then she did a PhD in Mechanical Engineering Design. 00:14:39.625 --> 00:14:42.000 But that must have been back in the day when the D School 00:14:42.000 --> 00:14:45.090 at Stanford accepted non-engineers 00:14:45.090 --> 00:14:47.820 and non-scientists, because now they don't. 00:14:47.820 --> 00:14:51.840 So it would be curious to think about the ways in which 00:14:51.840 --> 00:14:54.750 her background, in particular, might give her that insight 00:14:54.750 --> 00:14:57.870 that an engineer might not have about this particular field, 00:14:57.870 --> 00:15:01.080 going back to the question of multidisciplinary great groups. 00:15:01.080 --> 00:15:03.070 But I don't know to what extent, for example, 00:15:03.070 --> 00:15:05.840 someone like me might be accepted in maybe Lilly's lab. 00:15:05.840 --> 00:15:08.250 LILLY: One of the most famous women in my field 00:15:08.250 --> 00:15:13.500 has an undergraduate degree in German. 00:15:13.500 --> 00:15:14.840 So you never know. 00:15:14.840 --> 00:15:19.340 STUDENT: Lloyd Blankfein, the CEO of J P Morgan, I think-- 00:15:19.340 --> 00:15:20.490 his degree's in history. 00:15:20.490 --> 00:15:21.975 And he almost barely didn't pass. 00:15:21.975 --> 00:15:23.350 He said he still gets nightmares, 00:15:23.350 --> 00:15:26.965 because he didn't fulfill all the requirements. 00:15:26.965 --> 00:15:28.590 WILLIAM BONVILLIAN: All right, so we're 00:15:28.590 --> 00:15:31.410 going to move out of the testing world back to the DARPA world. 00:15:31.410 --> 00:15:33.870 MAX: Sure. 00:15:33.870 --> 00:15:36.420 So one of the things that Carleton pointed out 00:15:36.420 --> 00:15:42.510 was that as DARPA has aged, so, too, have its project managers. 00:15:42.510 --> 00:15:46.710 Even though they cycle between new managers every four or five 00:15:46.710 --> 00:15:50.820 years, the average age of the managers that they would pick 00:15:50.820 --> 00:15:53.610 was going up steadily. 00:15:53.610 --> 00:15:55.200 So I'm curious if anyone has thoughts 00:15:55.200 --> 00:15:58.230 on what the effect of that would be on their managerial ability, 00:15:58.230 --> 00:16:02.280 or on DARPA's direction as a whole. 00:16:02.280 --> 00:16:04.030 STUDENT: Could you repeat that first part? 00:16:04.030 --> 00:16:04.530 MAX: Yeah. 00:16:04.530 --> 00:16:07.080 So the average age of the project managers 00:16:07.080 --> 00:16:08.807 has been increasing, as DARPA has-- 00:16:08.807 --> 00:16:10.140 STUDENT: What's the average age? 00:16:10.140 --> 00:16:11.557 MAX: I don't remember the numbers. 00:16:11.557 --> 00:16:12.680 STUDENT: Is it 27? 00:16:12.680 --> 00:16:14.240 WILLIAM BONVILLIAN: No, it's a lot older than that. 00:16:14.240 --> 00:16:15.323 STUDENT: It's a lot older? 00:16:15.323 --> 00:16:16.820 MAX: Yeah. 00:16:16.820 --> 00:16:20.180 WILLIAM BONVILLIAN: One of the reasons, Max, 00:16:20.180 --> 00:16:22.760 may be that it's-- 00:16:26.230 --> 00:16:31.070 I mean, watching MIT, it's very hard for younger faculty 00:16:31.070 --> 00:16:34.460 to walk out of the place before they get tenure. 00:16:34.460 --> 00:16:37.430 It's hard enough to walk out once you get tenure, 00:16:37.430 --> 00:16:41.330 but it's impossible to walk out before you get tenure. 00:16:41.330 --> 00:16:43.710 And because you're establishing a whole-- 00:16:43.710 --> 00:16:46.730 your research portfolio and your whole researching system, 00:16:46.730 --> 00:16:49.080 and it's very hard to walk out of that. 00:16:49.080 --> 00:16:52.835 In addition, when you go to DARPA, you've got a conflict. 00:16:52.835 --> 00:16:54.710 Once you've worked there, you have a conflict 00:16:54.710 --> 00:16:55.970 of interest with DARPA. 00:16:55.970 --> 00:16:57.428 You're not going to be able to deal 00:16:57.428 --> 00:17:00.270 with the agency for a certain period of time. 00:17:00.270 --> 00:17:02.478 So if they happen to be funding your research before, 00:17:02.478 --> 00:17:05.062 they're not going to be able to continue funding your research 00:17:05.062 --> 00:17:06.480 once you've worked there. 00:17:06.480 --> 00:17:08.480 So it's gotten more complicated for DARPA 00:17:08.480 --> 00:17:10.760 to recruit at the university level. 00:17:10.760 --> 00:17:12.210 It's gotten a lot more difficult. 00:17:12.210 --> 00:17:14.660 So the route between MIT and DARPA, 00:17:14.660 --> 00:17:17.480 frankly, used to be a lot easier to manage. 00:17:17.480 --> 00:17:23.460 And when I look at my colleagues here who have gone to DARPA, 00:17:23.460 --> 00:17:25.988 they are more typically fairly far along, 00:17:25.988 --> 00:17:28.280 and very assured about their career, and their research 00:17:28.280 --> 00:17:29.530 foundation, and their writing. 00:17:29.530 --> 00:17:31.340 They've done that stuff, and they 00:17:31.340 --> 00:17:33.890 can take a three or five year break, 00:17:33.890 --> 00:17:36.830 and get that really big, visionary thing 00:17:36.830 --> 00:17:39.800 that they've wanted to work on forever done 00:17:39.800 --> 00:17:40.730 when they go to DARPA. 00:17:40.730 --> 00:17:42.730 But it's harder when you're trying to administer 00:17:42.730 --> 00:17:44.930 a whole research portfolio. 00:17:44.930 --> 00:17:47.230 Again, with the IPA authority, you've 00:17:47.230 --> 00:17:49.760 got a certain freedom to come back to your organization 00:17:49.760 --> 00:17:51.530 and keep the day-to-day stuff going. 00:17:51.530 --> 00:17:53.240 But it's still not simple. 00:17:53.240 --> 00:17:57.140 So that may be one part of the reason 00:17:57.140 --> 00:18:00.170 that you tend to get somewhat older faculty than they 00:18:00.170 --> 00:18:01.832 probably used to at the beginning. 00:18:01.832 --> 00:18:03.838 STUDENT 1: Licklider wasn't especially young 00:18:03.838 --> 00:18:04.880 when he was the director. 00:18:04.880 --> 00:18:06.710 WILLIAM BONVILLIAN: Right, he wasn't young. 00:18:06.710 --> 00:18:08.380 STUDENT 1: Or the project manager. 00:18:08.380 --> 00:18:11.940 So I think age does affect the perspective that you bring in. 00:18:11.940 --> 00:18:15.710 But having older people isn't a bad thing. 00:18:15.710 --> 00:18:17.610 I think in the tech world, it's seen 00:18:17.610 --> 00:18:20.590 as like if you're not young and hip, 00:18:20.590 --> 00:18:23.440 you're going to be out of touch with what's going to work. 00:18:23.440 --> 00:18:24.770 But I don't know if that's a-- 00:18:24.770 --> 00:18:26.240 I think the points that Bill made 00:18:26.240 --> 00:18:28.598 about having to kind of establish yourself and figure 00:18:28.598 --> 00:18:30.890 out what your vision is, you have more time when you're 00:18:30.890 --> 00:18:33.730 older to have a vision. 00:18:33.730 --> 00:18:35.470 LILLY: Yeah, and perspective as well. 00:18:35.470 --> 00:18:38.620 Just thinking about advisors I've had over the years, 00:18:38.620 --> 00:18:41.800 and younger or newer advisors versus more 00:18:41.800 --> 00:18:43.570 established advisors-- they definitely 00:18:43.570 --> 00:18:47.810 have a different perspective. 00:18:47.810 --> 00:18:53.290 And yeah, NASA is actually having the same phenomenon, 00:18:53.290 --> 00:18:57.690 with having directors who are progressively older and older. 00:18:57.690 --> 00:19:00.360 So that's very sim-- they can't-- 00:19:00.360 --> 00:19:02.940 it's very difficult to draw someone away 00:19:02.940 --> 00:19:06.210 from a mid-range career if they're being very successful, 00:19:06.210 --> 00:19:07.780 and having success getting grants. 00:19:07.780 --> 00:19:12.470 And it's hard to get them into administration or directorships 00:19:12.470 --> 00:19:13.053 or management. 00:19:13.053 --> 00:19:14.470 STUDENT: What would the difference 00:19:14.470 --> 00:19:16.125 between young and older directors be? 00:19:20.780 --> 00:19:25.170 LILLY: In general, actually, the more established and older 00:19:25.170 --> 00:19:28.710 advisors that I've had have had a lot more 00:19:28.710 --> 00:19:35.409 perspicacity with respect to what projects are worthwhile, 00:19:35.409 --> 00:19:39.467 actually, or-- 00:19:39.467 --> 00:19:41.300 WILLIAM BONVILLIAN: Through long experience, 00:19:41.300 --> 00:19:44.780 you're suggesting, because they've seen a lot. 00:19:44.902 --> 00:19:46.610 LILLY: Yeah, that's the impression I get. 00:19:46.610 --> 00:19:47.693 WILLIAM BONVILLIAN: Right. 00:19:49.630 --> 00:19:51.010 Next, how about another question? 00:19:51.010 --> 00:19:51.728 MAX: Sure. 00:19:51.728 --> 00:19:52.664 All right. 00:19:58.280 --> 00:20:02.750 So Carleton mentioned that DARPA, 00:20:02.750 --> 00:20:05.222 in order to save time when they are trying 00:20:05.222 --> 00:20:06.680 to develop these projects, they try 00:20:06.680 --> 00:20:10.040 to discourage consensus, which I didn't really see as a-- 00:20:10.040 --> 00:20:13.050 I think it was the second, the next slide you have. 00:20:13.050 --> 00:20:13.550 Yeah. 00:20:13.550 --> 00:20:17.362 So I understand why, because then it 00:20:17.362 --> 00:20:19.320 ensures that whatever you're trying to develop, 00:20:19.320 --> 00:20:20.630 it gets done quickly. 00:20:20.630 --> 00:20:22.370 But just because it gets done quickly 00:20:22.370 --> 00:20:24.410 doesn't really mean that it gets done right. 00:20:24.410 --> 00:20:28.820 So I guess I was curious what people thought on that, 00:20:28.820 --> 00:20:31.028 specifically that decision. 00:20:31.028 --> 00:20:33.320 WILLIAM BONVILLIAN: It's a very interesting point, Max. 00:20:33.320 --> 00:20:35.750 And actually, ARPA-E-- and we'll talk about it 00:20:35.750 --> 00:20:38.770 when we get to ARPA-E and the energy class, 00:20:38.770 --> 00:20:42.390 but ARPA-E does have a process. 00:20:42.390 --> 00:20:46.010 It's a much smaller agency, so it's 300 million, not 3 billion 00:20:46.010 --> 00:20:47.120 like DARPA. 00:20:47.120 --> 00:20:52.370 So it's like a big DARPA office, sizable DARPA office, 00:20:52.370 --> 00:20:54.290 one of its five or six offices. 00:20:54.290 --> 00:20:58.280 So ARPA-E does have a consensus process. 00:20:58.280 --> 00:21:01.730 They bring in-- they call them "project directors" 00:21:01.730 --> 00:21:03.830 but they're program managers. 00:21:03.830 --> 00:21:05.660 And that community, really, you have 00:21:05.660 --> 00:21:08.900 to present your vision to that community. 00:21:08.900 --> 00:21:12.890 And that's a very tough-minded group, and a very sophisticated 00:21:12.890 --> 00:21:13.700 group. 00:21:13.700 --> 00:21:17.150 And the director is quite sophisticated at ARPA-E 00:21:17.150 --> 00:21:18.530 historically, as well. 00:21:18.530 --> 00:21:20.900 So they did a variation on this. 00:21:20.900 --> 00:21:24.680 At DARPA, it's much more, convince your office director, 00:21:24.680 --> 00:21:27.950 and then the office director will work with you 00:21:27.950 --> 00:21:29.660 on convincing the director of DARPA. 00:21:29.660 --> 00:21:32.330 It's a simpler, more straightforward process. 00:21:32.330 --> 00:21:33.980 You do get support from other program 00:21:33.980 --> 00:21:36.870 managers, who will have a lot of advice and ideas for you. 00:21:36.870 --> 00:21:39.813 So it's a pretty supportive operation. 00:21:39.813 --> 00:21:41.480 Nobody's in competition with each other. 00:21:41.480 --> 00:21:42.855 They're there to help each other, 00:21:42.855 --> 00:21:46.970 so that's a positive, although again, they're 00:21:46.970 --> 00:21:48.530 competing to achieve their vision. 00:21:48.530 --> 00:21:49.905 And everybody kind of knows who's 00:21:49.905 --> 00:21:52.362 getting their vision done, and kind of who is 00:21:52.362 --> 00:21:53.570 in trouble with their vision. 00:21:53.570 --> 00:21:55.640 So that's kind of known and understood. 00:21:55.640 --> 00:21:58.070 There's a certain kind of competition in that way, 00:21:58.070 --> 00:22:01.860 but it's not a direct one-on-one competition with each other. 00:22:01.860 --> 00:22:05.030 But at ARPA-E, there is more of a consensus decision making. 00:22:05.030 --> 00:22:10.970 And Luyao when she was telling us about Xerox PARC, 00:22:10.970 --> 00:22:15.140 remember, your whole discussion about the dealer process 00:22:15.140 --> 00:22:18.440 and having to bring your idea to the whole team sitting around 00:22:18.440 --> 00:22:21.050 on beanbags, who would tear it apart-- 00:22:21.050 --> 00:22:24.590 that's kind of a consensus process. 00:22:24.590 --> 00:22:27.740 It's a definite, tough-minded review process 00:22:27.740 --> 00:22:30.087 that DARPA doesn't really quite have. 00:22:30.087 --> 00:22:32.420 So there may be strengths and weaknesses in both models. 00:22:37.668 --> 00:22:38.960 Max, what do you think on that? 00:22:42.040 --> 00:22:44.990 MAX: Yeah, I could see there'd be strengths and weaknesses 00:22:44.990 --> 00:22:45.490 to both. 00:22:45.490 --> 00:22:54.083 But it just feels if you have less consensus, than you have-- 00:22:54.083 --> 00:22:55.750 if you have fewer people that are trying 00:22:55.750 --> 00:22:57.640 to discuss these ideas, then there 00:22:57.640 --> 00:22:59.960 might be some critical flaw that you might not-- 00:22:59.960 --> 00:23:02.502 that you might just miss, just because there are fewer people 00:23:02.502 --> 00:23:06.670 with more diverse experiences. 00:23:06.670 --> 00:23:10.240 Of course, if you have a decent idea of the science behind it, 00:23:10.240 --> 00:23:12.220 and it's not something as untested 00:23:12.220 --> 00:23:15.370 as fusion, for example, then I could 00:23:15.370 --> 00:23:17.210 see it working pretty well. 00:23:17.210 --> 00:23:20.980 I mean, if you had to do that for-- 00:23:20.980 --> 00:23:24.220 I could see it having problems with something untested 00:23:24.220 --> 00:23:25.630 like stealth. 00:23:25.630 --> 00:23:27.640 But apparently, they made it work. 00:23:27.640 --> 00:23:34.888 So maybe they're passing my expectations. 00:23:34.888 --> 00:23:38.890 STUDENT: [INAUDIBLE] Silicon Valley perspective, 00:23:38.890 --> 00:23:42.520 like it's a pretty common mantra to say 00:23:42.520 --> 00:23:44.980 you want to have an idea that people kind of say 00:23:44.980 --> 00:23:46.690 is not a good idea. 00:23:46.690 --> 00:23:48.550 Because the idea is, if you're coming up 00:23:48.550 --> 00:23:49.900 with something that's pretty innovative, 00:23:49.900 --> 00:23:51.220 most likely people won't agree with it, 00:23:51.220 --> 00:23:52.900 or it doesn't make sense, or you have an insight 00:23:52.900 --> 00:23:54.245 that other people don't get. 00:23:54.245 --> 00:23:55.870 Also, if you're an expert, you probably 00:23:55.870 --> 00:23:58.020 know some things that other people don't know, 00:23:58.020 --> 00:24:00.687 that you'd have to bring them up to speed on, especially if it's 00:24:00.687 --> 00:24:02.520 in a completely new area. 00:24:02.520 --> 00:24:05.582 And so the history has been, they're 00:24:05.582 --> 00:24:08.825 just saying that for new fields, who is the expert? 00:24:08.825 --> 00:24:10.450 It wouldn't be a PhD, because the field 00:24:10.450 --> 00:24:11.880 hasn't been invented yet. 00:24:11.880 --> 00:24:15.360 It's this person who just started working on it. 00:24:15.360 --> 00:24:20.100 So Bill Gates was an expert on software for personal computers 00:24:20.100 --> 00:24:21.860 before that even became a thing. 00:24:21.860 --> 00:24:23.400 So it's very common to-- consensus 00:24:23.400 --> 00:24:26.940 is actually really bad, because you put down these ideas based 00:24:26.940 --> 00:24:27.730 on new insights. 00:24:27.730 --> 00:24:28.230 So. 00:24:28.230 --> 00:24:30.240 LILLY: Yeah, and another issue with consensus 00:24:30.240 --> 00:24:32.670 is you have to assume, or you have 00:24:32.670 --> 00:24:37.800 to have a group in which all members have the personality 00:24:37.800 --> 00:24:43.380 type that they will concede to someone else's idea 00:24:43.380 --> 00:24:45.620 to get consensus, even if it's not 00:24:45.620 --> 00:24:47.415 their pet project or their favorite. 00:24:47.415 --> 00:24:48.290 You know what I mean? 00:24:48.290 --> 00:24:50.978 Some people aren't disposed to do that, no matter what. 00:24:53.935 --> 00:24:56.310 WILLIAM BONVILLIAN: About a closing thought on this, Max? 00:24:56.310 --> 00:24:57.460 MAX: Sure. 00:24:57.460 --> 00:25:01.600 So overall, I feel that DARPA seems 00:25:01.600 --> 00:25:03.890 to have been a pretty great success. 00:25:03.890 --> 00:25:06.070 They've made lots of really cool projects. 00:25:06.070 --> 00:25:08.170 They do it very quickly. 00:25:08.170 --> 00:25:12.550 Not sure how cheaply they do it, but they get it done. 00:25:12.550 --> 00:25:14.500 And because of that continued success, 00:25:14.500 --> 00:25:18.580 I feel that that's probably why they continue 00:25:18.580 --> 00:25:21.250 to get so much funding. 00:25:21.250 --> 00:25:25.840 I really try to emphasize that technology transfer aspect, 00:25:25.840 --> 00:25:29.740 because it ensures that the technology that they develop 00:25:29.740 --> 00:25:33.190 isn't just it's a nice, new toy for the military to have. 00:25:33.190 --> 00:25:35.830 It ensures that whatever they make, from the internet, 00:25:35.830 --> 00:25:38.990 GPS, et cetera, that everyone can use it, 00:25:38.990 --> 00:25:42.310 and that it makes our entire society better. 00:25:42.310 --> 00:25:48.890 So I guess that's pretty much just why I like DARPA. 00:25:48.890 --> 00:25:50.150 WILLIAM BONVILLIAN: Great. 00:25:50.150 --> 00:25:53.500 There's a lot of MIT folks and people saying, what do you do? 00:25:53.500 --> 00:25:58.180 All right, let me do a quick wrap up of today's class. 00:25:58.180 --> 00:26:01.990 So we did Mitch Waldrop's book. 00:26:01.990 --> 00:26:04.210 And I really wanted to kind of portray 00:26:04.210 --> 00:26:07.500 the governmental role in supporting the earlier stage 00:26:07.500 --> 00:26:09.820 of the evolution of computing. 00:26:09.820 --> 00:26:13.420 It was a proving ground for new concepts, 00:26:13.420 --> 00:26:17.200 and designs, and architectures. 00:26:17.200 --> 00:26:19.270 The Defense Department created an initial market 00:26:19.270 --> 00:26:21.440 for a lot of the new products, and services, 00:26:21.440 --> 00:26:24.070 and in fact, whole industries. 00:26:24.070 --> 00:26:27.580 This greatly expanded university research capabilities, 00:26:27.580 --> 00:26:31.190 the computing revolution. 00:26:31.190 --> 00:26:36.920 And it was done to further governmental missions. 00:26:36.920 --> 00:26:40.950 But it had, obviously, dramatic societal effects. 00:26:40.950 --> 00:26:43.910 So it featured governmental agencies 00:26:43.910 --> 00:26:45.800 playing a pretty central role. 00:26:45.800 --> 00:26:47.670 DARPA, as we discussed was critical, 00:26:47.670 --> 00:26:51.230 but some of the others were necessary and needed. 00:26:51.230 --> 00:26:53.300 And the governmental agency sponsored 00:26:53.300 --> 00:26:56.990 these industry-university collaborations 00:26:56.990 --> 00:27:00.050 that led to great strength in the university side, 00:27:00.050 --> 00:27:02.330 and focus areas around them. 00:27:02.330 --> 00:27:07.990 So in the reading that we did from the textbook, 00:27:07.990 --> 00:27:09.740 the central point here was that DARPA 00:27:09.740 --> 00:27:14.600 was uniquely able to combine an innovation institutional role, 00:27:14.600 --> 00:27:17.150 as well as a great groups innovation 00:27:17.150 --> 00:27:21.470 at the face-to-face level kind of role. 00:27:21.470 --> 00:27:22.940 That's a remarkable accomplishment. 00:27:22.940 --> 00:27:28.370 And then DARPA's ability, as we talked about in that reading 00:27:28.370 --> 00:27:34.070 as well, to innovate in a legacy sector, 00:27:34.070 --> 00:27:37.610 to do a frontier sector like IT, but also 00:27:37.610 --> 00:27:41.060 innovate within a legacy sector, and do things like stealth, 00:27:41.060 --> 00:27:42.560 and UAVs-- 00:27:42.560 --> 00:27:46.670 that's a pretty fascinating organization, 00:27:46.670 --> 00:27:48.920 that's flexible, and interesting, 00:27:48.920 --> 00:27:51.140 and able to do great stuff. 00:27:51.140 --> 00:27:53.450 Glenn Fong's piece was really about DARPA 00:27:53.450 --> 00:27:55.820 playing the central institutional mobilization 00:27:55.820 --> 00:27:58.340 role for the IT revolution. 00:27:58.340 --> 00:28:01.160 And Tammy Carleton's piece really 00:28:01.160 --> 00:28:04.070 told us about this kind of central importance 00:28:04.070 --> 00:28:09.980 of technology visioning as a key modus operandi, that DARPA 00:28:09.980 --> 00:28:11.712 been able to operate in. 00:28:11.712 --> 00:28:14.762 Any closing questions? 00:28:14.762 --> 00:28:16.220 Good luck with your paper outlines. 00:28:16.220 --> 00:28:18.170 Let me know if you have questions. 00:28:18.170 --> 00:28:20.950 And I look forward to seeing you on Tuesday.