WEBVTT 00:00:00.000 --> 00:00:01.443 align:middle line:90% [SQUEAKING] 00:00:01.443 --> 00:00:02.867 align:middle line:90% [RUSTLING] 00:00:02.867 --> 00:00:03.367 align:middle line:90% 00:00:03.367 --> 00:00:04.329 align:middle line:90% [CLICKING] 00:00:04.329 --> 00:00:12.297 align:middle line:90% 00:00:12.297 --> 00:00:13.880 align:middle line:84% JUSTIN REICH: Let's get right into it. 00:00:13.880 --> 00:00:16.790 align:middle line:84% There's a bunch that I want to try to catch up from yesterday. 00:00:16.790 --> 00:00:18.870 align:middle line:84% You've just turned in paper number one. 00:00:18.870 --> 00:00:22.140 align:middle line:84% But I'm a big fan of trying to have people do work, 00:00:22.140 --> 00:00:24.280 align:middle line:84% like do the daily work in the class, 00:00:24.280 --> 00:00:26.280 align:middle line:84% thinking towards what the major assignments are. 00:00:26.280 --> 00:00:28.640 align:middle line:84% So now, I have to tell you about paper number two. 00:00:28.640 --> 00:00:30.500 align:middle line:84% For paper number two, you are going 00:00:30.500 --> 00:00:33.270 align:middle line:84% to identify a specific learning environment. 00:00:33.270 --> 00:00:36.980 align:middle line:84% You're going to identify the Physics Department 00:00:36.980 --> 00:00:41.100 align:middle line:84% at MIT, your middle school, your elementary school, 00:00:41.100 --> 00:00:44.990 align:middle line:84% your high school World Languages Department, 00:00:44.990 --> 00:00:47.840 align:middle line:84% an after school program you work with a library but something 00:00:47.840 --> 00:00:52.970 align:middle line:84% very specific, not like schools in general. 00:00:52.970 --> 00:00:54.260 align:middle line:90% A place. 00:00:54.260 --> 00:00:58.370 align:middle line:84% And you're going to reflect upon a particular education 00:00:58.370 --> 00:00:59.160 align:middle line:90% technology. 00:00:59.160 --> 00:01:01.820 align:middle line:84% And you are either going to recommend or not 00:01:01.820 --> 00:01:04.982 align:middle line:84% recommend that they adopt this particular technology. 00:01:04.982 --> 00:01:06.440 align:middle line:84% So part of the argument that you'll 00:01:06.440 --> 00:01:10.130 align:middle line:84% read about in, hopefully, the first half of Failure 00:01:10.130 --> 00:01:13.490 align:middle line:84% to Disrupt, is that while we can get 00:01:13.490 --> 00:01:17.400 align:middle line:84% a general sense of the efficacy of different technologies-- 00:01:17.400 --> 00:01:20.450 align:middle line:84% we could say, this class of technologies 00:01:20.450 --> 00:01:22.490 align:middle line:84% seems to have better research evidence behind it 00:01:22.490 --> 00:01:24.787 align:middle line:84% than this class of technologies-- there's really 00:01:24.787 --> 00:01:26.370 align:middle line:84% nothing that anyone has ever invented, 00:01:26.370 --> 00:01:28.460 align:middle line:84% which is really every school or every department 00:01:28.460 --> 00:01:31.910 align:middle line:84% ought to be using X. All of those decisions 00:01:31.910 --> 00:01:36.360 align:middle line:84% have to do with particular times, particular places, 00:01:36.360 --> 00:01:37.760 align:middle line:90% particular people. 00:01:37.760 --> 00:01:41.640 align:middle line:84% So what you'll have to do is identify some technology. 00:01:41.640 --> 00:01:45.830 align:middle line:84% Classify it like what genre is it from, or what is it like. 00:01:45.830 --> 00:01:47.580 align:middle line:90% What are its potential benefits? 00:01:47.580 --> 00:01:48.750 align:middle line:90% What are its affordances? 00:01:48.750 --> 00:01:49.890 align:middle line:90% What are its limitations? 00:01:49.890 --> 00:01:52.130 align:middle line:84% And then what is the characteristic of the place 00:01:52.130 --> 00:01:53.480 align:middle line:90% that you're recommending it to. 00:01:53.480 --> 00:01:57.440 align:middle line:84% And do the benefits, and needs, and characteristics 00:01:57.440 --> 00:02:00.680 align:middle line:84% of a place all line up, such that you recommend adoption? 00:02:00.680 --> 00:02:02.540 align:middle line:84% Or, do they not line up such that you 00:02:02.540 --> 00:02:04.550 align:middle line:90% don't recommend adoption? 00:02:04.550 --> 00:02:09.240 align:middle line:84% It should be a memo directed to a particular stakeholder-- 00:02:09.240 --> 00:02:11.900 align:middle line:84% a department head, an assistant superintendent for curriculum 00:02:11.900 --> 00:02:14.070 align:middle line:84% and instruction, the library director, 00:02:14.070 --> 00:02:16.920 align:middle line:84% the President of the Boys and Girls Club, whatever it is. 00:02:16.920 --> 00:02:19.160 align:middle line:84% I think I had one student who was 00:02:19.160 --> 00:02:22.290 align:middle line:84% building a program for Ukrainian refugees last year. 00:02:22.290 --> 00:02:25.130 align:middle line:84% And he basically wrote a memo to himself, which was fine. 00:02:25.130 --> 00:02:28.905 align:middle line:84% On pages 74 through 76, in Failure to Disrupt which-- 00:02:28.905 --> 00:02:30.030 align:middle line:90% at least in the hardcover-- 00:02:30.030 --> 00:02:32.383 align:middle line:84% I think it's the same in the paperback-- 00:02:32.383 --> 00:02:34.800 align:middle line:84% there's a section called What a Department Head Should Do, 00:02:34.800 --> 00:02:37.020 align:middle line:84% which is not exactly this assignment, 00:02:37.020 --> 00:02:38.540 align:middle line:84% but a pretty close worked example. 00:02:38.540 --> 00:02:41.400 align:middle line:90% 00:02:41.400 --> 00:02:43.970 align:middle line:84% So I think that specific example is 00:02:43.970 --> 00:02:46.320 align:middle line:84% about using intelligent tutors in math class. 00:02:46.320 --> 00:02:47.737 align:middle line:84% So you should pick something other 00:02:47.737 --> 00:02:50.780 align:middle line:84% than that specific example for you to use. 00:02:50.780 --> 00:02:55.040 align:middle line:84% But you should think about doing things, 00:02:55.040 --> 00:02:58.187 align:middle line:84% like assessing the claims that the developers 00:02:58.187 --> 00:03:00.770 align:middle line:84% of this technology make, seeing what kind of research evidence 00:03:00.770 --> 00:03:03.680 align:middle line:84% there is around that, seeing under what context 00:03:03.680 --> 00:03:06.920 align:middle line:84% this seems to work or not, what kind of enabling conditions 00:03:06.920 --> 00:03:09.810 align:middle line:84% are required, whether or not those enabling conditions exist. 00:03:09.810 --> 00:03:11.560 align:middle line:84% We'll talk about all these kinds of things 00:03:11.560 --> 00:03:12.890 align:middle line:90% over the next few weeks. 00:03:12.890 --> 00:03:15.772 align:middle line:84% So you don't have to understand what 00:03:15.772 --> 00:03:16.980 align:middle line:90% you're going to do right now. 00:03:16.980 --> 00:03:19.730 align:middle line:84% But I you to have a sense of this assignment. 00:03:19.730 --> 00:03:23.240 align:middle line:84% For some of you, you will pick a context 00:03:23.240 --> 00:03:26.933 align:middle line:84% that you can work with that you actually know a ton about. 00:03:26.933 --> 00:03:29.100 align:middle line:84% Or, like even if you didn't know something about it, 00:03:29.100 --> 00:03:33.260 align:middle line:84% you would be able to like, oh, the head of AeroAstro 00:03:33.260 --> 00:03:34.560 align:middle line:90% is a good colleague of mine. 00:03:34.560 --> 00:03:36.080 align:middle line:84% I'm going to just go ask him what he thinks about this. 00:03:36.080 --> 00:03:38.220 align:middle line:84% Or, like what capacity are and things like that. 00:03:38.220 --> 00:03:41.900 align:middle line:84% There may be other things that you need to fabricate, 00:03:41.900 --> 00:03:43.650 align:middle line:84% which you would be allowed to fabricate. 00:03:43.650 --> 00:03:45.470 align:middle line:84% You just have to somewhere stipulate along the way 00:03:45.470 --> 00:03:46.470 align:middle line:90% that you're fabricating them. 00:03:46.470 --> 00:03:46.970 align:middle line:90% So I know. 00:03:46.970 --> 00:03:49.160 align:middle line:84% I have to be like the characteristics 00:03:49.160 --> 00:03:51.798 align:middle line:84% of the world language teachers in my high school are this. 00:03:51.798 --> 00:03:53.840 align:middle line:84% And you might not actually know who the new staff 00:03:53.840 --> 00:03:54.882 align:middle line:90% are and things like that. 00:03:54.882 --> 00:03:59.390 align:middle line:90% But that would be OK. 00:03:59.390 --> 00:04:01.100 align:middle line:84% There's pages written about what you're 00:04:01.100 --> 00:04:02.730 align:middle line:90% supposed to do in the syllabus. 00:04:02.730 --> 00:04:04.910 align:middle line:84% But does that give you a general sense of the target 00:04:04.910 --> 00:04:07.147 align:middle line:84% that you should be thinking about over the-- 00:04:07.147 --> 00:04:08.730 align:middle line:84% I don't even remember the exact dates, 00:04:08.730 --> 00:04:10.380 align:middle line:84% but it's roughly a month from now. 00:04:10.380 --> 00:04:12.900 align:middle line:84% Yeah, each assignment takes about a third of the class. 00:04:12.900 --> 00:04:15.530 align:middle line:90% So does that all seem good? 00:04:15.530 --> 00:04:17.871 align:middle line:90% Great. 00:04:17.871 --> 00:04:19.579 align:middle line:84% Anybody have any questions that they want 00:04:19.579 --> 00:04:20.880 align:middle line:90% to ask about that assignment? 00:04:20.880 --> 00:04:23.900 align:middle line:90% 00:04:23.900 --> 00:04:26.970 align:middle line:90% OK, let's wrap up MOOCs. 00:04:26.970 --> 00:04:29.090 align:middle line:84% So last time we were talking about MOOCs 00:04:29.090 --> 00:04:30.710 align:middle line:90% made these three big bets. 00:04:30.710 --> 00:04:32.270 align:middle line:84% That they could provide new pathways 00:04:32.270 --> 00:04:34.610 align:middle line:84% into higher education for people with limited access. 00:04:34.610 --> 00:04:37.670 align:middle line:84% That they could reorganize, disrupt, unbundle 00:04:37.670 --> 00:04:38.490 align:middle line:90% higher education. 00:04:38.490 --> 00:04:40.972 align:middle line:84% They could create new structures across higher education. 00:04:40.972 --> 00:04:42.680 align:middle line:84% And then the one that I was most involved 00:04:42.680 --> 00:04:47.420 align:middle line:84% in is that they create a new data driven science of learning. 00:04:47.420 --> 00:04:51.980 align:middle line:84% Maybe part of the second one, but maybe 00:04:51.980 --> 00:04:54.140 align:middle line:84% was clearly a bet for some actors, 00:04:54.140 --> 00:04:56.200 align:middle line:84% less so for nonprofits, more so for for-profits 00:04:56.200 --> 00:04:58.810 align:middle line:84% was that you could make a whole bunch of money doing this. 00:04:58.810 --> 00:05:02.960 align:middle line:84% That has not panned out as well as some people had hoped. 00:05:02.960 --> 00:05:05.950 align:middle line:84% So here's an announcement today that Accenture 00:05:05.950 --> 00:05:08.170 align:middle line:90% is purchasing Udacity. 00:05:08.170 --> 00:05:12.370 align:middle line:84% So Udacity was the company that was founded by Sebastian Thrun. 00:05:12.370 --> 00:05:13.870 align:middle line:84% Sebastian Thrun is one of the people 00:05:13.870 --> 00:05:17.080 align:middle line:84% who taught this Intro to AI class that made the whole MOOC 00:05:17.080 --> 00:05:18.225 align:middle line:90% phenomenon start. 00:05:18.225 --> 00:05:21.730 align:middle line:90% 00:05:21.730 --> 00:05:23.570 align:middle line:84% Accenture is like a consulting firm, 00:05:23.570 --> 00:05:25.790 align:middle line:84% and they're going to use it to do AI training. 00:05:25.790 --> 00:05:30.140 align:middle line:84% So this is not like transforming all of education. 00:05:30.140 --> 00:05:32.410 align:middle line:84% This is doing pretty well-established 00:05:32.410 --> 00:05:34.090 align:middle line:90% executive education. 00:05:34.090 --> 00:05:37.150 align:middle line:84% And there may be ways that you could do executive education 00:05:37.150 --> 00:05:39.333 align:middle line:84% better with the MOOC platform than you could 00:05:39.333 --> 00:05:41.500 align:middle line:84% with other pre-existing technologies, or strategies, 00:05:41.500 --> 00:05:42.333 align:middle line:90% or things like this. 00:05:42.333 --> 00:05:45.340 align:middle line:84% But like, this is not going to profoundly changed the way 00:05:45.340 --> 00:05:46.850 align:middle line:90% executive education unfolds. 00:05:46.850 --> 00:05:49.940 align:middle line:84% This is just like somebody who sells executive education, 00:05:49.940 --> 00:05:52.210 align:middle line:84% buying a potential provider of it. 00:05:52.210 --> 00:05:58.900 align:middle line:84% In the notes here, it says that their best guess is that they-- 00:05:58.900 --> 00:06:00.850 align:middle line:84% a deal fell through with an asking price 00:06:00.850 --> 00:06:02.300 align:middle line:90% of about $80 million. 00:06:02.300 --> 00:06:03.910 align:middle line:84% And so nobody knows for sure, but they 00:06:03.910 --> 00:06:07.090 align:middle line:84% think that this deal was probably about that size. 00:06:07.090 --> 00:06:11.990 align:middle line:84% But Udacity took in $300 million of investment. 00:06:11.990 --> 00:06:14.200 align:middle line:84% So I don't go to the Sloan School of Management 00:06:14.200 --> 00:06:15.200 align:middle line:90% like some of you do. 00:06:15.200 --> 00:06:17.590 align:middle line:84% But I'm pretty sure that if you take $300 million 00:06:17.590 --> 00:06:20.500 align:middle line:84% and sell a thing for $80 million, you didn't do it right. 00:06:20.500 --> 00:06:24.460 align:middle line:84% Or, at least your investors don't think so. 00:06:24.460 --> 00:06:31.420 align:middle line:84% But I think that's, to me, it's a great indication of where 00:06:31.420 --> 00:06:34.330 align:middle line:84% this class of technologies have settled out, 00:06:34.330 --> 00:06:36.860 align:middle line:84% which is like $80 million, a lot of money. 00:06:36.860 --> 00:06:39.110 align:middle line:84% If any of you were offered $80 million for a business, 00:06:39.110 --> 00:06:40.250 align:middle line:84% you'd probably be like, yeah, that's pretty good. 00:06:40.250 --> 00:06:41.167 align:middle line:90% I'll take $80 million. 00:06:41.167 --> 00:06:43.240 align:middle line:90% That seems great. 00:06:43.240 --> 00:06:49.420 align:middle line:84% But what it's accomplishing is like a very well established 00:06:49.420 --> 00:06:52.300 align:middle line:84% existing thing in the education system. 00:06:52.300 --> 00:06:55.300 align:middle line:84% Or, a way that I've phrased it in the book, which 00:06:55.300 --> 00:06:57.880 align:middle line:84% a colleague of mine really came up with, 00:06:57.880 --> 00:07:02.000 align:middle line:84% is that these technologies did not disrupt education systems. 00:07:02.000 --> 00:07:04.400 align:middle line:84% They were domesticated by education systems. 00:07:04.400 --> 00:07:06.580 align:middle line:84% Education systems said, like you fit there. 00:07:06.580 --> 00:07:08.297 align:middle line:84% Like, you, little company, you would 00:07:08.297 --> 00:07:10.130 align:middle line:84% be pretty good at doing executive education. 00:07:10.130 --> 00:07:11.290 align:middle line:90% Why you don't you do that? 00:07:11.290 --> 00:07:14.200 align:middle line:84% That is not at all what Sebastian Thrun was envisioning 00:07:14.200 --> 00:07:17.410 align:middle line:84% when he said that there might be only 50 educational institutions 00:07:17.410 --> 00:07:19.990 align:middle line:84% in higher education in the world. 00:07:19.990 --> 00:07:23.770 align:middle line:84% But to me, this is very representative of how 00:07:23.770 --> 00:07:26.470 align:middle line:84% I think education technology often works. 00:07:26.470 --> 00:07:29.583 align:middle line:84% We very rarely build things that are totally transformative. 00:07:29.583 --> 00:07:31.000 align:middle line:84% And we can often build things that 00:07:31.000 --> 00:07:33.255 align:middle line:84% are useful in pretty specific contexts. 00:07:33.255 --> 00:07:37.464 align:middle line:90% 00:07:37.464 --> 00:07:39.850 align:middle line:84% I and lots of other people had really 00:07:39.850 --> 00:07:44.830 align:middle line:84% hoped that there would be very new kinds of science that 00:07:44.830 --> 00:07:46.300 align:middle line:90% would be possible. 00:07:46.300 --> 00:07:49.430 align:middle line:84% So we had never really, in higher education, 00:07:49.430 --> 00:07:53.140 align:middle line:84% had a system where you just had, all of a sudden, 00:07:53.140 --> 00:07:56.110 align:middle line:84% millions and millions of people doing online learning 00:07:56.110 --> 00:07:58.450 align:middle line:84% and all of that was being recorded. 00:07:58.450 --> 00:08:01.300 align:middle line:84% So when I got hired to help build the research 00:08:01.300 --> 00:08:04.690 align:middle line:84% infrastructure for HarvardX and MITx and EdX, 00:08:04.690 --> 00:08:07.515 align:middle line:84% one of the most exciting things was like, all of these people 00:08:07.515 --> 00:08:08.140 align:middle line:90% are signing up. 00:08:08.140 --> 00:08:09.220 align:middle line:90% And they're all clicking. 00:08:09.220 --> 00:08:11.200 align:middle line:90% And we're recording everything. 00:08:11.200 --> 00:08:15.500 align:middle line:84% You might think of all of the things that you did in school. 00:08:15.500 --> 00:08:17.000 align:middle line:84% Probably even better, as an example, 00:08:17.000 --> 00:08:19.570 align:middle line:84% like think about all the things that I did in school. 00:08:19.570 --> 00:08:22.280 align:middle line:84% So I filled out zillions of worksheets. 00:08:22.280 --> 00:08:23.890 align:middle line:90% I typed all kinds of papers. 00:08:23.890 --> 00:08:25.900 align:middle line:84% All that stuff just ended up in my mom's attic 00:08:25.900 --> 00:08:27.230 align:middle line:90% and was eventually thrown away. 00:08:27.230 --> 00:08:29.620 align:middle line:84% The whole records of my learning were lost. 00:08:29.620 --> 00:08:31.450 align:middle line:84% But young people today have their records 00:08:31.450 --> 00:08:34.190 align:middle line:84% of learning stored in all of these online platforms. 00:08:34.190 --> 00:08:36.820 align:middle line:84% If you saw what millions of learners 00:08:36.820 --> 00:08:40.750 align:middle line:84% were doing in really granular, detailed ways over days, 00:08:40.750 --> 00:08:42.610 align:middle line:84% or months, or years, shouldn't you 00:08:42.610 --> 00:08:44.953 align:middle line:84% be able to figure out all kinds of exciting things 00:08:44.953 --> 00:08:46.870 align:middle line:84% about the science of learning, about like what 00:08:46.870 --> 00:08:48.330 align:middle line:84% are the most effective teaching techniques, 00:08:48.330 --> 00:08:49.400 align:middle line:90% how do people learn best? 00:08:49.400 --> 00:08:51.590 align:middle line:84% All kinds of things that might be helpless. 00:08:51.590 --> 00:08:56.850 align:middle line:84% And it proved to be much, much harder than people expected. 00:08:56.850 --> 00:09:00.080 align:middle line:84% I would say, the core reason why it proved to be really hard 00:09:00.080 --> 00:09:03.750 align:middle line:84% is that you can have terabytes of data about what people click, 00:09:03.750 --> 00:09:07.730 align:middle line:84% and still not really know what's going on in their heads. 00:09:07.730 --> 00:09:11.210 align:middle line:84% What people click on a screen or what they add as an answer 00:09:11.210 --> 00:09:14.450 align:middle line:84% to a problem doesn't necessarily tell you all that much 00:09:14.450 --> 00:09:17.540 align:middle line:84% about the cognition that they went through to get there. 00:09:17.540 --> 00:09:20.030 align:middle line:84% It doesn't tell you a whole lot about which 00:09:20.030 --> 00:09:22.370 align:middle line:84% parts of the instruction that they receive to that point 00:09:22.370 --> 00:09:24.080 align:middle line:90% helped them do that. 00:09:24.080 --> 00:09:25.790 align:middle line:84% For years, there was a whole class 00:09:25.790 --> 00:09:29.733 align:middle line:84% of papers that just correlated everything to everything else. 00:09:29.733 --> 00:09:30.900 align:middle line:90% How many times do you click? 00:09:30.900 --> 00:09:32.660 align:middle line:90% How many times did you log in? 00:09:32.660 --> 00:09:34.080 align:middle line:90% How many units did you do? 00:09:34.080 --> 00:09:35.427 align:middle line:90% How far apart they were? 00:09:35.427 --> 00:09:36.510 align:middle line:90% All these kinds of things. 00:09:36.510 --> 00:09:39.830 align:middle line:84% And basically, like there was a whole series of papers 00:09:39.830 --> 00:09:41.210 align:middle line:84% that were all published that were 00:09:41.210 --> 00:09:44.690 align:middle line:84% all along the lines of people who do more stuff do better 00:09:44.690 --> 00:09:47.283 align:middle line:90% than people who do less stuff. 00:09:47.283 --> 00:09:49.450 align:middle line:84% People who clicked more often in the systems, people 00:09:49.450 --> 00:09:52.510 align:middle line:84% who watched all the videos, did better on the problems. 00:09:52.510 --> 00:09:54.010 align:middle line:84% People who did all the problems were 00:09:54.010 --> 00:09:55.700 align:middle line:90% more likely to pass the course. 00:09:55.700 --> 00:09:58.420 align:middle line:84% People who logged in more time, people who spent more hours 00:09:58.420 --> 00:10:00.075 align:middle line:84% were more likely to pass the course. 00:10:00.075 --> 00:10:02.110 align:middle line:90% None of that is interesting. 00:10:02.110 --> 00:10:04.330 align:middle line:84% You didn't need a $60 million research enterprise 00:10:04.330 --> 00:10:07.210 align:middle line:84% to figure out that doing more stuff helps people learn. 00:10:07.210 --> 00:10:11.530 align:middle line:84% There are a few things that we did that-- 00:10:11.530 --> 00:10:14.080 align:middle line:84% and I'll talk about them more as the semester goes on-- that 00:10:14.080 --> 00:10:16.510 align:middle line:90% tried to go beyond that. 00:10:16.510 --> 00:10:18.130 align:middle line:84% There were a bunch of us who were 00:10:18.130 --> 00:10:21.970 align:middle line:84% really interested in this idea that like-- 00:10:21.970 --> 00:10:23.470 align:middle line:84% a bunch of people have done research 00:10:23.470 --> 00:10:26.800 align:middle line:84% that suggests that the attitudes that you bring to a learning 00:10:26.800 --> 00:10:30.340 align:middle line:84% experience really matter for your success in that learning 00:10:30.340 --> 00:10:31.450 align:middle line:90% experience. 00:10:31.450 --> 00:10:34.450 align:middle line:84% And of you ever heard of stereotype threat? 00:10:34.450 --> 00:10:36.280 align:middle line:84% So stereotype threat is this idea 00:10:36.280 --> 00:10:40.240 align:middle line:84% that if we trigger stereotypes that people 00:10:40.240 --> 00:10:42.640 align:middle line:84% have as they enter a learning environment 00:10:42.640 --> 00:10:44.680 align:middle line:84% or as they're in a learning environment, 00:10:44.680 --> 00:10:48.230 align:middle line:84% those stereotypes can negatively impact them. 00:10:48.230 --> 00:10:52.515 align:middle line:84% So if you ask people to describe their gender before a math test, 00:10:52.515 --> 00:10:53.890 align:middle line:84% then people who identify as girls 00:10:53.890 --> 00:10:55.800 align:middle line:90% are more likely to do less well. 00:10:55.800 --> 00:10:58.300 align:middle line:84% You can do the same thing in different racial configurations 00:10:58.300 --> 00:11:00.220 align:middle line:84% and other kinds of things like that. 00:11:00.220 --> 00:11:02.998 align:middle line:84% And the way that you can inoculate people, in theory, 00:11:02.998 --> 00:11:04.540 align:middle line:84% against some of these kinds of things 00:11:04.540 --> 00:11:07.390 align:middle line:84% is by emphasizing how they belong. 00:11:07.390 --> 00:11:09.432 align:middle line:84% Is by saying, like you belong in here. 00:11:09.432 --> 00:11:11.390 align:middle line:84% Like, you have a space, you deserve to be here. 00:11:11.390 --> 00:11:14.320 align:middle line:84% And so you could imagine that if people go into a learning 00:11:14.320 --> 00:11:17.090 align:middle line:84% environment, and they're like, I'm not sure if this is for me. 00:11:17.090 --> 00:11:20.160 align:middle line:84% Then you're paying attention to not feeling like this is for me. 00:11:20.160 --> 00:11:22.160 align:middle line:84% And you're not paying attention to the learning. 00:11:22.160 --> 00:11:23.170 align:middle line:90% And then a test comes along. 00:11:23.170 --> 00:11:24.650 align:middle line:84% And you don't do as well on that test. 00:11:24.650 --> 00:11:26.690 align:middle line:84% And then that makes you feel like you belong even less. 00:11:26.690 --> 00:11:28.560 align:middle line:84% And then you're feeling more unbelonging. 00:11:28.560 --> 00:11:29.810 align:middle line:90% And then you don't do as well. 00:11:29.810 --> 00:11:31.160 align:middle line:90% And you get a negative cycle. 00:11:31.160 --> 00:11:32.800 align:middle line:84% You could imagine the opposite, where you feel like, 00:11:32.800 --> 00:11:34.760 align:middle line:84% oh, I definitely belong in this learning environment. 00:11:34.760 --> 00:11:36.500 align:middle line:84% This is going to be a good growth experience for me. 00:11:36.500 --> 00:11:37.930 align:middle line:84% And then you are focused on learning. 00:11:37.930 --> 00:11:38.810 align:middle line:90% And then you do better. 00:11:38.810 --> 00:11:41.102 align:middle line:84% And that makes you feel like you belong, and on and on. 00:11:41.102 --> 00:11:44.350 align:middle line:84% So we were really interested in these entry points. 00:11:44.350 --> 00:11:48.345 align:middle line:84% And basically, every class at Harvard and MIT 00:11:48.345 --> 00:11:50.470 align:middle line:84% offered had a little pre-course survey at the beg-- 00:11:50.470 --> 00:11:51.640 align:middle line:84% have any of you ever taken a class that 00:11:51.640 --> 00:11:53.630 align:middle line:84% has a little survey at the beginning of it for your MOOCs? 00:11:53.630 --> 00:11:55.005 align:middle line:84% If you did it for Harvard or MIT, 00:11:55.005 --> 00:12:00.790 align:middle line:84% I probably wrote it, especially if you did it a few years ago. 00:12:00.790 --> 00:12:04.600 align:middle line:84% And so we started randomly assigning people 00:12:04.600 --> 00:12:07.355 align:middle line:90% these little interventions. 00:12:07.355 --> 00:12:08.980 align:middle line:84% Like, you'd answer a bunch of questions 00:12:08.980 --> 00:12:10.480 align:middle line:84% about what you know about the topic. 00:12:10.480 --> 00:12:12.280 align:middle line:84% And then we'd say things like, write 00:12:12.280 --> 00:12:14.800 align:middle line:90% a letter to a future student-- 00:12:14.800 --> 00:12:16.540 align:middle line:90% one was a values affirmation. 00:12:16.540 --> 00:12:18.550 align:middle line:84% Pick from this list which of these values 00:12:18.550 --> 00:12:19.900 align:middle line:90% are really important to you. 00:12:19.900 --> 00:12:20.900 align:middle line:90% And they-- I don't know. 00:12:20.900 --> 00:12:24.350 align:middle line:84% They could be like loyalty, personal growth, family, 00:12:24.350 --> 00:12:25.660 align:middle line:90% all kinds of things. 00:12:25.660 --> 00:12:29.800 align:middle line:84% Write a short letter to yourself about why taking this class 00:12:29.800 --> 00:12:31.750 align:middle line:90% aligns with those values. 00:12:31.750 --> 00:12:35.350 align:middle line:84% So asking people to just write for a few minutes about why-- 00:12:35.350 --> 00:12:37.270 align:middle line:84% we do other things about belonging, right? 00:12:37.270 --> 00:12:38.650 align:middle line:84% Write a short letter to a future student 00:12:38.650 --> 00:12:41.180 align:middle line:84% in this class about why they should feel like they belong. 00:12:41.180 --> 00:12:42.530 align:middle line:90% Things like that. 00:12:42.530 --> 00:12:46.361 align:middle line:84% And our hope was is that it would 00:12:46.361 --> 00:12:48.880 align:middle line:84% trigger this positive cascade at the beginning of the class. 00:12:48.880 --> 00:12:50.130 align:middle line:90% You feel like you belong more. 00:12:50.130 --> 00:12:51.120 align:middle line:90% You feel like you value it more. 00:12:51.120 --> 00:12:51.870 align:middle line:90% You do better. 00:12:51.870 --> 00:12:53.453 align:middle line:84% You pay attention more, you do better, 00:12:53.453 --> 00:12:55.280 align:middle line:84% you learn more, all those kinds of things. 00:12:55.280 --> 00:13:02.030 align:middle line:84% We started by trying these in just a couple of classes. 00:13:02.030 --> 00:13:05.910 align:middle line:84% And even if we were picking a couple of classes, 00:13:05.910 --> 00:13:09.530 align:middle line:84% we were still doing some of the largest experiments that 00:13:09.530 --> 00:13:11.810 align:middle line:84% had ever been conducted in higher education. 00:13:11.810 --> 00:13:15.173 align:middle line:84% We could pick a couple of classes that might have 10,000 00:13:15.173 --> 00:13:17.090 align:middle line:84% people that were actively participating in it, 00:13:17.090 --> 00:13:18.882 align:middle line:84% and doing our things, and things like that. 00:13:18.882 --> 00:13:19.880 align:middle line:90% Kind of a lot of folks. 00:13:19.880 --> 00:13:22.160 align:middle line:84% So here are four things that we tried. 00:13:22.160 --> 00:13:26.030 align:middle line:84% One was plan making, having people write down 00:13:26.030 --> 00:13:28.620 align:middle line:84% exactly how they plan to complete the class. 00:13:28.620 --> 00:13:31.522 align:middle line:84% Like, when they were going to do it, who was going to help them, 00:13:31.522 --> 00:13:33.480 align:middle line:84% what they were going to do when they get stuck? 00:13:33.480 --> 00:13:34.520 align:middle line:90% Those kinds of things. 00:13:34.520 --> 00:13:37.400 align:middle line:84% This value relevance things that you just described. 00:13:37.400 --> 00:13:40.240 align:middle line:84% And then mental contrasting with implementation impatient 00:13:40.240 --> 00:13:42.220 align:middle line:84% intentions is basically like, what 00:13:42.220 --> 00:13:45.080 align:middle line:84% are the kinds of things that might make you stuck? 00:13:45.080 --> 00:13:48.730 align:middle line:84% And how would you get unstuck when you encounter these things? 00:13:48.730 --> 00:13:52.700 align:middle line:84% When we did them in one or two classes at a time, 00:13:52.700 --> 00:13:53.942 align:middle line:90% they worked pretty well. 00:13:53.942 --> 00:13:56.150 align:middle line:84% So I don't know how familiar you are with statistics. 00:13:56.150 --> 00:13:58.220 align:middle line:84% But the beta parameters are the one you look at. 00:13:58.220 --> 00:14:00.970 align:middle line:84% So this plan making things people were four percentage 00:14:00.970 --> 00:14:03.400 align:middle line:84% points more likely to finish the class if they got 00:14:03.400 --> 00:14:05.350 align:middle line:84% the intervention than they didn't. 00:14:05.350 --> 00:14:06.640 align:middle line:90% That's huge. 00:14:06.640 --> 00:14:09.642 align:middle line:84% If you ask you ask people to do something for five minutes 00:14:09.642 --> 00:14:11.350 align:middle line:84% at the beginning of the class and they're 00:14:11.350 --> 00:14:13.450 align:middle line:90% 4% more likely to finish it-- 00:14:13.450 --> 00:14:15.830 align:middle line:84% just like the cost-benefit of that is off the charts. 00:14:15.830 --> 00:14:17.890 align:middle line:90% That's fabulous. 00:14:17.890 --> 00:14:21.050 align:middle line:84% This one was getting people to do more activities, 00:14:21.050 --> 00:14:22.460 align:middle line:84% getting people to persist longer, 00:14:22.460 --> 00:14:24.167 align:middle line:84% getting people to complete better, 00:14:24.167 --> 00:14:25.250 align:middle line:90% all these kinds of things. 00:14:25.250 --> 00:14:26.930 align:middle line:84% And some of these studies were huge-- 00:14:26.930 --> 00:14:30.310 align:middle line:84% 5,000 people, 3,000 people, 2000 people. 00:14:30.310 --> 00:14:32.530 align:middle line:90% That's pretty good. 00:14:32.530 --> 00:14:36.380 align:middle line:84% We then were like, we were pretty excited about ourselves. 00:14:36.380 --> 00:14:38.440 align:middle line:84% We were like, we've come up with some stuff 00:14:38.440 --> 00:14:41.820 align:middle line:84% that doesn't cost anything and really helps learners. 00:14:41.820 --> 00:14:43.680 align:middle line:84% Let's go ahead and take these things 00:14:43.680 --> 00:14:46.270 align:middle line:84% and use them in all of the classes. 00:14:46.270 --> 00:14:51.240 align:middle line:84% Let's let Harvard and MIT let us test things in every class. 00:14:51.240 --> 00:14:56.220 align:middle line:84% And it is not uncommon in educational research, 00:14:56.220 --> 00:14:58.660 align:middle line:84% that if something works in a little space, 00:14:58.660 --> 00:15:02.070 align:middle line:84% that when you try it at much bigger scales, it doesn't work. 00:15:02.070 --> 00:15:04.523 align:middle line:84% That is a very frustrating finding, but also 00:15:04.523 --> 00:15:05.440 align:middle line:90% a very common finding. 00:15:05.440 --> 00:15:07.470 align:middle line:84% And that is exactly what happened to us. 00:15:07.470 --> 00:15:09.850 align:middle line:84% So we took these things which, again, 00:15:09.850 --> 00:15:12.370 align:middle line:84% would have been considered like huge studies in education. 00:15:12.370 --> 00:15:16.710 align:middle line:84% Randomly assigning 4,000 people to a treatment or not is 00:15:16.710 --> 00:15:17.650 align:middle line:90% enormous. 00:15:17.650 --> 00:15:20.940 align:middle line:84% And we did it with a quarter of a million people, 00:15:20.940 --> 00:15:22.560 align:middle line:90% like a lot of learners. 00:15:22.560 --> 00:15:25.375 align:middle line:84% And basically, everything that we had tried and wrote 00:15:25.375 --> 00:15:27.000 align:middle line:84% a bunch of papers that said, hey, these 00:15:27.000 --> 00:15:28.060 align:middle line:90% are pretty good ideas-- 00:15:28.060 --> 00:15:29.670 align:middle line:84% we had to be like, actually, they 00:15:29.670 --> 00:15:31.530 align:middle line:90% don't really work that well. 00:15:31.530 --> 00:15:39.130 align:middle line:84% And part of the problem was they appeared 00:15:39.130 --> 00:15:42.200 align:middle line:84% to work in some classes, but not others. 00:15:42.200 --> 00:15:45.190 align:middle line:84% And we couldn't make any sense of why. 00:15:45.190 --> 00:15:47.920 align:middle line:84% Part of that is that it's hard for us to know a whole lot-- 00:15:47.920 --> 00:15:49.750 align:middle line:84% if we did it in about 250 classes. 00:15:49.750 --> 00:15:51.250 align:middle line:84% And it's hard to know the difference 00:15:51.250 --> 00:15:53.800 align:middle line:84% between those different kinds of things. 00:15:53.800 --> 00:15:58.660 align:middle line:84% But we listed characteristics of all the classes. 00:15:58.660 --> 00:16:01.310 align:middle line:84% And we put them into big AI models to try to figure out why. 00:16:01.310 --> 00:16:01.690 align:middle line:90% And all these other kind-- 00:16:01.690 --> 00:16:03.280 align:middle line:90% I mean, you can read the paper. 00:16:03.280 --> 00:16:05.600 align:middle line:90% But it was an enormous effort. 00:16:05.600 --> 00:16:08.710 align:middle line:84% And at the end of it, a thing that we 00:16:08.710 --> 00:16:10.210 align:middle line:84% had lots of people in there, but we 00:16:10.210 --> 00:16:13.480 align:middle line:84% weren't able to identify things that worked particularly well. 00:16:13.480 --> 00:16:16.935 align:middle line:84% And this was pretty consistent across this enterprise. 00:16:16.935 --> 00:16:18.310 align:middle line:84% In fact, to my knowledge, there's 00:16:18.310 --> 00:16:21.850 align:middle line:84% virtually no one who's doing research on Harvard or MIT-- 00:16:21.850 --> 00:16:24.610 align:middle line:84% HarvardX or MITx classes anymore. 00:16:24.610 --> 00:16:29.475 align:middle line:84% And so this was the third of these big bets. 00:16:29.475 --> 00:16:32.290 align:middle line:90% 00:16:32.290 --> 00:16:34.150 align:middle line:84% There are people like Peter Norvig 00:16:34.150 --> 00:16:38.550 align:middle line:84% and who's a woman, Daphne Koller, who formed Coursera, who 00:16:38.550 --> 00:16:40.920 align:middle line:84% thought that data-driven education would 00:16:40.920 --> 00:16:43.390 align:middle line:84% be very similar to data-driven medicine. 00:16:43.390 --> 00:16:45.870 align:middle line:84% Like, just like the Human Genome Project 00:16:45.870 --> 00:16:48.570 align:middle line:84% revolutionized a data-driven form of medicine, 00:16:48.570 --> 00:16:52.350 align:middle line:84% that Coursera, EdX could revolutionize a data driven form 00:16:52.350 --> 00:16:53.880 align:middle line:90% of the science of learning. 00:16:53.880 --> 00:16:56.530 align:middle line:84% And we knew a lot about what people clicked, 00:16:56.530 --> 00:16:58.440 align:middle line:84% but we never really got much more data 00:16:58.440 --> 00:17:01.620 align:middle line:84% about what was going on inside people's heads. 00:17:01.620 --> 00:17:03.900 align:middle line:84% The other huge issue is that if you really 00:17:03.900 --> 00:17:08.760 align:middle line:84% want to do educational research, that studies how to help people 00:17:08.760 --> 00:17:13.082 align:middle line:84% learn better, you need variation in the course that's deeper 00:17:13.082 --> 00:17:14.290 align:middle line:90% than what we were attempting. 00:17:14.290 --> 00:17:15.660 align:middle line:84% You need to be able to do more than just 00:17:15.660 --> 00:17:17.140 align:middle line:84% change a little thing at the beginning of the course. 00:17:17.140 --> 00:17:18.515 align:middle line:84% You have to be like, well, here's 00:17:18.515 --> 00:17:19.900 align:middle line:84% one good way of teaching a class. 00:17:19.900 --> 00:17:21.849 align:middle line:84% Here's another good way of teaching the class. 00:17:21.849 --> 00:17:23.619 align:middle line:90% Let's try both of them. 00:17:23.619 --> 00:17:26.220 align:middle line:84% Well, it turns out that it was very difficult to go to faculty 00:17:26.220 --> 00:17:29.160 align:middle line:84% and be like, will you make two versions of this course, 00:17:29.160 --> 00:17:32.320 align:middle line:84% so that we can test some of these different ideas in them-- 00:17:32.320 --> 00:17:34.590 align:middle line:84% because people had barely enough time and resources 00:17:34.590 --> 00:17:36.795 align:middle line:84% to make one version of the course, and so forth. 00:17:36.795 --> 00:17:39.760 align:middle line:90% 00:17:39.760 --> 00:17:42.970 align:middle line:84% So like the other two big bets of MOOCs, 00:17:42.970 --> 00:17:46.030 align:middle line:84% they, for the most part didn't pan out. 00:17:46.030 --> 00:17:49.180 align:middle line:84% And I think, if you go back 10 years from now 00:17:49.180 --> 00:17:53.647 align:middle line:84% and read the op-eds that David Brooks and Thomas Friedman were 00:17:53.647 --> 00:17:56.230 align:middle line:84% writing in The New York Times about the totally transformative 00:17:56.230 --> 00:17:59.440 align:middle line:84% impact of online education, people would be pretty 00:17:59.440 --> 00:18:01.300 align:middle line:84% surprised, 10 years later, to find out 00:18:01.300 --> 00:18:05.470 align:middle line:84% that it's good for things like executive education 00:18:05.470 --> 00:18:07.518 align:middle line:84% or helping people earn a second master's degree, 00:18:07.518 --> 00:18:08.810 align:middle line:90% which are not bad things to do. 00:18:08.810 --> 00:18:13.810 align:middle line:84% But again, they're smaller components of it. 00:18:13.810 --> 00:18:16.670 align:middle line:84% I've thought a lot over the last couple of years about, OK, 00:18:16.670 --> 00:18:20.950 align:middle line:84% so if we're not going to totally reform the educational system 00:18:20.950 --> 00:18:23.770 align:middle line:84% with independent, self-paced online learning, 00:18:23.770 --> 00:18:25.780 align:middle line:90% what else might we do? 00:18:25.780 --> 00:18:29.750 align:middle line:84% Well, so for instance, I've told you that already affluent, 00:18:29.750 --> 00:18:33.070 align:middle line:84% already educated people are reasonably likely to persist 00:18:33.070 --> 00:18:34.170 align:middle line:90% in online courses. 00:18:34.170 --> 00:18:36.420 align:middle line:84% Well, there are all kinds of already affluent, already 00:18:36.420 --> 00:18:38.410 align:middle line:84% educated people who work in the public interest. 00:18:38.410 --> 00:18:39.640 align:middle line:84% There are people who are teachers. 00:18:39.640 --> 00:18:41.020 align:middle line:84% There are people who work in public health. 00:18:41.020 --> 00:18:42.520 align:middle line:84% There are people who work in nurses. 00:18:42.520 --> 00:18:44.490 align:middle line:84% Lots of them work in contexts where 00:18:44.490 --> 00:18:46.260 align:middle line:84% they don't have a lot of learning 00:18:46.260 --> 00:18:49.630 align:middle line:84% after their college degree or their advanced degree. 00:18:49.630 --> 00:18:52.170 align:middle line:84% So what if we created online courses 00:18:52.170 --> 00:18:54.180 align:middle line:84% that a good fraction of people would be 00:18:54.180 --> 00:18:56.250 align:middle line:90% able to persist in and finish? 00:18:56.250 --> 00:18:57.610 align:middle line:90% That would be able to help them. 00:18:57.610 --> 00:18:59.820 align:middle line:84% There was one course that we built 00:18:59.820 --> 00:19:02.430 align:middle line:84% called Launching Innovation in schools, which 00:19:02.430 --> 00:19:08.730 align:middle line:84% relates to the stuff in Iterate that you 00:19:08.730 --> 00:19:10.560 align:middle line:90% read a couple classes ago. 00:19:10.560 --> 00:19:12.465 align:middle line:84% And in our course, we make all of our-- not 00:19:12.465 --> 00:19:15.090 align:middle line:84% everybody do this-- we make all of our courses openly licensed. 00:19:15.090 --> 00:19:17.500 align:middle line:84% So we make it so that it's Creative Commons. 00:19:17.500 --> 00:19:20.100 align:middle line:84% Anybody can take our things and do anything with them. 00:19:20.100 --> 00:19:21.593 align:middle line:84% There were some folks in Ukraine, 00:19:21.593 --> 00:19:23.010 align:middle line:84% before the war, who had translated 00:19:23.010 --> 00:19:24.960 align:middle line:84% our courses into Ukrainian and put them 00:19:24.960 --> 00:19:26.820 align:middle line:90% on a Ukrainian platform. 00:19:26.820 --> 00:19:28.750 align:middle line:84% And for a while, at the beginning of the war-- 00:19:28.750 --> 00:19:30.210 align:middle line:84% I don't know if this is still true-- teachers still 00:19:30.210 --> 00:19:32.950 align:middle line:84% had to get recertified, and do training, and things like that. 00:19:32.950 --> 00:19:36.840 align:middle line:84% So some enormous fraction of the Ukrainian teaching force 00:19:36.840 --> 00:19:38.880 align:middle line:84% had at least signed up and plausibly 00:19:38.880 --> 00:19:41.280 align:middle line:84% taken one of these courses and things like that. 00:19:41.280 --> 00:19:44.040 align:middle line:84% So for us, it was of a proof of concept 00:19:44.040 --> 00:19:47.700 align:middle line:84% that people can benefit from some of these kinds of things. 00:19:47.700 --> 00:19:53.640 align:middle line:84% With Mariana's advisor, José, we got really interested in what 00:19:53.640 --> 00:19:55.110 align:middle line:84% if the thing you want is not really 00:19:55.110 --> 00:19:59.370 align:middle line:84% big MOOC providers, but very small local MOOC providers. 00:19:59.370 --> 00:20:00.960 align:middle line:84% So there's an organization called 00:20:00.960 --> 00:20:05.190 align:middle line:84% Bedrock in Jordan run by the Queen Rania Foundation 00:20:05.190 --> 00:20:08.430 align:middle line:84% that makes MOOCs in Arabic for people in the Arabic world. 00:20:08.430 --> 00:20:12.230 align:middle line:84% There are lots of providers that offer courses 00:20:12.230 --> 00:20:13.980 align:middle line:84% in the local language of their environment 00:20:13.980 --> 00:20:15.980 align:middle line:84% that are tailored to the particular environments 00:20:15.980 --> 00:20:16.830 align:middle line:90% that they're in. 00:20:16.830 --> 00:20:21.810 align:middle line:84% And these tables are unreadable, except if you go in them 00:20:21.810 --> 00:20:22.650 align:middle line:90% and look at them. 00:20:22.650 --> 00:20:25.770 align:middle line:84% But there's evidence that you could actually 00:20:25.770 --> 00:20:28.560 align:middle line:84% do a better job getting people from more 00:20:28.560 --> 00:20:31.860 align:middle line:84% diverse educational backgrounds if you offer them 00:20:31.860 --> 00:20:38.022 align:middle line:84% in local languages, with local providers with the topic area 00:20:38.022 --> 00:20:40.230 align:middle line:84% and content areas that would be interesting to people 00:20:40.230 --> 00:20:41.520 align:middle line:90% in those local places. 00:20:41.520 --> 00:20:48.180 align:middle line:84% So what we needed was not like giant, global, elite-driven 00:20:48.180 --> 00:20:52.950 align:middle line:84% online learning, but maybe another pathway with lots 00:20:52.950 --> 00:20:54.840 align:middle line:84% of distributed local online learning 00:20:54.840 --> 00:20:58.530 align:middle line:84% that was particularly tailored to people's cultural interests, 00:20:58.530 --> 00:21:01.290 align:middle line:84% and labor markets, and civic spheres, and needs, 00:21:01.290 --> 00:21:02.165 align:middle line:90% and things like that. 00:21:02.165 --> 00:21:06.270 align:middle line:90% 00:21:06.270 --> 00:21:09.630 align:middle line:84% I mean, I always try to remain open to the idea 00:21:09.630 --> 00:21:12.210 align:middle line:84% that any of these technologies could 00:21:12.210 --> 00:21:15.120 align:middle line:84% have new people that come along and refresh 00:21:15.120 --> 00:21:16.680 align:middle line:90% the ideas around them. 00:21:16.680 --> 00:21:20.130 align:middle line:84% But we should probably think of those claims 00:21:20.130 --> 00:21:21.970 align:middle line:90% with a pretty high bar. 00:21:21.970 --> 00:21:23.490 align:middle line:84% If someone comes along, and says, 00:21:23.490 --> 00:21:26.250 align:middle line:84% they have a new AI-powered MOOC platform or something like 00:21:26.250 --> 00:21:26.955 align:middle line:90% that-- 00:21:26.955 --> 00:21:28.830 align:middle line:84% now, you could be like, well, let me tell you 00:21:28.830 --> 00:21:32.610 align:middle line:84% what the research suggests about the original MOOC platform, what 00:21:32.610 --> 00:21:34.968 align:middle line:84% some of the limitations were, what it accomplished, 00:21:34.968 --> 00:21:36.510 align:middle line:84% why it tackled some big problems that 00:21:36.510 --> 00:21:37.760 align:middle line:90% turned out to be really hard-- 00:21:37.760 --> 00:21:41.880 align:middle line:90% 00:21:41.880 --> 00:21:45.840 align:middle line:84% and then what is it that a new effort would be 00:21:45.840 --> 00:21:47.900 align:middle line:90% able to build on top of that. 00:21:47.900 --> 00:21:50.400 align:middle line:84% Those are some of the things I want to tell you about MOOCs. 00:21:50.400 --> 00:21:53.080 align:middle line:84% What other things struck you about them? 00:21:53.080 --> 00:21:54.610 align:middle line:90% Or, what questions did you have? 00:21:54.610 --> 00:21:57.630 align:middle line:84% Or, what have we discussed in the last few classes 00:21:57.630 --> 00:21:59.606 align:middle line:84% that fit or don't fit with your experiences? 00:21:59.606 --> 00:22:17.573 align:middle line:90% 00:22:17.573 --> 00:22:19.990 align:middle line:84% AUDIENCE: I mean, I guess just a general question is like, 00:22:19.990 --> 00:22:24.510 align:middle line:84% what is your real motivation for higher educational institutions 00:22:24.510 --> 00:22:26.700 align:middle line:84% to want MOOCs to exist, considering 00:22:26.700 --> 00:22:30.830 align:middle line:84% that their entire business is based off of them, not like-- 00:22:30.830 --> 00:22:32.140 align:middle line:90% JUSTIN REICH: It's weird right? 00:22:32.140 --> 00:22:36.530 align:middle line:84% [LAUGHS] Yeah, like people got super enthusiastic 00:22:36.530 --> 00:22:40.558 align:middle line:84% for a long time about digitizing faculty labor. 00:22:40.558 --> 00:22:42.350 align:middle line:84% So what were some of the things that people 00:22:42.350 --> 00:22:44.240 align:middle line:90% would have said, at the time? 00:22:44.240 --> 00:22:48.500 align:middle line:84% In 2013, people would have said something along the lines of-- 00:22:48.500 --> 00:22:50.640 align:middle line:84% particularly for the last 40 or 50 years, 00:22:50.640 --> 00:22:54.020 align:middle line:84% but maybe for 100-plus years, there's a class of universities 00:22:54.020 --> 00:22:57.800 align:middle line:84% that brags every year about how many students they don't serve. 00:22:57.800 --> 00:23:02.478 align:middle line:84% Around this time every year, Stanford and MIT and Harvard 00:23:02.478 --> 00:23:04.520 align:middle line:84% will publish statistics about how many applicants 00:23:04.520 --> 00:23:06.020 align:middle line:90% they rejected. 00:23:06.020 --> 00:23:08.270 align:middle line:84% But one of the things that they could do with MOOCs 00:23:08.270 --> 00:23:11.330 align:middle line:84% is start bragging about how many people they served. 00:23:11.330 --> 00:23:14.550 align:middle line:84% For the faculty, who are especially early on, 00:23:14.550 --> 00:23:17.520 align:middle line:84% who got millions of people to sign up for their courses, 00:23:17.520 --> 00:23:18.950 align:middle line:90% it was exhilarating. 00:23:18.950 --> 00:23:21.350 align:middle line:84% It was incredibly exciting to be able to say-- 00:23:21.350 --> 00:23:23.210 align:middle line:84% I mean, Naman Aggarwal used to say, 00:23:23.210 --> 00:23:29.360 align:middle line:84% if I taught 6.11 at MIT for the next 40 years, 00:23:29.360 --> 00:23:32.070 align:middle line:84% I still couldn't have as many students who showed up 00:23:32.070 --> 00:23:34.070 align:middle line:84% in the first round of this class that I offered, 00:23:34.070 --> 00:23:34.945 align:middle line:90% and things like that. 00:23:34.945 --> 00:23:36.680 align:middle line:90% So that was exciting. 00:23:36.680 --> 00:23:41.480 align:middle line:84% I have a hunch that when people were afraid 00:23:41.480 --> 00:23:44.820 align:middle line:84% that this might profoundly changed the education landscape, 00:23:44.820 --> 00:23:47.570 align:middle line:84% that elite universities wanted to be in front of it, 00:23:47.570 --> 00:23:48.765 align:middle line:90% rather than behind it. 00:23:48.765 --> 00:23:51.140 align:middle line:84% Like, as soon as there was some commercial offerings that 00:23:51.140 --> 00:23:52.640 align:middle line:84% were available, there were folks who 00:23:52.640 --> 00:23:58.377 align:middle line:84% said, man, we really ought to have a nonprofit alternative. 00:23:58.377 --> 00:24:00.710 align:middle line:84% We ought to have universities that are involved in this. 00:24:00.710 --> 00:24:03.410 align:middle line:84% I mean, another really important structural factor 00:24:03.410 --> 00:24:09.348 align:middle line:84% is that university revenue sources are constantly 00:24:09.348 --> 00:24:09.890 align:middle line:90% under threat. 00:24:09.890 --> 00:24:12.165 align:middle line:84% And their costs are constantly going up. 00:24:12.165 --> 00:24:14.040 align:middle line:84% So health care costs are constantly going up. 00:24:14.040 --> 00:24:16.550 align:middle line:84% Technology costs are constantly going up. 00:24:16.550 --> 00:24:18.470 align:middle line:84% And if you're a state university, 00:24:18.470 --> 00:24:19.970 align:middle line:84% like across the United States, we've 00:24:19.970 --> 00:24:23.480 align:middle line:84% been dramatically reducing the amount that the taxpaying 00:24:23.480 --> 00:24:25.762 align:middle line:84% public support schools and asking students 00:24:25.762 --> 00:24:28.220 align:middle line:84% to pay more tuition, and so forth-- or, a higher percentage 00:24:28.220 --> 00:24:29.840 align:middle line:90% of their tuition. 00:24:29.840 --> 00:24:32.390 align:middle line:84% And so a bunch of folks thought they could make money 00:24:32.390 --> 00:24:34.520 align:middle line:84% doing it-- not make money, in the sense of putting 00:24:34.520 --> 00:24:37.740 align:middle line:84% their pocket-- but make money doing that. 00:24:37.740 --> 00:24:41.300 align:middle line:84% So those are some of the kinds of reasons that people 00:24:41.300 --> 00:24:43.490 align:middle line:90% got really excited about it. 00:24:43.490 --> 00:24:49.295 align:middle line:84% But yeah, in retrospect, in some ways, you're like, why-- 00:24:49.295 --> 00:24:51.920 align:middle line:84% I mean, at the time, there were definitely plenty of people who 00:24:51.920 --> 00:24:54.950 align:middle line:84% were like, you want to take this thing that we sell to people 00:24:54.950 --> 00:24:57.850 align:middle line:84% for $80,000 a year and give them to them for free? 00:24:57.850 --> 00:25:00.830 align:middle line:90% Like, what kind of plan is that? 00:25:00.830 --> 00:25:04.730 align:middle line:84% Or, for 35 bucks or other kinds of things like that. 00:25:04.730 --> 00:25:06.890 align:middle line:84% AUDIENCE: One thing that I was thinking about, 00:25:06.890 --> 00:25:11.360 align:middle line:84% with respect to presenting books with like helping 00:25:11.360 --> 00:25:13.310 align:middle line:84% underserved communities and stuff 00:25:13.310 --> 00:25:18.210 align:middle line:90% is like the issue of space. 00:25:18.210 --> 00:25:21.890 align:middle line:84% Because to me, I feel like having 00:25:21.890 --> 00:25:24.300 align:middle line:84% been in like a pretty underserved high school 00:25:24.300 --> 00:25:24.800 align:middle line:90% and stuff-- 00:25:24.800 --> 00:25:27.620 align:middle line:84% I was like, OK, well, even like in a position of higher 00:25:27.620 --> 00:25:32.720 align:middle line:84% education, space is a very big issue for a lot of these spaces. 00:25:32.720 --> 00:25:35.490 align:middle line:84% Where it's like, you simply can't just sit at home. 00:25:35.490 --> 00:25:38.340 align:middle line:84% And even if you have access to a laptop or something to do that. 00:25:38.340 --> 00:25:41.000 align:middle line:84% And it was like, I just wanted to know if-- 00:25:41.000 --> 00:25:43.730 align:middle line:84% I mean, this might kind of also just 00:25:43.730 --> 00:25:45.990 align:middle line:84% connect to the larger social aspect of it-- 00:25:45.990 --> 00:25:50.360 align:middle line:84% but I was wondering if this even came up 00:25:50.360 --> 00:25:54.380 align:middle line:84% when they were talking about the accessibility of these systems. 00:25:54.380 --> 00:25:57.600 align:middle line:84% JUSTIN REICH: I mean, I used to joke that something like anyone, 00:25:57.600 --> 00:26:01.520 align:middle line:84% even the poorest peasant in the most remote African village, who 00:26:01.520 --> 00:26:04.070 align:middle line:84% has high-speed internet and a basic understanding of matrix 00:26:04.070 --> 00:26:06.860 align:middle line:84% algebra, can take the Introduction Aeronautics class. 00:26:06.860 --> 00:26:08.360 align:middle line:84% And I was like, yeah, there actually 00:26:08.360 --> 00:26:11.830 align:middle line:84% are not that many peasants, who meet both of those criteria. 00:26:11.830 --> 00:26:12.620 align:middle line:90% [LAUGHTER] 00:26:12.620 --> 00:26:16.800 align:middle line:90% But it was always the case. 00:26:16.800 --> 00:26:19.503 align:middle line:84% There was a very famous case of-- there was the article 00:26:19.503 --> 00:26:21.920 align:middle line:84% in The New York Times about him was called "The Boy Genius 00:26:21.920 --> 00:26:23.490 align:middle line:90% of Ulan Bator." 00:26:23.490 --> 00:26:26.100 align:middle line:84% So there was a student who was in Mongolia, 00:26:26.100 --> 00:26:29.100 align:middle line:84% who got a perfect score on the first MITx class 00:26:29.100 --> 00:26:30.900 align:middle line:90% and was admitted to MIT. 00:26:30.900 --> 00:26:34.770 align:middle line:84% And this was taken as proof that, like anybody, anywhere 00:26:34.770 --> 00:26:37.270 align:middle line:84% in the world can get access to our courses. 00:26:37.270 --> 00:26:39.788 align:middle line:84% And we're going to be able to find the best high school 00:26:39.788 --> 00:26:41.080 align:middle line:90% students and bring them to MIT. 00:26:41.080 --> 00:26:43.300 align:middle line:84% Well, there were some more details to the story. 00:26:43.300 --> 00:26:45.930 align:middle line:84% One is, the principal of that school in Mongolia 00:26:45.930 --> 00:26:47.770 align:middle line:84% is, as I understood, at the time, 00:26:47.770 --> 00:26:50.363 align:middle line:84% the only person from Mongolia to have graduated from MIT. 00:26:50.363 --> 00:26:52.030 align:middle line:84% So there was kind of a connection there. 00:26:52.030 --> 00:26:55.440 align:middle line:84% Second, there was an advanced physics class 00:26:55.440 --> 00:26:57.360 align:middle line:84% that was offered that used the material. 00:26:57.360 --> 00:26:58.860 align:middle line:84% But there was a postdoc, who I think 00:26:58.860 --> 00:27:00.600 align:middle line:84% was at Stanford or some elite school 00:27:00.600 --> 00:27:04.260 align:middle line:84% in California, who went and studied with these kids 00:27:04.260 --> 00:27:05.940 align:middle line:90% for a bunch of months. 00:27:05.940 --> 00:27:07.900 align:middle line:84% So one thing you could say is like, oh, 00:27:07.900 --> 00:27:10.090 align:middle line:84% we just need to get MOOCs into every school. 00:27:10.090 --> 00:27:11.770 align:middle line:84% And then we'll help people find their way to MIT. 00:27:11.770 --> 00:27:12.780 align:middle line:84% The other thing you can say is like, we just 00:27:12.780 --> 00:27:15.010 align:middle line:84% need MIT grads being the principal of every school. 00:27:15.010 --> 00:27:15.510 align:middle line:90% [LAUGHTER] 00:27:15.510 --> 00:27:17.290 align:middle line:84% And then we can get people coming to MIT. 00:27:17.290 --> 00:27:18.520 align:middle line:84% But this kid, his name was Battushig. 00:27:18.520 --> 00:27:20.530 align:middle line:84% And he did a bunch of education classes at MIT. 00:27:20.530 --> 00:27:21.920 align:middle line:90% He was a remarkable kid. 00:27:21.920 --> 00:27:23.960 align:middle line:84% He used to watch two videos at the same time 00:27:23.960 --> 00:27:26.872 align:middle line:90% when he was doing these classes. 00:27:26.872 --> 00:27:29.330 align:middle line:84% It doesn't make any sense, but there's a great news article 00:27:29.330 --> 00:27:31.280 align:middle line:84% that shows his little setup of him watching 00:27:31.280 --> 00:27:32.460 align:middle line:90% two videos at the same time. 00:27:32.460 --> 00:27:34.460 align:middle line:84% I mean, he's clearly a smart guy. 00:27:34.460 --> 00:27:39.050 align:middle line:84% But it was really important to the narrative of MOOCs 00:27:39.050 --> 00:27:40.650 align:middle line:90% that, look-- 00:27:40.650 --> 00:27:42.590 align:middle line:84% look at, look at how far into the periphery 00:27:42.590 --> 00:27:45.620 align:middle line:84% that we've reached to bring people into the core. 00:27:45.620 --> 00:27:49.100 align:middle line:84% And so folks like me, who are writing papers, are like, 00:27:49.100 --> 00:27:50.400 align:middle line:90% that is neat. 00:27:50.400 --> 00:27:52.040 align:middle line:90% That is a true story. 00:27:52.040 --> 00:27:53.990 align:middle line:84% Battushig is an awesome kid that is not 00:27:53.990 --> 00:27:59.750 align:middle line:84% representative of the broader system that we've built here. 00:27:59.750 --> 00:28:04.440 align:middle line:90% And yeah, it was interesting. 00:28:04.440 --> 00:28:06.500 align:middle line:84% People really wanted these things 00:28:06.500 --> 00:28:08.720 align:middle line:90% to be democratizing education. 00:28:08.720 --> 00:28:10.250 align:middle line:84% But then, when it was pretty obvious 00:28:10.250 --> 00:28:12.660 align:middle line:84% they weren't, they were still like, yeah, well, OK. 00:28:12.660 --> 00:28:14.510 align:middle line:90% We'll just use them anyway. 00:28:14.510 --> 00:28:15.590 align:middle line:90% I know. 00:28:15.590 --> 00:28:17.880 align:middle line:90% I helped support-- 00:28:17.880 --> 00:28:21.270 align:middle line:84% I did an evaluation of the Supply Chain Management program, 00:28:21.270 --> 00:28:24.050 align:middle line:84% which was one of the first programs that said, 00:28:24.050 --> 00:28:26.280 align:middle line:90% do half the program online. 00:28:26.280 --> 00:28:30.380 align:middle line:84% So you have to take and pass proctored tests in four or five 00:28:30.380 --> 00:28:31.500 align:middle line:90% online classes. 00:28:31.500 --> 00:28:33.535 align:middle line:84% And then we'll give you half a degree credit. 00:28:33.535 --> 00:28:34.910 align:middle line:84% And you come to campus and finish 00:28:34.910 --> 00:28:36.770 align:middle line:90% the other half of the degree. 00:28:36.770 --> 00:28:41.190 align:middle line:84% And there are a lot of good things about it, actually. 00:28:41.190 --> 00:28:42.920 align:middle line:84% So one is supply chain management 00:28:42.920 --> 00:28:44.310 align:middle line:90% is a relatively new field. 00:28:44.310 --> 00:28:45.500 align:middle line:84% And so there aren't a lot of universities 00:28:45.500 --> 00:28:46.710 align:middle line:90% around the world that offer it. 00:28:46.710 --> 00:28:47.918 align:middle line:90% And MIT is pretty good at it. 00:28:47.918 --> 00:28:51.290 align:middle line:84% So this was a good way of having more people be able to do it. 00:28:51.290 --> 00:28:52.625 align:middle line:90% It's a relatively small program. 00:28:52.625 --> 00:28:54.000 align:middle line:84% They don't have a ton of faculty. 00:28:54.000 --> 00:28:56.593 align:middle line:84% So in theory, you could have twice as many people 00:28:56.593 --> 00:28:58.760 align:middle line:84% come through campus if they do half the stuff online 00:28:58.760 --> 00:28:59.880 align:middle line:90% and half the stuff here. 00:28:59.880 --> 00:29:02.005 align:middle line:84% And there were, actually, some really cool findings 00:29:02.005 --> 00:29:03.920 align:middle line:90% where the folks who-- 00:29:03.920 --> 00:29:06.980 align:middle line:84% the way they did it is they kept an in-person cohort 00:29:06.980 --> 00:29:12.500 align:middle line:84% and added an online cohort that showed up with in-person cohort 00:29:12.500 --> 00:29:13.910 align:middle line:90% in January. 00:29:13.910 --> 00:29:17.960 align:middle line:84% And we did some surveys about how both groups felt, in terms 00:29:17.960 --> 00:29:19.163 align:middle line:90% of their connection to MIT. 00:29:19.163 --> 00:29:21.330 align:middle line:84% And they both felt really strongly connected to MIT. 00:29:21.330 --> 00:29:24.210 align:middle line:84% Actually, in sort of funny ways, like on campus people were like, 00:29:24.210 --> 00:29:25.260 align:middle line:90% we've been here all year. 00:29:25.260 --> 00:29:26.260 align:middle line:90% Like, this is our place. 00:29:26.260 --> 00:29:28.927 align:middle line:84% And the online people were like, we've been taking these classes 00:29:28.927 --> 00:29:30.180 align:middle line:90% together for three years. 00:29:30.180 --> 00:29:31.350 align:middle line:90% Like, this is our place. 00:29:31.350 --> 00:29:32.767 align:middle line:84% So they both felt really connected 00:29:32.767 --> 00:29:34.010 align:middle line:90% in interesting kind of ways. 00:29:34.010 --> 00:29:36.650 align:middle line:84% The grades of the online folks were actually slightly better 00:29:36.650 --> 00:29:37.980 align:middle line:90% at the end of the first cohort. 00:29:37.980 --> 00:29:41.040 align:middle line:84% Although, who knows if that would have persisted into time. 00:29:41.040 --> 00:29:43.970 align:middle line:84% But the only people who persisted 00:29:43.970 --> 00:29:47.000 align:middle line:84% through the online courses were people who already 00:29:47.000 --> 00:29:48.500 align:middle line:84% had the jobs in logistics industries 00:29:48.500 --> 00:29:50.833 align:middle line:84% were getting their second degree, were already affluent. 00:29:50.833 --> 00:29:52.280 align:middle line:84% They were disproportionately guys. 00:29:52.280 --> 00:29:55.700 align:middle line:84% There's a whole line of research that 00:29:55.700 --> 00:29:59.570 align:middle line:84% really focuses on people's ability 00:29:59.570 --> 00:30:01.170 align:middle line:84% to conduct self-regulated learning, 00:30:01.170 --> 00:30:03.980 align:middle line:84% people's ability to persist through difficulty, 00:30:03.980 --> 00:30:05.183 align:middle line:90% and things like that. 00:30:05.183 --> 00:30:06.350 align:middle line:90% And so you talk to learners. 00:30:06.350 --> 00:30:09.230 align:middle line:84% And they would talk about how they took their lunch breaks 00:30:09.230 --> 00:30:11.540 align:middle line:84% and they studied, how they took weekends and studied, 00:30:11.540 --> 00:30:13.623 align:middle line:84% how they stayed up late at night and they studied. 00:30:13.623 --> 00:30:15.740 align:middle line:84% And especially if you think about it 00:30:15.740 --> 00:30:18.570 align:middle line:84% through an individual psychological perspective, 00:30:18.570 --> 00:30:21.380 align:middle line:84% you're like, yeah, those are all great qualities. 00:30:21.380 --> 00:30:23.130 align:middle line:84% Another thing we realized, though, 00:30:23.130 --> 00:30:27.150 align:middle line:84% is like if you're studying in the evenings, 00:30:27.150 --> 00:30:30.032 align:middle line:84% you're probably shirking family responsibilities. 00:30:30.032 --> 00:30:31.740 align:middle line:84% And in lots of cultures across the world, 00:30:31.740 --> 00:30:34.250 align:middle line:84% it is way easier for guys to shirk family responsibilities 00:30:34.250 --> 00:30:36.500 align:middle line:84% than it is for women to shirk family responsibilities. 00:30:36.500 --> 00:30:38.390 align:middle line:84% So in some ways, self-regulated learning 00:30:38.390 --> 00:30:40.830 align:middle line:84% was the same thing as shirking family responsibility. 00:30:40.830 --> 00:30:43.100 align:middle line:84% Like a thing that you'd study from a psychological perspective 00:30:43.100 --> 00:30:45.225 align:middle line:84% is self-regulated learning would be shirking family 00:30:45.225 --> 00:30:47.870 align:middle line:84% responsibilities, from a sociological point of view. 00:30:47.870 --> 00:30:52.130 align:middle line:84% And so, again, just giving us more ways to think about, as we 00:30:52.130 --> 00:30:55.088 align:middle line:84% create learning environments, who will benefit from them? 00:30:55.088 --> 00:30:57.380 align:middle line:84% What are the enabling conditions to help lots of people 00:30:57.380 --> 00:30:59.420 align:middle line:84% from lots of different kinds of backgrounds and places 00:30:59.420 --> 00:31:01.087 align:middle line:84% in society be able to benefit from them? 00:31:01.087 --> 00:31:03.240 align:middle line:90% Things like that. 00:31:03.240 --> 00:31:05.240 align:middle line:84% AUDIENCE: I think something that stood out to me 00:31:05.240 --> 00:31:07.260 align:middle line:84% was the business model of the MOOCs. 00:31:07.260 --> 00:31:09.590 align:middle line:84% I think I don't really understand it 00:31:09.590 --> 00:31:11.390 align:middle line:84% because, I think, for me, as someone 00:31:11.390 --> 00:31:14.355 align:middle line:84% who went to international school in Africa-- and I would say, 00:31:14.355 --> 00:31:17.450 align:middle line:84% I was what, maybe like top five, like in my country or something. 00:31:17.450 --> 00:31:19.280 align:middle line:90% And pretty big, like in Africa. 00:31:19.280 --> 00:31:22.070 align:middle line:84% Like I'd never heard of any MIT or Harvard course, 00:31:22.070 --> 00:31:22.750 align:middle line:90% like in my life. 00:31:22.750 --> 00:31:25.250 align:middle line:84% I'd probably say, none of my teachers have ever heard of it. 00:31:25.250 --> 00:31:29.750 align:middle line:84% And I think it made me wonder, who exactly they're targeting. 00:31:29.750 --> 00:31:31.480 align:middle line:84% Because I don't think it's necessarily 00:31:31.480 --> 00:31:34.790 align:middle line:84% young high schoolers, like trying to get into college. 00:31:34.790 --> 00:31:37.160 align:middle line:84% Because what I did realize when I came to MIT, 00:31:37.160 --> 00:31:39.370 align:middle line:84% and when I Googled like how to do linear algebra, 00:31:39.370 --> 00:31:41.590 align:middle line:84% or all these things is then I would 00:31:41.590 --> 00:31:44.050 align:middle line:84% see MIT courses, and MIT OpenCourseWare, 00:31:44.050 --> 00:31:46.210 align:middle line:84% and all these things pop up on YouTube. 00:31:46.210 --> 00:31:47.710 align:middle line:84% But then, back in high school, and I 00:31:47.710 --> 00:31:49.100 align:middle line:84% didn't know what linear algebra is-- 00:31:49.100 --> 00:31:50.480 align:middle line:84% JUSTIN REICH: If you don't know what linear algebra was, 00:31:50.480 --> 00:31:51.913 align:middle line:84% how are you going to search for a linear algebra course--- 00:31:51.913 --> 00:31:52.160 align:middle line:84% AUDIENCE: Yeah, it was something that came to my mind-- 00:31:52.160 --> 00:31:52.640 align:middle line:90% JUSTIN REICH: Yeah, yeah. 00:31:52.640 --> 00:31:54.290 align:middle line:90% No, that's a terrific example. 00:31:54.290 --> 00:31:56.030 align:middle line:90% I mean, the business model-- 00:31:56.030 --> 00:31:57.760 align:middle line:84% I mean Daphne Koller, and I think, 00:31:57.760 --> 00:32:00.070 align:middle line:84% in one of her first talks, refers to it as a blue ocean 00:32:00.070 --> 00:32:01.810 align:middle line:84% business model, which is, basically, 00:32:01.810 --> 00:32:05.507 align:middle line:84% the potential addressable market is like six billion people 00:32:05.507 --> 00:32:06.590 align:middle line:90% in the world or something. 00:32:06.590 --> 00:32:09.040 align:middle line:84% Anybody who's got an internet connection, 00:32:09.040 --> 00:32:12.730 align:middle line:84% if you can sell $35 certificates a couple of times 00:32:12.730 --> 00:32:14.480 align:middle line:84% to six billion people over and over again, 00:32:14.480 --> 00:32:18.142 align:middle line:84% you can make a lot of money doing that. 00:32:18.142 --> 00:32:21.700 align:middle line:84% And there were efforts to try to create 00:32:21.700 --> 00:32:23.030 align:middle line:90% institutional connections. 00:32:23.030 --> 00:32:25.850 align:middle line:84% So CS50, who's Harvard's computer science class, 00:32:25.850 --> 00:32:29.210 align:middle line:84% has a whole program to help high schools 00:32:29.210 --> 00:32:31.630 align:middle line:84% to adopt the online CS50 materials 00:32:31.630 --> 00:32:33.880 align:middle line:90% in their in-person instruction. 00:32:33.880 --> 00:32:37.263 align:middle line:84% So there's some fraction that you, hanging out in Africa, 00:32:37.263 --> 00:32:38.680 align:middle line:84% would have encountered this course 00:32:38.680 --> 00:32:40.840 align:middle line:90% and just decided to take it. 00:32:40.840 --> 00:32:43.090 align:middle line:84% There's some sense that your school 00:32:43.090 --> 00:32:45.342 align:middle line:84% might have made-- connected with one 00:32:45.342 --> 00:32:46.550 align:middle line:90% of these formal institutions. 00:32:46.550 --> 00:32:49.150 align:middle line:84% There was another great story of the guy who founded 00:32:49.150 --> 00:32:52.870 align:middle line:90% stripe was from St. Louis. 00:32:52.870 --> 00:32:55.550 align:middle line:84% And was like, I want to live in St. Louis. 00:32:55.550 --> 00:32:57.500 align:middle line:84% And I can't hire any developers in St. Louis. 00:32:57.500 --> 00:32:58.280 align:middle line:90% There aren't enough developers. 00:32:58.280 --> 00:32:59.090 align:middle line:90% I'm pretty sure it was St Louis. 00:32:59.090 --> 00:33:00.257 align:middle line:90% I might have the town wrong. 00:33:00.257 --> 00:33:04.240 align:middle line:84% So when MOOCs came out, he was like, oh, 00:33:04.240 --> 00:33:06.770 align:middle line:90% let's offer CS50 in St. Louis. 00:33:06.770 --> 00:33:09.020 align:middle line:84% And we'll start like building infrastructure for this. 00:33:09.020 --> 00:33:11.380 align:middle line:84% And they started some pilot program. 00:33:11.380 --> 00:33:14.140 align:middle line:84% And it completely filled the library. 00:33:14.140 --> 00:33:16.540 align:middle line:84% There's hundreds of people, who with a little additional 00:33:16.540 --> 00:33:18.790 align:middle line:84% support-- so they ended up they ended up like creating 00:33:18.790 --> 00:33:20.120 align:middle line:90% a community space for it. 00:33:20.120 --> 00:33:21.800 align:middle line:90% So you use the online materials. 00:33:21.800 --> 00:33:23.710 align:middle line:84% That was-- I think there were models 00:33:23.710 --> 00:33:26.980 align:middle line:84% like that to try to bridge the gap that you're describing. 00:33:26.980 --> 00:33:31.270 align:middle line:84% That it's not realistic to just have kids 00:33:31.270 --> 00:33:32.740 align:middle line:84% encounter these classes and decided 00:33:32.740 --> 00:33:34.740 align:middle line:84% to get excited about them, and things like that. 00:33:34.740 --> 00:33:38.680 align:middle line:84% Still, almost all of those intermediate bridging layers 00:33:38.680 --> 00:33:41.000 align:middle line:90% require human capital. 00:33:41.000 --> 00:33:42.590 align:middle line:90% And humans are super expensive. 00:33:42.590 --> 00:33:47.590 align:middle line:84% And then you erase the kind of financial advantages 00:33:47.590 --> 00:33:49.640 align:middle line:84% that the MOOCs were promoting in the beginning. 00:33:49.640 --> 00:33:51.730 align:middle line:84% Lots of online programs have found this. 00:33:51.730 --> 00:33:54.190 align:middle line:84% It's pretty cheap to offer online classes, 00:33:54.190 --> 00:33:56.450 align:middle line:84% but a lot of times people just drop out of them. 00:33:56.450 --> 00:33:58.158 align:middle line:84% If you want them to not drop out of them, 00:33:58.158 --> 00:33:59.860 align:middle line:84% you have to hire coaches, and advisors, 00:33:59.860 --> 00:34:02.210 align:middle line:84% and assistant instructors, and things like that. 00:34:02.210 --> 00:34:04.720 align:middle line:84% And then you've just-- you brought back all the costs 00:34:04.720 --> 00:34:06.922 align:middle line:84% that you got rid of from teaching. 00:34:06.922 --> 00:34:08.380 align:middle line:84% AUDIENCE: Is there like a statistic 00:34:08.380 --> 00:34:11.800 align:middle line:84% on how often state universities create MOOCs 00:34:11.800 --> 00:34:16.489 align:middle line:90% versus these Ivy-esque schools? 00:34:16.489 --> 00:34:19.690 align:middle line:84% Because the thought occurs to me that maybe the reason why 00:34:19.690 --> 00:34:22.870 align:middle line:84% it doesn't matter if Harvard or MIT or Stanford puts out 00:34:22.870 --> 00:34:24.730 align:middle line:84% a MOOC is because you're not coming 00:34:24.730 --> 00:34:27.328 align:middle line:84% to Harvard or MIT or Stanford just for the information. 00:34:27.328 --> 00:34:28.120 align:middle line:90% JUSTIN REICH: Yeah. 00:34:28.120 --> 00:34:29.620 align:middle line:84% AUDIENCE: Because, obviously, what's 00:34:29.620 --> 00:34:33.820 align:middle line:84% associated with that is kind of like this title and this class 00:34:33.820 --> 00:34:37.219 align:middle line:84% stratification thing that isn't accessible through a MOOC. 00:34:37.219 --> 00:34:40.179 align:middle line:84% So it doesn't matter if the entire world has access 00:34:40.179 --> 00:34:43.179 align:middle line:84% to a course that's meant for learning-- because part of what 00:34:43.179 --> 00:34:45.760 align:middle line:84% you pay for and part of the real target 00:34:45.760 --> 00:34:47.830 align:middle line:84% audience of these universities is people 00:34:47.830 --> 00:34:49.389 align:middle line:90% who want the title associated. 00:34:49.389 --> 00:34:51.639 align:middle line:84% JUSTIN REICH: Well, there was an economist at Stanford 00:34:51.639 --> 00:34:53.530 align:middle line:84% who wrote a cool paper that says, 00:34:53.530 --> 00:34:55.989 align:middle line:84% the way you have to understand elite education funding 00:34:55.989 --> 00:35:00.610 align:middle line:84% models is that they roughly offer their education for free, 00:35:00.610 --> 00:35:04.270 align:middle line:84% in the hopes of having a very small percentage of people 00:35:04.270 --> 00:35:07.480 align:middle line:84% pay very large amounts of money for their education later. 00:35:07.480 --> 00:35:13.485 align:middle line:84% Basically, we subsidize people's tuition in all kinds of ways. 00:35:13.485 --> 00:35:15.110 align:middle line:84% Even if your tuition feels really high, 00:35:15.110 --> 00:35:17.920 align:middle line:84% your tuition is subsidized far beyond what it actually 00:35:17.920 --> 00:35:19.040 align:middle line:90% costs to educate you. 00:35:19.040 --> 00:35:21.820 align:middle line:84% And then, hopefully, 1% of you will give tens of millions 00:35:21.820 --> 00:35:23.900 align:middle line:90% of dollars back to MIT later. 00:35:23.900 --> 00:35:28.210 align:middle line:84% And so this person was trying to think about like given that's 00:35:28.210 --> 00:35:31.870 align:middle line:84% actually how elite universities are funded, 00:35:31.870 --> 00:35:34.610 align:middle line:84% what effect will MOOCs have on those kinds of relationships? 00:35:34.610 --> 00:35:37.960 align:middle line:84% But there were plenty of public institutions. 00:35:37.960 --> 00:35:40.210 align:middle line:84% One reason that public institutions got really excited 00:35:40.210 --> 00:35:42.040 align:middle line:84% about it is their job is to serve 00:35:42.040 --> 00:35:44.020 align:middle line:84% as many people in the state as possible. 00:35:44.020 --> 00:35:47.230 align:middle line:84% And if you're in a geographically large state, 00:35:47.230 --> 00:35:49.630 align:middle line:84% like Virginia, and you're the University of Virginia 00:35:49.630 --> 00:35:52.005 align:middle line:84% in the middle of it, like it's pretty hard to help people 00:35:52.005 --> 00:35:54.940 align:middle line:84% in the Tidewater, and in the southwest area, 00:35:54.940 --> 00:35:57.580 align:middle line:84% and in the DC area, and other things like that. 00:35:57.580 --> 00:36:00.020 align:middle line:84% So Michigan made big investments. 00:36:00.020 --> 00:36:01.570 align:middle line:90% Texas made big investments. 00:36:01.570 --> 00:36:05.710 align:middle line:84% There were lots of flagship public schools. 00:36:05.710 --> 00:36:08.352 align:middle line:84% Probably the most distinctive one, 00:36:08.352 --> 00:36:10.810 align:middle line:84% maybe not in MOOCs, but just in online learning, generally, 00:36:10.810 --> 00:36:13.300 align:middle line:84% is Arizona State University, which has really 00:36:13.300 --> 00:36:15.310 align:middle line:84% transformed itself through online learning 00:36:15.310 --> 00:36:18.010 align:middle line:84% into a global institution-- which they really 00:36:18.010 --> 00:36:19.210 align:middle line:90% see as mission-driven. 00:36:19.210 --> 00:36:22.378 align:middle line:84% Like how do we just have as many people as possible in Arizona 00:36:22.378 --> 00:36:24.670 align:middle line:84% beyond Arizona get access to the learning opportunities 00:36:24.670 --> 00:36:25.870 align:middle line:90% we create? 00:36:25.870 --> 00:36:29.060 align:middle line:84% And there's something very noble that I find about that. 00:36:29.060 --> 00:36:31.870 align:middle line:84% AUDIENCE: I wanted to follow up on your first question 00:36:31.870 --> 00:36:32.960 align:middle line:90% about the business model. 00:36:32.960 --> 00:36:36.850 align:middle line:84% How have for-profit universities reacted to MOOCs, 00:36:36.850 --> 00:36:40.480 align:middle line:84% since their entire model is trying 00:36:40.480 --> 00:36:43.030 align:middle line:84% to serve as many underserved people as possible 00:36:43.030 --> 00:36:45.230 align:middle line:84% and just like charging them a lot? 00:36:45.230 --> 00:36:48.560 align:middle line:84% And the fact is that a lot of them are also online on the-- 00:36:48.560 --> 00:36:49.510 align:middle line:90% JUSTIN REICH: Yeah. 00:36:49.510 --> 00:36:50.230 align:middle line:90% Yeah. 00:36:50.230 --> 00:36:52.313 align:middle line:84% AUDIENCE: Like the University of Phoenix, I think. 00:36:52.313 --> 00:36:55.930 align:middle line:84% JUSTIN REICH: Yeah, University of Phoenix, Kaplan, Southern New 00:36:55.930 --> 00:36:58.580 align:middle line:84% Hampshire University, Western Governors University. 00:36:58.580 --> 00:37:01.390 align:middle line:84% I mean, I think there are some Southern New Hampshire 00:37:01.390 --> 00:37:04.690 align:middle line:84% University would have really welcomed-- 00:37:04.690 --> 00:37:07.940 align:middle line:84% In some way, some of their reactions were like, hey team, 00:37:07.940 --> 00:37:08.930 align:middle line:90% this is not new. 00:37:08.930 --> 00:37:11.240 align:middle line:84% This is exactly what we've been doing for decades. 00:37:11.240 --> 00:37:13.810 align:middle line:84% So a weird thing was that there was a status 00:37:13.810 --> 00:37:14.980 align:middle line:90% flip in online learning. 00:37:14.980 --> 00:37:19.517 align:middle line:84% In 2010, online learning was a very low-status activity. 00:37:19.517 --> 00:37:21.100 align:middle line:84% It would have been primarily conducted 00:37:21.100 --> 00:37:22.900 align:middle line:84% at community colleges, for-profit colleges, state 00:37:22.900 --> 00:37:23.540 align:middle line:90% colleges. 00:37:23.540 --> 00:37:27.550 align:middle line:84% And overnight, instantly, Stanford, MIT, and Harvard 00:37:27.550 --> 00:37:29.260 align:middle line:90% made it an elite activity. 00:37:29.260 --> 00:37:31.390 align:middle line:84% They instantly-- there's actually 00:37:31.390 --> 00:37:33.400 align:middle line:84% all kinds of sociological theory about this, 00:37:33.400 --> 00:37:37.330 align:middle line:84% where the middle class people have 00:37:37.330 --> 00:37:39.720 align:middle line:84% to avoid the activities of lower class people 00:37:39.720 --> 00:37:41.720 align:middle line:84% and pursue the activities of upper class people. 00:37:41.720 --> 00:37:42.970 align:middle line:84% And lower class people pursue the activity 00:37:42.970 --> 00:37:43.730 align:middle line:90% of upper class people. 00:37:43.730 --> 00:37:45.188 align:middle line:84% But upper class people can actually 00:37:45.188 --> 00:37:47.140 align:middle line:84% pursue the activities of lower class people 00:37:47.140 --> 00:37:48.440 align:middle line:90% because their status is safe. 00:37:48.440 --> 00:37:49.590 align:middle line:84% Like, every once in a while, there 00:37:49.590 --> 00:37:51.160 align:middle line:84% are trends come along of people who 00:37:51.160 --> 00:37:59.370 align:middle line:84% dress like trash or other kinds of adopting lower class 00:37:59.370 --> 00:37:59.870 align:middle line:90% activities. 00:37:59.870 --> 00:38:01.328 align:middle line:84% So there's some sociological theory 00:38:01.328 --> 00:38:03.910 align:middle line:90% that is behind some of that. 00:38:03.910 --> 00:38:06.370 align:middle line:84% But it was one of the things that 00:38:06.370 --> 00:38:08.350 align:middle line:90% was striking about that moment. 00:38:08.350 --> 00:38:10.850 align:middle line:84% So a bunch of these universities were like, give me a break. 00:38:10.850 --> 00:38:11.570 align:middle line:90% We've been doing this forever. 00:38:11.570 --> 00:38:13.248 align:middle line:84% I gave a talk at Oakland University, 00:38:13.248 --> 00:38:15.290 align:middle line:84% which is kind of a community college in Michigan. 00:38:15.290 --> 00:38:20.050 align:middle line:84% And they were like, they'd be like, why are you using video? 00:38:20.050 --> 00:38:22.077 align:middle line:90% Like text is so much better. 00:38:22.077 --> 00:38:23.410 align:middle line:90% We've been doing this for years. 00:38:23.410 --> 00:38:25.700 align:middle line:84% Like, lots of people can't download the video. 00:38:25.700 --> 00:38:27.290 align:middle line:90% Like bandwidth is not that good. 00:38:27.290 --> 00:38:30.040 align:middle line:84% And there's not really a benefit to watching someone 00:38:30.040 --> 00:38:32.290 align:middle line:84% say something versus reading what they would say. 00:38:32.290 --> 00:38:33.530 align:middle line:90% Like, what are you all doing? 00:38:33.530 --> 00:38:35.280 align:middle line:84% And be like, you might have a point there. 00:38:35.280 --> 00:38:36.040 align:middle line:90% [LAUGHTER] 00:38:36.040 --> 00:38:38.230 align:middle line:84% Because they were doing something exciting. 00:38:38.230 --> 00:38:41.200 align:middle line:84% But on the other hand, if you're really fired up 00:38:41.200 --> 00:38:44.122 align:middle line:84% about online learning, and Harvard, and MIT, and Stanford, 00:38:44.122 --> 00:38:45.830 align:middle line:84% and Johns Hopkins, and Michigan are like, 00:38:45.830 --> 00:38:47.980 align:middle line:84% no, online learning is cool now-- 00:38:47.980 --> 00:38:50.530 align:middle line:84% there's ways that you can use that to raise 00:38:50.530 --> 00:38:52.420 align:middle line:84% your status, to legitimize your activities, 00:38:52.420 --> 00:38:53.630 align:middle line:90% and other things like that. 00:38:53.630 --> 00:38:54.645 align:middle line:90% So existing players. 00:38:54.645 --> 00:38:57.670 align:middle line:90% 00:38:57.670 --> 00:39:02.710 align:middle line:84% And they weren't really pursuing similar kinds of markets, 00:39:02.710 --> 00:39:03.830 align:middle line:90% for the most part. 00:39:03.830 --> 00:39:06.340 align:middle line:84% I mean, there are a handful of MOOC efforts 00:39:06.340 --> 00:39:09.440 align:middle line:84% that tried to help people earn bachelor's degrees. 00:39:09.440 --> 00:39:12.100 align:middle line:84% And they were mostly totally unsuccessful. 00:39:12.100 --> 00:39:14.620 align:middle line:84% And there weren't even really that many of them that were 00:39:14.620 --> 00:39:17.230 align:middle line:84% tried, which that is kind of like the sweet spot of most 00:39:17.230 --> 00:39:20.860 align:middle line:90% for-profit colleges-- 00:39:20.860 --> 00:39:23.140 align:middle line:84% which in many respects, there's a lot of evidence 00:39:23.140 --> 00:39:25.420 align:middle line:84% that they're just a disaster for-- 00:39:25.420 --> 00:39:26.840 align:middle line:90% a lot of people don't finish. 00:39:26.840 --> 00:39:28.870 align:middle line:84% People finish with a lot of debt. 00:39:28.870 --> 00:39:31.450 align:middle line:84% Tressie McMillan Cottom wrote a book called Lower Ed, where 00:39:31.450 --> 00:39:34.750 align:middle line:84% she described them as a negative social insurance program, where 00:39:34.750 --> 00:39:36.820 align:middle line:84% on average, people are worse after finding 00:39:36.820 --> 00:39:38.778 align:middle line:84% their ways into them than they were beforehand. 00:39:38.778 --> 00:39:40.950 align:middle line:84% Her book, Lower Ed is pretty fabulous. 00:39:40.950 --> 00:39:41.450 align:middle line:90% All right. 00:39:41.450 --> 00:39:42.950 align:middle line:84% Let's make sure that we get a chance 00:39:42.950 --> 00:39:46.600 align:middle line:84% to talk a little bit about part two, which is Algorithm-Guided 00:39:46.600 --> 00:39:47.690 align:middle line:90% Learning At Scale. 00:39:47.690 --> 00:39:50.560 align:middle line:84% So it sounds like a bunch of you-- 00:39:50.560 --> 00:39:53.090 align:middle line:84% we're heading into new territory for me, which is like-- 00:39:53.090 --> 00:39:57.470 align:middle line:84% well, raise your hand if in your K-12 experience, 00:39:57.470 --> 00:39:59.890 align:middle line:84% you did something that you think was algorithm-guided 00:39:59.890 --> 00:40:00.710 align:middle line:90% learning at scale. 00:40:00.710 --> 00:40:01.670 align:middle line:84% Like you sat in front of a thing-- 00:40:01.670 --> 00:40:02.490 align:middle line:90% OK. 00:40:02.490 --> 00:40:04.890 align:middle line:84% Like this, by far, the highest percentage that I've had 00:40:04.890 --> 00:40:07.200 align:middle line:90% in this class, thus far. 00:40:07.200 --> 00:40:12.305 align:middle line:84% So I don't know, five years ago, like a handful of people 00:40:12.305 --> 00:40:13.680 align:middle line:84% wouldn't have done it or had done 00:40:13.680 --> 00:40:14.883 align:middle line:90% a little bit as a curiosity. 00:40:14.883 --> 00:40:16.800 align:middle line:84% But like, clearly, if you all know what IXL is 00:40:16.800 --> 00:40:20.010 align:middle line:84% and things like that, this is becoming a much more mainstream 00:40:20.010 --> 00:40:25.080 align:middle line:90% activity in people's classes. 00:40:25.080 --> 00:40:27.112 align:middle line:90% What was your sense of them? 00:40:27.112 --> 00:40:29.320 align:middle line:84% So Dana's sense was that they were kind of pointless. 00:40:29.320 --> 00:40:31.320 align:middle line:84% And she could just have her mom do them for her. 00:40:31.320 --> 00:40:32.790 align:middle line:90% [LAUGHTER] 00:40:32.790 --> 00:40:36.670 align:middle line:84% What were other people's sentiments about them? 00:40:36.670 --> 00:40:38.310 align:middle line:90% Emily. 00:40:38.310 --> 00:40:40.300 align:middle line:84% AUDIENCE: Well, for me, like in high school, 00:40:40.300 --> 00:40:42.300 align:middle line:84% we used this thing called Moonbeam, where 00:40:42.300 --> 00:40:44.100 align:middle line:84% it was like, you would take an assessment 00:40:44.100 --> 00:40:46.600 align:middle line:84% and it would kind of like rank you like where you're reading 00:40:46.600 --> 00:40:48.070 align:middle line:90% and your vocab skills are at. 00:40:48.070 --> 00:40:52.440 align:middle line:84% And then you log in every week to do it, and learn, 00:40:52.440 --> 00:40:53.620 align:middle line:90% and progress, I guess. 00:40:53.620 --> 00:40:57.240 align:middle line:84% And I thought it was useful for us 00:40:57.240 --> 00:41:00.000 align:middle line:84% because, even though we had like regular classes 00:41:00.000 --> 00:41:03.910 align:middle line:84% or honors classes for English, like within that class, 00:41:03.910 --> 00:41:05.198 align:middle line:90% like everybody's vocab like-- 00:41:05.198 --> 00:41:06.490 align:middle line:90% JUSTIN REICH: It was different. 00:41:06.490 --> 00:41:07.573 align:middle line:90% AUDIENCE: It varied a lot. 00:41:07.573 --> 00:41:10.290 align:middle line:84% And it was very hard for the teachers 00:41:10.290 --> 00:41:12.970 align:middle line:84% to figure out what kind of vocab to teach, 00:41:12.970 --> 00:41:15.555 align:middle line:84% what materials to provide for us because everybody 00:41:15.555 --> 00:41:16.430 align:middle line:90% had different things. 00:41:16.430 --> 00:41:16.570 align:middle line:90% JUSTIN REICH: Yeah. 00:41:16.570 --> 00:41:17.340 align:middle line:84% AUDIENCE: And like a lot of times, 00:41:17.340 --> 00:41:19.180 align:middle line:84% we would just sit in class, like, I already know this word. 00:41:19.180 --> 00:41:19.650 align:middle line:84% JUSTIN REICH: Yeah, this is dumb. 00:41:19.650 --> 00:41:20.950 align:middle line:84% AUDIENCE: And like, you just stop paying attention. 00:41:20.950 --> 00:41:22.497 align:middle line:84% But then other kids are like, well, 00:41:22.497 --> 00:41:23.830 align:middle line:90% I have never heard of this word. 00:41:23.830 --> 00:41:24.790 align:middle line:90% I need to learn it. 00:41:24.790 --> 00:41:24.990 align:middle line:90% JUSTIN REICH: Yeah. 00:41:24.990 --> 00:41:25.490 align:middle line:90% Yeah. 00:41:25.490 --> 00:41:26.770 align:middle line:90% AUDIENCE: For us it was cool. 00:41:26.770 --> 00:41:30.900 align:middle line:84% But then, one of my friend, she assessed it. 00:41:30.900 --> 00:41:36.040 align:middle line:84% And she started off in the highest level ranking. 00:41:36.040 --> 00:41:39.240 align:middle line:84% And then after she finished, they just bumped her back down 00:41:39.240 --> 00:41:40.510 align:middle line:90% to level one. 00:41:40.510 --> 00:41:44.700 align:middle line:84% So then she just went back to learning like blue, 00:41:44.700 --> 00:41:47.073 align:middle line:84% green in like high school and sophomore year. 00:41:47.073 --> 00:41:47.740 align:middle line:90% So it was like-- 00:41:47.740 --> 00:41:49.450 align:middle line:84% JUSTIN REICH: That sounds like a flawed system. 00:41:49.450 --> 00:41:49.770 align:middle line:90% AUDIENCE: Yeah. 00:41:49.770 --> 00:41:51.353 align:middle line:84% JUSTIN REICH: I will tell you, though, 00:41:51.353 --> 00:41:53.250 align:middle line:84% when I was in the fourth grade, we 00:41:53.250 --> 00:41:54.870 align:middle line:90% had a similar version of that. 00:41:54.870 --> 00:41:56.620 align:middle line:84% They were called SRI reading cards. 00:41:56.620 --> 00:42:01.020 align:middle line:84% It was like a file box, and it was a bunch of short stories 00:42:01.020 --> 00:42:04.270 align:middle line:84% or nonfiction passages that were arranged in order of difficulty. 00:42:04.270 --> 00:42:07.040 align:middle line:84% And you would just go and read one and answer 00:42:07.040 --> 00:42:08.290 align:middle line:90% a bunch of questions about it. 00:42:08.290 --> 00:42:09.570 align:middle line:84% And if you got them all right, you'd 00:42:09.570 --> 00:42:11.237 align:middle line:84% go on to the next one or a couple later. 00:42:11.237 --> 00:42:12.930 align:middle line:84% And if you got them less, well, then you 00:42:12.930 --> 00:42:15.180 align:middle line:84% went back to earlier ones, or your teacher helped you, 00:42:15.180 --> 00:42:17.150 align:middle line:90% or something like that. 00:42:17.150 --> 00:42:19.650 align:middle line:84% So I mean, it's something that will come up, but this notion 00:42:19.650 --> 00:42:21.150 align:middle line:84% that like, oh, what would it look 00:42:21.150 --> 00:42:24.030 align:middle line:84% like to figure out what kinds of learning experiences people 00:42:24.030 --> 00:42:26.490 align:middle line:84% need, in that moment, and try to meet that need, instead 00:42:26.490 --> 00:42:29.280 align:middle line:84% of teaching everybody the same thing at the same time, 00:42:29.280 --> 00:42:30.910 align:middle line:84% has been compelling for a long time. 00:42:30.910 --> 00:42:32.618 align:middle line:84% It sounds like it was compelling for you, 00:42:32.618 --> 00:42:34.170 align:middle line:84% at least until the system freaked out 00:42:34.170 --> 00:42:36.700 align:middle line:84% and made you learn kindergarten words again. 00:42:36.700 --> 00:42:39.430 align:middle line:84% AUDIENCE: Yeah, we had something that like reading as well. 00:42:39.430 --> 00:42:41.040 align:middle line:84% Well, I think one of the issues was 00:42:41.040 --> 00:42:45.310 align:middle line:84% it was not nearly as consistent because I mean, 00:42:45.310 --> 00:42:47.340 align:middle line:84% it was meant to be done once a week 00:42:47.340 --> 00:42:49.990 align:middle line:84% or twice a week, like a lot of the examples we saw in class. 00:42:49.990 --> 00:42:51.900 align:middle line:84% But it'd be like, oh, like whenever 00:42:51.900 --> 00:42:55.725 align:middle line:84% the teacher didn't want to deal with us to kind of bring 00:42:55.725 --> 00:42:56.560 align:middle line:90% in the cards. 00:42:56.560 --> 00:42:57.660 align:middle line:90% It's like, OK, do this. 00:42:57.660 --> 00:42:59.190 align:middle line:90% So it was so inconsistent. 00:42:59.190 --> 00:43:02.130 align:middle line:84% That I was like, OK, well, I don't exactly 00:43:02.130 --> 00:43:04.440 align:middle line:84% know if I'm improving or I'm staying stagnant 00:43:04.440 --> 00:43:06.690 align:middle line:84% because I haven't seen this program for two weeks now. 00:43:06.690 --> 00:43:09.273 align:middle line:84% JUSTIN REICH: Good, you've said two things there that I really 00:43:09.273 --> 00:43:11.070 align:middle line:90% want you to hang on to. 00:43:11.070 --> 00:43:14.520 align:middle line:84% One is you said something which Larry Cuban would have said 00:43:14.520 --> 00:43:16.380 align:middle line:84% about television things or things 00:43:16.380 --> 00:43:18.870 align:middle line:84% like that, which is that the designers of this program 00:43:18.870 --> 00:43:21.453 align:middle line:84% have a vision, but the teachers have a different set of needs. 00:43:21.453 --> 00:43:24.037 align:middle line:84% So the teachers are like, I need to do some grading today or I 00:43:24.037 --> 00:43:26.280 align:middle line:84% don't feel well, we're going to do this thing-- which 00:43:26.280 --> 00:43:28.905 align:middle line:84% would have been the exact same thing as rolling into television 00:43:28.905 --> 00:43:30.900 align:middle line:84% and putting on a movie for people to watch. 00:43:30.900 --> 00:43:34.080 align:middle line:84% And also, that the systems have some kind 00:43:34.080 --> 00:43:37.590 align:middle line:84% of pedagogical model behind them and their effectiveness 00:43:37.590 --> 00:43:41.310 align:middle line:84% may have to do with how well that pedagogical model is 00:43:41.310 --> 00:43:43.560 align:middle line:90% implemented. 00:43:43.560 --> 00:43:45.570 align:middle line:84% So if the system is designed to be 00:43:45.570 --> 00:43:49.140 align:middle line:84% used with some consistency, twice a week in six week chunks 00:43:49.140 --> 00:43:51.660 align:middle line:84% or something like that-- and you don't use it 00:43:51.660 --> 00:43:54.973 align:middle line:84% with that program design in mind-- 00:43:54.973 --> 00:43:56.890 align:middle line:84% it might not accomplish what you wanted to do. 00:43:56.890 --> 00:43:59.580 align:middle line:84% So it's not just a matter of like downloading the software 00:43:59.580 --> 00:44:01.050 align:middle line:90% onto kids' computers. 00:44:01.050 --> 00:44:03.482 align:middle line:84% It's a matter of getting whole systems to implement them. 00:44:03.482 --> 00:44:05.190 align:middle line:84% We're going to look at some research I'll 00:44:05.190 --> 00:44:06.215 align:middle line:90% show you on Monday. 00:44:06.215 --> 00:44:07.840 align:middle line:84% I don't know if I'll make you read it-- 00:44:07.840 --> 00:44:09.660 align:middle line:84% but I'll just summarize it for you-- 00:44:09.660 --> 00:44:11.460 align:middle line:84% about some of these intelligent tutors 00:44:11.460 --> 00:44:15.270 align:middle line:84% that are really focused on letting people 00:44:15.270 --> 00:44:18.070 align:middle line:84% work on problems that are challenging for them, 00:44:18.070 --> 00:44:20.310 align:middle line:84% even if it's from material that is from earlier 00:44:20.310 --> 00:44:21.990 align:middle line:90% in the year or previous years. 00:44:21.990 --> 00:44:25.230 align:middle line:84% And that system only works if teachers actually 00:44:25.230 --> 00:44:26.850 align:middle line:90% let their students do that. 00:44:26.850 --> 00:44:28.822 align:middle line:84% If teachers force their students along, 00:44:28.822 --> 00:44:30.780 align:middle line:84% against the will of the system, then the system 00:44:30.780 --> 00:44:32.640 align:middle line:84% doesn't do what it's supposed to do. 00:44:32.640 --> 00:44:33.540 align:middle line:90% AUDIENCE: I think-- 00:44:33.540 --> 00:44:35.730 align:middle line:84% I guess to elaborate a little bit, why 00:44:35.730 --> 00:44:37.697 align:middle line:84% on like my experience was the way it was, was 00:44:37.697 --> 00:44:40.390 align:middle line:84% like not only were teachers like in your experience, 00:44:40.390 --> 00:44:43.230 align:middle line:84% just like doing it randomly, they weren't incorporating it 00:44:43.230 --> 00:44:45.060 align:middle line:90% into their curriculum. 00:44:45.060 --> 00:44:47.260 align:middle line:84% They were just like slapping it on top of it. 00:44:47.260 --> 00:44:49.620 align:middle line:84% And when students are already overwhelmed 00:44:49.620 --> 00:44:52.300 align:middle line:84% with after school activities and things like that, 00:44:52.300 --> 00:44:55.320 align:middle line:84% it then like-- and it was like more of a math-based thing 00:44:55.320 --> 00:45:00.870 align:middle line:84% than English, where you see some of the benefits early on. 00:45:00.870 --> 00:45:02.975 align:middle line:84% So it was just kind of thrown in there, 00:45:02.975 --> 00:45:04.600 align:middle line:84% but not incorporated in the curriculum. 00:45:04.600 --> 00:45:05.940 align:middle line:84% And then it's just like, essentially, 00:45:05.940 --> 00:45:06.705 align:middle line:90% became a low priority-- 00:45:06.705 --> 00:45:07.538 align:middle line:90% JUSTIN REICH: Good-- 00:45:07.538 --> 00:45:09.820 align:middle line:84% AUDIENCE: And like for a lot of people. 00:45:09.820 --> 00:45:12.520 align:middle line:84% JUSTIN REICH: So if a system like depends on some hybridity, 00:45:12.520 --> 00:45:14.250 align:middle line:84% if it depends on some things that are happening in class 00:45:14.250 --> 00:45:15.550 align:middle line:84% and some things are happening online-- 00:45:15.550 --> 00:45:17.830 align:middle line:84% like if those two pieces are not fitting together, 00:45:17.830 --> 00:45:19.205 align:middle line:84% it's not going to accomplish what 00:45:19.205 --> 00:45:20.630 align:middle line:90% the system wants to accomplish. 00:45:20.630 --> 00:45:21.130 align:middle line:90% All right. 00:45:21.130 --> 00:45:22.588 align:middle line:84% I'll give you a funny little piece, 00:45:22.588 --> 00:45:25.180 align:middle line:84% a little personal piece that is behind some of the technology. 00:45:25.180 --> 00:45:26.680 align:middle line:84% I explain some of this a little bit, 00:45:26.680 --> 00:45:29.730 align:middle line:84% but my father has more or less done one cool thing 00:45:29.730 --> 00:45:36.400 align:middle line:84% in his life, which was to get a fake person into Princeton. 00:45:36.400 --> 00:45:43.140 align:middle line:84% He and his frat buddies invented a person named J David Osnat 00:45:43.140 --> 00:45:47.130 align:middle line:90% or Joe D Osnat or Joe Does Not. 00:45:47.130 --> 00:45:52.237 align:middle line:84% And they found a friend of theirs to take the SAT for him. 00:45:52.237 --> 00:45:54.570 align:middle line:84% They found some friends who had a frat house in Michigan 00:45:54.570 --> 00:45:56.500 align:middle line:90% to make up a fake transcript. 00:45:56.500 --> 00:45:58.950 align:middle line:84% They got one of them to go, and do an interview, 00:45:58.950 --> 00:45:59.980 align:middle line:90% and things like that. 00:45:59.980 --> 00:46:02.490 align:middle line:84% And they got this fake guy into Princeton. 00:46:02.490 --> 00:46:05.490 align:middle line:84% And they eventually had to admit it. 00:46:05.490 --> 00:46:09.210 align:middle line:84% And at some point, somebody was taking one of them 00:46:09.210 --> 00:46:14.010 align:middle line:84% to the admissions office to go explain to the admissions 00:46:14.010 --> 00:46:16.570 align:middle line:84% officer that they had gotten a fake person into Princeton. 00:46:16.570 --> 00:46:19.028 align:middle line:84% And this guy started asking them a whole bunch of questions 00:46:19.028 --> 00:46:21.000 align:middle line:90% about how they took the SAT. 00:46:21.000 --> 00:46:23.640 align:middle line:84% And it turned out this guy worked for educational testing 00:46:23.640 --> 00:46:26.590 align:middle line:84% services, which was also in Princeton, New Jersey, 00:46:26.590 --> 00:46:29.430 align:middle line:84% and was trying to figure out some of the test security things 00:46:29.430 --> 00:46:31.020 align:middle line:90% that were going on there. 00:46:31.020 --> 00:46:34.620 align:middle line:84% So as I mentioned last time, one of the ways 00:46:34.620 --> 00:46:36.990 align:middle line:84% that you can make tests more secure 00:46:36.990 --> 00:46:40.600 align:middle line:84% is that you can make it so that tests are of equal difficulty, 00:46:40.600 --> 00:46:42.660 align:middle line:84% but they use different questions. 00:46:42.660 --> 00:46:47.130 align:middle line:84% So the students in Boston are not 00:46:47.130 --> 00:46:49.290 align:middle line:84% taking the exact same test that students in Hawaii 00:46:49.290 --> 00:46:51.180 align:middle line:84% will take six hours later, or students 00:46:51.180 --> 00:46:52.680 align:middle line:84% in other parts of the world, or things like that. 00:46:52.680 --> 00:46:54.138 align:middle line:84% In order to be able to do that, you 00:46:54.138 --> 00:46:58.020 align:middle line:84% have to be able to create a mathematical representation 00:46:58.020 --> 00:47:01.620 align:middle line:84% for any given item of how difficult that item is. 00:47:01.620 --> 00:47:04.410 align:middle line:84% What people basically decided to do 00:47:04.410 --> 00:47:08.850 align:middle line:84% to solve that is to model the difficulty of items 00:47:08.850 --> 00:47:10.920 align:middle line:90% as logistic functions. 00:47:10.920 --> 00:47:13.840 align:middle line:84% So on the x-axis, you have this thing, 00:47:13.840 --> 00:47:15.845 align:middle line:90% which is a proficiency score. 00:47:15.845 --> 00:47:18.810 align:middle line:84% Roughly kind of how hard in any domain this would be. 00:47:18.810 --> 00:47:21.790 align:middle line:84% And then the probability of a correct answer. 00:47:21.790 --> 00:47:27.030 align:middle line:84% So these are three different possible distributions of items. 00:47:27.030 --> 00:47:31.340 align:middle line:84% So for instance, which of these items 00:47:31.340 --> 00:47:33.840 align:middle line:84% would be the most difficult for individuals with proficiency 00:47:33.840 --> 00:47:34.655 align:middle line:90% score of 1. 00:47:34.655 --> 00:47:40.110 align:middle line:90% 00:47:40.110 --> 00:47:41.430 align:middle line:90% AUDIENCE: The blue. 00:47:41.430 --> 00:47:44.670 align:middle line:84% JUSTIN REICH: So people on the blue have almost 100% 00:47:44.670 --> 00:47:46.505 align:middle line:84% probability of getting this one correct. 00:47:46.505 --> 00:47:47.850 align:middle line:90% AUDIENCE: Oh, I was wondering. 00:47:47.850 --> 00:47:49.600 align:middle line:84% JUSTIN REICH: So it will be the green one. 00:47:49.600 --> 00:47:51.080 align:middle line:84% So this is the probability answer. 00:47:51.080 --> 00:47:55.640 align:middle line:84% So for this particular item, people with proficiency of 1 00:47:55.640 --> 00:47:57.540 align:middle line:84% are about 50% likely to get it right. 00:47:57.540 --> 00:48:01.050 align:middle line:84% These are the people who are about 90% to get it right, 00:48:01.050 --> 00:48:03.470 align:middle line:84% almost exactly likely to get it right. 00:48:03.470 --> 00:48:05.820 align:middle line:84% So that seems like a useful thing to know about an item, 00:48:05.820 --> 00:48:06.300 align:middle line:90% right? 00:48:06.300 --> 00:48:07.590 align:middle line:84% Now you can figure out, like, oh, 00:48:07.590 --> 00:48:09.500 align:middle line:84% let's give them the blue item first, and then 00:48:09.500 --> 00:48:13.010 align:middle line:84% the red item, and then the green item, or something like that. 00:48:13.010 --> 00:48:15.900 align:middle line:84% Not only that, but if you model it this way, 00:48:15.900 --> 00:48:19.940 align:middle line:84% you can also figure out how discriminating an item is, 00:48:19.940 --> 00:48:23.930 align:middle line:84% which is how likely is it that people around a given 00:48:23.930 --> 00:48:28.230 align:middle line:84% proficiency are likely to get it wrong or not get it wrong? 00:48:28.230 --> 00:48:33.290 align:middle line:84% Like, how tightly clustered are the answers versus be 00:48:33.290 --> 00:48:37.040 align:middle line:84% or how spread out are the answers going to be? 00:48:37.040 --> 00:48:43.790 align:middle line:84% So around 1, you would imagine for this one, 00:48:43.790 --> 00:48:48.810 align:middle line:84% that a whole bunch of people with scores slightly less than 1 00:48:48.810 --> 00:48:50.268 align:middle line:90% would be less likely to write. 00:48:50.268 --> 00:48:51.810 align:middle line:84% And then more likely to get it right. 00:48:51.810 --> 00:48:55.730 align:middle line:84% Whereas, around here, because the lines are more horizontal 00:48:55.730 --> 00:48:58.010 align:middle line:84% there, just about everybody who gets those items are 00:48:58.010 --> 00:48:58.760 align:middle line:90% going to be right. 00:48:58.760 --> 00:49:01.100 align:middle line:84% So it's going to give you that much information 00:49:01.100 --> 00:49:03.883 align:middle line:90% about how well people do. 00:49:03.883 --> 00:49:05.300 align:middle line:84% Another way that you could do this 00:49:05.300 --> 00:49:08.960 align:middle line:84% is draw a distribution of where you think a group of learners 00:49:08.960 --> 00:49:09.800 align:middle line:90% would be. 00:49:09.800 --> 00:49:13.880 align:middle line:84% So this is like the various skill levels in the class 00:49:13.880 --> 00:49:15.950 align:middle line:84% and the distribution of probabilities of people 00:49:15.950 --> 00:49:17.150 align:middle line:90% getting it right. 00:49:17.150 --> 00:49:19.310 align:middle line:84% And you can see that the item that 00:49:19.310 --> 00:49:22.310 align:middle line:84% traverse through the heart of that group 00:49:22.310 --> 00:49:24.980 align:middle line:84% is going to be the one that's best at efficiently measuring 00:49:24.980 --> 00:49:26.960 align:middle line:90% the proficiency of that group. 00:49:26.960 --> 00:49:29.635 align:middle line:84% You don't have to totally understand all the math or all 00:49:29.635 --> 00:49:30.510 align:middle line:90% the ideas about this. 00:49:30.510 --> 00:49:32.843 align:middle line:84% Probably, like the intuition that you need to develop is 00:49:32.843 --> 00:49:36.140 align:middle line:84% in the 1980s, a bunch of people came up with a pretty good 00:49:36.140 --> 00:49:39.560 align:middle line:84% statistical toolkit for figuring out how difficult an item was. 00:49:39.560 --> 00:49:42.020 align:middle line:84% The first thing that they did with it was 00:49:42.020 --> 00:49:45.630 align:middle line:84% make different kinds of tests for test security purposes. 00:49:45.630 --> 00:49:47.840 align:middle line:84% But almost the next thing they did with it was 00:49:47.840 --> 00:49:50.600 align:middle line:84% start developing systems that use computers 00:49:50.600 --> 00:49:53.250 align:middle line:84% to evaluate people's performance on an item, 00:49:53.250 --> 00:49:55.665 align:middle line:84% and then figure out what is the next good item 00:49:55.665 --> 00:49:56.790 align:middle line:90% to give them in a sequence. 00:49:56.790 --> 00:49:58.498 align:middle line:84% And all the things that you're describing 00:49:58.498 --> 00:49:59.990 align:middle line:84% IXL, Khan Academy, all other things 00:49:59.990 --> 00:50:03.360 align:middle line:84% like that, they may not use this exact statistical toolkit, 00:50:03.360 --> 00:50:06.685 align:middle line:84% but they all use something that's similar to that. 00:50:06.685 --> 00:50:10.460 align:middle line:90% 00:50:10.460 --> 00:50:12.710 align:middle line:84% I mean, part of the reason why in the book that I take 00:50:12.710 --> 00:50:15.380 align:middle line:84% a couple of pages to describe item response theory to people 00:50:15.380 --> 00:50:18.770 align:middle line:84% is, again, to try to give the sense it's not magic. 00:50:18.770 --> 00:50:21.860 align:middle line:84% It's not like some impossibly difficult thing 00:50:21.860 --> 00:50:25.730 align:middle line:84% that computers and scientists on the very cutting 00:50:25.730 --> 00:50:27.830 align:middle line:84% edge of mathematics have figured out. 00:50:27.830 --> 00:50:30.620 align:middle line:84% This is a pretty well-established set of tools 00:50:30.620 --> 00:50:33.080 align:middle line:84% that we've studied for a long time. 00:50:33.080 --> 00:50:37.080 align:middle line:84% If people come up with new AI-powered intelligent tutors, 00:50:37.080 --> 00:50:40.020 align:middle line:84% we should absolutely be able to ask the question like, well, 00:50:40.020 --> 00:50:42.395 align:middle line:84% how are they really different from the intelligent tutors 00:50:42.395 --> 00:50:44.910 align:middle line:84% that we've had for decades, in various formats? 00:50:44.910 --> 00:50:48.830 align:middle line:84% How different are they from the box of SRI reading cards 00:50:48.830 --> 00:50:50.850 align:middle line:84% that I had when I was in the fourth grade? 00:50:50.850 --> 00:50:52.490 align:middle line:84% And there may be some ways that they're 00:50:52.490 --> 00:50:53.510 align:middle line:84% different, in which case, we would 00:50:53.510 --> 00:50:54.840 align:middle line:90% expect much better outcomes. 00:50:54.840 --> 00:50:56.923 align:middle line:84% There may be some ways that they're quite similar, 00:50:56.923 --> 00:51:00.470 align:middle line:84% in which case, we might not expect that many differences. 00:51:00.470 --> 00:51:04.730 align:middle line:84% But the heart of the reason why people 00:51:04.730 --> 00:51:10.490 align:middle line:84% wanted these kinds of tools to exist was personalization. 00:51:10.490 --> 00:51:12.680 align:middle line:84% That a goal that you can get a lot of people 00:51:12.680 --> 00:51:14.810 align:middle line:84% around is the idea that wouldn't it 00:51:14.810 --> 00:51:18.230 align:middle line:84% be great if every learner could have an experience that's 00:51:18.230 --> 00:51:23.420 align:middle line:84% tailored to their own personal needs, and interests, 00:51:23.420 --> 00:51:25.370 align:middle line:84% and experiences, and things like that? 00:51:25.370 --> 00:51:29.270 align:middle line:84% I noticed, I don't know, around 2008, 2009, 2010 00:51:29.270 --> 00:51:31.400 align:middle line:84% that lots of people in education got interested 00:51:31.400 --> 00:51:32.460 align:middle line:90% in personalization. 00:51:32.460 --> 00:51:34.295 align:middle line:84% A thing that was weird to me is that there 00:51:34.295 --> 00:51:36.420 align:middle line:84% were people who were interested in personalization, 00:51:36.420 --> 00:51:39.920 align:middle line:84% who disagreed about everything else. 00:51:39.920 --> 00:51:44.720 align:middle line:84% So there would be people who are really into school choice, 00:51:44.720 --> 00:51:49.170 align:middle line:84% and traditional instruction, and conservative curriculum values, 00:51:49.170 --> 00:51:51.800 align:middle line:84% who typically were more associated 00:51:51.800 --> 00:51:54.230 align:middle line:84% with people on the political right, though not always 00:51:54.230 --> 00:51:57.090 align:middle line:84% in America, who got really interested in personalization. 00:51:57.090 --> 00:52:00.980 align:middle line:84% There were also people who were educational progressives, who 00:52:00.980 --> 00:52:03.980 align:middle line:84% were really interested in alternate, project-based school 00:52:03.980 --> 00:52:04.650 align:middle line:90% kinds of models. 00:52:04.650 --> 00:52:07.490 align:middle line:84% They also were talking about personalization. 00:52:07.490 --> 00:52:09.260 align:middle line:90% And I thought to myself, OK. 00:52:09.260 --> 00:52:12.530 align:middle line:84% If these groups who don't agree about anything 00:52:12.530 --> 00:52:14.670 align:middle line:84% are all using the word personalization, 00:52:14.670 --> 00:52:16.910 align:middle line:84% they can't possibly mean the same thing, 00:52:16.910 --> 00:52:18.750 align:middle line:84% because they have really different values. 00:52:18.750 --> 00:52:21.110 align:middle line:84% And so they probably don't agree about this either. 00:52:21.110 --> 00:52:24.470 align:middle line:84% Oh, this was a slide that my colleague Dan Meyer put together 00:52:24.470 --> 00:52:27.710 align:middle line:84% of this is some technology from the 1970s. 00:52:27.710 --> 00:52:30.950 align:middle line:84% Right at the dawn of personal computers, where 00:52:30.950 --> 00:52:34.020 align:middle line:84% we were trying to give kids math problems, evaluate them, 00:52:34.020 --> 00:52:35.130 align:middle line:90% give them harder problems. 00:52:35.130 --> 00:52:37.110 align:middle line:90% This is a slide from 2012. 00:52:37.110 --> 00:52:38.580 align:middle line:84% But the thing that you can see is 00:52:38.580 --> 00:52:41.270 align:middle line:84% there's 40 years between these pictures. 00:52:41.270 --> 00:52:48.230 align:middle line:84% And they are strikingly similar in their setup, in their outfit. 00:52:48.230 --> 00:52:50.960 align:middle line:84% Trying to personalize education with computers 00:52:50.960 --> 00:52:55.525 align:middle line:84% is one of the oldest enterprises in educational computing. 00:52:55.525 --> 00:52:57.650 align:middle line:84% It is one of the main things that we've been trying 00:52:57.650 --> 00:53:01.400 align:middle line:90% to do for 40 or 50 years. 00:53:01.400 --> 00:53:04.490 align:middle line:84% If there is any single paper that 00:53:04.490 --> 00:53:08.240 align:middle line:84% got people extremely excited about this, 00:53:08.240 --> 00:53:10.677 align:middle line:84% it was called "The 2 Sigma Problem-- 00:53:10.677 --> 00:53:13.010 align:middle line:84% The Search for Methods of Group Instruction as Effective 00:53:13.010 --> 00:53:15.770 align:middle line:84% as One-to-One Tutoring," where this guy Benjamin-- 00:53:15.770 --> 00:53:18.860 align:middle line:84% any of you ever heard of Bloom's Taxonomy? 00:53:18.860 --> 00:53:21.710 align:middle line:84% It's this little pyramid that has 00:53:21.710 --> 00:53:25.147 align:middle line:84% different kinds of learning activities, 00:53:25.147 --> 00:53:26.730 align:middle line:84% instructional activities you could do. 00:53:26.730 --> 00:53:28.850 align:middle line:84% It turns out that Benjamin Bloom did more than one thing 00:53:28.850 --> 00:53:29.423 align:middle line:90% in his life. 00:53:29.423 --> 00:53:30.840 align:middle line:84% He didn't just make this taxonomy, 00:53:30.840 --> 00:53:32.180 align:middle line:90% but he other kinds of studies. 00:53:32.180 --> 00:53:33.680 align:middle line:84% He didn't actually do these studies. 00:53:33.680 --> 00:53:36.560 align:middle line:84% He was reporting on some doctoral students of his 00:53:36.560 --> 00:53:39.750 align:middle line:90% that were studying tutoring. 00:53:39.750 --> 00:53:41.850 align:middle line:84% And in these tutoring studies that they did, 00:53:41.850 --> 00:53:44.888 align:middle line:84% you just get astronomically good results. 00:53:44.888 --> 00:53:46.430 align:middle line:84% They said that they could get results 00:53:46.430 --> 00:53:50.520 align:middle line:84% two standard deviations better what you'd typically find. 00:53:50.520 --> 00:53:52.430 align:middle line:90% That's the 2 sigma problem. 00:53:52.430 --> 00:53:54.710 align:middle line:84% A two standard deviation improvement in education 00:53:54.710 --> 00:53:57.110 align:middle line:90% is impossibly good. 00:53:57.110 --> 00:53:59.690 align:middle line:84% Most of the things that we do get 00:53:59.690 --> 00:54:01.670 align:middle line:84% a tenth of a standard deviation, or a hundredth 00:54:01.670 --> 00:54:03.080 align:middle line:84% of a standard deviation improvement, 00:54:03.080 --> 00:54:03.750 align:middle line:90% or something like that. 00:54:03.750 --> 00:54:04.890 align:middle line:84% We usually get really small gains, 00:54:04.890 --> 00:54:05.940 align:middle line:90% but tutoring works really well. 00:54:05.940 --> 00:54:07.315 align:middle line:84% I mean, to some extent, you could 00:54:07.315 --> 00:54:09.410 align:middle line:84% be like, I'm pretty sure medieval lords knew 00:54:09.410 --> 00:54:10.750 align:middle line:90% that tutoring works pretty well. 00:54:10.750 --> 00:54:11.300 align:middle line:90% [LAUGHTER] 00:54:11.300 --> 00:54:14.100 align:middle line:84% The reason why we don't do it is it's enormously expensive. 00:54:14.100 --> 00:54:16.670 align:middle line:84% But what if, instead of medieval lords 00:54:16.670 --> 00:54:19.610 align:middle line:84% hiring governesses to tutor their students, 00:54:19.610 --> 00:54:22.160 align:middle line:90% what if computers could do that? 00:54:22.160 --> 00:54:24.658 align:middle line:84% What if computers, which are relatively inexpensive, 00:54:24.658 --> 00:54:25.950 align:middle line:90% could do this kind of tutoring? 00:54:25.950 --> 00:54:27.030 align:middle line:90% So this has been a dream. 00:54:27.030 --> 00:54:29.430 align:middle line:84% This was Daphne Coller's dream, Saul Cotton's dream, 00:54:29.430 --> 00:54:31.640 align:middle line:84% lots of other people's dream as they were imagining 00:54:31.640 --> 00:54:33.340 align:middle line:90% what's possible here. 00:54:33.340 --> 00:54:36.530 align:middle line:90% 00:54:36.530 --> 00:54:39.500 align:middle line:84% Around 2010, there were probably two different groups 00:54:39.500 --> 00:54:45.240 align:middle line:84% of people, who made personalized learning central to their ideas 00:54:45.240 --> 00:54:46.620 align:middle line:90% and their philosophies. 00:54:46.620 --> 00:54:48.870 align:middle line:84% One group of people were really interested in what 00:54:48.870 --> 00:54:51.728 align:middle line:84% they called blended learning-- a formal educational program 00:54:51.728 --> 00:54:53.520 align:middle line:84% in which a student learns, at least in part 00:54:53.520 --> 00:54:56.490 align:middle line:84% through online delivery, with some element of student control 00:54:56.490 --> 00:55:00.090 align:middle line:90% over time, place, path, or pace. 00:55:00.090 --> 00:55:04.800 align:middle line:84% Some of it may be supervised, and then some of it at home. 00:55:04.800 --> 00:55:08.280 align:middle line:84% The alternative view was called connected 00:55:08.280 --> 00:55:10.842 align:middle line:84% learning-- socially embedded, interest-driven, oriented 00:55:10.842 --> 00:55:13.050 align:middle line:84% towards educational, economic, political opportunity. 00:55:13.050 --> 00:55:14.508 align:middle line:84% Connected learning is realized when 00:55:14.508 --> 00:55:16.830 align:middle line:84% a person is able to pursue a personal interest 00:55:16.830 --> 00:55:20.790 align:middle line:84% or passion with the support of friends and caring adults. 00:55:20.790 --> 00:55:24.395 align:middle line:84% So what else do these two ideas sound like? 00:55:24.395 --> 00:55:30.015 align:middle line:90% 00:55:30.015 --> 00:55:31.158 align:middle line:90% AUDIENCE: CLT. 00:55:31.158 --> 00:55:32.200 align:middle line:90% JUSTIN REICH: Yeah, good. 00:55:32.200 --> 00:55:34.073 align:middle line:84% So you could have said cognitive load theory 00:55:34.073 --> 00:55:34.990 align:middle line:90% and situated learning. 00:55:34.990 --> 00:55:36.220 align:middle line:84% You could have said doing Thorndike. 00:55:36.220 --> 00:55:37.637 align:middle line:84% You could have said instructionism 00:55:37.637 --> 00:55:38.870 align:middle line:90% and constructionism. 00:55:38.870 --> 00:55:41.570 align:middle line:84% This is like the reoccurrence of these two themes 00:55:41.570 --> 00:55:44.000 align:middle line:84% over-- like there's always two ideas. 00:55:44.000 --> 00:55:44.510 align:middle line:90% And they're. 00:55:44.510 --> 00:55:45.830 align:middle line:84% And they're always being reformatted 00:55:45.830 --> 00:55:47.010 align:middle line:84% and they're always getting connected to each other. 00:55:47.010 --> 00:55:49.370 align:middle line:84% So this is like the Dewey and Thorndike version 00:55:49.370 --> 00:55:51.830 align:middle line:84% of how we're going to make personalization 00:55:51.830 --> 00:55:54.380 align:middle line:90% come to life in the world. 00:55:54.380 --> 00:55:57.345 align:middle line:84% We'll talk more about connected learning in a week. 00:55:57.345 --> 00:55:58.970 align:middle line:84% But mostly, what we'll talk about today 00:55:58.970 --> 00:56:02.160 align:middle line:84% is what this blended learning might look like. 00:56:02.160 --> 00:56:04.670 align:middle line:84% So Khan Academy doesn't have this anymore. 00:56:04.670 --> 00:56:09.350 align:middle line:84% But they used to represent their mathematics content in K 00:56:09.350 --> 00:56:12.920 align:middle line:84% through 12 as a knowledge map, as kind of a galaxy. 00:56:12.920 --> 00:56:15.740 align:middle line:84% One of the things that this blended learning 00:56:15.740 --> 00:56:18.110 align:middle line:84% version of personalization requires 00:56:18.110 --> 00:56:19.910 align:middle line:84% is you have to know all of the stuff 00:56:19.910 --> 00:56:22.020 align:middle line:84% that you want people to learn in advance. 00:56:22.020 --> 00:56:24.800 align:middle line:84% So if you want to create an algorithmically optimized 00:56:24.800 --> 00:56:27.840 align:middle line:84% pathway for people through a set of learning materials, 00:56:27.840 --> 00:56:30.913 align:middle line:84% you have to know what those learning materials are, 00:56:30.913 --> 00:56:33.080 align:middle line:84% so that you can write the algorithms that drive them 00:56:33.080 --> 00:56:33.800 align:middle line:90% through it. 00:56:33.800 --> 00:56:36.680 align:middle line:84% That's very different from this connected learning view, which 00:56:36.680 --> 00:56:38.710 align:middle line:84% says we should figure out what kids are really interested in 00:56:38.710 --> 00:56:41.230 align:middle line:84% and help them create personal pathways through whatever it 00:56:41.230 --> 00:56:43.090 align:middle line:90% is they want. 00:56:43.090 --> 00:56:45.850 align:middle line:84% So you can imagine that the goal of Khan Academy 00:56:45.850 --> 00:56:49.220 align:middle line:84% is to take some individual kid and figure out the fastest, 00:56:49.220 --> 00:56:51.760 align:middle line:84% most efficient way for them to traverse pathways 00:56:51.760 --> 00:56:53.180 align:middle line:90% through this kind of thing. 00:56:53.180 --> 00:56:57.100 align:middle line:84% People were very compelled by that model. 00:56:57.100 --> 00:56:59.180 align:middle line:90% Partially, it was interesting. 00:56:59.180 --> 00:57:01.010 align:middle line:90% They were contrasting it. 00:57:01.010 --> 00:57:03.760 align:middle line:84% A lot of times people will critique 00:57:03.760 --> 00:57:06.790 align:middle line:84% what they call the factory model of schooling, which 00:57:06.790 --> 00:57:09.040 align:middle line:84% is that students come in, in cohorts, they're 00:57:09.040 --> 00:57:10.542 align:middle line:90% stamped at the same time. 00:57:10.542 --> 00:57:12.250 align:middle line:84% They're moved through the same materials, 00:57:12.250 --> 00:57:14.210 align:middle line:84% despite all their individual differences. 00:57:14.210 --> 00:57:16.300 align:middle line:84% It's not really true, but there are 00:57:16.300 --> 00:57:18.790 align:middle line:84% a lot of folks who will say that schools were organized 00:57:18.790 --> 00:57:21.615 align:middle line:84% around factories, in order to generate factory laborers 00:57:21.615 --> 00:57:22.490 align:middle line:90% and things like that. 00:57:22.490 --> 00:57:24.640 align:middle line:84% And there are certainly ideas about how 00:57:24.640 --> 00:57:27.610 align:middle line:84% we use time and other kinds of things like that go 00:57:27.610 --> 00:57:29.090 align:middle line:90% across institutions in society. 00:57:29.090 --> 00:57:32.590 align:middle line:84% This is a representation from a keynote. 00:57:32.590 --> 00:57:34.210 align:middle line:90% Oh, Sir Ken. 00:57:34.210 --> 00:57:35.170 align:middle line:90% I'm forgetting. 00:57:35.170 --> 00:57:39.895 align:middle line:84% He's the most popular keynote on TED. 00:57:39.895 --> 00:57:41.770 align:middle line:84% I still think it's true, the most popular TED 00:57:41.770 --> 00:57:43.550 align:middle line:90% Talk that was ever written. 00:57:43.550 --> 00:57:45.550 align:middle line:84% Somebody will have to look up Ken and figure out 00:57:45.550 --> 00:57:46.960 align:middle line:90% who I mean by this. 00:57:46.960 --> 00:57:50.080 align:middle line:84% But I don't know if this is actually from that or a follow 00:57:50.080 --> 00:57:51.940 align:middle line:90% up representation of slide. 00:57:51.940 --> 00:57:58.540 align:middle line:84% So then the question is does this blended learning model have 00:57:58.540 --> 00:58:00.830 align:middle line:90% truly individualized thing? 00:58:00.830 --> 00:58:03.130 align:middle line:84% Or, are you just kind of like stamping each child 00:58:03.130 --> 00:58:05.680 align:middle line:84% with the same stuff in their own rate? 00:58:05.680 --> 00:58:07.685 align:middle line:84% They all still are going to get kind of stamped 00:58:07.685 --> 00:58:09.310 align:middle line:84% with these materials, but maybe they'll 00:58:09.310 --> 00:58:13.552 align:middle line:84% get stamped through the factory at their own individual path. 00:58:13.552 --> 00:58:14.510 align:middle line:90% AUDIENCE: Ken Robinson. 00:58:14.510 --> 00:58:15.820 align:middle line:90% JUSTIN REICH: Ken Robinson. 00:58:15.820 --> 00:58:16.600 align:middle line:90% Robinson. 00:58:16.600 --> 00:58:19.450 align:middle line:90% Sir Ken Robinson. 00:58:19.450 --> 00:58:24.200 align:middle line:84% People make all kinds of crazy claims about these systems. 00:58:24.200 --> 00:58:26.830 align:middle line:84% And sometimes I like to share these crazy claims 00:58:26.830 --> 00:58:30.260 align:middle line:84% 10 years later because they sound silly afterwards. 00:58:30.260 --> 00:58:32.970 align:middle line:84% So in 2009, Clay Christensen, who 00:58:32.970 --> 00:58:35.368 align:middle line:84% was one of the people who defined blended learning, 00:58:35.368 --> 00:58:37.660 align:middle line:84% wrote this book called Disrupting Class in the business 00:58:37.660 --> 00:58:41.030 align:middle line:84% school, do you study still the theory of disruptive innovation? 00:58:41.030 --> 00:58:43.210 align:middle line:90% Is that a thing that-- 00:58:43.210 --> 00:58:44.990 align:middle line:84% it was like crazy popular a while ago? 00:58:44.990 --> 00:58:46.130 align:middle line:90% And I think it has faded. 00:58:46.130 --> 00:58:48.130 align:middle line:84% But they apply this idea called the theory 00:58:48.130 --> 00:58:49.250 align:middle line:90% of disruptive innovation. 00:58:49.250 --> 00:58:51.820 align:middle line:84% And in 2009, they made the prediction 00:58:51.820 --> 00:58:55.360 align:middle line:84% that by 2019, 50% of all grades six through 12 00:58:55.360 --> 00:58:56.360 align:middle line:90% courses would be online. 00:58:56.360 --> 00:58:58.870 align:middle line:84% That they would cost a third as much to deliver. 00:58:58.870 --> 00:59:01.390 align:middle line:84% That your school system would cost like about a third 00:59:01.390 --> 00:59:03.500 align:middle line:84% as much to operate as it does right now. 00:59:03.500 --> 00:59:05.447 align:middle line:84% And they would have better learning outcomes. 00:59:05.447 --> 00:59:07.030 align:middle line:84% How many of you were in a school where 00:59:07.030 --> 00:59:10.120 align:middle line:84% 50% of all of your classes in secondary school 00:59:10.120 --> 00:59:12.890 align:middle line:84% were online or predominantly online? 00:59:12.890 --> 00:59:13.700 align:middle line:90% Yeah, zero. 00:59:13.700 --> 00:59:15.580 align:middle line:84% Pretty much didn't work out that way. 00:59:15.580 --> 00:59:19.330 align:middle line:84% Sebastian Thrun, the quote that Dana saw, was up there. 00:59:19.330 --> 00:59:23.140 align:middle line:84% Sal Khan had an extremely popular TED Talk called 00:59:23.140 --> 00:59:25.460 align:middle line:84% "Let's Use Video to Reinvent Education," 00:59:25.460 --> 00:59:29.290 align:middle line:84% where he described this vision where kids would sit down 00:59:29.290 --> 00:59:30.800 align:middle line:84% in front of individual terminals. 00:59:30.800 --> 00:59:33.560 align:middle line:84% They would learn at an algorithmically optimized rate 00:59:33.560 --> 00:59:34.890 align:middle line:90% through mathematical content. 00:59:34.890 --> 00:59:36.470 align:middle line:84% They still might do things together. 00:59:36.470 --> 00:59:38.060 align:middle line:84% A teacher might come and collect students 00:59:38.060 --> 00:59:40.470 align:middle line:84% who are stuck in the same thing and remediate them together. 00:59:40.470 --> 00:59:43.053 align:middle line:84% Or, take advanced students and do projects with them or things 00:59:43.053 --> 00:59:43.590 align:middle line:90% like that. 00:59:43.590 --> 00:59:45.530 align:middle line:84% But the heart of their mathematical learning 00:59:45.530 --> 00:59:47.330 align:middle line:90% would be individual. 00:59:47.330 --> 00:59:49.200 align:middle line:84% There's one guy who was actually, 00:59:49.200 --> 00:59:51.600 align:middle line:84% I think, probably the most wrong of them, 00:59:51.600 --> 00:59:54.270 align:middle line:90% who won the 2013 Ted Prize. 00:59:54.270 --> 00:59:56.780 align:middle line:90% He got $1 million for this idea. 00:59:56.780 --> 01:00:00.090 align:middle line:84% That you don't even need schools or educational systems. 01:00:00.090 --> 01:00:01.820 align:middle line:84% If you just give kids laptops, they 01:00:01.820 --> 01:00:04.280 align:middle line:84% can teach anything by themselves. 01:00:04.280 --> 01:00:07.220 align:middle line:84% And he had this whole program called The Hole in the Wall 01:00:07.220 --> 01:00:09.740 align:middle line:84% Program, where they put computers in public spaces 01:00:09.740 --> 01:00:11.810 align:middle line:90% in Southeast Asia. 01:00:11.810 --> 01:00:12.550 align:middle line:90% Anyway. 01:00:12.550 --> 01:00:15.050 align:middle line:84% There are millions of parents who lived through the pandemic 01:00:15.050 --> 01:00:17.772 align:middle line:84% that can confirm that kids with laptops cannot teach anything 01:00:17.772 --> 01:00:18.355 align:middle line:90% to themselves. 01:00:18.355 --> 01:00:20.030 align:middle line:90% [LAUGHTER] 01:00:20.030 --> 01:00:22.070 align:middle line:84% Sal Khan is interesting to me because there 01:00:22.070 --> 01:00:26.630 align:middle line:84% are millions of people who have watched his 2011 TED 01:00:26.630 --> 01:00:28.020 align:middle line:90% Talk about this model. 01:00:28.020 --> 01:00:29.750 align:middle line:84% There are many, many fewer people 01:00:29.750 --> 01:00:32.840 align:middle line:84% who read an interview that he did in 2019 with District 01:00:32.840 --> 01:00:34.370 align:middle line:90% Administration Magazine. 01:00:34.370 --> 01:00:37.160 align:middle line:84% So District Administration is like a little trade magazine 01:00:37.160 --> 01:00:38.810 align:middle line:90% for superintendents. 01:00:38.810 --> 01:00:42.690 align:middle line:84% In between starting Khan Academy in 2019, 01:00:42.690 --> 01:00:45.800 align:middle line:84% he built his own school called the Khan Lab School. 01:00:45.800 --> 01:00:47.262 align:middle line:90% It's a private school. 01:00:47.262 --> 01:00:48.720 align:middle line:84% It's in the Bay Area of California. 01:00:48.720 --> 01:00:51.303 align:middle line:84% It costs $20,000 or $30,000 or something like that to go there 01:00:51.303 --> 01:00:51.930 align:middle line:90% for a year. 01:00:51.930 --> 01:00:53.347 align:middle line:84% But it's basically his laboratory. 01:00:53.347 --> 01:00:55.443 align:middle line:84% He can implement any model he wanted. 01:00:55.443 --> 01:00:56.610 align:middle line:90% He can pick his own faculty. 01:00:56.610 --> 01:00:57.902 align:middle line:90% He can pick his own curriculum. 01:00:57.902 --> 01:01:00.065 align:middle line:84% He can recruit whatever students he wants. 01:01:00.065 --> 01:01:01.440 align:middle line:84% And he does this for a few years. 01:01:01.440 --> 01:01:03.120 align:middle line:84% And he says, now that I run a school, 01:01:03.120 --> 01:01:04.250 align:middle line:84% I see that some of this stuff is not 01:01:04.250 --> 01:01:06.720 align:middle line:84% as easy to accomplish compared to how it sounds theoretically. 01:01:06.720 --> 01:01:08.940 align:middle line:84% It's taken me and the Khan Academy longer to realize this, 01:01:08.940 --> 01:01:11.618 align:middle line:84% but not everyone can easily move to a mastery-based, self-paced 01:01:11.618 --> 01:01:12.660 align:middle line:90% learning model overnight. 01:01:12.660 --> 01:01:13.560 align:middle line:90% That's hard to do. 01:01:13.560 --> 01:01:15.780 align:middle line:84% There are a bunch of efficacy studies for that type of model, 01:01:15.780 --> 01:01:17.405 align:middle line:84% and they're robust, but change is hard. 01:01:17.405 --> 01:01:19.850 align:middle line:84% More recently, we're seeing that students put 30 minutes 01:01:19.850 --> 01:01:22.107 align:middle line:84% to an hour per week, or one class period per week 01:01:22.107 --> 01:01:23.940 align:middle line:84% towards software-based, self-paced learning. 01:01:23.940 --> 01:01:26.270 align:middle line:84% Schools will see a 20% to 30% greater-than-expected gain 01:01:26.270 --> 01:01:27.060 align:middle line:90% on state assessments. 01:01:27.060 --> 01:01:28.977 align:middle line:84% That's exciting because that's a dosage that's 01:01:28.977 --> 01:01:30.662 align:middle line:84% very doable in mainstream classrooms. 01:01:30.662 --> 01:01:33.370 align:middle line:84% We tell schools to give students 30 to 60 minutes of Khan Academy 01:01:33.370 --> 01:01:35.650 align:middle line:84% per week, with teachers doing traditional curriculum four 01:01:35.650 --> 01:01:36.170 align:middle line:90% days a week. 01:01:36.170 --> 01:01:38.045 align:middle line:84% And you're going to get dramatic improvement. 01:01:38.045 --> 01:01:39.940 align:middle line:84% You'll get the best of both worlds. 01:01:39.940 --> 01:01:45.280 align:middle line:84% So this is a pretty big step away from let's reinvent video 01:01:45.280 --> 01:01:46.570 align:middle line:90% with education. 01:01:46.570 --> 01:01:48.850 align:middle line:84% It's saying, like instead of reinventing education 01:01:48.850 --> 01:01:51.910 align:middle line:84% with video, let's do practice problems one day a week. 01:01:51.910 --> 01:01:55.780 align:middle line:84% And again, you can think about, is this disrupting education 01:01:55.780 --> 01:01:57.560 align:middle line:84% or is education domesticating the system? 01:01:57.560 --> 01:01:59.102 align:middle line:84% But there's something which struck me 01:01:59.102 --> 01:02:00.670 align:middle line:84% about this argument, which is like, I 01:02:00.670 --> 01:02:02.560 align:middle line:90% think I've heard this before. 01:02:02.560 --> 01:02:06.580 align:middle line:84% And one of the earliest places you can find it is in 1994. 01:02:06.580 --> 01:02:10.030 align:middle line:84% So there are a group of folks at Carnegie Mellon, who 01:02:10.030 --> 01:02:13.035 align:middle line:84% took a bunch of computers that had digital-- 01:02:13.035 --> 01:02:14.660 align:middle line:84% they wouldn't have been online network. 01:02:14.660 --> 01:02:16.618 align:middle line:84% They would have been digital practice problems. 01:02:16.618 --> 01:02:18.700 align:middle line:84% Brought them into the Pittsburgh Public Schools. 01:02:18.700 --> 01:02:21.320 align:middle line:84% And said to the teachers, hey, teach regularly three days 01:02:21.320 --> 01:02:21.820 align:middle line:90% a week. 01:02:21.820 --> 01:02:23.173 align:middle line:90% And do this two days a week. 01:02:23.173 --> 01:02:24.590 align:middle line:84% And apparently, for the most part, 01:02:24.590 --> 01:02:27.160 align:middle line:84% the teachers were like, eh, we'll do it one day a week. 01:02:27.160 --> 01:02:32.830 align:middle line:84% And they got the experimental classes outperformed students 01:02:32.830 --> 01:02:36.100 align:middle line:84% in comparison classes by 15% on standardized tests and 100% 01:02:36.100 --> 01:02:37.760 align:middle line:84% on tests targeting these objectives. 01:02:37.760 --> 01:02:39.970 align:middle line:84% So roughly the same order of magnitude 01:02:39.970 --> 01:02:41.980 align:middle line:84% kind of gains that Khan Academy thought. 01:02:41.980 --> 01:02:44.200 align:middle line:84% So anyway, one of the ways that you could frame that 01:02:44.200 --> 01:02:47.590 align:middle line:84% is that with 10 years and $150 million 01:02:47.590 --> 01:02:49.630 align:middle line:84% of philanthropic investment, Sal Khan 01:02:49.630 --> 01:02:51.588 align:middle line:84% learned what you could have learned with a trip 01:02:51.588 --> 01:02:52.630 align:middle line:90% to the library-- 01:02:52.630 --> 01:02:55.690 align:middle line:84% which is that they're already existing models for how 01:02:55.690 --> 01:02:58.270 align:middle line:84% you can incorporate intelligent tutors and practice problems 01:02:58.270 --> 01:02:59.228 align:middle line:90% into what you're doing. 01:02:59.228 --> 01:03:00.418 align:middle line:90% And they work pretty well. 01:03:00.418 --> 01:03:01.960 align:middle line:84% AUDIENCE: I'm curious to know-- so we 01:03:01.960 --> 01:03:05.980 align:middle line:84% discussed that this individualized tutoring has 01:03:05.980 --> 01:03:07.863 align:middle line:90% many benefits. 01:03:07.863 --> 01:03:09.280 align:middle line:84% But I'm curious to know if there's 01:03:09.280 --> 01:03:12.910 align:middle line:84% any studies that show to which particular subjects 01:03:12.910 --> 01:03:14.525 align:middle line:84% they might be more effective or not? 01:03:14.525 --> 01:03:15.650 align:middle line:90% JUSTIN REICH: That's great. 01:03:15.650 --> 01:03:15.790 align:middle line:90% Yeah. 01:03:15.790 --> 01:03:16.295 align:middle line:90% Yeah. 01:03:16.295 --> 01:03:16.795 align:middle line:90% Many. 01:03:16.795 --> 01:03:19.660 align:middle line:90% 01:03:19.660 --> 01:03:21.765 align:middle line:90% It really mostly works in math. 01:03:21.765 --> 01:03:24.670 align:middle line:90% 01:03:24.670 --> 01:03:29.560 align:middle line:84% The current evidence based for reading is very weak. 01:03:29.560 --> 01:03:31.370 align:middle line:84% And we'll show more examples of that 01:03:31.370 --> 01:03:35.770 align:middle line:84% and read some more papers about that for next class. 01:03:35.770 --> 01:03:36.520 align:middle line:90% So here's a trick. 01:03:36.520 --> 01:03:38.740 align:middle line:84% When we try to evaluate these tools, 01:03:38.740 --> 01:03:41.110 align:middle line:84% you don't want to look at any individual study, A, 01:03:41.110 --> 01:03:44.392 align:middle line:84% because any individual study can be wrong-- 01:03:44.392 --> 01:03:45.850 align:middle line:84% well, maybe that's the main reason. 01:03:45.850 --> 01:03:47.267 align:middle line:84% Any individual study can be wrong. 01:03:47.267 --> 01:03:49.910 align:middle line:84% So what you do is you do what's called a meta analysis. 01:03:49.910 --> 01:03:51.290 align:middle line:90% You look at groups of studies. 01:03:51.290 --> 01:03:53.020 align:middle line:84% You say, what are all the studies 01:03:53.020 --> 01:03:55.250 align:middle line:84% that have been done with intelligent tutors in math? 01:03:55.250 --> 01:03:56.170 align:middle line:84% What are all the studies that have 01:03:56.170 --> 01:03:58.100 align:middle line:84% been done with intelligent tutors in reading? 01:03:58.100 --> 01:04:00.700 align:middle line:84% And generally speaking, the only places 01:04:00.700 --> 01:04:05.020 align:middle line:84% that we find consistently more positive things, which are not 01:04:05.020 --> 01:04:07.030 align:middle line:90% hugely positive, is in math. 01:04:07.030 --> 01:04:09.220 align:middle line:84% What complicates that a little bit 01:04:09.220 --> 01:04:14.020 align:middle line:84% is that some of the strongest studies in evidence 01:04:14.020 --> 01:04:16.360 align:middle line:84% for mathematics have been more recently. 01:04:16.360 --> 01:04:18.650 align:middle line:84% Now, one possibility is that that's a fluke. 01:04:18.650 --> 01:04:21.050 align:middle line:84% Another possibility is that we're figuring stuff out. 01:04:21.050 --> 01:04:21.970 align:middle line:84% So you have to decide to yourself, 01:04:21.970 --> 01:04:24.512 align:middle line:84% like do you want to average over the whole length of studies? 01:04:24.512 --> 01:04:26.400 align:middle line:84% Or, do you want to reinforce your ideas 01:04:26.400 --> 01:04:27.580 align:middle line:90% based on more recent things. 01:04:27.580 --> 01:04:28.680 align:middle line:84% I'll make an argument next class. 01:04:28.680 --> 01:04:30.940 align:middle line:84% That actually some of the recent studies in math 01:04:30.940 --> 01:04:32.550 align:middle line:90% seem to be pretty robust. 01:04:32.550 --> 01:04:35.460 align:middle line:84% People try this stuff in early language acquisition. 01:04:35.460 --> 01:04:37.048 align:middle line:84% Duolingo, I think, has some evidence 01:04:37.048 --> 01:04:39.090 align:middle line:84% that they can teach you about as much in Duolingo 01:04:39.090 --> 01:04:41.310 align:middle line:84% as you learn in a first year college class. 01:04:41.310 --> 01:04:42.420 align:middle line:84% Although, you should probably note that 01:04:42.420 --> 01:04:44.490 align:middle line:84% you really don't learn that much in a first year 01:04:44.490 --> 01:04:45.965 align:middle line:90% college class in the language. 01:04:45.965 --> 01:04:46.920 align:middle line:90% [LAUGHTER] 01:04:46.920 --> 01:04:49.180 align:middle line:84% Computer science uses tons of things like this. 01:04:49.180 --> 01:04:50.880 align:middle line:84% Probably many of you who've done computer programming 01:04:50.880 --> 01:04:52.922 align:middle line:84% with Cat Soup and things like that, they probably 01:04:52.922 --> 01:04:57.280 align:middle line:84% have hinting and other sorts of things that are baked into that. 01:04:57.280 --> 01:05:01.050 align:middle line:84% Doesn't seem to be very involved in later language acquisition. 01:05:01.050 --> 01:05:04.500 align:middle line:84% People don't use it that much in science or in social studies. 01:05:04.500 --> 01:05:06.113 align:middle line:84% A little bit in chemistry, physics. 01:05:06.113 --> 01:05:08.530 align:middle line:84% Some places have some practice problems, things like that, 01:05:08.530 --> 01:05:11.370 align:middle line:84% but not about the conceptual understanding more 01:05:11.370 --> 01:05:15.570 align:middle line:84% about the procedural fluency kinds of things. 01:05:15.570 --> 01:05:22.020 align:middle line:84% And so what they require, in addition to these item response 01:05:22.020 --> 01:05:24.395 align:middle line:84% theory models, which figure out how difficult an item is, 01:05:24.395 --> 01:05:26.270 align:middle line:84% is it has to be able to be auto-graded. 01:05:26.270 --> 01:05:29.352 align:middle line:84% And because it has to be able to instantly-- 01:05:29.352 --> 01:05:31.060 align:middle line:84% it has to be able to instantly assign you 01:05:31.060 --> 01:05:32.530 align:middle line:90% what the next item is. 01:05:32.530 --> 01:05:35.860 align:middle line:84% And so the only way we can do that is we choose items 01:05:35.860 --> 01:05:38.553 align:middle line:84% where we can be pretty confident that a machine can ascertain 01:05:38.553 --> 01:05:40.220 align:middle line:84% whether or not you've done it correctly. 01:05:40.220 --> 01:05:42.303 align:middle line:84% You can do that in any field with multiple choice. 01:05:42.303 --> 01:05:44.480 align:middle line:84% But multiple choice is-- it's good for some things. 01:05:44.480 --> 01:05:45.670 align:middle line:90% It's kind of boring. 01:05:45.670 --> 01:05:47.230 align:middle line:84% AUDIENCE: Oh, I was going to ask. 01:05:47.230 --> 01:05:49.670 align:middle line:84% When you said that it's good with working with math, 01:05:49.670 --> 01:05:53.600 align:middle line:84% I guess, you answered it later in your speech just now. 01:05:53.600 --> 01:05:57.070 align:middle line:84% But what do you mean by it's good with math 01:05:57.070 --> 01:05:59.290 align:middle line:84% because there's a difference, a really strong one, 01:05:59.290 --> 01:06:02.330 align:middle line:84% between procedural fluency and like conceptual understanding. 01:06:02.330 --> 01:06:05.870 align:middle line:84% And when I was thinking about IXL writing this thing, 01:06:05.870 --> 01:06:06.820 align:middle line:90% I actually loved IXL. 01:06:06.820 --> 01:06:07.570 align:middle line:90% JUSTIN REICH: Yep. 01:06:07.570 --> 01:06:11.680 align:middle line:84% AUDIENCE: And the reason was because my math teachers were 01:06:11.680 --> 01:06:14.130 align:middle line:84% trying really hard, but it was boring. 01:06:14.130 --> 01:06:14.630 align:middle line:90% [LAUGHTER] 01:06:14.630 --> 01:06:17.110 align:middle line:84% And I felt it was like something I could use, 01:06:17.110 --> 01:06:19.580 align:middle line:84% where you don't have to do the same grade. 01:06:19.580 --> 01:06:21.223 align:middle line:90% You can go past your grade. 01:06:21.223 --> 01:06:21.530 align:middle line:90% JUSTIN REICH: Yeah, yeah. 01:06:21.530 --> 01:06:23.280 align:middle line:84% AUDIENCE: And then you learn a bunch of procedural stuff 01:06:23.280 --> 01:06:25.197 align:middle line:84% and you feel like you're so smart or whatever. 01:06:25.197 --> 01:06:27.675 align:middle line:84% But that's not true, because procedural fluency is not-- 01:06:27.675 --> 01:06:28.290 align:middle line:84% JUSTIN REICH: It's not mathematics. 01:06:28.290 --> 01:06:29.860 align:middle line:84% AUDIENCE: I mean, it's definitely-- yeah, 01:06:29.860 --> 01:06:30.730 align:middle line:90% it's not really math. 01:06:30.730 --> 01:06:32.370 align:middle line:84% JUSTIN REICH: I mean, a riff that I'll give you a little bit 01:06:32.370 --> 01:06:34.800 align:middle line:84% later in the semester-- so in the Common Core State 01:06:34.800 --> 01:06:38.070 align:middle line:84% Standards for Mathematics describe mathematical modeling 01:06:38.070 --> 01:06:39.130 align:middle line:90% as five steps. 01:06:39.130 --> 01:06:41.010 align:middle line:84% There are five things mathematicians do. 01:06:41.010 --> 01:06:42.970 align:middle line:84% They find a problem in the real world. 01:06:42.970 --> 01:06:45.730 align:middle line:84% They arrange that problem in some kind of structure, 01:06:45.730 --> 01:06:48.490 align:middle line:84% like an equation, or a table, or a schematic. 01:06:48.490 --> 01:06:51.340 align:middle line:84% They resolve the computation in that structure. 01:06:51.340 --> 01:06:54.390 align:middle line:84% They make a claim about how the answer relates 01:06:54.390 --> 01:06:55.210 align:middle line:90% to the real world. 01:06:55.210 --> 01:06:58.920 align:middle line:84% So it's 47 board feet, or 47 kittens, or 47 acres. 01:06:58.920 --> 01:07:01.440 align:middle line:84% And then they explain their reasoning. 01:07:01.440 --> 01:07:05.280 align:middle line:84% Our auto-graders can only assess one of those five things. 01:07:05.280 --> 01:07:06.960 align:middle line:84% Our auto-graders basically can only 01:07:06.960 --> 01:07:10.430 align:middle line:84% assess whether or not you like address the computational part 01:07:10.430 --> 01:07:10.930 align:middle line:90% of things. 01:07:10.930 --> 01:07:12.805 align:middle line:84% They basically don't work for the other four. 01:07:12.805 --> 01:07:17.370 align:middle line:84% If you use mathematics professionally, 01:07:17.370 --> 01:07:19.590 align:middle line:84% the only value you have, as a human being, 01:07:19.590 --> 01:07:22.047 align:middle line:84% are in the four things the graders can't evaluate. 01:07:22.047 --> 01:07:23.630 align:middle line:84% If you use mathematics professionally, 01:07:23.630 --> 01:07:25.210 align:middle line:84% you probably have machines compute 01:07:25.210 --> 01:07:27.250 align:middle line:84% all the answers of the things that you're doing. 01:07:27.250 --> 01:07:28.808 align:middle line:84% The value that you, as a human, bring 01:07:28.808 --> 01:07:30.850 align:middle line:84% is in figuring out what problems are interesting, 01:07:30.850 --> 01:07:33.010 align:middle line:84% and setting them up, and explaining them to other people, 01:07:33.010 --> 01:07:34.880 align:middle line:84% and putting the answer into some kind of context. 01:07:34.880 --> 01:07:36.338 align:middle line:84% So I think you're absolutely right. 01:07:36.338 --> 01:07:39.130 align:middle line:84% Now, what do we mean when we say it improves math? 01:07:39.130 --> 01:07:42.460 align:middle line:84% Almost always what we mean is that in a randomized controlled 01:07:42.460 --> 01:07:47.360 align:middle line:84% trial, people in the control condition 01:07:47.360 --> 01:07:51.200 align:middle line:84% get some average score, usually on an annual state test. 01:07:51.200 --> 01:07:52.960 align:middle line:84% And people in the experimental condition 01:07:52.960 --> 01:07:56.510 align:middle line:84% get a higher average score, again, on the same state test. 01:07:56.510 --> 01:07:58.810 align:middle line:84% That's what we usually mean by math that works-- 01:07:58.810 --> 01:08:01.150 align:middle line:84% by having a thing work in math, or any other subject, 01:08:01.150 --> 01:08:01.700 align:middle line:90% or things like that. 01:08:01.700 --> 01:08:02.750 align:middle line:90% We might not use a state test. 01:08:02.750 --> 01:08:04.750 align:middle line:84% We might use some other particular evaluation or things 01:08:04.750 --> 01:08:05.180 align:middle line:90% like that. 01:08:05.180 --> 01:08:07.700 align:middle line:84% But that's almost always the way these things are evaluated. 01:08:07.700 --> 01:08:08.840 align:middle line:90% And if those tests-- 01:08:08.840 --> 01:08:10.557 align:middle line:84% those tests are called math tests-- 01:08:10.557 --> 01:08:12.140 align:middle line:84% but they don't test you on everything, 01:08:12.140 --> 01:08:13.470 align:middle line:90% that's important in math. 01:08:13.470 --> 01:08:15.470 align:middle line:84% And we'll get into that more as we go along too. 01:08:15.470 --> 01:08:17.390 align:middle line:84% So you've identified a crucial issue. 01:08:17.390 --> 01:08:19.098 align:middle line:84% Let me try to give you a few other things 01:08:19.098 --> 01:08:21.390 align:middle line:90% to send you on your way with. 01:08:21.390 --> 01:08:24.240 align:middle line:84% There is a wonderful book by Morgan Ames called The Charisma 01:08:24.240 --> 01:08:28.327 align:middle line:84% Machine, which studies the One Laptop Per Child Program. 01:08:28.327 --> 01:08:29.910 align:middle line:84% It studies a particular implementation 01:08:29.910 --> 01:08:31.785 align:middle line:84% in Paraguay, which was kind of a cool country 01:08:31.785 --> 01:08:35.873 align:middle line:84% to study because you basically can know everything that's 01:08:35.873 --> 01:08:36.790 align:middle line:90% happening in Paraguay. 01:08:36.790 --> 01:08:38.310 align:middle line:90% It's kind of a small place. 01:08:38.310 --> 01:08:40.740 align:middle line:90% And Morgan was embedded there. 01:08:40.740 --> 01:08:44.399 align:middle line:84% She argues that there are two kinds of stances that people 01:08:44.399 --> 01:08:45.670 align:middle line:90% can take towards technology. 01:08:45.670 --> 01:08:47.520 align:middle line:84% One is the charismatic stance, the idea 01:08:47.520 --> 01:08:50.250 align:middle line:84% that these technologies are going to be new, and different, 01:08:50.250 --> 01:08:51.122 align:middle line:90% and transformative. 01:08:51.122 --> 01:08:53.580 align:middle line:84% And that the future will be brand new and different because 01:08:53.580 --> 01:08:54.279 align:middle line:90% of these things. 01:08:54.279 --> 01:08:57.210 align:middle line:84% And that's what Sal Khan sounds like in his TED Talk. 01:08:57.210 --> 01:08:59.800 align:middle line:84% An alternative stance is the tinkering stance, 01:08:59.800 --> 01:09:02.040 align:middle line:84% which is what Sal Khan stands like sounds 01:09:02.040 --> 01:09:06.420 align:middle line:84% like in his district administrative interview. 01:09:06.420 --> 01:09:09.060 align:middle line:84% That existing systems domesticate new technologies 01:09:09.060 --> 01:09:11.880 align:middle line:84% that a pretty good guide for the future of schooling 01:09:11.880 --> 01:09:12.520 align:middle line:90% and education. 01:09:12.520 --> 01:09:13.979 align:middle line:90% Technology is the past. 01:09:13.979 --> 01:09:16.390 align:middle line:84% And it's not that we can't make things better. 01:09:16.390 --> 01:09:18.090 align:middle line:84% But that making things better is hard, 01:09:18.090 --> 01:09:20.842 align:middle line:84% and it takes a lot of time, and it involves building capacity 01:09:20.842 --> 01:09:21.550 align:middle line:90% of the community. 01:09:21.550 --> 01:09:24.840 align:middle line:84% You could think of failure to disrupt 01:09:24.840 --> 01:09:28.142 align:middle line:84% is basically a celebration of the tinkering stance, which 01:09:28.142 --> 01:09:29.350 align:middle line:90% you don't have to agree with. 01:09:29.350 --> 01:09:31.830 align:middle line:84% But one of the hearts of the argument 01:09:31.830 --> 01:09:35.250 align:middle line:84% is that charismatics don't really help education systems. 01:09:35.250 --> 01:09:38.649 align:middle line:84% Making huge promises that we can't deliver on is not helpful. 01:09:38.649 --> 01:09:41.734 align:middle line:84% It diverts resources away from where they could be most useful. 01:09:41.734 --> 01:09:43.109 align:middle line:84% A much more useful thing to do is 01:09:43.109 --> 01:09:45.779 align:middle line:84% to be real realistic with school systems about what's 01:09:45.779 --> 01:09:49.710 align:middle line:84% possible to accept the fact that usually the things we improve 01:09:49.710 --> 01:09:52.952 align:middle line:84% are only in some subjects, but not others. 01:09:52.952 --> 01:09:54.910 align:middle line:84% And not even in the entirety of those subjects. 01:09:54.910 --> 01:09:56.260 align:middle line:84% But only in part of those subjects. 01:09:56.260 --> 01:09:57.660 align:middle line:84% And only for certain kinds of students 01:09:57.660 --> 01:09:58.868 align:middle line:90% in certain kinds of contexts. 01:09:58.868 --> 01:10:01.020 align:middle line:84% And it's still worth pursuing those things 01:10:01.020 --> 01:10:03.990 align:middle line:84% because the way you make education systems better is you 01:10:03.990 --> 01:10:07.290 align:middle line:84% just relentlessly try to improve little bits of them 01:10:07.290 --> 01:10:09.460 align:middle line:84% by a little amount over, and over, and over again. 01:10:09.460 --> 01:10:11.400 align:middle line:84% If you do that long enough, they get better. 01:10:11.400 --> 01:10:16.380 align:middle line:84% But if you want the underlying argument behind the whole book 01:10:16.380 --> 01:10:19.320 align:middle line:84% is like stop trying to disrupt things and instead, 01:10:19.320 --> 01:10:22.380 align:middle line:84% accept the limits of what can be accomplished 01:10:22.380 --> 01:10:23.860 align:middle line:90% with education technology. 01:10:23.860 --> 01:10:26.820 align:middle line:84% But also, recognize that even if the improvements we can make 01:10:26.820 --> 01:10:28.927 align:middle line:84% are small, small improvements compounded 01:10:28.927 --> 01:10:31.260 align:middle line:84% over lots of students, over a long period of time really 01:10:31.260 --> 01:10:31.760 align:middle line:90% matters. 01:10:31.760 --> 01:10:35.190 align:middle line:90% 01:10:35.190 --> 01:10:37.630 align:middle line:84% I want to say, briefly, and maybe I won't finish this, 01:10:37.630 --> 01:10:42.340 align:middle line:84% but personalization was a pedagogical idea. 01:10:42.340 --> 01:10:45.510 align:middle line:84% But it's also connected to broader political economies. 01:10:45.510 --> 01:10:47.010 align:middle line:84% I think I first brought this up when 01:10:47.010 --> 01:10:49.800 align:middle line:84% we talked about instructionism and constructionism too. 01:10:49.800 --> 01:10:53.050 align:middle line:84% That these pedagogies can be aligned with values. 01:10:53.050 --> 01:10:55.240 align:middle line:84% The instructionists say learning is hard. 01:10:55.240 --> 01:10:58.350 align:middle line:84% The constructionists say learning is natural and easy. 01:10:58.350 --> 01:11:01.270 align:middle line:84% That therefore, schools should be disciplined, 01:11:01.270 --> 01:11:02.880 align:middle line:90% rigorous, controlled. 01:11:02.880 --> 01:11:05.668 align:middle line:84% Schools should be free, democracy-inducing. 01:11:05.668 --> 01:11:07.960 align:middle line:84% There are values that get associated with these things. 01:11:07.960 --> 01:11:10.290 align:middle line:84% And there were values that got associated 01:11:10.290 --> 01:11:13.410 align:middle line:84% with personalization, in particular, the emphasis 01:11:13.410 --> 01:11:17.850 align:middle line:84% on the individual as the main unit of analysis 01:11:17.850 --> 01:11:20.040 align:middle line:84% when thinking about how to improve schools. 01:11:20.040 --> 01:11:22.000 align:middle line:84% To me, it's not at all clear that that's right. 01:11:22.000 --> 01:11:24.190 align:middle line:84% So one thing that people thought about a lot-- 01:11:24.190 --> 01:11:26.732 align:middle line:84% and this related to MOOCs, and intelligent tutors, and things 01:11:26.732 --> 01:11:28.980 align:middle line:84% like that-- is folks have said like, it's crazy 01:11:28.980 --> 01:11:31.270 align:middle line:84% how many different things we ask teachers to do. 01:11:31.270 --> 01:11:32.970 align:middle line:84% They need to design assessments, design 01:11:32.970 --> 01:11:35.760 align:middle line:84% lessons, grade assessments, give instruction, coach people, 01:11:35.760 --> 01:11:37.860 align:middle line:84% provide what the British call pastoral care, which 01:11:37.860 --> 01:11:39.750 align:middle line:84% is like taking care of people, and making them feel good, 01:11:39.750 --> 01:11:41.340 align:middle line:84% and helping them when they cry, and put on Band-Aids, 01:11:41.340 --> 01:11:42.215 align:middle line:90% and things like that. 01:11:42.215 --> 01:11:43.920 align:middle line:84% Why on Earth would we expect an adult 01:11:43.920 --> 01:11:45.660 align:middle line:90% to be good at all those things? 01:11:45.660 --> 01:11:48.790 align:middle line:84% What if some people are just really good instructors? 01:11:48.790 --> 01:11:50.790 align:middle line:84% What if we just have these rock star teachers, 01:11:50.790 --> 01:11:52.470 align:middle line:84% like Sal Khan and Sebastian Thrun, who 01:11:52.470 --> 01:11:54.370 align:middle line:84% are just better at doing direct instruction? 01:11:54.370 --> 01:11:56.460 align:middle line:84% Why are we asking every teacher in the country 01:11:56.460 --> 01:11:59.040 align:middle line:84% to be able to explain math, when Sal Khan is 01:11:59.040 --> 01:12:00.780 align:middle line:90% really good at explaining math? 01:12:00.780 --> 01:12:05.550 align:middle line:84% And then, we could unbundle that part of their work out. 01:12:05.550 --> 01:12:07.772 align:middle line:84% And instead, just have them focus on the things 01:12:07.772 --> 01:12:10.230 align:middle line:84% that Sal Khan can't do, which is putting Band-Aids on kids, 01:12:10.230 --> 01:12:12.000 align:middle line:84% and grading their feedback, and talking to their parents, 01:12:12.000 --> 01:12:13.083 align:middle line:90% and those kinds of things. 01:12:13.083 --> 01:12:16.140 align:middle line:90% So not crazy at the start of it. 01:12:16.140 --> 01:12:18.850 align:middle line:84% People have taken this idea even further, 01:12:18.850 --> 01:12:21.540 align:middle line:84% which is something like, what if we just unbundled schools 01:12:21.540 --> 01:12:22.240 align:middle line:90% entirely? 01:12:22.240 --> 01:12:24.890 align:middle line:84% Why do we pack all these different functions in schools? 01:12:24.890 --> 01:12:29.260 align:middle line:90% 01:12:29.260 --> 01:12:32.550 align:middle line:84% Why don't why don't we let people gain learning experiences 01:12:32.550 --> 01:12:33.840 align:middle line:90% wherever they want to? 01:12:33.840 --> 01:12:37.213 align:middle line:84% There is a great leftist thinker named Ivan Illich. 01:12:37.213 --> 01:12:39.630 align:middle line:84% We're going to read some of his book, Deschooling Society, 01:12:39.630 --> 01:12:42.450 align:middle line:84% later, which was written at the dawn of personal computing 01:12:42.450 --> 01:12:45.390 align:middle line:84% as an incredible prediction about the future of learning 01:12:45.390 --> 01:12:45.910 align:middle line:90% online. 01:12:45.910 --> 01:12:47.340 align:middle line:90% Totally extraordinary. 01:12:47.340 --> 01:12:50.370 align:middle line:84% But he was very taken by the ideas of this guy named 01:12:50.370 --> 01:12:53.670 align:middle line:84% Milton Friedman, who's a very famous economist, who 01:12:53.670 --> 01:12:56.075 align:middle line:84% believed really strongly in the power of markets. 01:12:56.075 --> 01:12:58.450 align:middle line:84% That governments weren't really good at solving problems. 01:12:58.450 --> 01:13:00.533 align:middle line:84% That markets were really good at solving problems. 01:13:00.533 --> 01:13:02.670 align:middle line:84% What if we got rid of schools and just 01:13:02.670 --> 01:13:06.570 align:middle line:84% had people engage in a marketplace of learning 01:13:06.570 --> 01:13:07.210 align:middle line:90% activities? 01:13:07.210 --> 01:13:10.000 align:middle line:84% What if we unbundled the different functions of schools? 01:13:10.000 --> 01:13:13.430 align:middle line:84% Why do you have to take math and English and physical education 01:13:13.430 --> 01:13:15.630 align:middle line:84% and sex education from the same institution? 01:13:15.630 --> 01:13:18.950 align:middle line:84% Why don't you just let people buy those learning experiences 01:13:18.950 --> 01:13:21.830 align:middle line:90% from wherever they want to? 01:13:21.830 --> 01:13:24.320 align:middle line:84% The horseshoe is here because this 01:13:24.320 --> 01:13:27.840 align:middle line:84% is part of the arguments where people on the left, 01:13:27.840 --> 01:13:29.960 align:middle line:84% people on the right, suddenly found themselves 01:13:29.960 --> 01:13:34.790 align:middle line:84% in of strange convergence around personalization. 01:13:34.790 --> 01:13:37.100 align:middle line:90% Maybe I'll pass this just to-- 01:13:37.100 --> 01:13:41.660 align:middle line:84% so the pedagogical ideas of personalization 01:13:41.660 --> 01:13:45.900 align:middle line:84% got connected to a broader political project-- 01:13:45.900 --> 01:13:48.120 align:middle line:84% which is enormously successful in the United States, 01:13:48.120 --> 01:13:50.078 align:middle line:84% particularly in conservative states right now-- 01:13:50.078 --> 01:13:53.540 align:middle line:84% of trying to refashion school systems 01:13:53.540 --> 01:13:56.120 align:middle line:84% with less of a focus on schools and much more of a focus 01:13:56.120 --> 01:13:59.090 align:middle line:90% on individual students. 01:13:59.090 --> 01:14:04.520 align:middle line:84% A very popular idea among people who advocate for this thing 01:14:04.520 --> 01:14:08.250 align:middle line:84% is to say, why do we give monies to schools to educate people? 01:14:08.250 --> 01:14:12.380 align:middle line:84% What if we took that money and gave it directly to students? 01:14:12.380 --> 01:14:15.680 align:middle line:84% Sometimes these systems were called vouchers. 01:14:15.680 --> 01:14:17.880 align:middle line:84% Like, don't send people to public school. 01:14:17.880 --> 01:14:20.240 align:middle line:84% Give them a voucher to buy any school they want. 01:14:20.240 --> 01:14:23.750 align:middle line:84% It turned out the word voucher was extremely unpopular 01:14:23.750 --> 01:14:26.120 align:middle line:84% and evoked a voucher that you would 01:14:26.120 --> 01:14:28.550 align:middle line:84% get like in an airport when your flight is 01:14:28.550 --> 01:14:31.160 align:middle line:84% canceled that doesn't pay for your hotel and things like that. 01:14:31.160 --> 01:14:33.020 align:middle line:84% For a while they experimented with this idea 01:14:33.020 --> 01:14:35.338 align:middle line:90% of student backpacks. 01:14:35.338 --> 01:14:37.380 align:middle line:84% Like, the money would go into a student backpack. 01:14:37.380 --> 01:14:39.810 align:middle line:84% But that evoked this idea of backpacks full of cash, 01:14:39.810 --> 01:14:41.570 align:middle line:90% which was kind of unseemly. 01:14:41.570 --> 01:14:46.340 align:middle line:84% So the term that folks use right now 01:14:46.340 --> 01:14:49.790 align:middle line:84% is to fund students instead of systems. 01:14:49.790 --> 01:14:51.330 align:middle line:84% Give money to individual students. 01:14:51.330 --> 01:14:54.980 align:middle line:84% These are all the states in 2023 that passed or were considering 01:14:54.980 --> 01:14:56.410 align:middle line:84% legislation that would say, we're 01:14:56.410 --> 01:14:57.900 align:middle line:84% going to take your property taxes. 01:14:57.900 --> 01:14:59.567 align:middle line:84% We're not going to give them to schools. 01:14:59.567 --> 01:15:01.610 align:middle line:84% We're going to give them to individual kids. 01:15:01.610 --> 01:15:03.290 align:middle line:84% And the individual kids can then use 01:15:03.290 --> 01:15:06.330 align:middle line:84% those dollars to buy whatever public school they want. 01:15:06.330 --> 01:15:07.830 align:middle line:90% Or, to buy a parochial school. 01:15:07.830 --> 01:15:10.490 align:middle line:84% Or, to pay more money and buy a private school. 01:15:10.490 --> 01:15:12.680 align:middle line:84% Or, any kind of things like that. 01:15:12.680 --> 01:15:15.590 align:middle line:84% AUDIENCE: Were those funds given to all students? 01:15:15.590 --> 01:15:17.240 align:middle line:90% Is there a criteria? 01:15:17.240 --> 01:15:19.100 align:middle line:84% JUSTIN REICH: The ideal, the ideal model 01:15:19.100 --> 01:15:21.080 align:middle line:90% is that everyone would get them. 01:15:21.080 --> 01:15:24.140 align:middle line:84% Like, if you believe in this free market system, 01:15:24.140 --> 01:15:25.970 align:middle line:84% the only way public schools should exist 01:15:25.970 --> 01:15:28.875 align:middle line:84% is if they can compete in the market 01:15:28.875 --> 01:15:30.000 align:middle line:90% for other kinds of schools. 01:15:30.000 --> 01:15:31.705 align:middle line:84% You can also imagine you don't even need schools. 01:15:31.705 --> 01:15:33.980 align:middle line:84% Like, what if you bought your math from Khan Academy, 01:15:33.980 --> 01:15:36.410 align:middle line:84% and your health class from CVS, and your language 01:15:36.410 --> 01:15:39.350 align:middle line:84% class from Rosetta Stone, and your sex ed class from the Big 01:15:39.350 --> 01:15:41.780 align:middle line:84% Valley Creation Science Museum, and your English class 01:15:41.780 --> 01:15:44.990 align:middle line:84% from the National Writing Project, and things like that? 01:15:44.990 --> 01:15:48.050 align:middle line:84% And this is like a broader political vision 01:15:48.050 --> 01:15:49.920 align:middle line:84% of what schooling could be in the future 01:15:49.920 --> 01:15:51.930 align:middle line:90% that online learning enabled. 01:15:51.930 --> 01:15:55.203 align:middle line:84% And to some extent, parts of it are enormously successful. 01:15:55.203 --> 01:15:57.620 align:middle line:84% I would say, the pandemic-- because people basically hated 01:15:57.620 --> 01:15:59.330 align:middle line:84% online learning, as a whole-- threw 01:15:59.330 --> 01:16:01.580 align:middle line:84% cold water on a whole bunch of these things. 01:16:01.580 --> 01:16:03.980 align:middle line:84% But a thing I want us to keep coming back 01:16:03.980 --> 01:16:06.590 align:middle line:84% to as we talk about things, like personalized learning, 01:16:06.590 --> 01:16:10.260 align:middle line:84% is that they can be construed as pedagogies. 01:16:10.260 --> 01:16:12.260 align:middle line:84% But these pedagogies are almost always connected 01:16:12.260 --> 01:16:16.580 align:middle line:84% to broader political movements about how we organize schools. 01:16:16.580 --> 01:16:19.040 align:middle line:84% The most political question you could possibly ask 01:16:19.040 --> 01:16:21.200 align:middle line:84% is how should we raise our children? 01:16:21.200 --> 01:16:24.110 align:middle line:84% In some ways, it seems like an extremely technical question, 01:16:24.110 --> 01:16:27.450 align:middle line:84% but it is a deeply moral, value-laden question as well. 01:16:27.450 --> 01:16:30.740 align:middle line:84% OK, we will continue this a little bit, 01:16:30.740 --> 01:16:33.360 align:middle line:84% and say some more things about it, and look at some more-- 01:16:33.360 --> 01:16:35.630 align:middle line:84% We'll probably get pretty deep into the evidence in the paper 01:16:35.630 --> 01:16:37.850 align:middle line:84% that you read, trying to get at some of the questions 01:16:37.850 --> 01:16:40.790 align:middle line:84% that Mari asked before, of what is the evidence base for how 01:16:40.790 --> 01:16:41.760 align:middle line:90% well these things work? 01:16:41.760 --> 01:16:44.177 align:middle line:84% Some of Avril's questions, like what does it mean to work? 01:16:44.177 --> 01:16:47.780 align:middle line:84% We'll do a bunch of that with intelligent tutors on Monday. 01:16:47.780 --> 01:16:49.140 align:middle line:90% Have a wonderful weekend. 01:16:49.140 --> 01:16:50.720 align:middle line:84% I look forward to getting and reading all your papers. 01:16:50.720 --> 01:16:52.762 align:middle line:84% And if I can help you with anything, let me know. 01:16:52.762 --> 01:16:55.040 align:middle line:90% Thanks, everybody. 01:16:55.040 --> 01:17:08.000 align:middle line:90%