WEBVTT 00:00:00.000 --> 00:00:01.992 align:middle line:90% [SQUEAKING] 00:00:01.992 --> 00:00:03.486 align:middle line:90% [RUSTLING] 00:00:03.486 --> 00:00:04.980 align:middle line:90% [CLICKING] 00:00:04.980 --> 00:00:12.950 align:middle line:90% 00:00:12.950 --> 00:00:14.900 align:middle line:84% JUSTIN REICH: I'm very interested to know 00:00:14.900 --> 00:00:18.590 align:middle line:84% your experiences from last class, what your thoughts were 00:00:18.590 --> 00:00:21.470 align:middle line:84% on the NETP discussion and the small group conversations 00:00:21.470 --> 00:00:22.937 align:middle line:90% that you had. 00:00:22.937 --> 00:00:24.770 align:middle line:84% Did anybody find themselves in conversations 00:00:24.770 --> 00:00:25.687 align:middle line:90% that were of interest? 00:00:25.687 --> 00:00:31.850 align:middle line:90% 00:00:31.850 --> 00:00:34.190 align:middle line:84% AUDIENCE: I think my group, it was 00:00:34.190 --> 00:00:37.550 align:middle line:84% a lot of discussion on how they themselves 00:00:37.550 --> 00:00:40.490 align:middle line:84% are trying to bring about these new innovations. 00:00:40.490 --> 00:00:44.030 align:middle line:84% But a lot of it's like teacher lashback. 00:00:44.030 --> 00:00:47.780 align:middle line:84% And some people are having a really hard time moving 00:00:47.780 --> 00:00:50.520 align:middle line:84% forwards, and then some people are doing a good job. 00:00:50.520 --> 00:00:52.640 align:middle line:84% And it was all of them agreeing that they 00:00:52.640 --> 00:00:56.393 align:middle line:84% need to get those who are proactive about it all together. 00:00:56.393 --> 00:00:57.810 align:middle line:84% I also thought it was interesting, 00:00:57.810 --> 00:01:00.480 align:middle line:84% I kind of proposed a question to the group 00:01:00.480 --> 00:01:05.400 align:middle line:84% about how they see all of these ideologies playing 00:01:05.400 --> 00:01:08.550 align:middle line:90% a part in higher education. 00:01:08.550 --> 00:01:11.370 align:middle line:84% A lot of the times, specifically the research that you guys do, 00:01:11.370 --> 00:01:13.710 align:middle line:84% is kind of focused on K through 12. 00:01:13.710 --> 00:01:16.710 align:middle line:84% But how can this really be implemented in colleges? 00:01:16.710 --> 00:01:18.510 align:middle line:84% And especially at a university like 00:01:18.510 --> 00:01:22.440 align:middle line:84% MIT, where you have professors being hired not because they're 00:01:22.440 --> 00:01:25.590 align:middle line:84% good teachers but because they're experts in the field. 00:01:25.590 --> 00:01:27.330 align:middle line:84% And I think that it just kind of prompted 00:01:27.330 --> 00:01:31.500 align:middle line:84% an interesting discussion about where schools 00:01:31.500 --> 00:01:35.430 align:middle line:84% need to prioritize, whether they're research institutions 00:01:35.430 --> 00:01:38.130 align:middle line:90% or educational facilities. 00:01:38.130 --> 00:01:40.020 align:middle line:90% JUSTIN REICH: Cool. 00:01:40.020 --> 00:01:40.960 align:middle line:90% Other folks? 00:01:40.960 --> 00:01:45.540 align:middle line:84% What else did you end up talking about with your new education 00:01:45.540 --> 00:01:47.790 align:middle line:84% school colleagues, or what else was interesting to you 00:01:47.790 --> 00:01:50.640 align:middle line:84% about the National Education Tech Plan discussion or anything 00:01:50.640 --> 00:01:51.140 align:middle line:90% else? 00:01:51.140 --> 00:01:58.510 align:middle line:90% 00:01:58.510 --> 00:01:59.890 align:middle line:90% Ty, and then Omar. 00:01:59.890 --> 00:02:01.630 align:middle line:84% AUDIENCE: Yeah, it was interesting. 00:02:01.630 --> 00:02:03.310 align:middle line:84% Omar and I were in the same group. 00:02:03.310 --> 00:02:04.300 align:middle line:90% AUDIENCE: Yeah. 00:02:04.300 --> 00:02:05.967 align:middle line:84% AUDIENCE: So we'll see if this overlaps. 00:02:05.967 --> 00:02:09.410 align:middle line:84% But there wasn't a ton of conversation in our group. 00:02:09.410 --> 00:02:13.120 align:middle line:84% But they did ask me about my time as a software engineer 00:02:13.120 --> 00:02:15.843 align:middle line:84% and how we thought about iteration 00:02:15.843 --> 00:02:17.260 align:middle line:84% in the software development space. 00:02:17.260 --> 00:02:19.360 align:middle line:84% And then they tried to start making parallels 00:02:19.360 --> 00:02:22.067 align:middle line:84% to how they try to implement it in their own schools. 00:02:22.067 --> 00:02:24.400 align:middle line:84% And we talked a little bit about some of the differences 00:02:24.400 --> 00:02:27.058 align:middle line:84% and where things might be more complex or harder. 00:02:27.058 --> 00:02:28.600 align:middle line:84% So it was an interesting conversation 00:02:28.600 --> 00:02:30.680 align:middle line:84% to try to take my experience and reapply it. 00:02:30.680 --> 00:02:31.763 align:middle line:90% JUSTIN REICH: That's cool. 00:02:31.763 --> 00:02:33.015 align:middle line:90% That's great. 00:02:33.015 --> 00:02:33.640 align:middle line:90% AUDIENCE: Yeah. 00:02:33.640 --> 00:02:36.143 align:middle line:84% So again, there wasn't that much discussion. 00:02:36.143 --> 00:02:38.060 align:middle line:84% You kind of went over what I was going to say. 00:02:38.060 --> 00:02:41.950 align:middle line:84% So I guess I'll touch on the first part of the session, which 00:02:41.950 --> 00:02:44.120 align:middle line:90% is the big conversation. 00:02:44.120 --> 00:02:45.110 align:middle line:90% I was reading the chat. 00:02:45.110 --> 00:02:47.050 align:middle line:84% And it was pretty cool to see everybody 00:02:47.050 --> 00:02:49.930 align:middle line:84% commenting on everything and seeing a bunch of examples 00:02:49.930 --> 00:02:53.590 align:middle line:84% and seeing either frustrations or revelations that they never 00:02:53.590 --> 00:02:59.200 align:middle line:84% realized about the message of focusing 00:02:59.200 --> 00:03:05.350 align:middle line:84% on not the technology itself, but what we can do, 00:03:05.350 --> 00:03:07.090 align:middle line:84% and then how technology can help and what 00:03:07.090 --> 00:03:08.770 align:middle line:90% we're trying to do first. 00:03:08.770 --> 00:03:11.710 align:middle line:84% And that revelation for a lot of professionals 00:03:11.710 --> 00:03:14.068 align:middle line:84% was pretty cool to see develop in the chat. 00:03:14.068 --> 00:03:14.860 align:middle line:90% JUSTIN REICH: Yeah. 00:03:14.860 --> 00:03:15.360 align:middle line:90% Yeah. 00:03:15.360 --> 00:03:17.960 align:middle line:84% You get to watch human development in real time. 00:03:17.960 --> 00:03:20.290 align:middle line:90% People encounter where they're-- 00:03:20.290 --> 00:03:24.430 align:middle line:84% I didn't follow it that closely because I was talking. 00:03:24.430 --> 00:03:26.320 align:middle line:84% Were there frustrations that were-- 00:03:26.320 --> 00:03:29.440 align:middle line:84% anything on the frustration side that stuck out to you 00:03:29.440 --> 00:03:30.285 align:middle line:90% or seemed memorable? 00:03:30.285 --> 00:03:31.910 align:middle line:84% AUDIENCE: Not necessarily frustrations, 00:03:31.910 --> 00:03:36.610 align:middle line:84% but sort of like, oh, that's why this is happening, 00:03:36.610 --> 00:03:39.470 align:middle line:84% or, OK, now this is starting to make sense. 00:03:39.470 --> 00:03:43.150 align:middle line:84% So it's sort of like their frustrations were cleared up, 00:03:43.150 --> 00:03:43.847 align:middle line:90% so to speak. 00:03:43.847 --> 00:03:44.680 align:middle line:90% JUSTIN REICH: Great. 00:03:44.680 --> 00:03:46.810 align:middle line:90% Good. 00:03:46.810 --> 00:03:47.520 align:middle line:90% Hi, Ramon. 00:03:47.520 --> 00:03:49.520 align:middle line:84% You've got to come up here and hang out with me. 00:03:49.520 --> 00:03:51.530 align:middle line:84% That's what happens when you get towards the end. 00:03:51.530 --> 00:03:51.940 align:middle line:90% Yeah. 00:03:51.940 --> 00:03:52.270 align:middle line:90% Asa. 00:03:52.270 --> 00:03:53.645 align:middle line:84% Asa and I were in the same group. 00:03:53.645 --> 00:03:55.495 align:middle line:90% We had a spicy one. 00:03:55.495 --> 00:03:56.120 align:middle line:90% AUDIENCE: Yeah. 00:03:56.120 --> 00:03:59.360 align:middle line:84% So a lot of the discussion was about-- 00:03:59.360 --> 00:04:04.160 align:middle line:84% a lot of the people in our group are the people 00:04:04.160 --> 00:04:07.410 align:middle line:84% designing the structures that they were trying to implement. 00:04:07.410 --> 00:04:11.180 align:middle line:84% And something that was interesting was this repeated, 00:04:11.180 --> 00:04:14.270 align:middle line:84% oh, we design something, we send it to the administration, 00:04:14.270 --> 00:04:17.040 align:middle line:84% and they're like, no, that's not exactly what we're looking for. 00:04:17.040 --> 00:04:17.579 align:middle line:90% Try again. 00:04:17.579 --> 00:04:19.970 align:middle line:84% And it's just kind of constantly trying 00:04:19.970 --> 00:04:22.790 align:middle line:84% to get something that they're like, OK, I 00:04:22.790 --> 00:04:28.910 align:middle line:84% like this, as opposed to them being able to effectively 00:04:28.910 --> 00:04:33.470 align:middle line:84% communicate the points that they want included, 00:04:33.470 --> 00:04:34.950 align:middle line:90% which was very interesting. 00:04:34.950 --> 00:04:35.870 align:middle line:84% JUSTIN REICH: Well, part of the way I 00:04:35.870 --> 00:04:37.745 align:middle line:84% read our group is that we had some people who 00:04:37.745 --> 00:04:41.120 align:middle line:84% were kind of middle-level leaders, 00:04:41.120 --> 00:04:44.960 align:middle line:84% and they were trying to convince more senior leaders 00:04:44.960 --> 00:04:45.930 align:middle line:90% to make change. 00:04:45.930 --> 00:04:47.900 align:middle line:84% And the senior leaders were much slower 00:04:47.900 --> 00:04:49.370 align:middle line:84% and less interested in that, which 00:04:49.370 --> 00:04:52.110 align:middle line:84% happens in basically every kind of institution. 00:04:52.110 --> 00:04:54.340 align:middle line:84% When you go and visit an institution, 00:04:54.340 --> 00:04:58.140 align:middle line:84% one way to think of it is it is designed exactly 00:04:58.140 --> 00:05:01.575 align:middle line:90% to do what it's doing right now. 00:05:01.575 --> 00:05:02.950 align:middle line:84% It functions the way it functions 00:05:02.950 --> 00:05:04.980 align:middle line:84% because there's some set of actors who 00:05:04.980 --> 00:05:06.660 align:middle line:90% want it to function that way. 00:05:06.660 --> 00:05:09.130 align:middle line:84% Often, those folks are people with power and control. 00:05:09.130 --> 00:05:11.680 align:middle line:84% And it exists the way it does because that's what they want. 00:05:11.680 --> 00:05:13.768 align:middle line:84% And then new people come along-- new people live 00:05:13.768 --> 00:05:15.310 align:middle line:84% in that environment and they're like, 00:05:15.310 --> 00:05:16.930 align:middle line:84% well, this could be better, and this could be better, 00:05:16.930 --> 00:05:18.400 align:middle line:84% and this could be better, and this could be better. 00:05:18.400 --> 00:05:20.020 align:middle line:84% And then more senior folks are like, eh, 00:05:20.020 --> 00:05:21.760 align:middle line:84% but we like the way we've done it all the time. 00:05:21.760 --> 00:05:23.718 align:middle line:84% And so it's not uncommon for people coming up-- 00:05:23.718 --> 00:05:25.860 align:middle line:84% I think really healthy organizations 00:05:25.860 --> 00:05:28.980 align:middle line:84% recognize, oh, we need to be constantly improving. 00:05:28.980 --> 00:05:32.430 align:middle line:84% And constantly improving means throwing away and destroying 00:05:32.430 --> 00:05:34.420 align:middle line:84% things that I have built over my career, 00:05:34.420 --> 00:05:36.300 align:middle line:84% because that's the only way you get better. 00:05:36.300 --> 00:05:38.940 align:middle line:84% But it's pretty understandable that people 00:05:38.940 --> 00:05:39.898 align:middle line:90% wouldn't feel that way. 00:05:39.898 --> 00:05:42.190 align:middle line:84% It'd be like, I've spent my career building this thing. 00:05:42.190 --> 00:05:44.010 align:middle line:84% Why don't we just keep doing this thing? 00:05:44.010 --> 00:05:46.820 align:middle line:84% And so they were struggling with-- 00:05:46.820 --> 00:05:48.582 align:middle line:84% and an interesting place where lots 00:05:48.582 --> 00:05:50.040 align:middle line:84% of people and lots of organizations 00:05:50.040 --> 00:05:52.970 align:middle line:84% find themselves is, OK, we need to do something different, 00:05:52.970 --> 00:05:55.520 align:middle line:84% but the people in charge aren't really supportive. 00:05:55.520 --> 00:05:58.790 align:middle line:90% How do we do it anyway? 00:05:58.790 --> 00:06:00.440 align:middle line:90% Where change comes from-- 00:06:00.440 --> 00:06:01.000 align:middle line:90% I don't know. 00:06:01.000 --> 00:06:02.920 align:middle line:90% Maybe I mentioned this. 00:06:02.920 --> 00:06:05.180 align:middle line:84% I can't remember if I mentioned this particular story, 00:06:05.180 --> 00:06:07.180 align:middle line:84% but one of the people that I worked with on some 00:06:07.180 --> 00:06:09.263 align:middle line:84% of the content that's in Iterate is this guy Peter 00:06:09.263 --> 00:06:12.722 align:middle line:84% Senge, who's at the Management School at Sloan. 00:06:12.722 --> 00:06:14.680 align:middle line:84% And a neat thing about working with him is he's 00:06:14.680 --> 00:06:16.840 align:middle line:84% done a ton of stuff in education, but lots of stuff 00:06:16.840 --> 00:06:18.670 align:middle line:90% in many other sectors. 00:06:18.670 --> 00:06:22.445 align:middle line:84% And it comes up periodically, like, man, teachers, 00:06:22.445 --> 00:06:24.070 align:middle line:84% lots of people in education, just don't 00:06:24.070 --> 00:06:25.840 align:middle line:90% feel empowered to make change. 00:06:25.840 --> 00:06:28.420 align:middle line:84% They feel like, look, I got hired in this particular role. 00:06:28.420 --> 00:06:29.463 align:middle line:90% I'm a math teacher. 00:06:29.463 --> 00:06:30.380 align:middle line:90% I'm a history teacher. 00:06:30.380 --> 00:06:31.463 align:middle line:90% I'm a chemistry professor. 00:06:31.463 --> 00:06:33.628 align:middle line:84% It's not my job to fix the university. 00:06:33.628 --> 00:06:35.920 align:middle line:84% And Peter once said something to me along the lines of, 00:06:35.920 --> 00:06:39.620 align:middle line:84% everyone in every industry feels that way. 00:06:39.620 --> 00:06:41.920 align:middle line:84% CEOs don't feel like they're empowered to make change 00:06:41.920 --> 00:06:42.470 align:middle line:90% in the institution. 00:06:42.470 --> 00:06:43.630 align:middle line:84% Nobody feels like they're empowered 00:06:43.630 --> 00:06:44.750 align:middle line:90% to make change in institutions. 00:06:44.750 --> 00:06:46.000 align:middle line:84% But change still happens all the time 00:06:46.000 --> 00:06:48.490 align:middle line:84% because people who don't necessarily feel empowered still 00:06:48.490 --> 00:06:50.670 align:middle line:84% take it upon themselves to try new things, 00:06:50.670 --> 00:06:52.430 align:middle line:90% put new things in the world. 00:06:52.430 --> 00:06:54.270 align:middle line:90% And I don't know. 00:06:54.270 --> 00:06:56.978 align:middle line:84% It's a story I like to retell educators 00:06:56.978 --> 00:06:59.520 align:middle line:84% because it's like, this is not an education-specific problem. 00:06:59.520 --> 00:07:00.840 align:middle line:90% This is a human problem. 00:07:00.840 --> 00:07:01.490 align:middle line:90% AUDIENCE: Yeah. 00:07:01.490 --> 00:07:03.865 align:middle line:84% I thought what Lou said, from the University of Kentucky, 00:07:03.865 --> 00:07:06.620 align:middle line:84% was really fascinating because I'm actually from Kentucky. 00:07:06.620 --> 00:07:09.680 align:middle line:84% And I worked at an organization with her before 00:07:09.680 --> 00:07:11.460 align:middle line:84% called the Kentucky Student Voice Team. 00:07:11.460 --> 00:07:11.780 align:middle line:90% JUSTIN REICH: Oh, really? 00:07:11.780 --> 00:07:12.270 align:middle line:90% Cool. 00:07:12.270 --> 00:07:14.145 align:middle line:84% AUDIENCE: And the way that they impact change 00:07:14.145 --> 00:07:16.860 align:middle line:84% is by putting students in stakeholder positions. 00:07:16.860 --> 00:07:20.570 align:middle line:84% So instead of influencing change through ineffective ways, 00:07:20.570 --> 00:07:23.250 align:middle line:84% like administration, they put students at the forefront. 00:07:23.250 --> 00:07:26.233 align:middle line:84% So she's a really big leader in driving, 00:07:26.233 --> 00:07:27.900 align:middle line:84% I guess, educational change in Kentucky. 00:07:27.900 --> 00:07:28.700 align:middle line:90% So I thought it was cool. 00:07:28.700 --> 00:07:29.370 align:middle line:90% JUSTIN REICH: That's great. 00:07:29.370 --> 00:07:30.510 align:middle line:84% Oh, I'm glad you had that connection. 00:07:30.510 --> 00:07:31.010 align:middle line:90% That's fun. 00:07:31.010 --> 00:07:33.890 align:middle line:84% How did you-- so she, at the university, 00:07:33.890 --> 00:07:36.530 align:middle line:84% ran the student voice thing that high school 00:07:36.530 --> 00:07:39.210 align:middle line:84% students across Kentucky could participate in and stuff or? 00:07:39.210 --> 00:07:39.470 align:middle line:90% AUDIENCE: Yeah. 00:07:39.470 --> 00:07:41.700 align:middle line:84% So it was born out of something called the Prichard Committee, 00:07:41.700 --> 00:07:43.620 align:middle line:84% but now it's an independent organization. 00:07:43.620 --> 00:07:48.410 align:middle line:84% And the, I guess, organization has adults and younger students 00:07:48.410 --> 00:07:50.610 align:middle line:90% as co-peers. 00:07:50.610 --> 00:07:52.800 align:middle line:84% They put them on the same level of the hierarchy 00:07:52.800 --> 00:07:56.130 align:middle line:84% to work together to, I guess, craft a bunch of initiatives 00:07:56.130 --> 00:07:57.460 align:middle line:90% to improve Kentucky schools. 00:07:57.460 --> 00:07:57.630 align:middle line:90% JUSTIN REICH: Yeah. 00:07:57.630 --> 00:07:58.130 align:middle line:90% Yeah. 00:07:58.130 --> 00:08:00.750 align:middle line:84% There are all kinds of cool-- education technology 00:08:00.750 --> 00:08:03.510 align:middle line:84% has tons of cool stuff like that, in part 00:08:03.510 --> 00:08:06.600 align:middle line:84% because lots of young people can get interested and excited 00:08:06.600 --> 00:08:08.940 align:middle line:84% by technology and so bring them in into leadership 00:08:08.940 --> 00:08:10.290 align:middle line:90% roles, partnership roles. 00:08:10.290 --> 00:08:11.232 align:middle line:90% Cities have that too. 00:08:11.232 --> 00:08:13.440 align:middle line:84% There's a group in Boston called the Hyde Square Task 00:08:13.440 --> 00:08:17.370 align:middle line:84% Force, which does all kinds of-- they usually do more, 00:08:17.370 --> 00:08:20.430 align:middle line:84% I don't know, community organizing, kind of adversarial, 00:08:20.430 --> 00:08:22.430 align:middle line:84% like why are you doing this to us? 00:08:22.430 --> 00:08:23.680 align:middle line:90% Why are you doing it this way? 00:08:23.680 --> 00:08:25.200 align:middle line:84% They have a whole campaign they're 00:08:25.200 --> 00:08:27.717 align:middle line:84% doing right now against either Shaw's or Stop & Shop, 00:08:27.717 --> 00:08:29.550 align:middle line:84% which is separate from their education work. 00:08:29.550 --> 00:08:32.070 align:middle line:84% But they've gone to a bunch of grocery stores 00:08:32.070 --> 00:08:34.799 align:middle line:84% in neighborhoods around Boston and bought the same thing 00:08:34.799 --> 00:08:37.140 align:middle line:84% in each store and basically found 00:08:37.140 --> 00:08:41.179 align:middle line:84% that in some minoritized, poverty-impacted neighborhoods, 00:08:41.179 --> 00:08:42.929 align:middle line:84% the groceries are more expensive than they 00:08:42.929 --> 00:08:44.887 align:middle line:84% are in Newton and Wellesley or stuff like that. 00:08:44.887 --> 00:08:46.020 align:middle line:90% And they're holding a-- 00:08:46.020 --> 00:08:48.640 align:middle line:84% Stop & Shop is not happy with these kids 00:08:48.640 --> 00:08:49.807 align:middle line:90% being a thorn in their side. 00:08:49.807 --> 00:08:50.307 align:middle line:90% It's great. 00:08:50.307 --> 00:08:51.190 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:08:51.190 --> 00:08:52.370 align:middle line:84% JUSTIN REICH: Yeah, yeah, yeah, yeah. 00:08:52.370 --> 00:08:54.760 align:middle line:84% They're pretty good at getting press for what they do 00:08:54.760 --> 00:08:56.110 align:middle line:90% and things like that. 00:08:56.110 --> 00:08:58.660 align:middle line:84% They'll probably have to respond at some point. 00:08:58.660 --> 00:08:59.830 align:middle line:90% Good. 00:08:59.830 --> 00:09:03.440 align:middle line:84% Well, I enjoyed letting you all connect with them. 00:09:03.440 --> 00:09:05.740 align:middle line:84% And it sounds like the meeting-- every time 00:09:05.740 --> 00:09:07.460 align:middle line:84% you go into breakout rooms, it's uneven. 00:09:07.460 --> 00:09:09.085 align:middle line:84% But hopefully it's a good sense for you 00:09:09.085 --> 00:09:11.770 align:middle line:84% to meet some of the people who are in higher education, who 00:09:11.770 --> 00:09:15.040 align:middle line:84% are trying to make the changes that we're going to talk about, 00:09:15.040 --> 00:09:17.510 align:middle line:84% in the past few weeks and for the rest of the semester. 00:09:17.510 --> 00:09:20.230 align:middle line:84% So hopefully that was a neat opportunity. 00:09:20.230 --> 00:09:21.970 align:middle line:84% Why don't you take a couple of minutes, 00:09:21.970 --> 00:09:25.010 align:middle line:84% before I launch into things, just to chat with each other, 00:09:25.010 --> 00:09:26.950 align:middle line:84% and then we can surface some of these ideas. 00:09:26.950 --> 00:09:29.140 align:middle line:84% But the learning journal prompt that I 00:09:29.140 --> 00:09:31.570 align:middle line:84% had for you was to talk about, what role 00:09:31.570 --> 00:09:37.600 align:middle line:84% does technology play in your MIT education thus far? 00:09:37.600 --> 00:09:39.548 align:middle line:84% And so talk to the person next to you. 00:09:39.548 --> 00:09:41.590 align:middle line:84% Some of you might not be from MIT, or some of you 00:09:41.590 --> 00:09:42.760 align:middle line:84% might be in graduate school and so you 00:09:42.760 --> 00:09:44.870 align:middle line:84% have some undergraduate connections to this as well. 00:09:44.870 --> 00:09:45.953 align:middle line:90% Those things are all fine. 00:09:45.953 --> 00:09:49.210 align:middle line:90% Technology and higher education. 00:09:49.210 --> 00:09:53.080 align:middle line:84% Typically, the things that are most useful to talk about 00:09:53.080 --> 00:09:57.130 align:middle line:84% are either systems or tools or things 00:09:57.130 --> 00:09:59.740 align:middle line:84% that you think are particularly helpful or useful, 00:09:59.740 --> 00:10:02.470 align:middle line:84% systems or tools or things that are particularly 00:10:02.470 --> 00:10:05.980 align:middle line:84% damaging or harmful or pointless or a waste of your time, 00:10:05.980 --> 00:10:10.180 align:middle line:84% and then also anything that has started 00:10:10.180 --> 00:10:12.640 align:middle line:84% to seem really familiar to you, but actually you 00:10:12.640 --> 00:10:15.910 align:middle line:84% think has interesting, unfamiliar qualities or things 00:10:15.910 --> 00:10:18.592 align:middle line:84% that feel standard and boring, but actually they're 00:10:18.592 --> 00:10:21.050 align:middle line:84% important, maybe in ways that we don't want to think about. 00:10:21.050 --> 00:10:23.150 align:middle line:84% So turn to the two or three people next to you, 00:10:23.150 --> 00:10:24.650 align:middle line:84% talk for a few minutes about, if you 00:10:24.650 --> 00:10:27.040 align:middle line:84% were to explain to someone not from your undergraduate 00:10:27.040 --> 00:10:29.367 align:middle line:84% institution the role that technology plays there, 00:10:29.367 --> 00:10:31.700 align:middle line:84% what would be the couple of stories that you would tell? 00:10:31.700 --> 00:10:32.620 align:middle line:90% Ready? 00:10:32.620 --> 00:10:34.246 align:middle line:90% Go. 00:10:34.246 --> 00:10:35.920 align:middle line:90% [SIDE CONVERSATION] 00:10:35.920 --> 00:10:37.090 align:middle line:90% JUSTIN REICH: OK. 00:10:37.090 --> 00:10:38.180 align:middle line:90% Come on back. 00:10:38.180 --> 00:10:40.208 align:middle line:84% AUDIENCE: You need to learn the technology. 00:10:40.208 --> 00:10:41.500 align:middle line:90% Because of that, they believe-- 00:10:41.500 --> 00:10:43.610 align:middle line:84% JUSTIN REICH: Be great to keep trying to hear from everybody. 00:10:43.610 --> 00:10:45.990 align:middle line:84% So would love to have some folks who haven't shared yet. 00:10:45.990 --> 00:10:50.480 align:middle line:84% Would love to hear, what are some MIT highlights or lowlights 00:10:50.480 --> 00:10:52.760 align:middle line:90% or distinctive parts of-- 00:10:52.760 --> 00:10:55.640 align:middle line:84% or in your undergraduate program, distinctive parts 00:10:55.640 --> 00:10:57.860 align:middle line:90% of technologies? 00:10:57.860 --> 00:10:59.300 align:middle line:90% What do you got, Jaron? 00:10:59.300 --> 00:11:03.980 align:middle line:84% AUDIENCE: My group was discussing MIT's use of Canvas. 00:11:03.980 --> 00:11:09.200 align:middle line:84% And pre-COVID, MIT had this weird proprietary thing, 00:11:09.200 --> 00:11:10.290 align:middle line:90% where they had their own-- 00:11:10.290 --> 00:11:11.130 align:middle line:84% JUSTIN REICH: What was that called? 00:11:11.130 --> 00:11:11.720 align:middle line:90% AUDIENCE: Stellar. 00:11:11.720 --> 00:11:12.180 align:middle line:90% JUSTIN REICH: Stellar. 00:11:12.180 --> 00:11:12.410 align:middle line:90% Yeah. 00:11:12.410 --> 00:11:13.035 align:middle line:90% AUDIENCE: Yeah. 00:11:13.035 --> 00:11:16.670 align:middle line:84% So they had-- each class had their own different Stellar 00:11:16.670 --> 00:11:18.990 align:middle line:84% website, and it wasn't standardized at all. 00:11:18.990 --> 00:11:20.670 align:middle line:90% And it was kind of a mess. 00:11:20.670 --> 00:11:24.550 align:middle line:84% It had this weird, outdated, 1990s internet-era aesthetic 00:11:24.550 --> 00:11:25.050 align:middle line:90% to it. 00:11:25.050 --> 00:11:26.660 align:middle line:90% It was pretty horrible. 00:11:26.660 --> 00:11:29.240 align:middle line:84% And then they had this proprietary thing, 00:11:29.240 --> 00:11:31.400 align:middle line:84% called the Learning Module, which 00:11:31.400 --> 00:11:35.610 align:middle line:84% was basically Canvas, like MIT was just trying to make Canvas. 00:11:35.610 --> 00:11:37.790 align:middle line:84% But then during the pandemic, I guess 00:11:37.790 --> 00:11:40.290 align:middle line:84% they decided to make Canvas a much more standardized thing. 00:11:40.290 --> 00:11:44.040 align:middle line:84% And so I find that to be very helpful for keeping everything 00:11:44.040 --> 00:11:46.810 align:middle line:84% organized and having assignments done through Canvas, 00:11:46.810 --> 00:11:48.610 align:middle line:84% as opposed to being like, oh crap, 00:11:48.610 --> 00:11:50.430 align:middle line:84% what's that website I have to go to? 00:11:50.430 --> 00:11:51.870 align:middle line:90% Search up Stellar MIT. 00:11:51.870 --> 00:11:53.850 align:middle line:90% And it was weird. 00:11:53.850 --> 00:11:57.300 align:middle line:84% But I feel like, because of that, a lot of professors 00:11:57.300 --> 00:12:00.060 align:middle line:84% don't fully utilize all the different things that Canvas 00:12:00.060 --> 00:12:02.580 align:middle line:84% has to offer, especially when it comes to grades 00:12:02.580 --> 00:12:03.630 align:middle line:90% and putting grades. 00:12:03.630 --> 00:12:09.790 align:middle line:84% Professors and TAs don't care about inputting grades. 00:12:09.790 --> 00:12:12.480 align:middle line:84% And a lot of students, I feel, would really 00:12:12.480 --> 00:12:15.480 align:middle line:84% like to see real-time feedback on their grades 00:12:15.480 --> 00:12:18.910 align:middle line:84% and how that actively affects their final grade in the class. 00:12:18.910 --> 00:12:21.630 align:middle line:90% But that just isn't a-- 00:12:21.630 --> 00:12:22.810 align:middle line:90% professors don't do that. 00:12:22.810 --> 00:12:22.950 align:middle line:90% JUSTIN REICH: Yeah. 00:12:22.950 --> 00:12:23.940 align:middle line:90% Yeah. 00:12:23.940 --> 00:12:26.235 align:middle line:84% And I mean, another thing that's unusual-- 00:12:26.235 --> 00:12:28.360 align:middle line:84% we'll talk about this more as the semester goes on, 00:12:28.360 --> 00:12:30.027 align:middle line:84% but it does seem a little weird that you 00:12:30.027 --> 00:12:33.750 align:middle line:84% would have the exact same Canvas template for a graduate seminar 00:12:33.750 --> 00:12:36.930 align:middle line:84% on chemistry synthesis and an undergraduate seminar 00:12:36.930 --> 00:12:38.740 align:middle line:90% on Renaissance poetry. 00:12:38.740 --> 00:12:43.095 align:middle line:84% And if you're studying really different kinds of things, 00:12:43.095 --> 00:12:44.470 align:middle line:84% does it really make sense to have 00:12:44.470 --> 00:12:46.970 align:middle line:84% the same structure for all of those different kinds of things? 00:12:46.970 --> 00:12:48.650 align:middle line:84% It seems like there should be some differences there. 00:12:48.650 --> 00:12:50.090 align:middle line:84% What were you going to say, Emily? 00:12:50.090 --> 00:12:53.260 align:middle line:84% AUDIENCE: We talked about MITx, which is like-- 00:12:53.260 --> 00:12:56.180 align:middle line:84% physics use it, and then a lot of math classes use it. 00:12:56.180 --> 00:12:59.020 align:middle line:90% And I like how it's-- 00:12:59.020 --> 00:13:01.527 align:middle line:84% it's sometimes recordings of a concept, 00:13:01.527 --> 00:13:03.860 align:middle line:84% introducing you to a concept, or it could be in writing, 00:13:03.860 --> 00:13:05.590 align:middle line:84% but it's not as dense as a textbook, 00:13:05.590 --> 00:13:06.610 align:middle line:84% where you're kind of like, oh, I don't 00:13:06.610 --> 00:13:07.780 align:middle line:90% want to start reading this. 00:13:07.780 --> 00:13:10.940 align:middle line:84% And then it's like, right after, when they introduce it, 00:13:10.940 --> 00:13:13.073 align:middle line:84% you have check questions, where it's 00:13:13.073 --> 00:13:14.740 align:middle line:84% kind of checking for your understanding. 00:13:14.740 --> 00:13:16.823 align:middle line:84% And it immediately tells if you're right or wrong, 00:13:16.823 --> 00:13:19.330 align:middle line:84% which is really nice, because for math, I feel like-- 00:13:19.330 --> 00:13:22.023 align:middle line:84% so when you're told, oh, this is the right answer, 00:13:22.023 --> 00:13:23.440 align:middle line:84% you're more focused on the process 00:13:23.440 --> 00:13:25.150 align:middle line:84% of getting to it rather than the answer 00:13:25.150 --> 00:13:27.340 align:middle line:84% itself, which is a lot more beneficial when 00:13:27.340 --> 00:13:28.090 align:middle line:90% you're doing math. 00:13:28.090 --> 00:13:29.465 align:middle line:84% You care about the process of how 00:13:29.465 --> 00:13:31.490 align:middle line:84% to get to that answer and not the answer itself. 00:13:31.490 --> 00:13:33.550 align:middle line:84% So I think it's really, really nice 00:13:33.550 --> 00:13:36.760 align:middle line:84% to have that platform for especially math classes. 00:13:36.760 --> 00:13:38.260 align:middle line:84% JUSTIN REICH: And you could imagine, 00:13:38.260 --> 00:13:40.430 align:middle line:84% for all of those autograded problems 00:13:40.430 --> 00:13:44.300 align:middle line:84% you've ever done, five years ago, you very likely would 00:13:44.300 --> 00:13:46.650 align:middle line:90% have done those Psets on paper. 00:13:46.650 --> 00:13:48.492 align:middle line:84% You would have turned the Psets into a TA. 00:13:48.492 --> 00:13:51.110 align:middle line:84% A TA would have gotten hundreds of them. 00:13:51.110 --> 00:13:53.260 align:middle line:84% It would have taken them a week, 10 days, 00:13:53.260 --> 00:13:55.260 align:middle line:84% two weeks to turn around or something like that. 00:13:55.260 --> 00:13:57.630 align:middle line:84% And so, by the time it gets handed back to you-- 00:13:57.630 --> 00:14:00.080 align:middle line:84% you might get the answer key sooner. 00:14:00.080 --> 00:14:01.460 align:middle line:90% By the time you get-- 00:14:01.460 --> 00:14:03.710 align:middle line:84% it can be days or weeks after doing 00:14:03.710 --> 00:14:05.370 align:middle line:84% a problem that you figure out whether you got it right or not. 00:14:05.370 --> 00:14:06.660 align:middle line:84% And you just forgot how you did it 00:14:06.660 --> 00:14:08.070 align:middle line:84% or what you were thinking about at the time. 00:14:08.070 --> 00:14:10.310 align:middle line:84% So you could definitely imagine some real advantages 00:14:10.310 --> 00:14:12.980 align:middle line:84% to that kind of instant-- it can change the way study 00:14:12.980 --> 00:14:16.020 align:middle line:84% groups operate too, both for good and bad. 00:14:16.020 --> 00:14:18.380 align:middle line:84% But you could imagine that study groups previously 00:14:18.380 --> 00:14:20.925 align:middle line:84% would have been like, OK, let's all write this Pset. 00:14:20.925 --> 00:14:22.550 align:middle line:84% They all write the Pset and turn it in. 00:14:22.550 --> 00:14:24.380 align:middle line:90% You're like, I hope that worked. 00:14:24.380 --> 00:14:27.990 align:middle line:84% But when you have the autograding, you can be like, 00:14:27.990 --> 00:14:29.790 align:middle line:84% wait, half of us did not get this right. 00:14:29.790 --> 00:14:31.423 align:middle line:90% Why didn't we? 00:14:31.423 --> 00:14:32.840 align:middle line:84% What can we learn from each other? 00:14:32.840 --> 00:14:35.720 align:middle line:84% There's all kinds of ways that that instant feedback 00:14:35.720 --> 00:14:38.030 align:middle line:84% can change the relationships that you 00:14:38.030 --> 00:14:40.550 align:middle line:90% have with that learning. 00:14:40.550 --> 00:14:41.520 align:middle line:90% Any other thoughts? 00:14:41.520 --> 00:14:42.578 align:middle line:90% Yeah, Sabrina. 00:14:42.578 --> 00:14:44.870 align:middle line:84% AUDIENCE: I was going to say, something like Notability 00:14:44.870 --> 00:14:49.130 align:middle line:84% has really transformed the way I, for example, organize all 00:14:49.130 --> 00:14:51.260 align:middle line:84% of my classwork and the fact that everyone 00:14:51.260 --> 00:14:55.400 align:middle line:84% uses their iPads now instead of going to paper and pencil. 00:14:55.400 --> 00:14:58.315 align:middle line:84% I think, yeah, just the fact that we 00:14:58.315 --> 00:14:59.690 align:middle line:84% can have all of our lecture notes 00:14:59.690 --> 00:15:01.065 align:middle line:84% and then upload it to Notability. 00:15:01.065 --> 00:15:03.810 align:middle line:84% We can annotate it, and then do all of our homework on here too, 00:15:03.810 --> 00:15:06.450 align:middle line:84% and be able to compare screens and things like that. 00:15:06.450 --> 00:15:07.790 align:middle line:90% And I think that's really-- 00:15:07.790 --> 00:15:10.430 align:middle line:84% before, I only ever really used paper and pencil. 00:15:10.430 --> 00:15:12.530 align:middle line:84% And then switching-- the fact that MIT gave out 00:15:12.530 --> 00:15:13.920 align:middle line:90% iPads to everyone. 00:15:13.920 --> 00:15:17.900 align:middle line:84% And then, because we have the resources to have Notability 00:15:17.900 --> 00:15:23.090 align:middle line:84% for free and the fact that we submit everything electronically 00:15:23.090 --> 00:15:25.980 align:middle line:84% now has kind of been a little bit of a game changer. 00:15:25.980 --> 00:15:28.640 align:middle line:84% JUSTIN REICH: Does Notability have any social features? 00:15:28.640 --> 00:15:31.610 align:middle line:84% Do you just have your own Notability, or are there ways-- 00:15:31.610 --> 00:15:34.880 align:middle line:84% AUDIENCE: They've recently added some share your notes 00:15:34.880 --> 00:15:38.763 align:middle line:90% kind of gallery search thing. 00:15:38.763 --> 00:15:39.555 align:middle line:90% JUSTIN REICH: Good. 00:15:39.555 --> 00:15:39.810 align:middle line:90% Cool. 00:15:39.810 --> 00:15:41.810 align:middle line:84% AUDIENCE: There's also the good old screenshot-- 00:15:41.810 --> 00:15:44.250 align:middle line:90% the iPads are usually iCloud. 00:15:44.250 --> 00:15:45.450 align:middle line:90% You can just text it. 00:15:45.450 --> 00:15:46.290 align:middle line:84% JUSTIN REICH: To the next person. 00:15:46.290 --> 00:15:47.350 align:middle line:90% Yeah, yeah, yeah, yeah. 00:15:47.350 --> 00:15:47.850 align:middle line:90% Good. 00:15:47.850 --> 00:15:48.780 align:middle line:90% Goo.d 00:15:48.780 --> 00:15:50.850 align:middle line:84% AUDIENCE: You can dual screen on the iPad, which 00:15:50.850 --> 00:15:52.510 align:middle line:90% isn't necessarily an aspect-- 00:15:52.510 --> 00:15:54.400 align:middle line:84% well, Notability you can also dual screen. 00:15:54.400 --> 00:15:56.040 align:middle line:84% So you pull up someone else's work. 00:15:56.040 --> 00:15:57.870 align:middle line:90% [LAUGHTER] 00:15:57.870 --> 00:16:01.350 align:middle line:84% JUSTIN REICH: There's some very specific applications for some 00:16:01.350 --> 00:16:02.590 align:middle line:90% of those kinds of things. 00:16:02.590 --> 00:16:03.360 align:middle line:90% Good. 00:16:03.360 --> 00:16:04.562 align:middle line:90% Other thoughts? 00:16:04.562 --> 00:16:05.520 align:middle line:90% What do you have, Omar? 00:16:05.520 --> 00:16:09.810 align:middle line:84% AUDIENCE: We had the same thought of CAT-SOOP has been 00:16:09.810 --> 00:16:12.945 align:middle line:84% kind of revolutionary, I guess, in a way that-- 00:16:12.945 --> 00:16:14.320 align:middle line:84% JUSTIN REICH: I've heard of this, 00:16:14.320 --> 00:16:17.170 align:middle line:84% but you've got to tell us what it is again. 00:16:17.170 --> 00:16:19.140 align:middle line:84% AUDIENCE: CAT-SOOP is an automatic tutor 00:16:19.140 --> 00:16:20.770 align:middle line:90% for 6.01 problems. 00:16:20.770 --> 00:16:22.170 align:middle line:90% AUDIENCE: For 6.01? 00:16:22.170 --> 00:16:24.870 align:middle line:84% AUDIENCE: It used to be an old introductory computer science 00:16:24.870 --> 00:16:27.580 align:middle line:84% course that was required for most physics majors, 00:16:27.580 --> 00:16:31.680 align:middle line:84% but they phased that out three years ago. 00:16:31.680 --> 00:16:33.580 align:middle line:84% I don't remember what 6.01 specifically was, 00:16:33.580 --> 00:16:35.230 align:middle line:84% but that's what CAT-SOOP stands for. 00:16:35.230 --> 00:16:37.970 align:middle line:90% It's a recursive acronym. 00:16:37.970 --> 00:16:41.110 align:middle line:90% But I guess, in a way-- 00:16:41.110 --> 00:16:42.250 align:middle line:90% I never got to use Stellar. 00:16:42.250 --> 00:16:45.430 align:middle line:84% But in a way, that's sort of become the new Stellar 00:16:45.430 --> 00:16:48.320 align:middle line:84% in the sense that each class has their own different website 00:16:48.320 --> 00:16:50.278 align:middle line:84% and each upload problems on there. 00:16:50.278 --> 00:16:52.570 align:middle line:84% And I have found it to be kind of nice, because it also 00:16:52.570 --> 00:16:58.030 align:middle line:84% has an autograder, and it lets you submit stuff 00:16:58.030 --> 00:16:59.570 align:middle line:90% for recitation in class. 00:16:59.570 --> 00:17:01.040 align:middle line:90% And I found it's pretty great. 00:17:01.040 --> 00:17:03.970 align:middle line:84% I like the interface, but I just hate the fact 00:17:03.970 --> 00:17:06.680 align:middle line:84% that I have to bookmark every single different website or else 00:17:06.680 --> 00:17:07.355 align:middle line:90% I forget them. 00:17:07.355 --> 00:17:08.230 align:middle line:90% JUSTIN REICH: Uh-huh. 00:17:08.230 --> 00:17:08.630 align:middle line:90% Good. 00:17:08.630 --> 00:17:09.963 align:middle line:90% Some of those logistical things. 00:17:09.963 --> 00:17:12.200 align:middle line:84% AUDIENCE: I mean, to add to that, different course-- 00:17:12.200 --> 00:17:14.530 align:middle line:84% I mean, I've seen it mostly in course 6. 00:17:14.530 --> 00:17:19.609 align:middle line:84% Yeah, different Course 6 classes use it to different degrees. 00:17:19.609 --> 00:17:24.099 align:middle line:84% And there is a little call for it there. 00:17:24.099 --> 00:17:27.260 align:middle line:84% There are some classes where the entire class is on there. 00:17:27.260 --> 00:17:32.150 align:middle line:84% And then you go to your class in the basement with 40 people, 00:17:32.150 --> 00:17:34.320 align:middle line:84% and they're all trying to log in at the same time 00:17:34.320 --> 00:17:36.050 align:middle line:84% so that they're marked as present, 00:17:36.050 --> 00:17:39.840 align:middle line:84% and they're starting their lab, and everything just crashes. 00:17:39.840 --> 00:17:42.620 align:middle line:84% JUSTIN REICH: That is so comforting to me 00:17:42.620 --> 00:17:45.000 align:middle line:84% because I run this system called Teacher Moments. 00:17:45.000 --> 00:17:46.410 align:middle line:90% And we have similar problems. 00:17:46.410 --> 00:17:49.280 align:middle line:84% But I'm not an MIT Computer Science professor. 00:17:49.280 --> 00:17:54.750 align:middle line:84% And the fact that their shit doesn't work either is very-- 00:17:54.750 --> 00:17:56.943 align:middle line:90% that's very comforting to me. 00:17:56.943 --> 00:17:58.610 align:middle line:84% AUDIENCE: I think that there's one thing 00:17:58.610 --> 00:18:03.320 align:middle line:84% that we kind of discussed, which is an application of the MITx, 00:18:03.320 --> 00:18:05.900 align:middle line:84% is typically, most freshmen at MIT 00:18:05.900 --> 00:18:09.440 align:middle line:90% take the 8.01, 8.02 TEAL series. 00:18:09.440 --> 00:18:12.560 align:middle line:84% And I feel like that gives this false advertisement of how 00:18:12.560 --> 00:18:15.920 align:middle line:84% education technology is used here at MIT, where they give you 00:18:15.920 --> 00:18:18.440 align:middle line:90% this flipped classroom idea. 00:18:18.440 --> 00:18:20.130 align:middle line:90% And some kids love it. 00:18:20.130 --> 00:18:22.550 align:middle line:84% And then you switch into another class, 00:18:22.550 --> 00:18:25.920 align:middle line:84% and you never see that type of learning again. 00:18:25.920 --> 00:18:29.480 align:middle line:84% And you're like, well, I liked that, so where is it? 00:18:29.480 --> 00:18:32.400 align:middle line:84% And for me, who struggled through it 00:18:32.400 --> 00:18:36.690 align:middle line:84% because I was a student that was kind of behind going in, 00:18:36.690 --> 00:18:39.420 align:middle line:84% then I switched to my Course 16 classes, 00:18:39.420 --> 00:18:42.098 align:middle line:84% where it's a professor lecturing in front of you, 00:18:42.098 --> 00:18:44.140 align:middle line:84% and all of a sudden I now understand the concept. 00:18:44.140 --> 00:18:46.290 align:middle line:84% It's like, why did I lose a whole year? 00:18:46.290 --> 00:18:50.646 align:middle line:84% So it's kind of like that back and forth between, 00:18:50.646 --> 00:18:53.320 align:middle line:84% are you going to continue to implement this in other ways, 00:18:53.320 --> 00:18:56.838 align:middle line:84% or are we just stagnant in this one idea behind it? 00:18:56.838 --> 00:18:57.630 align:middle line:90% JUSTIN REICH: Yeah. 00:18:57.630 --> 00:18:57.940 align:middle line:90% Yeah. 00:18:57.940 --> 00:18:58.140 align:middle line:90% Yeah. 00:18:58.140 --> 00:19:00.720 align:middle line:84% That's really-- I mean, TEAL is an experience that a lot of you 00:19:00.720 --> 00:19:01.220 align:middle line:90% have had. 00:19:01.220 --> 00:19:03.030 align:middle line:84% If you're not from here, TEAL is a way 00:19:03.030 --> 00:19:05.730 align:middle line:84% that they do introductory physics teaching. 00:19:05.730 --> 00:19:09.220 align:middle line:84% It came about because of high failure rates, 00:19:09.220 --> 00:19:11.670 align:middle line:84% particularly from people who came from backgrounds where 00:19:11.670 --> 00:19:13.830 align:middle line:84% they were less well prepared in physics. 00:19:13.830 --> 00:19:17.527 align:middle line:84% And so 8.01, 8.02, these introductory physics classes, 00:19:17.527 --> 00:19:19.860 align:middle line:84% would be taught as lectures with problem sets and things 00:19:19.860 --> 00:19:20.700 align:middle line:90% like that. 00:19:20.700 --> 00:19:22.752 align:middle line:84% It also actually-- there's a long-- 00:19:22.752 --> 00:19:24.960 align:middle line:84% we won't get into it-- a really interesting tradition 00:19:24.960 --> 00:19:28.260 align:middle line:84% of research in physics education, which, 00:19:28.260 --> 00:19:30.040 align:middle line:84% I don't know, three decades ago maybe, 00:19:30.040 --> 00:19:32.830 align:middle line:84% discovered that when students, even 00:19:32.830 --> 00:19:35.900 align:middle line:84% at really good universities, take introductory physics, 00:19:35.900 --> 00:19:37.720 align:middle line:84% historically they were actually less 00:19:37.720 --> 00:19:40.150 align:middle line:84% Newtonian in their thinking at the end of the class 00:19:40.150 --> 00:19:42.190 align:middle line:90% than at the beginning. 00:19:42.190 --> 00:19:43.330 align:middle line:90% They got good grades. 00:19:43.330 --> 00:19:46.490 align:middle line:84% They could solve all the kinds of physics problems. 00:19:46.490 --> 00:19:48.830 align:middle line:84% But then they gave them this test, 00:19:48.830 --> 00:19:51.250 align:middle line:84% this assessment called the Force Concept Inventory, which 00:19:51.250 --> 00:19:53.700 align:middle line:84% was a bunch of conceptual questions about how force works, 00:19:53.700 --> 00:19:56.200 align:middle line:84% and students would actually do worse at the end of the class 00:19:56.200 --> 00:19:57.700 align:middle line:84% than they would do at the beginning. 00:19:57.700 --> 00:20:00.258 align:middle line:84% So there was this sense that you could 00:20:00.258 --> 00:20:02.050 align:middle line:84% spend a whole bunch of time teaching people 00:20:02.050 --> 00:20:04.133 align:middle line:84% the procedures of physics, and they would actually 00:20:04.133 --> 00:20:06.730 align:middle line:84% come to slightly more misunderstand 00:20:06.730 --> 00:20:10.030 align:middle line:84% the concepts as a result of that, which is bad. 00:20:10.030 --> 00:20:15.303 align:middle line:84% And so active learning, learning environments 00:20:15.303 --> 00:20:17.470 align:middle line:84% where students are asked to not just listen and take 00:20:17.470 --> 00:20:19.390 align:middle line:84% notes and then do problem sets, but to engage 00:20:19.390 --> 00:20:21.490 align:middle line:84% during the conversations about the material, 00:20:21.490 --> 00:20:22.850 align:middle line:90% were shown to improve that. 00:20:22.850 --> 00:20:25.120 align:middle line:84% And then TEAL stuck around because, while people 00:20:25.120 --> 00:20:27.010 align:middle line:84% have individual experiences with it, 00:20:27.010 --> 00:20:30.910 align:middle line:84% there are fewer students who failed 8.01 and 8.02 after TEAL 00:20:30.910 --> 00:20:33.760 align:middle line:84% was implemented and given a few years to ramp up and work 00:20:33.760 --> 00:20:35.710 align:middle line:90% than there were beforehand. 00:20:35.710 --> 00:20:39.790 align:middle line:84% However, just because, on average, 00:20:39.790 --> 00:20:43.720 align:middle line:84% the TEAL system is more likely to have more students persist, 00:20:43.720 --> 00:20:46.480 align:middle line:84% doesn't mean that for any individual student-- 00:20:46.480 --> 00:20:48.640 align:middle line:84% individual students can still not like it, 00:20:48.640 --> 00:20:50.003 align:middle line:90% and it can not serve them well. 00:20:50.003 --> 00:20:52.420 align:middle line:84% And it can not serve them well because of their background 00:20:52.420 --> 00:20:53.740 align:middle line:90% preparation or because-- 00:20:53.740 --> 00:20:55.870 align:middle line:84% I mean, lots of people who come to MIT end 00:20:55.870 --> 00:20:58.287 align:middle line:84% up here because they're pretty good at teaching themselves 00:20:58.287 --> 00:20:59.410 align:middle line:90% stuff through lectures. 00:20:59.410 --> 00:21:02.933 align:middle line:84% And so alternate methods, when the physics department 00:21:02.933 --> 00:21:04.850 align:middle line:84% is trying to think about what to do with that, 00:21:04.850 --> 00:21:05.950 align:middle line:84% these are all the kinds of trade-offs 00:21:05.950 --> 00:21:07.220 align:middle line:90% that they're considering. 00:21:07.220 --> 00:21:09.280 align:middle line:84% What were you going to say, Howard? 00:21:09.280 --> 00:21:10.780 align:middle line:84% AUDIENCE: Slides, especially when 00:21:10.780 --> 00:21:13.850 align:middle line:84% you're taking classes that are Course 6 or math. 00:21:13.850 --> 00:21:15.880 align:middle line:84% When professors use slides, I feel 00:21:15.880 --> 00:21:17.950 align:middle line:84% like it affects my learning because I like 00:21:17.950 --> 00:21:21.680 align:middle line:90% when they use chalk and board. 00:21:21.680 --> 00:21:23.740 align:middle line:84% And it's just walking through it because, one, it 00:21:23.740 --> 00:21:24.680 align:middle line:90% slows them down. 00:21:24.680 --> 00:21:24.940 align:middle line:90% JUSTIN REICH: Yeah. 00:21:24.940 --> 00:21:25.225 align:middle line:90% Yeah. 00:21:25.225 --> 00:21:27.517 align:middle line:84% AUDIENCE: And two, you also kind of catch up with them. 00:21:27.517 --> 00:21:29.850 align:middle line:84% But if they're using slides, they go very fast. 00:21:29.850 --> 00:21:31.502 align:middle line:90% And second, you just have to-- 00:21:31.502 --> 00:21:32.960 align:middle line:84% I don't know, it makes the learning 00:21:32.960 --> 00:21:34.088 align:middle line:90% process a different thing. 00:21:34.088 --> 00:21:34.880 align:middle line:90% JUSTIN REICH: Yeah. 00:21:34.880 --> 00:21:35.380 align:middle line:90% Yeah. 00:21:35.380 --> 00:21:37.110 align:middle line:84% No, I think it's really interesting. 00:21:37.110 --> 00:21:39.170 align:middle line:84% They're probably like-- they're less 00:21:39.170 --> 00:21:40.940 align:middle line:84% likely to have errors on the slides 00:21:40.940 --> 00:21:43.223 align:middle line:84% than they are when they're working things out by hand. 00:21:43.223 --> 00:21:45.140 align:middle line:84% However, it's not totally clear whether or not 00:21:45.140 --> 00:21:46.863 align:middle line:90% errors are a bad thing in that. 00:21:46.863 --> 00:21:48.780 align:middle line:84% Sometimes people note the errors in real time, 00:21:48.780 --> 00:21:49.940 align:middle line:84% or they note them in real time, and they 00:21:49.940 --> 00:21:51.023 align:middle line:90% fix them and discuss them. 00:21:51.023 --> 00:21:53.870 align:middle line:84% But definitely, if you can only talk and move 00:21:53.870 --> 00:21:56.480 align:middle line:84% at the pace at which someone can write, 00:21:56.480 --> 00:21:58.590 align:middle line:84% that can have different kind of advantages. 00:21:58.590 --> 00:22:00.590 align:middle line:84% I mean, the slides are kind of cool in that they 00:22:00.590 --> 00:22:02.442 align:middle line:90% get automatically sent to you. 00:22:02.442 --> 00:22:03.900 align:middle line:84% They can be sent to you in advance. 00:22:03.900 --> 00:22:05.580 align:middle line:84% You can take notes directly on them. 00:22:05.580 --> 00:22:07.650 align:middle line:84% Is taking notes directly on them actually better, 00:22:07.650 --> 00:22:10.610 align:middle line:84% or is it better to have to hand copy, pay attention to what's 00:22:10.610 --> 00:22:11.710 align:middle line:90% going on there? 00:22:11.710 --> 00:22:13.460 align:middle line:84% These are all great questions, and they're 00:22:13.460 --> 00:22:15.420 align:middle line:84% all questions that actually have research behind them. 00:22:15.420 --> 00:22:18.220 align:middle line:84% There are folks who come up with experiments where they test all 00:22:18.220 --> 00:22:19.470 align:middle line:90% of these different conditions. 00:22:19.470 --> 00:22:20.720 align:middle line:84% What do we do when we tell students 00:22:20.720 --> 00:22:22.080 align:middle line:90% to try to copy everything down? 00:22:22.080 --> 00:22:23.580 align:middle line:84% What do we do when we tell students to summarize? 00:22:23.580 --> 00:22:24.840 align:middle line:84% What if they give them stuff in advance? 00:22:24.840 --> 00:22:25.950 align:middle line:84% What if we let them have computers? 00:22:25.950 --> 00:22:27.760 align:middle line:84% What if we take away their computers? 00:22:27.760 --> 00:22:29.530 align:middle line:84% Actually, some of the best places 00:22:29.530 --> 00:22:32.950 align:middle line:84% to do this research end up being military schools, 00:22:32.950 --> 00:22:35.117 align:middle line:84% like West Point and the Naval Academy 00:22:35.117 --> 00:22:37.450 align:middle line:84% and stuff like that, because they have a lot of required 00:22:37.450 --> 00:22:39.280 align:middle line:84% classes, and because you can be pretty 00:22:39.280 --> 00:22:42.100 align:middle line:84% sure that the students will do what you tell them to do. 00:22:42.100 --> 00:22:43.600 align:middle line:84% You assign them to either condition, 00:22:43.600 --> 00:22:47.170 align:middle line:90% and they do what they want to. 00:22:47.170 --> 00:22:48.430 align:middle line:90% Good. 00:22:48.430 --> 00:22:51.587 align:middle line:84% So an interesting thing about our own experiences 00:22:51.587 --> 00:22:53.170 align:middle line:84% is that our own experiences are really 00:22:53.170 --> 00:22:55.360 align:middle line:84% important sources of data for understanding 00:22:55.360 --> 00:22:56.510 align:middle line:90% education technology. 00:22:56.510 --> 00:22:59.650 align:middle line:84% It's important for us to look through the lens 00:22:59.650 --> 00:23:01.670 align:middle line:90% of our own learning experiences. 00:23:01.670 --> 00:23:03.470 align:middle line:84% And they can be very informative. 00:23:03.470 --> 00:23:06.080 align:middle line:84% The other thing to remember is that if you're studying at MIT, 00:23:06.080 --> 00:23:07.840 align:middle line:90% you're probably weird. 00:23:07.840 --> 00:23:09.640 align:middle line:84% There are probably characteristics of you 00:23:09.640 --> 00:23:10.850 align:middle line:90% which are not typical. 00:23:10.850 --> 00:23:15.190 align:middle line:84% Like if you generally find classroom learning pleasant, 00:23:15.190 --> 00:23:16.870 align:middle line:90% you are unusual. 00:23:16.870 --> 00:23:19.360 align:middle line:84% If you think to yourself, man, I really 00:23:19.360 --> 00:23:22.450 align:middle line:84% like just kind of learning stuff and figuring out on my own 00:23:22.450 --> 00:23:25.060 align:middle line:84% in academic material, that is unusual. 00:23:25.060 --> 00:23:26.830 align:middle line:84% Most people are not good at those things 00:23:26.830 --> 00:23:28.250 align:middle line:90% and don't find them enjoyable. 00:23:28.250 --> 00:23:30.530 align:middle line:84% So it can be helpful to bring your own experience. 00:23:30.530 --> 00:23:32.447 align:middle line:84% But it's also important to recognize the way-- 00:23:32.447 --> 00:23:35.207 align:middle line:84% and I went to private schools my whole life. 00:23:35.207 --> 00:23:36.790 align:middle line:84% I've had an extraordinarily privileged 00:23:36.790 --> 00:23:39.100 align:middle line:90% educational upbringing. 00:23:39.100 --> 00:23:41.530 align:middle line:84% In lots of ways, I have to recognize 00:23:41.530 --> 00:23:43.870 align:middle line:84% how that upbringing does not prepare me well 00:23:43.870 --> 00:23:46.180 align:middle line:84% to understand the learning experiences of most students 00:23:46.180 --> 00:23:48.130 align:middle line:84% who go to K-12 schools and public schools in the United 00:23:48.130 --> 00:23:48.980 align:middle line:90% States and things like that. 00:23:48.980 --> 00:23:51.200 align:middle line:84% So be conscientious of that as we do some thinking. 00:23:51.200 --> 00:23:51.950 align:middle line:90% But this stuff is good. 00:23:51.950 --> 00:23:53.408 align:middle line:84% All the things you're talking about 00:23:53.408 --> 00:23:55.930 align:middle line:84% are things we'll keep trying to make sense of. 00:23:55.930 --> 00:23:59.530 align:middle line:90% Failure to Disrupt. 00:23:59.530 --> 00:24:01.810 align:middle line:84% This is a book about large-scale learning, 00:24:01.810 --> 00:24:04.930 align:middle line:84% learning environments with many, many learners and few experts 00:24:04.930 --> 00:24:06.520 align:middle line:90% to guide them. 00:24:06.520 --> 00:24:10.570 align:middle line:84% And I would say, the heart of what 00:24:10.570 --> 00:24:17.470 align:middle line:84% I'm trying to do in this book is to demystify education 00:24:17.470 --> 00:24:21.370 align:middle line:84% technology for people and to say, 00:24:21.370 --> 00:24:23.500 align:middle line:84% with every generation of new technologies, 00:24:23.500 --> 00:24:26.340 align:middle line:84% there will be folks who come along that try to go to schools 00:24:26.340 --> 00:24:28.420 align:middle line:84% and say, this thing is totally brand new. 00:24:28.420 --> 00:24:30.690 align:middle line:90% This is disruptive. 00:24:30.690 --> 00:24:31.960 align:middle line:90% This is innovative. 00:24:31.960 --> 00:24:33.940 align:middle line:90% This is a game changer. 00:24:33.940 --> 00:24:36.090 align:middle line:84% This is going to transform education. 00:24:36.090 --> 00:24:39.540 align:middle line:84% And my job is to be like, eh, probably not. 00:24:39.540 --> 00:24:42.180 align:middle line:84% It might be helpful, but it's very, very unlikely 00:24:42.180 --> 00:24:45.390 align:middle line:84% that it will be any of those kinds of things. 00:24:45.390 --> 00:24:48.840 align:middle line:84% One of the ways to demystify technologies 00:24:48.840 --> 00:24:51.750 align:middle line:84% is to show where they come from, is to say, 00:24:51.750 --> 00:24:53.290 align:middle line:90% these things have patterns. 00:24:53.290 --> 00:24:55.530 align:middle line:90% The patterns are recognizable. 00:24:55.530 --> 00:24:58.450 align:middle line:84% They are patterns, not just in how these things are designed, 00:24:58.450 --> 00:25:01.720 align:middle line:84% but how they get implemented and what kind of effects they have. 00:25:01.720 --> 00:25:03.460 align:middle line:84% When new technologies come along, 00:25:03.460 --> 00:25:06.300 align:middle line:84% we don't have to guess out of the blue 00:25:06.300 --> 00:25:07.660 align:middle line:90% how well they might work. 00:25:07.660 --> 00:25:10.092 align:middle line:84% We know roughly how well education technology 00:25:10.092 --> 00:25:11.800 align:middle line:84% works in different kinds of circumstance. 00:25:11.800 --> 00:25:13.590 align:middle line:84% Now, new things-- there will be new things 00:25:13.590 --> 00:25:15.548 align:middle line:84% that are invented that are, in fact, different. 00:25:15.548 --> 00:25:18.460 align:middle line:84% But most new things are just a little bit new. 00:25:18.460 --> 00:25:21.410 align:middle line:84% They make just small modifications, small advances 00:25:21.410 --> 00:25:25.010 align:middle line:90% off of past things. 00:25:25.010 --> 00:25:28.280 align:middle line:84% And then the second half of the book 00:25:28.280 --> 00:25:32.180 align:middle line:84% confronts the challenge, why is it so hard to make progress 00:25:32.180 --> 00:25:33.480 align:middle line:90% with education technology? 00:25:33.480 --> 00:25:35.370 align:middle line:90% Why are things not disruptive? 00:25:35.370 --> 00:25:36.720 align:middle line:90% Why are they not transformative? 00:25:36.720 --> 00:25:38.690 align:middle line:90% Why are they not game changers? 00:25:38.690 --> 00:25:41.040 align:middle line:84% And I think there are answers to those questions. 00:25:41.040 --> 00:25:43.850 align:middle line:84% I think there are challenges that we have as a field yet 00:25:43.850 --> 00:25:45.920 align:middle line:90% to figure out how to tackle. 00:25:45.920 --> 00:25:48.780 align:middle line:84% And if we study those challenges and make sense of them, 00:25:48.780 --> 00:25:51.540 align:middle line:84% then we could probably do a better job moving past them. 00:25:51.540 --> 00:25:54.860 align:middle line:84% So the first half of the book looks at these three genres 00:25:54.860 --> 00:25:56.010 align:middle line:90% of learning at scale. 00:25:56.010 --> 00:25:58.640 align:middle line:84% It says that most education technologies, especially 00:25:58.640 --> 00:26:00.740 align:middle line:84% the ones where you try to teach a lot of people 00:26:00.740 --> 00:26:04.170 align:middle line:84% with very few experts, fall into one of three buckets. 00:26:04.170 --> 00:26:05.600 align:middle line:84% And you can figure out what bucket 00:26:05.600 --> 00:26:08.240 align:middle line:84% each thing is in by asking the question, who 00:26:08.240 --> 00:26:10.790 align:middle line:84% guides the sequence of learning activities? 00:26:10.790 --> 00:26:12.560 align:middle line:84% When learners do stuff, they happen 00:26:12.560 --> 00:26:14.840 align:middle line:84% in an order, one after the other after the other, 00:26:14.840 --> 00:26:17.330 align:middle line:84% and there are really three sorts of ways 00:26:17.330 --> 00:26:19.070 align:middle line:90% that those orders unfold. 00:26:19.070 --> 00:26:21.700 align:middle line:84% The first is instructor-guided learning at scale. 00:26:21.700 --> 00:26:24.480 align:middle line:84% So there can be some expert or team of experts who says, 00:26:24.480 --> 00:26:26.500 align:middle line:84% you should learn things in this order. 00:26:26.500 --> 00:26:29.500 align:middle line:84% Here's part 1, part 2, part 3, part 4, part 5. 00:26:29.500 --> 00:26:31.390 align:middle line:84% And maybe learners don't go in that order, 00:26:31.390 --> 00:26:33.100 align:middle line:90% but overwhelmingly they will. 00:26:33.100 --> 00:26:35.050 align:middle line:84% And there's some logical sequence to that. 00:26:35.050 --> 00:26:36.960 align:middle line:84% So I call those things instructor-guided 00:26:36.960 --> 00:26:38.490 align:middle line:90% learning at scale. 00:26:38.490 --> 00:26:40.240 align:middle line:84% A bunch of what you read about for today-- 00:26:40.240 --> 00:26:42.450 align:middle line:84% massive open online courses, any kind 00:26:42.450 --> 00:26:45.930 align:middle line:84% of self-paced online learning falls in that bucket. 00:26:45.930 --> 00:26:48.510 align:middle line:84% The second big genre is algorithm-guided 00:26:48.510 --> 00:26:49.600 align:middle line:90% learning at scale. 00:26:49.600 --> 00:26:53.920 align:middle line:84% So that's when a computer poses you a problem, 00:26:53.920 --> 00:26:56.040 align:middle line:84% you answer that problem, and the computer 00:26:56.040 --> 00:26:57.490 align:middle line:90% evaluates your performance. 00:26:57.490 --> 00:26:59.650 align:middle line:84% And then based on that performance, 00:26:59.650 --> 00:27:03.150 align:middle line:84% it gives you either more difficult problems or simpler 00:27:03.150 --> 00:27:04.950 align:middle line:84% problems, or the same kind of problem, 00:27:04.950 --> 00:27:07.240 align:middle line:84% or shunts you in some kind of different direction. 00:27:07.240 --> 00:27:10.300 align:middle line:84% But basically, the computer evaluates your performance, 00:27:10.300 --> 00:27:12.510 align:middle line:84% decides on the basis of that performance what 00:27:12.510 --> 00:27:15.190 align:middle line:84% the next item in a sequence of activities should be, 00:27:15.190 --> 00:27:19.210 align:middle line:90% and then you go do that forever. 00:27:19.210 --> 00:27:21.080 align:middle line:90% You are probably-- 00:27:21.080 --> 00:27:22.130 align:middle line:90% I don't know. 00:27:22.130 --> 00:27:24.550 align:middle line:84% When I taught this 10 years ago, maybe students 00:27:24.550 --> 00:27:26.240 align:middle line:84% would have dabbled with these things, 00:27:26.240 --> 00:27:28.180 align:middle line:84% but it's far, far more common now. 00:27:28.180 --> 00:27:34.240 align:middle line:84% If you in school ever used DreamBox or IXL or Khan Academy, 00:27:34.240 --> 00:27:36.310 align:middle line:84% almost all of those things are some version 00:27:36.310 --> 00:27:39.010 align:middle line:84% of algorithm-guided learning at scale. 00:27:39.010 --> 00:27:40.790 align:middle line:84% A bunch of the testing that you've done, 00:27:40.790 --> 00:27:42.560 align:middle line:84% if you've taken the computer-based GREs, 00:27:42.560 --> 00:27:45.340 align:middle line:84% other things like that, they use similar technologies. 00:27:45.340 --> 00:27:48.430 align:middle line:84% The third bucket of things we call peer-guided learning 00:27:48.430 --> 00:27:49.193 align:middle line:90% at scale. 00:27:49.193 --> 00:27:50.860 align:middle line:84% So if any of you have ever played around 00:27:50.860 --> 00:27:52.820 align:middle line:84% with the Scratch programming community, 00:27:52.820 --> 00:27:54.680 align:middle line:84% there's no instruction in Scratch. 00:27:54.680 --> 00:27:57.760 align:middle line:84% There's no Scratch University or Scratch lesson 00:27:57.760 --> 00:27:58.910 align:middle line:90% 1, lesson 2, lesson 3. 00:27:58.910 --> 00:28:00.860 align:middle line:84% And that's very intentional and by design. 00:28:00.860 --> 00:28:04.780 align:middle line:84% Instead, you're immersed in this community of peers. 00:28:04.780 --> 00:28:07.250 align:middle line:84% And if you go to the home page of the Scratch website, 00:28:07.250 --> 00:28:10.230 align:middle line:84% there are featured tutorials and featured projects and things 00:28:10.230 --> 00:28:10.730 align:middle line:90% like that. 00:28:10.730 --> 00:28:12.160 align:middle line:84% And the way you figure out Scratch 00:28:12.160 --> 00:28:14.030 align:middle line:84% is that you think about some things that you want to do, 00:28:14.030 --> 00:28:15.100 align:middle line:84% and then you find some other people who've 00:28:15.100 --> 00:28:17.010 align:middle line:84% done that kind of thing, and you copy them, 00:28:17.010 --> 00:28:20.070 align:middle line:84% you converse with them, you take their project and you modify it, 00:28:20.070 --> 00:28:23.360 align:middle line:84% but you basically design your own pathway. 00:28:23.360 --> 00:28:27.600 align:middle line:84% There are lots of other learning environments, Stack Overflow, 00:28:27.600 --> 00:28:29.660 align:middle line:84% other kinds of communities of practice. 00:28:29.660 --> 00:28:31.910 align:middle line:84% The one that I talk about at the beginning of the book 00:28:31.910 --> 00:28:33.050 align:middle line:90% is Rainbow Loom. 00:28:33.050 --> 00:28:35.550 align:middle line:84% Any of you ever played with Rainbow Loom when you were kids? 00:28:35.550 --> 00:28:36.230 align:middle line:90% Yeah. 00:28:36.230 --> 00:28:38.490 align:middle line:84% So when you bought a-- did I tell you this story yet? 00:28:38.490 --> 00:28:39.550 align:middle line:84% When you bought a Rainbow Loom, there 00:28:39.550 --> 00:28:40.900 align:middle line:90% were 10 cards in the packet. 00:28:40.900 --> 00:28:42.150 align:middle line:90% So you pulled open the packet. 00:28:42.150 --> 00:28:44.690 align:middle line:84% There's a bunch of these little rubber bands, a little-- 00:28:44.690 --> 00:28:45.960 align:middle line:90% what do they call that thing? 00:28:45.960 --> 00:28:48.270 align:middle line:84% Frame, to stitch the rubber bands on. 00:28:48.270 --> 00:28:51.498 align:middle line:84% And then there are 10 cards that show 10 different kinds 00:28:51.498 --> 00:28:52.790 align:middle line:90% of bracelets that you can make. 00:28:52.790 --> 00:28:54.950 align:middle line:84% If you get really into Rainbow Loom, 00:28:54.950 --> 00:28:56.697 align:middle line:84% 10 different kinds of bracelets is not 00:28:56.697 --> 00:28:57.780 align:middle line:90% enough kinds of bracelets. 00:28:57.780 --> 00:29:00.260 align:middle line:84% You have to learn more different ways of doing it. 00:29:00.260 --> 00:29:03.440 align:middle line:84% And the main way you'll learn that is through YouTube. 00:29:03.440 --> 00:29:04.585 align:middle line:90% You'll go to this-- 00:29:04.585 --> 00:29:05.960 align:middle line:84% and here's the thing that boggled 00:29:05.960 --> 00:29:07.085 align:middle line:90% my mind about Rainbow Loom. 00:29:07.085 --> 00:29:09.080 align:middle line:84% So there's some dude who is in Michigan. 00:29:09.080 --> 00:29:10.830 align:middle line:84% He was an automotive engineer or something 00:29:10.830 --> 00:29:12.413 align:middle line:84% like that who was like, in my country, 00:29:12.413 --> 00:29:13.880 align:middle line:84% this is the way we make bracelets. 00:29:13.880 --> 00:29:16.980 align:middle line:84% And he comes up with some CAD designs, 00:29:16.980 --> 00:29:18.280 align:middle line:90% and he sends them to China. 00:29:18.280 --> 00:29:20.290 align:middle line:84% And China manufactures zillions of these things. 00:29:20.290 --> 00:29:21.998 align:middle line:84% And they put them on shipping containers, 00:29:21.998 --> 00:29:24.340 align:middle line:84% and they send them to his distribution center. 00:29:24.340 --> 00:29:26.985 align:middle line:84% And eventually, these packets, these little boxes 00:29:26.985 --> 00:29:29.110 align:middle line:84% of Rainbow Loom things get sent all over the world. 00:29:29.110 --> 00:29:31.290 align:middle line:84% It's a total supply chain miracle. 00:29:31.290 --> 00:29:33.748 align:middle line:84% The fact that some dude in Michigan just has an idea 00:29:33.748 --> 00:29:35.790 align:middle line:84% and is like, let's have people all over the world 00:29:35.790 --> 00:29:37.630 align:middle line:84% buy these things, and they do, is amazing. 00:29:37.630 --> 00:29:41.220 align:middle line:84% But simultaneously to that physical supply chain 00:29:41.220 --> 00:29:46.050 align:middle line:84% infrastructure, this giant global community of Rainbow Loom 00:29:46.050 --> 00:29:48.540 align:middle line:84% learners emerges, where there are 00:29:48.540 --> 00:29:52.890 align:middle line:84% people who are running little classes at art shops, 00:29:52.890 --> 00:29:54.390 align:middle line:84% and they're posting videos online, 00:29:54.390 --> 00:29:56.140 align:middle line:84% and they're sharing ideas with each other, 00:29:56.140 --> 00:29:58.500 align:middle line:84% and they're creating little Facebook Marketplaces. 00:29:58.500 --> 00:30:01.080 align:middle line:84% And the only way Rainbow Loom could grow 00:30:01.080 --> 00:30:04.690 align:middle line:84% is if that learning network grows alongside it. 00:30:04.690 --> 00:30:06.390 align:middle line:90% That is miraculous. 00:30:06.390 --> 00:30:09.540 align:middle line:84% That is an amazing feature of online learning, 00:30:09.540 --> 00:30:14.050 align:middle line:84% that people could invent a tool to do this kind of practice 00:30:14.050 --> 00:30:17.800 align:middle line:84% and a community could develop to teach each other and themselves 00:30:17.800 --> 00:30:18.910 align:middle line:90% this practice. 00:30:18.910 --> 00:30:19.923 align:middle line:90% I think there were-- 00:30:19.923 --> 00:30:21.340 align:middle line:84% oh, the other thing I really liked 00:30:21.340 --> 00:30:22.923 align:middle line:84% about that, which I think I mentioned, 00:30:22.923 --> 00:30:25.368 align:middle line:84% is that there were these two girls-- 00:30:25.368 --> 00:30:27.160 align:middle line:84% I think their names were Ashley and Steph-- 00:30:27.160 --> 00:30:29.420 align:middle line:84% who made Rainbow Loom YouTube videos. 00:30:29.420 --> 00:30:31.690 align:middle line:90% And they got millions of views. 00:30:31.690 --> 00:30:35.620 align:middle line:84% And you could compare their videos to the actual Rainbow 00:30:35.620 --> 00:30:37.190 align:middle line:90% Loom manufacturer videos. 00:30:37.190 --> 00:30:39.580 align:middle line:84% And these girls were getting like 10x the number of views 00:30:39.580 --> 00:30:41.260 align:middle line:90% that the actual Rainbow-- 00:30:41.260 --> 00:30:43.480 align:middle line:84% basically, kids would rather learn 00:30:43.480 --> 00:30:44.757 align:middle line:90% from these two random girls. 00:30:44.757 --> 00:30:46.340 align:middle line:84% They ended up being from Philadelphia. 00:30:46.340 --> 00:30:48.280 align:middle line:84% They're very good at protecting their anonymity, at least 00:30:48.280 --> 00:30:48.780 align:middle line:90% for a while. 00:30:48.780 --> 00:30:51.010 align:middle line:84% One of them eventually became a makeup influencer, 00:30:51.010 --> 00:30:52.850 align:middle line:90% and then they revealed herself. 00:30:52.850 --> 00:30:58.565 align:middle line:84% But I find these global networks incredibly fascinating and very, 00:30:58.565 --> 00:31:00.190 align:middle line:84% very difficult to integrate in schools, 00:31:00.190 --> 00:31:03.460 align:middle line:84% for reasons that we'll explore as we go along. 00:31:03.460 --> 00:31:07.240 align:middle line:84% So if you buy that there are these three big categories, 00:31:07.240 --> 00:31:09.730 align:middle line:84% a thing that we'll do over the next few weeks 00:31:09.730 --> 00:31:12.640 align:middle line:84% is we'll look at how there are similarities 00:31:12.640 --> 00:31:14.800 align:middle line:90% within these categories. 00:31:14.800 --> 00:31:17.410 align:middle line:84% So for instance, in instructor-guided learning 00:31:17.410 --> 00:31:21.335 align:middle line:84% at scale, there typically are two key technologies. 00:31:21.335 --> 00:31:22.960 align:middle line:84% There are two technologies that show up 00:31:22.960 --> 00:31:24.610 align:middle line:90% over and over and over again. 00:31:24.610 --> 00:31:26.480 align:middle line:84% One is a learning management system. 00:31:26.480 --> 00:31:27.860 align:middle line:84% So a learning management system-- 00:31:27.860 --> 00:31:31.012 align:middle line:84% Canvas, Blackboard, edX, Moodle, Stellar, 00:31:31.012 --> 00:31:32.470 align:middle line:84% or a zillion other things-- they're 00:31:32.470 --> 00:31:36.760 align:middle line:84% platforms that organize learning materials. 00:31:36.760 --> 00:31:39.610 align:middle line:84% An unusual feature of learning management systems 00:31:39.610 --> 00:31:44.170 align:middle line:84% is that they have a feature that people who study products called 00:31:44.170 --> 00:31:45.440 align:middle line:90% feature convergence. 00:31:45.440 --> 00:31:49.600 align:middle line:84% Essentially, it's very difficult to patent or have 00:31:49.600 --> 00:31:52.240 align:middle line:84% intellectual property around components of a learning 00:31:52.240 --> 00:31:53.000 align:middle line:90% management system. 00:31:53.000 --> 00:31:54.830 align:middle line:84% So anytime anyone invents something good, 00:31:54.830 --> 00:31:56.530 align:middle line:90% everybody else copies it. 00:31:56.530 --> 00:31:59.510 align:middle line:84% So they're all basically the exact same system. 00:31:59.510 --> 00:32:01.010 align:middle line:84% You buy them from different vendors, 00:32:01.010 --> 00:32:04.030 align:middle line:84% but they're basically one learning management system. 00:32:04.030 --> 00:32:07.180 align:middle line:84% And then the second kind of crucial feature 00:32:07.180 --> 00:32:09.190 align:middle line:90% of online learning management-- 00:32:09.190 --> 00:32:11.990 align:middle line:84% or instructor-guided learning at scale are autograders. 00:32:11.990 --> 00:32:13.910 align:middle line:84% So you have to be able to give-- if you have 00:32:13.910 --> 00:32:16.160 align:middle line:84% lots and lots of learners, if you want to be able to give them 00:32:16.160 --> 00:32:17.870 align:middle line:84% feedback, if you want to be to certify 00:32:17.870 --> 00:32:19.872 align:middle line:84% their learning in any way, you either 00:32:19.872 --> 00:32:21.830 align:middle line:84% have to spend zillions of dollars hiring people 00:32:21.830 --> 00:32:25.610 align:middle line:84% to grade their work, or you find ways of having computers grade 00:32:25.610 --> 00:32:26.130 align:middle line:90% their work. 00:32:26.130 --> 00:32:28.620 align:middle line:84% Now, computers can-- we'll talk about this at length. 00:32:28.620 --> 00:32:31.220 align:middle line:84% Computers can grade some things, but not others. 00:32:31.220 --> 00:32:33.050 align:middle line:84% Computers are really good at answering-- 00:32:33.050 --> 00:32:36.050 align:middle line:84% at evaluating questions where the answer is 00:32:36.050 --> 00:32:37.670 align:middle line:90% well known and certain. 00:32:37.670 --> 00:32:39.710 align:middle line:84% They are much less good at evaluating 00:32:39.710 --> 00:32:43.670 align:middle line:84% anything which has a kind of subjective component to it. 00:32:43.670 --> 00:32:45.620 align:middle line:84% One way to think about this is, autograders 00:32:45.620 --> 00:32:48.350 align:middle line:84% are great in early language acquisition 00:32:48.350 --> 00:32:52.370 align:middle line:84% and very rapidly become useless in later language acquisition. 00:32:52.370 --> 00:32:56.187 align:middle line:84% So they can tell whether you've spelled "hola" correctly. 00:32:56.187 --> 00:32:58.520 align:middle line:84% We even have some pretty cool ones that can tell whether 00:32:58.520 --> 00:33:00.327 align:middle line:84% you've pronounced "hola" correctly. 00:33:00.327 --> 00:33:02.660 align:middle line:84% We can give you a multiple choice question to see if you 00:33:02.660 --> 00:33:05.750 align:middle line:84% can recognize the audio recording of "hola." 00:33:05.750 --> 00:33:08.240 align:middle line:84% Eventually, however, what we want to know are, 00:33:08.240 --> 00:33:10.680 align:middle line:84% what are the key themes in Don Quixote? 00:33:10.680 --> 00:33:12.990 align:middle line:84% And that we don't have very good autograders to be 00:33:12.990 --> 00:33:16.170 align:middle line:90% able to evaluate and so forth. 00:33:16.170 --> 00:33:19.390 align:middle line:84% I would say, in addition to this technological infrastructure, 00:33:19.390 --> 00:33:21.100 align:middle line:84% these genres of learning at scale, 00:33:21.100 --> 00:33:25.950 align:middle line:84% they tend to align more closely either with direct instruction 00:33:25.950 --> 00:33:30.722 align:middle line:84% kind of folks or apprenticeship constructivism kinds of folks. 00:33:30.722 --> 00:33:32.680 align:middle line:84% The instructor-guided learning at scale people, 00:33:32.680 --> 00:33:35.920 align:middle line:84% they tend to be Thorndike, direct instruction, 00:33:35.920 --> 00:33:37.840 align:middle line:84% cognitive load theory kind of people. 00:33:37.840 --> 00:33:42.180 align:middle line:84% They tend to be pail-filling kinds of folks. 00:33:42.180 --> 00:33:44.760 align:middle line:84% Algorithm-guided learning at scale 00:33:44.760 --> 00:33:47.220 align:middle line:84% has similarities but differences. 00:33:47.220 --> 00:33:52.170 align:middle line:84% So the crucial technology behind algorithm-guided learning 00:33:52.170 --> 00:33:55.590 align:middle line:84% at scale is a set of algorithms called item response theory. 00:33:55.590 --> 00:33:59.445 align:middle line:84% This was developed in the 1980s by Educational Testing Services, 00:33:59.445 --> 00:34:00.945 align:middle line:84% the people who make-- what do they-- 00:34:00.945 --> 00:34:02.940 align:middle line:90% do they do the SAT or the GRE? 00:34:02.940 --> 00:34:05.198 align:middle line:90% What test does ETS make? 00:34:05.198 --> 00:34:05.740 align:middle line:90% I don't know. 00:34:05.740 --> 00:34:07.210 align:middle line:90% They make tests. 00:34:07.210 --> 00:34:14.500 align:middle line:84% And the basic toolkit of item response theory 00:34:14.500 --> 00:34:17.330 align:middle line:90% has proven really quite robust. 00:34:17.330 --> 00:34:19.570 align:middle line:84% All item response theory does basically 00:34:19.570 --> 00:34:22.405 align:middle line:84% is it lets you figure out how difficult an item is. 00:34:22.405 --> 00:34:24.530 align:middle line:84% It's a way of quantifying how difficult an item is. 00:34:24.530 --> 00:34:26.469 align:middle line:84% If you can quantify how difficult an item is, if someone 00:34:26.469 --> 00:34:28.480 align:middle line:84% gets the item right, you can give them a more difficult item 00:34:28.480 --> 00:34:29.449 align:middle line:90% or less difficult item. 00:34:29.449 --> 00:34:30.949 align:middle line:84% It was actually originally developed 00:34:30.949 --> 00:34:34.510 align:middle line:84% so that you could give people tests of equivalent difficulty 00:34:34.510 --> 00:34:36.620 align:middle line:84% without giving them the same test. 00:34:36.620 --> 00:34:39.550 align:middle line:84% This proved important, like when we had national testing 00:34:39.550 --> 00:34:41.469 align:middle line:84% and we built communication networks, 00:34:41.469 --> 00:34:45.080 align:middle line:84% because, in theory, somebody could take a test in Portland, 00:34:45.080 --> 00:34:47.679 align:middle line:84% Maine, and later that same day, someone would take it 00:34:47.679 --> 00:34:49.058 align:middle line:90% in Honolulu, Hawaii. 00:34:49.058 --> 00:34:51.100 align:middle line:84% And there's just enough time in there for someone 00:34:51.100 --> 00:34:53.770 align:middle line:84% to call and send all the questions and answers 00:34:53.770 --> 00:34:55.570 align:middle line:90% and things like that. 00:34:55.570 --> 00:35:00.080 align:middle line:84% And so it, in part, came about from test security. 00:35:00.080 --> 00:35:04.415 align:middle line:84% You want to be able to make claims about people's ability. 00:35:04.415 --> 00:35:06.790 align:middle line:84% And one way to do that is to give everyone the same test. 00:35:06.790 --> 00:35:08.790 align:middle line:84% Another way to do that is to give everyone tests 00:35:08.790 --> 00:35:11.220 align:middle line:84% with different items but of the same difficulty. 00:35:11.220 --> 00:35:13.470 align:middle line:84% And so we had to come up with some statistical toolkit 00:35:13.470 --> 00:35:15.990 align:middle line:90% for doing that. 00:35:15.990 --> 00:35:19.620 align:middle line:84% What we came up with in the 1980s and 1990s, 00:35:19.620 --> 00:35:22.950 align:middle line:84% we've modified it a bit, but we still pretty much use today. 00:35:22.950 --> 00:35:25.860 align:middle line:84% We know a bunch about its strengths and weaknesses 00:35:25.860 --> 00:35:27.090 align:middle line:90% and how it works. 00:35:27.090 --> 00:35:29.565 align:middle line:84% Most of our autograders, just now maybe they're 00:35:29.565 --> 00:35:30.940 align:middle line:84% starting to improve a little bit. 00:35:30.940 --> 00:35:33.232 align:middle line:84% Most of them haven't improved that much in the last two 00:35:33.232 --> 00:35:34.950 align:middle line:90% or three decades. 00:35:34.950 --> 00:35:37.617 align:middle line:84% In some specific areas, some parts of language acquisition, 00:35:37.617 --> 00:35:40.200 align:middle line:84% some parts of computer science, maybe they've improved better. 00:35:40.200 --> 00:35:43.500 align:middle line:84% But these technologies, for the most part, are pretty stable. 00:35:43.500 --> 00:35:45.660 align:middle line:90% When someone has a new entrant-- 00:35:45.660 --> 00:35:48.160 align:middle line:84% I mean, you'll read more of this story in chapter 2. 00:35:48.160 --> 00:35:50.220 align:middle line:84% But there was a company called Newton, 00:35:50.220 --> 00:35:54.570 align:middle line:84% which said that they could take any body of educational content 00:35:54.570 --> 00:35:57.240 align:middle line:84% and turn it into an intelligent tutor. 00:35:57.240 --> 00:35:59.890 align:middle line:84% And they thought that their tutor was absolutely amazing. 00:35:59.890 --> 00:36:02.610 align:middle line:84% The founder described it as something 00:36:02.610 --> 00:36:04.410 align:middle line:84% like a magical robot tutor in the sky, 00:36:04.410 --> 00:36:06.850 align:middle line:84% knew exactly what you were thinking and so forth. 00:36:06.850 --> 00:36:09.610 align:middle line:84% So he was making all these totally outlandish claims. 00:36:09.610 --> 00:36:12.920 align:middle line:84% And then his engineers had a blog 00:36:12.920 --> 00:36:14.170 align:middle line:90% about what they were building. 00:36:14.170 --> 00:36:16.128 align:middle line:84% And one of their blog posts was something like, 00:36:16.128 --> 00:36:18.120 align:middle line:84% using two-parameter item response theory 00:36:18.120 --> 00:36:19.120 align:middle line:90% to evaluate items. 00:36:19.120 --> 00:36:20.477 align:middle line:90% They were basically like-- 00:36:20.477 --> 00:36:22.060 align:middle line:84% so the leader is out there being like, 00:36:22.060 --> 00:36:23.650 align:middle line:90% this thing is totally amazing. 00:36:23.650 --> 00:36:25.530 align:middle line:84% And the engineers are like, here's 00:36:25.530 --> 00:36:27.030 align:middle line:84% the very well-established technology 00:36:27.030 --> 00:36:31.530 align:middle line:84% that we're using to do the thing that we are selling you to do. 00:36:31.530 --> 00:36:33.720 align:middle line:84% The peer-guided learning at scale 00:36:33.720 --> 00:36:36.095 align:middle line:84% looks pretty different from the other two. 00:36:36.095 --> 00:36:37.470 align:middle line:84% Peer-guided learning environments 00:36:37.470 --> 00:36:39.460 align:middle line:84% tend to be more apprenticeship-oriented. 00:36:39.460 --> 00:36:42.120 align:middle line:84% They tend to be more wrapped into situated learning, things 00:36:42.120 --> 00:36:43.050 align:middle line:90% like that. 00:36:43.050 --> 00:36:47.370 align:middle line:84% They tend to have a different technical infrastructure. 00:36:47.370 --> 00:36:50.112 align:middle line:84% Instead of being based in some kind of content holding 00:36:50.112 --> 00:36:51.820 align:middle line:84% thing, like a learning management system, 00:36:51.820 --> 00:36:53.490 align:middle line:90% they usually exist on the web. 00:36:53.490 --> 00:36:56.490 align:middle line:84% And usually, their killer app is not an autograder, but a thing 00:36:56.490 --> 00:36:58.230 align:middle line:90% that pulls content together. 00:36:58.230 --> 00:37:01.470 align:middle line:84% How do you figure out what the right sequence of Rainbow Loom 00:37:01.470 --> 00:37:04.380 align:middle line:84% learning materials is to look at, and things like hashtags 00:37:04.380 --> 00:37:07.410 align:middle line:90% are really important for that. 00:37:07.410 --> 00:37:10.088 align:middle line:84% In Scratch, how do you pull together and come up 00:37:10.088 --> 00:37:12.630 align:middle line:84% with a set of featured projects, or a way of tagging projects 00:37:12.630 --> 00:37:14.700 align:middle line:84% so that people can find their way through them? 00:37:14.700 --> 00:37:17.310 align:middle line:84% So these are the three big buckets 00:37:17.310 --> 00:37:20.280 align:middle line:84% that we'll explore one bucket each 00:37:20.280 --> 00:37:23.880 align:middle line:90% over the next bunch of weeks. 00:37:23.880 --> 00:37:26.340 align:middle line:84% The second half of the book, which 00:37:26.340 --> 00:37:30.070 align:middle line:84% I guess if this project takes us a good chunk of March, 00:37:30.070 --> 00:37:32.070 align:middle line:84% this project will take us a good chunk of April, 00:37:32.070 --> 00:37:35.340 align:middle line:84% are four problems that educators and education technology 00:37:35.340 --> 00:37:37.440 align:middle line:84% designers run into over and over again, 00:37:37.440 --> 00:37:39.800 align:middle line:84% which I'll enumerate quickly just to orient you. 00:37:39.800 --> 00:37:42.780 align:middle line:90% 00:37:42.780 --> 00:37:43.420 align:middle line:90% One's missing. 00:37:43.420 --> 00:37:46.260 align:middle line:84% Anyway, the first one is the Curse of the Familiar. 00:37:46.260 --> 00:37:48.030 align:middle line:84% The idea of the Curse of the Familiar 00:37:48.030 --> 00:37:50.790 align:middle line:84% is that it's pretty easy to make technologies 00:37:50.790 --> 00:37:53.260 align:middle line:84% that do things that educators already are doing 00:37:53.260 --> 00:37:54.660 align:middle line:90% and get them adopted. 00:37:54.660 --> 00:37:57.300 align:middle line:84% How many of you have ever used Quizlet? 00:37:57.300 --> 00:37:59.028 align:middle line:90% OK. 00:37:59.028 --> 00:38:00.570 align:middle line:90% That's about as universal-- 00:38:00.570 --> 00:38:03.220 align:middle line:84% Quizlet was founded by MIT dropout Andy 00:38:03.220 --> 00:38:07.550 align:middle line:84% Sutherland, who took a bunch of education classes with us. 00:38:07.550 --> 00:38:13.240 align:middle line:84% And he built it to, I think, study French for himself. 00:38:13.240 --> 00:38:15.100 align:middle line:90% Quizlet is flashcards. 00:38:15.100 --> 00:38:17.980 align:middle line:84% If you went and you gathered a bunch of educational experts 00:38:17.980 --> 00:38:22.750 align:middle line:84% around the world and said, what do our kids really 00:38:22.750 --> 00:38:24.057 align:middle line:90% need to advance their learning? 00:38:24.057 --> 00:38:26.390 align:middle line:84% There is no one who would raise their hands and be like, 00:38:26.390 --> 00:38:28.040 align:middle line:84% there's a huge dearth of flashcards. 00:38:28.040 --> 00:38:30.290 align:middle line:84% We just don't have enough flashcards at school. 00:38:30.290 --> 00:38:33.100 align:middle line:84% We've run out of index cards, something like that. 00:38:33.100 --> 00:38:36.167 align:middle line:90% And yet here's this something-- 00:38:36.167 --> 00:38:38.500 align:middle line:84% I mean, they say things like, half of all high schoolers 00:38:38.500 --> 00:38:40.480 align:middle line:90% every month use Quizlet. 00:38:40.480 --> 00:38:43.400 align:middle line:84% How much does Quizlet improve teaching and learning? 00:38:43.400 --> 00:38:45.910 align:middle line:84% Well, if you're replacing regular flashcards 00:38:45.910 --> 00:38:48.370 align:middle line:84% with digital flashcards, how much would 00:38:48.370 --> 00:38:50.230 align:middle line:84% you expect that to improve learning? 00:38:50.230 --> 00:38:50.733 align:middle line:90% Maybe some. 00:38:50.733 --> 00:38:52.400 align:middle line:84% Maybe there's a little efficiency there, 00:38:52.400 --> 00:38:53.150 align:middle line:90% something like that. 00:38:53.150 --> 00:38:55.150 align:middle line:84% You also might say something like, actually, one 00:38:55.150 --> 00:38:56.830 align:middle line:84% of the best things about flashcards 00:38:56.830 --> 00:38:58.350 align:middle line:84% is the creation of the flashcards. 00:38:58.350 --> 00:39:00.350 align:middle line:84% There's actually something important of learning 00:39:00.350 --> 00:39:02.392 align:middle line:84% about writing down the prompts on one end, 00:39:02.392 --> 00:39:03.600 align:middle line:90% the answers on the other end. 00:39:03.600 --> 00:39:05.100 align:middle line:84% So maybe the paper ones are actually 00:39:05.100 --> 00:39:07.890 align:middle line:84% better than the Quizlet ones in different kinds of ways. 00:39:07.890 --> 00:39:09.650 align:middle line:84% But the point is, there's probably 00:39:09.650 --> 00:39:12.950 align:middle line:84% no one who would make a bet that human development would 00:39:12.950 --> 00:39:16.350 align:middle line:84% be profoundly transformed by digitizing flashcards. 00:39:16.350 --> 00:39:19.002 align:middle line:84% But if you look across many of the most successful education 00:39:19.002 --> 00:39:21.210 align:middle line:84% technology companies, they do something like Quizlet. 00:39:21.210 --> 00:39:23.700 align:middle line:84% They take a thing which is already happening in schools, 00:39:23.700 --> 00:39:26.690 align:middle line:84% and they digitize it, maybe make it a little bit more efficient. 00:39:26.690 --> 00:39:28.190 align:middle line:84% There are a handful of projects that 00:39:28.190 --> 00:39:30.860 align:middle line:84% try to do something totally different, and many of them 00:39:30.860 --> 00:39:31.590 align:middle line:90% fail. 00:39:31.590 --> 00:39:34.430 align:middle line:84% And the reason why they fail is it's really hard 00:39:34.430 --> 00:39:37.650 align:middle line:84% to convince people in schools to adopt totally new routines. 00:39:37.650 --> 00:39:40.910 align:middle line:84% So there are all kinds of visionary educators out there 00:39:40.910 --> 00:39:42.440 align:middle line:84% who think, man, wouldn't it be cool 00:39:42.440 --> 00:39:44.550 align:middle line:84% if we could teach math in this really different kind of way, 00:39:44.550 --> 00:39:46.670 align:middle line:84% if we can teach computer science in this really different kind 00:39:46.670 --> 00:39:47.180 align:middle line:90% of way? 00:39:47.180 --> 00:39:49.430 align:middle line:84% And then they build things that do that and people go, 00:39:49.430 --> 00:39:50.443 align:middle line:90% that's weird. 00:39:50.443 --> 00:39:52.860 align:middle line:84% I don't understand this really new, different kind of way. 00:39:52.860 --> 00:39:55.360 align:middle line:84% So a bunch of the things which might potentially lead us 00:39:55.360 --> 00:39:57.860 align:middle line:84% to the biggest gains or changes in our teaching and learning 00:39:57.860 --> 00:39:59.485 align:middle line:84% are really hard to get people to adopt. 00:39:59.485 --> 00:40:01.780 align:middle line:84% That's the Curse of the Familiar. 00:40:01.780 --> 00:40:05.170 align:middle line:84% The EdTech Matthew Effect is that we very often hope that new 00:40:05.170 --> 00:40:08.110 align:middle line:84% technologies, especially free technologies, will be-- 00:40:08.110 --> 00:40:10.690 align:middle line:84% the term people often use is "democratizing," 00:40:10.690 --> 00:40:13.960 align:middle line:84% that they will-- essentially what will happen-- 00:40:13.960 --> 00:40:15.430 align:middle line:84% people, generally speaking, agreed 00:40:15.430 --> 00:40:17.020 align:middle line:84% that folks who have more resources 00:40:17.020 --> 00:40:18.670 align:middle line:84% have had more learning experiences, 00:40:18.670 --> 00:40:20.462 align:middle line:84% and therefore have learned more than people 00:40:20.462 --> 00:40:22.030 align:middle line:90% with less experience. 00:40:22.030 --> 00:40:23.710 align:middle line:84% But we hope that there are technologies 00:40:23.710 --> 00:40:27.920 align:middle line:84% that will disproportionately benefit the less affluent, 00:40:27.920 --> 00:40:30.260 align:middle line:84% that they will help poor folks catch up to other people. 00:40:30.260 --> 00:40:31.730 align:middle line:84% They'll create new kinds of on ramps. 00:40:31.730 --> 00:40:33.730 align:middle line:84% Maybe they help everybody, but they particularly 00:40:33.730 --> 00:40:34.420 align:middle line:90% help this group. 00:40:34.420 --> 00:40:37.570 align:middle line:84% What we almost always find is the figure on the right, which 00:40:37.570 --> 00:40:39.700 align:middle line:84% is that new technologies disproportionately benefit 00:40:39.700 --> 00:40:41.530 align:middle line:84% the affluent, is that they're very 00:40:41.530 --> 00:40:43.600 align:middle line:84% likely to accelerate the learning of people 00:40:43.600 --> 00:40:45.880 align:middle line:90% who are already advantaged. 00:40:45.880 --> 00:40:48.130 align:middle line:84% Many people go into education because they're 00:40:48.130 --> 00:40:51.470 align:middle line:84% trying to build a more equitable, just society, 00:40:51.470 --> 00:40:54.550 align:middle line:84% and it can be quite frustrating if you keep building stuff 00:40:54.550 --> 00:40:56.920 align:middle line:84% and it keeps helping the most affluent people in society 00:40:56.920 --> 00:41:00.160 align:middle line:90% secure their position better. 00:41:00.160 --> 00:41:02.840 align:middle line:84% We can document this really well. 00:41:02.840 --> 00:41:04.377 align:middle line:84% It's actually, in a lot of cases, 00:41:04.377 --> 00:41:05.960 align:middle line:84% hard to understand why it's happening. 00:41:05.960 --> 00:41:08.050 align:middle line:84% And we have very little science or research 00:41:08.050 --> 00:41:11.708 align:middle line:84% that helps us explain what alternatives would be. 00:41:11.708 --> 00:41:13.750 align:middle line:84% We'll give you some examples of some technologies 00:41:13.750 --> 00:41:15.167 align:middle line:84% that I think don't do that, that I 00:41:15.167 --> 00:41:18.002 align:middle line:84% think do disproportionately benefit people further 00:41:18.002 --> 00:41:18.710 align:middle line:90% from opportunity. 00:41:18.710 --> 00:41:22.180 align:middle line:84% But it's really hard to figure out why they work. 00:41:22.180 --> 00:41:25.300 align:middle line:84% A third challenge that education technology folks 00:41:25.300 --> 00:41:27.880 align:middle line:84% come across all the time is that computers 00:41:27.880 --> 00:41:30.140 align:middle line:84% are good at evaluating certain kinds of things. 00:41:30.140 --> 00:41:32.557 align:middle line:84% The kinds of things that they're really good at evaluating 00:41:32.557 --> 00:41:36.260 align:middle line:84% are those that are routine, that have common features to them, 00:41:36.260 --> 00:41:38.800 align:middle line:90% that are highly structured. 00:41:38.800 --> 00:41:42.730 align:middle line:84% Computers are much less good at evaluating ill structured kinds 00:41:42.730 --> 00:41:45.770 align:middle line:84% of things, things that don't have much form to them, 00:41:45.770 --> 00:41:48.130 align:middle line:84% or we don't know what inputs need to go in them 00:41:48.130 --> 00:41:50.290 align:middle line:84% or what the final output should look like. 00:41:50.290 --> 00:41:52.450 align:middle line:84% For instance, one of the things that we 00:41:52.450 --> 00:41:55.240 align:middle line:84% don't have good autograders for is people's ability 00:41:55.240 --> 00:41:58.580 align:middle line:84% to reason from evidence in natural language, 00:41:58.580 --> 00:42:00.120 align:middle line:90% either written or auditory. 00:42:00.120 --> 00:42:02.990 align:middle line:84% The main thing that we teach you for 16 or 20 00:42:02.990 --> 00:42:05.810 align:middle line:84% years in the education system is how to reason from evidence. 00:42:05.810 --> 00:42:08.300 align:middle line:84% The most important things we're teaching you how to do we 00:42:08.300 --> 00:42:09.870 align:middle line:90% don't have good autograders for. 00:42:09.870 --> 00:42:13.280 align:middle line:84% Moreover, the kinds of things that computers can grade 00:42:13.280 --> 00:42:15.980 align:middle line:84% tend to overlap pretty closely with the kinds of things 00:42:15.980 --> 00:42:17.690 align:middle line:90% that computers are good at. 00:42:17.690 --> 00:42:19.483 align:middle line:84% If computers are good at things, it's 00:42:19.483 --> 00:42:20.900 align:middle line:84% going to be very difficult to make 00:42:20.900 --> 00:42:22.470 align:middle line:90% a living doing those things. 00:42:22.470 --> 00:42:25.610 align:middle line:84% So we have really good tests for the kinds of things 00:42:25.610 --> 00:42:28.670 align:middle line:84% that are probably not so important to teach people, 00:42:28.670 --> 00:42:30.713 align:middle line:84% and we have not very good tests on the kinds 00:42:30.713 --> 00:42:32.880 align:middle line:84% of things that are really important to teach people. 00:42:32.880 --> 00:42:36.270 align:middle line:84% That is a bad feature for an educational system, 00:42:36.270 --> 00:42:39.080 align:middle line:84% because autograding of all kinds is way cheaper 00:42:39.080 --> 00:42:40.460 align:middle line:90% than human grading. 00:42:40.460 --> 00:42:42.770 align:middle line:84% So we essentially build all of these systems 00:42:42.770 --> 00:42:46.010 align:middle line:84% where, conceivably, we're spending money 00:42:46.010 --> 00:42:49.308 align:middle line:84% to cheaply evaluate the least important parts of the system. 00:42:49.308 --> 00:42:51.350 align:middle line:84% And we're not evaluating the most important parts 00:42:51.350 --> 00:42:53.870 align:middle line:84% that people ought to be developing in those systems. 00:42:53.870 --> 00:42:57.020 align:middle line:84% So that is the trap of routine assessment. 00:42:57.020 --> 00:42:59.240 align:middle line:84% A really cool thing about computers 00:42:59.240 --> 00:43:01.370 align:middle line:84% is that they can collect enormous amounts of data 00:43:01.370 --> 00:43:05.030 align:middle line:84% about what folks do and that we can run experiments on people 00:43:05.030 --> 00:43:06.450 align:middle line:90% and improve those systems. 00:43:06.450 --> 00:43:08.900 align:middle line:84% So every time-- not every time, but 1% of the time 00:43:08.900 --> 00:43:11.252 align:middle line:84% you go to Google or Amazon or Facebook or whatever, 00:43:11.252 --> 00:43:13.460 align:middle line:84% they're doing some little randomized controlled trial 00:43:13.460 --> 00:43:13.980 align:middle line:90% on you. 00:43:13.980 --> 00:43:16.035 align:middle line:84% The Buy button is green instead of red. 00:43:16.035 --> 00:43:18.410 align:middle line:84% And we figure out whether you buy more from green buttons 00:43:18.410 --> 00:43:19.340 align:middle line:90% than red buttons. 00:43:19.340 --> 00:43:22.678 align:middle line:84% And eventually, Amazon, Facebook, they do millions. 00:43:22.678 --> 00:43:23.220 align:middle line:90% I don't know. 00:43:23.220 --> 00:43:25.550 align:middle line:84% They do many, many of those experiments every year, 00:43:25.550 --> 00:43:28.340 align:middle line:84% and their system continuously gets better. 00:43:28.340 --> 00:43:30.870 align:middle line:84% There are a couple of problems with doing that in education. 00:43:30.870 --> 00:43:32.450 align:middle line:84% The first problem is, if you go to folks 00:43:32.450 --> 00:43:33.740 align:middle line:84% and you say that what you want to do 00:43:33.740 --> 00:43:35.190 align:middle line:84% is experiment with their children, 00:43:35.190 --> 00:43:37.580 align:middle line:84% as we've joked before, people are not that enthusiastic 00:43:37.580 --> 00:43:39.290 align:middle line:90% about that as an idea. 00:43:39.290 --> 00:43:44.870 align:middle line:84% It's also not clear that surveilling people constantly 00:43:44.870 --> 00:43:48.560 align:middle line:84% in public schools is a good idea for civil society. 00:43:48.560 --> 00:43:50.540 align:middle line:84% An exercise maybe we'll do later on 00:43:50.540 --> 00:43:54.300 align:middle line:84% is start thinking about all of the data that MIT has about you. 00:43:54.300 --> 00:43:55.830 align:middle line:84% And we might start thinking, is it 00:43:55.830 --> 00:43:58.630 align:middle line:84% a good idea that MIT has all of this data about people? 00:43:58.630 --> 00:44:00.420 align:middle line:84% Is it a good idea for an elementary school 00:44:00.420 --> 00:44:03.292 align:middle line:84% to have all of this data about where people are moving, 00:44:03.292 --> 00:44:05.250 align:middle line:84% where they're logging in, how they're learning, 00:44:05.250 --> 00:44:07.470 align:middle line:84% their health care, all these other kinds of things? 00:44:07.470 --> 00:44:10.320 align:middle line:84% So it could be that collecting huge amounts of data 00:44:10.320 --> 00:44:12.190 align:middle line:84% lets us build really cool EdTech products. 00:44:12.190 --> 00:44:14.640 align:middle line:84% It also could be that it inducts people 00:44:14.640 --> 00:44:16.890 align:middle line:84% into a surveillance society, which ultimately is not 00:44:16.890 --> 00:44:20.700 align:middle line:84% good for democracies or civil society. 00:44:20.700 --> 00:44:22.350 align:middle line:84% So in the second half of the class, 00:44:22.350 --> 00:44:24.520 align:middle line:84% or the second half of our reading of this book, 00:44:24.520 --> 00:44:27.000 align:middle line:84% those are the four dilemmas that we'll 00:44:27.000 --> 00:44:30.330 align:middle line:84% spend some time messing around with and exploring. 00:44:30.330 --> 00:44:34.860 align:middle line:84% That is an overview of the task of the reading together. 00:44:34.860 --> 00:44:39.030 align:middle line:84% And then what we'll do for the last 20 minutes of class 00:44:39.030 --> 00:44:42.580 align:middle line:84% is to talk about massive open online courses in particular. 00:44:42.580 --> 00:44:50.220 align:middle line:84% But maybe I'll pause back here essentially on these two slides 00:44:50.220 --> 00:44:53.200 align:middle line:84% and see if people have any questions or things they 00:44:53.200 --> 00:44:54.360 align:middle line:90% want to interject here. 00:44:54.360 --> 00:44:58.600 align:middle line:90% 00:44:58.600 --> 00:45:00.490 align:middle line:84% AUDIENCE: For the next slide, when 00:45:00.490 --> 00:45:04.300 align:middle line:84% you mentioned the four as-yet intractable dilemmas, when 00:45:04.300 --> 00:45:07.510 align:middle line:84% you were studying technology and education, have 00:45:07.510 --> 00:45:12.130 align:middle line:84% you found issues that used to be dilemmas and that at some point 00:45:12.130 --> 00:45:14.800 align:middle line:84% we were actually able to solve and these 00:45:14.800 --> 00:45:16.350 align:middle line:90% are the four that are left? 00:45:16.350 --> 00:45:17.270 align:middle line:90% JUSTIN REICH: Huh. 00:45:17.270 --> 00:45:17.770 align:middle line:90% Yeah. 00:45:17.770 --> 00:45:22.780 align:middle line:84% So I think there would be some-- there would be access 00:45:22.780 --> 00:45:25.510 align:middle line:84% and distribution problems that you might have thought about 00:45:25.510 --> 00:45:29.480 align:middle line:84% as being really central to lots of different places 20 years ago 00:45:29.480 --> 00:45:30.790 align:middle line:90% that would still-- 00:45:30.790 --> 00:45:32.500 align:middle line:84% if we're talking about sub-Saharan Africa 00:45:32.500 --> 00:45:36.574 align:middle line:84% or other kinds of places in the world are still serious, 00:45:36.574 --> 00:45:39.280 align:middle line:84% refugee camps, places of protracted conflicts, 00:45:39.280 --> 00:45:41.165 align:middle line:84% lots of places where access is hard. 00:45:41.165 --> 00:45:43.540 align:middle line:84% But there's also lots of places where we'd be like, yeah, 00:45:43.540 --> 00:45:45.457 align:middle line:84% in the United States, pretty much every school 00:45:45.457 --> 00:45:46.790 align:middle line:90% has high-speed internet now. 00:45:46.790 --> 00:45:49.478 align:middle line:84% And so whereas-- we still have to think about how we do that 00:45:49.478 --> 00:45:51.270 align:middle line:84% and how we upgrade it and things like that. 00:45:51.270 --> 00:45:56.120 align:middle line:84% But that is a dilemma which has diminished quite a bit 00:45:56.120 --> 00:45:56.760 align:middle line:90% over time. 00:45:56.760 --> 00:46:00.720 align:middle line:84% And for sure, some of these dilemmas we've made progress on. 00:46:00.720 --> 00:46:03.440 align:middle line:84% I mean, I would particularly say the Trap of Routine Assessment 00:46:03.440 --> 00:46:07.050 align:middle line:84% is one where, in the last 10 or 15 years, 00:46:07.050 --> 00:46:11.180 align:middle line:84% there are definitely places where you could say, hey, 00:46:11.180 --> 00:46:13.670 align:middle line:84% we're doing better automated assessment 00:46:13.670 --> 00:46:15.540 align:middle line:84% now than we were before at different levels. 00:46:15.540 --> 00:46:17.990 align:middle line:84% So, I mean, computer science is probably 00:46:17.990 --> 00:46:21.140 align:middle line:84% the place where we have the most robust suite of autograding 00:46:21.140 --> 00:46:22.220 align:middle line:90% tools. 00:46:22.220 --> 00:46:23.960 align:middle line:84% It doesn't cover everything that's 00:46:23.960 --> 00:46:26.245 align:middle line:84% important in computer science, but boy, you 00:46:26.245 --> 00:46:27.620 align:middle line:84% can automatically evaluate people 00:46:27.620 --> 00:46:29.900 align:middle line:90% on a lot of stuff that matters. 00:46:29.900 --> 00:46:30.410 align:middle line:90% Yeah. 00:46:30.410 --> 00:46:34.760 align:middle line:84% I mean, there's an example in the book of autograding 00:46:34.760 --> 00:46:37.980 align:middle line:84% spoken language for language learners, 00:46:37.980 --> 00:46:39.950 align:middle line:84% that you can talk out loud and Duolingo 00:46:39.950 --> 00:46:42.533 align:middle line:84% can tell whether or not you've pronounced something correctly. 00:46:42.533 --> 00:46:44.420 align:middle line:90% That's pretty rad. 00:46:44.420 --> 00:46:46.917 align:middle line:90% And then, I think-- 00:46:46.917 --> 00:46:49.000 align:middle line:84% it'll be interesting to see in the next few years, 00:46:49.000 --> 00:46:52.410 align:middle line:84% but my hunch is, large language models will play an increasingly 00:46:52.410 --> 00:46:55.350 align:middle line:84% large role in at least giving people meaningful feedback 00:46:55.350 --> 00:46:57.060 align:middle line:90% on their written products. 00:46:57.060 --> 00:46:59.550 align:middle line:84% Whether or not we actually grade with them, 00:46:59.550 --> 00:47:02.887 align:middle line:90% that may be a bridge too far. 00:47:02.887 --> 00:47:04.720 align:middle line:84% I'm sure we'll use it in some circumstances. 00:47:04.720 --> 00:47:08.070 align:middle line:84% But I bet there'll be many, many circumstances-- if you told me, 00:47:08.070 --> 00:47:11.410 align:middle line:84% 10 years from now, every time a student turned in a paper, 00:47:11.410 --> 00:47:13.663 align:middle line:84% they ran it through an advanced version of something 00:47:13.663 --> 00:47:15.330 align:middle line:84% like Grammarly, which gave them not only 00:47:15.330 --> 00:47:17.170 align:middle line:84% feedback about their syntax, but was like, 00:47:17.170 --> 00:47:19.000 align:middle line:84% hey, I don't think you have any evidence in this paragraph, 00:47:19.000 --> 00:47:20.400 align:middle line:84% you might want to go back and look at that, that 00:47:20.400 --> 00:47:21.930 align:middle line:90% wouldn't surprise me at all. 00:47:21.930 --> 00:47:23.740 align:middle line:84% And that could help students a lot. 00:47:23.740 --> 00:47:27.390 align:middle line:84% If there's someone who can-- if there's an automated bot that 00:47:27.390 --> 00:47:31.030 align:middle line:84% can give you reasonable feedback generically about your writing, 00:47:31.030 --> 00:47:32.642 align:middle line:90% that would be cool. 00:47:32.642 --> 00:47:35.260 align:middle line:84% I'd rather focus on other things than giving you feedback. 00:47:35.260 --> 00:47:37.718 align:middle line:84% AUDIENCE: So I remember you mentioning in one of the slides 00:47:37.718 --> 00:47:40.590 align:middle line:84% that computers are not very good at things that 00:47:40.590 --> 00:47:44.550 align:middle line:84% don't have as much structure, but with things like LLMs, 00:47:44.550 --> 00:47:49.030 align:middle line:84% do you think that that's going to change? 00:47:49.030 --> 00:47:52.000 align:middle line:84% JUSTIN REICH: They will get better. 00:47:52.000 --> 00:47:54.410 align:middle line:90% I think there is still-- 00:47:54.410 --> 00:47:56.560 align:middle line:84% I think even some of our earliest evidence 00:47:56.560 --> 00:47:59.920 align:middle line:84% suggests that these are really thorny problems. 00:47:59.920 --> 00:48:02.470 align:middle line:84% So I feel like we're starting to get 00:48:02.470 --> 00:48:04.780 align:middle line:84% some feedback about, for instance, people 00:48:04.780 --> 00:48:08.110 align:middle line:84% using LLMs to help them in coding. 00:48:08.110 --> 00:48:09.880 align:middle line:84% And there's a joke, which is something 00:48:09.880 --> 00:48:16.690 align:middle line:84% like, using Copilot's encoding will reduce your initial coding 00:48:16.690 --> 00:48:19.900 align:middle line:84% time in half, and then it will double your quality assurance 00:48:19.900 --> 00:48:25.188 align:middle line:84% time, basically because not only will it maybe introduce errors, 00:48:25.188 --> 00:48:26.980 align:middle line:84% but also the people engineering things just 00:48:26.980 --> 00:48:29.060 align:middle line:84% won't understand what the bot is doing. 00:48:29.060 --> 00:48:32.320 align:middle line:84% And the bot is much better at very specific parts 00:48:32.320 --> 00:48:35.090 align:middle line:84% of building code than it is in engineering a whole system. 00:48:35.090 --> 00:48:37.245 align:middle line:84% And so it makes a bunch of decisions 00:48:37.245 --> 00:48:38.870 align:middle line:84% that make sense for a particular thing, 00:48:38.870 --> 00:48:40.370 align:middle line:84% but don't make sense for building a whole system. 00:48:40.370 --> 00:48:41.950 align:middle line:84% And then somebody has to go back and repair 00:48:41.950 --> 00:48:42.990 align:middle line:90% those kinds of things. 00:48:42.990 --> 00:48:45.580 align:middle line:90% 00:48:45.580 --> 00:48:47.620 align:middle line:90% So in terms of-- 00:48:47.620 --> 00:48:48.227 align:middle line:90% I don't know. 00:48:48.227 --> 00:48:50.810 align:middle line:84% We'll get into this more when we talk about autograders later. 00:48:50.810 --> 00:48:53.360 align:middle line:84% But Hal Abelson, who's been on the CS faculty forever, 00:48:53.360 --> 00:48:55.750 align:middle line:84% has this wonderful quotation in one of his earliest books 00:48:55.750 --> 00:48:57.670 align:middle line:84% on computer science, which says something 00:48:57.670 --> 00:49:05.020 align:middle line:84% like, computer programming is a language for communicating 00:49:05.020 --> 00:49:07.720 align:middle line:84% between two people about methods that's only incidentally meant 00:49:07.720 --> 00:49:09.670 align:middle line:90% to be run by a machine. 00:49:09.670 --> 00:49:12.430 align:middle line:84% And the idea is that really good code is actually 00:49:12.430 --> 00:49:16.510 align:middle line:84% talking amongst people and not talking to the machine. 00:49:16.510 --> 00:49:20.230 align:middle line:84% Our autograders are really good at figuring out whether or not 00:49:20.230 --> 00:49:22.660 align:middle line:84% code does things and machines understand, 00:49:22.660 --> 00:49:24.160 align:middle line:84% and autograders for computer science 00:49:24.160 --> 00:49:26.590 align:middle line:84% are not very good at evaluating things like, 00:49:26.590 --> 00:49:29.530 align:middle line:84% How well is this code designed for collaboration 00:49:29.530 --> 00:49:30.350 align:middle line:90% amongst people? 00:49:30.350 --> 00:49:33.480 align:middle line:84% How sensibly are the functions named? and other kinds of things 00:49:33.480 --> 00:49:33.980 align:middle line:90% like that. 00:49:33.980 --> 00:49:36.640 align:middle line:84% My hunch is, we'll make progress on it. 00:49:36.640 --> 00:49:40.060 align:middle line:84% My other hunch is that it will remain a pretty hard problem 00:49:40.060 --> 00:49:42.310 align:middle line:90% for a pretty long time. 00:49:42.310 --> 00:49:44.930 align:middle line:84% And also, it's a problem we'll solve in degrees. 00:49:44.930 --> 00:49:47.870 align:middle line:84% Will we give better automatic-- five years 00:49:47.870 --> 00:49:50.240 align:middle line:84% from now, will we give better automatic feedback 00:49:50.240 --> 00:49:52.280 align:middle line:84% to the typical middle school essay written 00:49:52.280 --> 00:49:55.010 align:middle line:84% in almost any topic, and will some of that feedback actually 00:49:55.010 --> 00:49:55.830 align:middle line:90% be useful? 00:49:55.830 --> 00:49:57.620 align:middle line:90% It seems almost certain to me. 00:49:57.620 --> 00:49:59.122 align:middle line:84% Will we be able to actually grade? 00:49:59.122 --> 00:50:00.830 align:middle line:84% Will we be able to actually say something 00:50:00.830 --> 00:50:04.850 align:middle line:84% like, this paper is better than this other paper, 00:50:04.850 --> 00:50:08.090 align:middle line:84% with all the consequences that are attendant to that? 00:50:08.090 --> 00:50:10.628 align:middle line:84% We already do that to some extent in the education system, 00:50:10.628 --> 00:50:12.170 align:middle line:84% but I think we're still going to find 00:50:12.170 --> 00:50:14.030 align:middle line:84% that we're not confident in the assessment 00:50:14.030 --> 00:50:15.560 align:middle line:90% that these machines make. 00:50:15.560 --> 00:50:18.450 align:middle line:84% But fortunately, these are all testable hypotheses. 00:50:18.450 --> 00:50:21.570 align:middle line:84% So if your predictions were more optimistic than mine, 00:50:21.570 --> 00:50:24.050 align:middle line:90% you could very well be right. 00:50:24.050 --> 00:50:27.580 align:middle line:84% Let's talk about massive open online courses for a bit. 00:50:27.580 --> 00:50:30.320 align:middle line:90% 00:50:30.320 --> 00:50:35.100 align:middle line:84% MOOCs burst on the scene in particular 2011, 2012. 00:50:35.100 --> 00:50:36.360 align:middle line:90% So almost 15 years ago now. 00:50:36.360 --> 00:50:38.998 align:middle line:84% How many of you have taken a massive open online course? 00:50:38.998 --> 00:50:40.790 align:middle line:84% How many of you have signed up for a class? 00:50:40.790 --> 00:50:42.670 align:middle line:84% How many of you signed up for one 00:50:42.670 --> 00:50:44.560 align:middle line:84% and finished it all the way through? 00:50:44.560 --> 00:50:45.200 align:middle line:90% OK. 00:50:45.200 --> 00:50:48.370 align:middle line:84% So people laugh because that is a much smaller number of folks. 00:50:48.370 --> 00:50:53.770 align:middle line:84% This is one of the distinctive features of online learning, 00:50:53.770 --> 00:50:57.610 align:middle line:84% is that online learning is really hard. 00:50:57.610 --> 00:51:00.220 align:middle line:84% If you were not alive or paying attention 00:51:00.220 --> 00:51:02.900 align:middle line:84% to education in 2011, 2012, 2013, 00:51:02.900 --> 00:51:07.900 align:middle line:84% it is hard to understand how much people assumed 00:51:07.900 --> 00:51:09.850 align:middle line:84% that these tools would completely 00:51:09.850 --> 00:51:11.920 align:middle line:90% change higher education. 00:51:11.920 --> 00:51:16.360 align:middle line:84% Not be a little bit useful maybe, but utterly rearrange 00:51:16.360 --> 00:51:17.320 align:middle line:90% educational systems. 00:51:17.320 --> 00:51:18.820 align:middle line:84% People thought there was going to be 00:51:18.820 --> 00:51:20.240 align:middle line:90% before MOOCs and after MOOCs. 00:51:20.240 --> 00:51:23.410 align:middle line:84% Part of the hype of this, the first really large-scale class 00:51:23.410 --> 00:51:25.997 align:middle line:84% like this, Sebastian Thrun and Peter Norvig, 00:51:25.997 --> 00:51:27.080 align:middle line:90% who are computer science-- 00:51:27.080 --> 00:51:28.940 align:middle line:84% Sebastian Thrun works on self-driving cars. 00:51:28.940 --> 00:51:30.350 align:middle line:84% He founded a company called Udacity. 00:51:30.350 --> 00:51:31.760 align:middle line:84% Peter Norvig worked at Google forever. 00:51:31.760 --> 00:51:33.400 align:middle line:84% They were both professors at Stanford. 00:51:33.400 --> 00:51:35.650 align:middle line:84% They created a class called Intro to AI. 00:51:35.650 --> 00:51:39.650 align:middle line:84% And they sent an email around to the IEEE mailing list saying, 00:51:39.650 --> 00:51:41.800 align:middle line:84% anybody want to take this Intro to AI class? 00:51:41.800 --> 00:51:44.230 align:middle line:84% They were particularly inspired by Sal Khan. 00:51:44.230 --> 00:51:45.040 align:middle line:90% I think it was-- 00:51:45.040 --> 00:51:48.700 align:middle line:84% who was talking about the MITx practice sets 00:51:48.700 --> 00:51:50.050 align:middle line:90% that you were going through? 00:51:50.050 --> 00:51:52.220 align:middle line:84% They were very taken by that model. 00:51:52.220 --> 00:51:53.660 align:middle line:84% You watch a little bit of a video, 00:51:53.660 --> 00:51:54.780 align:middle line:84% you solve a couple of problems, you 00:51:54.780 --> 00:51:57.363 align:middle line:84% watch a little bit more video, you solve a couple of problems, 00:51:57.363 --> 00:51:59.410 align:middle line:90% interleave them along. 00:51:59.410 --> 00:52:00.757 align:middle line:90% They send this email around. 00:52:00.757 --> 00:52:02.590 align:middle line:84% They're thinking, maybe a few hundred, maybe 00:52:02.590 --> 00:52:03.840 align:middle line:90% a few thousand people show up. 00:52:03.840 --> 00:52:05.750 align:middle line:84% 100,000 people sign up for this course. 00:52:05.750 --> 00:52:10.120 align:middle line:84% 100,000 people, multiples more than Peter and Sebastian 00:52:10.120 --> 00:52:12.820 align:middle line:84% combined would have ever taught on Stanford campus 00:52:12.820 --> 00:52:15.070 align:middle line:90% in their entire career there. 00:52:15.070 --> 00:52:18.040 align:middle line:84% And there was a vision like, wow, this 00:52:18.040 --> 00:52:20.170 align:middle line:84% could happen for everything everywhere. 00:52:20.170 --> 00:52:24.070 align:middle line:84% This could be all of higher education in the future. 00:52:24.070 --> 00:52:28.300 align:middle line:84% The amount of resources that ultimately people like Peter 00:52:28.300 --> 00:52:31.600 align:middle line:84% and Sebastian could put behind a course for 100,000 people, 00:52:31.600 --> 00:52:35.540 align:middle line:84% you could easily spend $100,000 building that course material, 00:52:35.540 --> 00:52:37.820 align:middle line:84% and then the cost of every marginal student, 00:52:37.820 --> 00:52:40.050 align:middle line:84% every additional student would be quite low. 00:52:40.050 --> 00:52:42.970 align:middle line:84% So most people who are teaching a class, you hire someone. 00:52:42.970 --> 00:52:45.730 align:middle line:84% You pay them $60,000 a year in a community college, maybe less. 00:52:45.730 --> 00:52:47.772 align:middle line:84% You pay them $100,000 a year at an elite college, 00:52:47.772 --> 00:52:48.627 align:middle line:90% maybe a little more. 00:52:48.627 --> 00:52:50.710 align:middle line:84% And they spend some of that money making a course, 00:52:50.710 --> 00:52:52.560 align:middle line:84% and 20 people take it, and that's that. 00:52:52.560 --> 00:52:54.720 align:middle line:84% But imagine if you could concentrate resources 00:52:54.720 --> 00:52:57.810 align:middle line:84% to make the best possible instructional sequence, the best 00:52:57.810 --> 00:52:58.830 align:middle line:90% course ever. 00:52:58.830 --> 00:53:02.130 align:middle line:84% I did my undergraduate at the University of Virginia. 00:53:02.130 --> 00:53:03.960 align:middle line:84% And the University of Virginia very briefly 00:53:03.960 --> 00:53:06.310 align:middle line:84% fired their president in, I think, 00:53:06.310 --> 00:53:09.810 align:middle line:84% 2013, because she was not transitioning fast enough 00:53:09.810 --> 00:53:11.790 align:middle line:90% to this new online world. 00:53:11.790 --> 00:53:17.520 align:middle line:84% And then the Board of Governors underestimated her popularity 00:53:17.520 --> 00:53:19.350 align:middle line:84% on campus, and all the faculty and students 00:53:19.350 --> 00:53:22.320 align:middle line:84% came out on the lawn and protested, and she got rehired. 00:53:22.320 --> 00:53:29.220 align:middle line:84% But imagine if MIT's president, UMass' president got 00:53:29.220 --> 00:53:31.020 align:middle line:84% fired this week because they weren't 00:53:31.020 --> 00:53:32.500 align:middle line:90% moving fast enough on AI. 00:53:32.500 --> 00:53:36.580 align:middle line:84% That was the equivalent of what was happening in 2013. 00:53:36.580 --> 00:53:38.060 align:middle line:90% It was just madness. 00:53:38.060 --> 00:53:40.700 align:middle line:84% And I had just finished my PhD, so I had just 00:53:40.700 --> 00:53:43.118 align:middle line:90% finished at Harvard in 2012. 00:53:43.118 --> 00:53:45.410 align:middle line:84% I spent a year working at an organization nearby called 00:53:45.410 --> 00:53:48.080 align:middle line:84% Facing History & Ourselves, and I got hired at Harvard 00:53:48.080 --> 00:53:50.570 align:middle line:84% as the first researcher whose responsibility 00:53:50.570 --> 00:53:54.620 align:middle line:84% was to create a research infrastructure for HarvardX. 00:53:54.620 --> 00:53:57.410 align:middle line:84% There were three big bets, that when 00:53:57.410 --> 00:54:02.840 align:middle line:84% Harvard and MIT came together to create MOOCs, to create edX-- 00:54:02.840 --> 00:54:04.820 align:middle line:84% MOOCs had been created in other places, too. 00:54:04.820 --> 00:54:06.810 align:middle line:84% Coursera at Stanford, most notably-- 00:54:06.810 --> 00:54:08.430 align:middle line:84% there were three things that they're going to try to do. 00:54:08.430 --> 00:54:10.370 align:middle line:84% They're going to provide new pathways into higher education 00:54:10.370 --> 00:54:12.740 align:middle line:84% for people who had limited access, that basically there 00:54:12.740 --> 00:54:14.990 align:middle line:84% are lots of places in the world where people can't get 00:54:14.990 --> 00:54:16.700 align:middle line:84% into higher education because they can't afford it, 00:54:16.700 --> 00:54:19.350 align:middle line:84% because it's not nearby, the programs they want aren't there. 00:54:19.350 --> 00:54:21.865 align:middle line:84% If all of that is online, then as soon as you can connect 00:54:21.865 --> 00:54:23.240 align:middle line:84% people to the internet, everybody 00:54:23.240 --> 00:54:26.000 align:middle line:90% can participate in everything. 00:54:26.000 --> 00:54:29.010 align:middle line:84% People thought that it would reorganize, disrupt, 00:54:29.010 --> 00:54:31.957 align:middle line:84% unbundle, profoundly change systems of higher education. 00:54:31.957 --> 00:54:33.540 align:middle line:84% So universities might still be around, 00:54:33.540 --> 00:54:36.228 align:middle line:84% but they could be organized in totally new ways. 00:54:36.228 --> 00:54:37.770 align:middle line:84% We were talking just at the beginning 00:54:37.770 --> 00:54:42.360 align:middle line:84% about how research universities often hire people who are not 00:54:42.360 --> 00:54:43.890 align:middle line:90% particularly good at teaching. 00:54:43.890 --> 00:54:45.690 align:middle line:84% Well, what if all the teaching just 00:54:45.690 --> 00:54:48.990 align:middle line:84% got provided by these companies that specialized in generating 00:54:48.990 --> 00:54:51.883 align:middle line:84% learning materials, and then your research professors could 00:54:51.883 --> 00:54:53.550 align:middle line:84% just go do their research and all of you 00:54:53.550 --> 00:54:56.320 align:middle line:84% could take 6.001 from an online course? 00:54:56.320 --> 00:54:58.780 align:middle line:84% AUDIENCE: Wasn't there a quote from the book that was like, 00:54:58.780 --> 00:55:02.760 align:middle line:84% in 30 years, there would only be 10 universities left 00:55:02.760 --> 00:55:03.580 align:middle line:90% in the country? 00:55:03.580 --> 00:55:05.140 align:middle line:84% JUSTIN REICH: And Udacity could be one of them. 00:55:05.140 --> 00:55:06.682 align:middle line:84% Yeah, yeah, that was Sebastian Thrun. 00:55:06.682 --> 00:55:09.013 align:middle line:84% And the idea probably from that-- it was probably not 00:55:09.013 --> 00:55:10.180 align:middle line:90% quite as crazy as it seemed. 00:55:10.180 --> 00:55:11.940 align:middle line:84% It was probably not that there would not 00:55:11.940 --> 00:55:13.630 align:middle line:84% be institutions of higher education, 00:55:13.630 --> 00:55:17.010 align:middle line:84% but you would go to UMass or Bunker Hill Community College 00:55:17.010 --> 00:55:20.700 align:middle line:84% and all of the classes being taught there would have been 00:55:20.700 --> 00:55:29.183 align:middle line:84% made at Tsinghua University, Stanford, Harvard, MIT, CMU, 00:55:29.183 --> 00:55:31.350 align:middle line:84% and a handful of other places, Udacity and a handful 00:55:31.350 --> 00:55:32.170 align:middle line:90% of other places. 00:55:32.170 --> 00:55:35.270 align:middle line:84% And going to something like UMass or Bunker Hill Community 00:55:35.270 --> 00:55:38.300 align:middle line:84% College would mean taking online courses that 00:55:38.300 --> 00:55:41.070 align:middle line:84% were created by elite institutions elsewhere. 00:55:41.070 --> 00:55:44.300 align:middle line:84% And you'd get a dorm room, and be in a fraternity, 00:55:44.300 --> 00:55:47.930 align:middle line:84% and be able to have a human do recitations and extra help 00:55:47.930 --> 00:55:49.100 align:middle line:90% and things like that. 00:55:49.100 --> 00:55:52.080 align:middle line:84% But people really-- it sounds ridiculous, 00:55:52.080 --> 00:55:53.580 align:middle line:84% but people absolutely believed that. 00:55:53.580 --> 00:55:57.440 align:middle line:84% People thought, for sure, that higher education would be-- 00:55:57.440 --> 00:56:00.740 align:middle line:84% if we went back into 2012 and said that higher education is 00:56:00.740 --> 00:56:02.570 align:middle line:84% basically the same, except having 00:56:02.570 --> 00:56:06.620 align:middle line:84% more financial difficulties, in 2024 than it was in 2012, 00:56:06.620 --> 00:56:09.380 align:middle line:84% there would be lots of people who would be very surprised 00:56:09.380 --> 00:56:12.680 align:middle line:90% by that. 00:56:12.680 --> 00:56:15.527 align:middle line:90% So I guess this presentation-- 00:56:15.527 --> 00:56:18.110 align:middle line:84% I don't know when I developed this, probably 2019 or something 00:56:18.110 --> 00:56:19.730 align:middle line:84% like that, but it was at the moment 00:56:19.730 --> 00:56:21.860 align:middle line:84% where it was pretty clear that none of these bets 00:56:21.860 --> 00:56:25.550 align:middle line:84% were going to pan out, that we were not going 00:56:25.550 --> 00:56:26.940 align:middle line:90% to accomplish these things. 00:56:26.940 --> 00:56:29.680 align:middle line:84% And I was trying to help a community-- 00:56:29.680 --> 00:56:31.430 align:middle line:84% I don't know, sometimes you'd be talking-- 00:56:31.430 --> 00:56:33.472 align:middle line:84% like the people we talked to last time, education 00:56:33.472 --> 00:56:35.150 align:middle line:84% deans and stuff like that, or provosts 00:56:35.150 --> 00:56:39.240 align:middle line:84% or funders or people educated in education, 00:56:39.240 --> 00:56:42.770 align:middle line:84% why didn't MOOCs do the things that we wanted them to do? 00:56:42.770 --> 00:56:43.645 align:middle line:90% Oh, first of all-- 00:56:43.645 --> 00:56:45.770 align:middle line:84% actually, I don't have to spend too much time on it 00:56:45.770 --> 00:56:47.930 align:middle line:84% because it sounds like everybody has taken a MOOC. 00:56:47.930 --> 00:56:50.570 align:middle line:84% But the most important thing that you need-- the two most 00:56:50.570 --> 00:56:52.880 align:middle line:84% important buttons in a MOOC are the Previous button 00:56:52.880 --> 00:56:53.813 align:middle line:90% and the Next button. 00:56:53.813 --> 00:56:55.730 align:middle line:84% That's kind of what makes it instructor-guided 00:56:55.730 --> 00:56:56.720 align:middle line:90% learning at scale. 00:56:56.720 --> 00:56:58.928 align:middle line:84% There's a bunch of stuff that we shove in the middle. 00:56:58.928 --> 00:57:00.900 align:middle line:84% And then you click Next, you do some stuff. 00:57:00.900 --> 00:57:03.390 align:middle line:84% If you get it wrong or get confused, you click Previous, 00:57:03.390 --> 00:57:06.950 align:middle line:84% but you linearly make your way through some instructor-guided 00:57:06.950 --> 00:57:08.160 align:middle line:90% set of materials. 00:57:08.160 --> 00:57:12.288 align:middle line:84% A bunch of it is video or animations or text explanation 00:57:12.288 --> 00:57:13.080 align:middle line:90% or stuff like that. 00:57:13.080 --> 00:57:15.455 align:middle line:84% And then periodically there are some assessment questions 00:57:15.455 --> 00:57:17.870 align:middle line:90% that come along. 00:57:17.870 --> 00:57:20.930 align:middle line:84% We realized, as education researchers, 00:57:20.930 --> 00:57:27.560 align:middle line:84% almost right away that folks who are furthest from opportunity 00:57:27.560 --> 00:57:31.010 align:middle line:84% were not the ones who are signing up for MOOCs first. 00:57:31.010 --> 00:57:33.330 align:middle line:84% This is a paper that was published in 2019 in Nature 00:57:33.330 --> 00:57:35.100 align:middle line:84% by Ezekiel Emanuel, who's actually 00:57:35.100 --> 00:57:37.020 align:middle line:90% mostly a medical researcher. 00:57:37.020 --> 00:57:39.367 align:middle line:84% But he was like, in the United States, 00:57:39.367 --> 00:57:41.200 align:middle line:84% here's the population with a college degree. 00:57:41.200 --> 00:57:42.617 align:middle line:84% Here's the MOOC student population 00:57:42.617 --> 00:57:43.770 align:middle line:90% with a college degree. 00:57:43.770 --> 00:57:46.300 align:middle line:84% In Brazil, Russia, India, China, South Africa, 00:57:46.300 --> 00:57:48.400 align:middle line:84% here's the population with a college degree. 00:57:48.400 --> 00:57:50.340 align:middle line:84% Here's the percent of people from those countries taking 00:57:50.340 --> 00:57:51.673 align:middle line:90% MOOCs who have a college degree. 00:57:51.673 --> 00:57:54.360 align:middle line:84% So basically, we could tell almost right away 00:57:54.360 --> 00:57:57.210 align:middle line:84% that the people that we were recruiting into MOOCs 00:57:57.210 --> 00:58:02.190 align:middle line:84% were folks who were already educated. 00:58:02.190 --> 00:58:05.370 align:middle line:84% Today, I don't know that we know everything 00:58:05.370 --> 00:58:07.470 align:middle line:90% about why that's the case. 00:58:07.470 --> 00:58:09.390 align:middle line:84% People who have earned an education 00:58:09.390 --> 00:58:12.180 align:middle line:84% probably enjoy education as leisure time activities 00:58:12.180 --> 00:58:14.410 align:middle line:84% more than people who don't pursue an education. 00:58:14.410 --> 00:58:15.872 align:middle line:90% That would be normal. 00:58:15.872 --> 00:58:17.580 align:middle line:84% But I would say that one thing that we've 00:58:17.580 --> 00:58:20.790 align:middle line:84% discovered in various formats is that online, self-paced learning 00:58:20.790 --> 00:58:24.040 align:middle line:84% is incredibly difficult for people. 00:58:24.040 --> 00:58:26.402 align:middle line:84% You all laughingly joked about that when I said, 00:58:26.402 --> 00:58:28.860 align:middle line:84% how many of you have started a MOOC that you didn't finish? 00:58:28.860 --> 00:58:31.090 align:middle line:84% And you all are some of the best people in the world 00:58:31.090 --> 00:58:32.210 align:middle line:90% at self-paced learning. 00:58:32.210 --> 00:58:33.490 align:middle line:90% And you found it hard. 00:58:33.490 --> 00:58:36.040 align:middle line:84% The entire population, the vast majority of people 00:58:36.040 --> 00:58:39.340 align:middle line:84% find online self-paced learning really, really difficult. 00:58:39.340 --> 00:58:41.630 align:middle line:84% They have other priorities in their life. 00:58:41.630 --> 00:58:42.452 align:middle line:90% It's boring. 00:58:42.452 --> 00:58:44.410 align:middle line:84% And actually, an interesting thing about people 00:58:44.410 --> 00:58:46.035 align:middle line:84% is that when they're studying something 00:58:46.035 --> 00:58:49.370 align:middle line:84% they really care about, they can be quite persistent in learning. 00:58:49.370 --> 00:58:53.230 align:middle line:84% So you study young people who are like-- right now, 00:58:53.230 --> 00:58:58.930 align:middle line:84% if you ask young tweens/teens to explore skincare routines 00:58:58.930 --> 00:59:02.750 align:middle line:84% online, they will pursue that with great passion, 00:59:02.750 --> 00:59:05.500 align:middle line:84% great determination, persist through lots of difficulties, 00:59:05.500 --> 00:59:06.820 align:middle line:90% and be successful. 00:59:06.820 --> 00:59:09.030 align:middle line:84% They will not do the same with algebra. 00:59:09.030 --> 00:59:11.530 align:middle line:84% You kind of wish as a society that they were better doing it 00:59:11.530 --> 00:59:12.950 align:middle line:84% in algebra than skincare routines, 00:59:12.950 --> 00:59:15.430 align:middle line:84% but that's just not the way human beings seem to be wired. 00:59:15.430 --> 00:59:19.870 align:middle line:84% So who gets good at online self-paced learning? 00:59:19.870 --> 00:59:23.260 align:middle line:84% How do you become good at being a good autodidact? 00:59:23.260 --> 00:59:25.127 align:middle line:84% I think the best guess we have is 00:59:25.127 --> 00:59:27.710 align:middle line:84% that you get an apprenticeship in the formal education system. 00:59:27.710 --> 00:59:29.950 align:middle line:84% The best way to be good at learning on your own 00:59:29.950 --> 00:59:31.960 align:middle line:84% is to learn with a bunch of other people. 00:59:31.960 --> 00:59:34.810 align:middle line:84% So that means the way to become really good at taking 00:59:34.810 --> 00:59:37.760 align:middle line:84% these online, self-paced MOOCs is to go get a college degree. 00:59:37.760 --> 00:59:41.770 align:middle line:84% But if your goal was to create new pathways into higher 00:59:41.770 --> 00:59:44.657 align:middle line:84% education for limited access, building a thing which 00:59:44.657 --> 00:59:46.990 align:middle line:84% you really have to have spent a lot of time in education 00:59:46.990 --> 00:59:49.660 align:middle line:84% to be good at is maybe not the best strategy. 00:59:49.660 --> 00:59:52.390 align:middle line:84% But people didn't know that at the time. 00:59:52.390 --> 00:59:55.010 align:middle line:84% This is an article that I wrote with Mariano's advisor, 00:59:55.010 --> 00:59:59.560 align:middle line:84% José, where we were trying to figure out 00:59:59.560 --> 01:00:01.540 align:middle line:90% why MOOCs were not working. 01:00:01.540 --> 01:00:04.583 align:middle line:84% And one of the things that we observed-- these 01:00:04.583 --> 01:00:07.000 align:middle line:84% are charts of enrollment, but they're charts of enrollment 01:00:07.000 --> 01:00:09.530 align:middle line:84% by human development index categories. 01:00:09.530 --> 01:00:11.380 align:middle line:84% So basically, way more people were 01:00:11.380 --> 01:00:13.445 align:middle line:84% coming from the world's most affluent countries 01:00:13.445 --> 01:00:15.070 align:middle line:84% than were coming from the world's least 01:00:15.070 --> 01:00:15.950 align:middle line:90% affluent countries. 01:00:15.950 --> 01:00:18.040 align:middle line:84% And we basically made no dent in it 01:00:18.040 --> 01:00:21.080 align:middle line:84% over the first eight years or so. 01:00:21.080 --> 01:00:24.440 align:middle line:84% You could imagine, oh, well, we started with a pretty big gap. 01:00:24.440 --> 01:00:25.870 align:middle line:84% If these three lines in the bottom 01:00:25.870 --> 01:00:28.740 align:middle line:84% had continued to traverse up, then maybe you could say, 01:00:28.740 --> 01:00:30.700 align:middle line:84% OK, we're making progress towards our goals. 01:00:30.700 --> 01:00:32.450 align:middle line:84% The other thing that was kind of a problem 01:00:32.450 --> 01:00:35.180 align:middle line:84% is, heading into the pandemic, enrollments 01:00:35.180 --> 01:00:38.070 align:middle line:84% and if you're trying to run a company or something like that, 01:00:38.070 --> 01:00:40.140 align:middle line:84% it's really not good to have this line go down. 01:00:40.140 --> 01:00:41.057 align:middle line:90% You'd really like it-- 01:00:41.057 --> 01:00:42.890 align:middle line:84% if this was Facebook's graph, it would just 01:00:42.890 --> 01:00:45.080 align:middle line:84% keep going up exponentially forever. 01:00:45.080 --> 01:00:46.970 align:middle line:84% That was another problem with MOOCs 01:00:46.970 --> 01:00:48.993 align:middle line:84% compared to other social media businesses. 01:00:48.993 --> 01:00:51.660 align:middle line:84% When you sign up for Facebook or Twitter or something like that, 01:00:51.660 --> 01:00:53.360 align:middle line:84% you're kind of stuck in there forever. 01:00:53.360 --> 01:00:56.270 align:middle line:84% When you sign up for a class, the class ends, and then 01:00:56.270 --> 01:00:59.300 align:middle line:84% you leave and go do something else with your life. 01:00:59.300 --> 01:01:01.070 align:middle line:90% So why didn't this work? 01:01:01.070 --> 01:01:06.147 align:middle line:84% Or actually, we'll talk about the second one next. 01:01:06.147 --> 01:01:07.730 align:middle line:84% But what were you going to say, Maria? 01:01:07.730 --> 01:01:09.390 align:middle line:90% AUDIENCE: I was just reflecting. 01:01:09.390 --> 01:01:10.910 align:middle line:84% One thing that I realized as well 01:01:10.910 --> 01:01:15.932 align:middle line:84% recently is that everyone now creates their own course. 01:01:15.932 --> 01:01:19.170 align:middle line:84% Every influencer has a course on something different, right? 01:01:19.170 --> 01:01:22.670 align:middle line:84% And I think that probably also affected this ability of MOOCs 01:01:22.670 --> 01:01:27.090 align:middle line:84% to spread because, originally, five or six organizations 01:01:27.090 --> 01:01:28.230 align:middle line:90% were creating content. 01:01:28.230 --> 01:01:31.000 align:middle line:84% And nowadays, this is very decentralized. 01:01:31.000 --> 01:01:33.300 align:middle line:84% So even if it's still hard for people 01:01:33.300 --> 01:01:36.210 align:middle line:84% to complete these courses online, et cetera, et cetera, 01:01:36.210 --> 01:01:39.312 align:middle line:84% there's also a lot more competition, right? 01:01:39.312 --> 01:01:41.520 align:middle line:84% JUSTIN REICH: There's a lot of competition out there. 01:01:41.520 --> 01:01:43.650 align:middle line:84% And traditional institutions prove 01:01:43.650 --> 01:01:45.370 align:middle line:84% to be very effective in that competition. 01:01:45.370 --> 01:01:47.340 align:middle line:84% So here's the thing which shocked even me. 01:01:47.340 --> 01:01:48.820 align:middle line:84% At the beginning of the pandemic-- 01:01:48.820 --> 01:01:51.160 align:middle line:84% you all remember this from March 2020, 01:01:51.160 --> 01:01:53.640 align:middle line:84% April 2020-- we sent you all home. 01:01:53.640 --> 01:01:56.820 align:middle line:84% And here's something that could have happened. 01:01:56.820 --> 01:02:00.150 align:middle line:84% There were thousands of professors in the United States 01:02:00.150 --> 01:02:03.010 align:middle line:84% who were teaching Introduction to Microeconomics. 01:02:03.010 --> 01:02:04.510 align:middle line:84% And a bunch of them could have said, 01:02:04.510 --> 01:02:07.110 align:middle line:84% there is no way I can do a good job continuing to teach you 01:02:07.110 --> 01:02:08.280 align:middle line:90% Introduction to Economics. 01:02:08.280 --> 01:02:09.720 align:middle line:84% My kids aren't in school, they're 01:02:09.720 --> 01:02:11.350 align:middle line:84% running underneath my feet right now. 01:02:11.350 --> 01:02:12.940 align:middle line:90% I've never taught online before. 01:02:12.940 --> 01:02:17.008 align:middle line:84% Just go to edX, go to Johns Hopkins, go to MIT, 01:02:17.008 --> 01:02:19.300 align:middle line:84% take their courses, and I'll give you a test at the end 01:02:19.300 --> 01:02:21.060 align:middle line:90% and help you out. 01:02:21.060 --> 01:02:25.210 align:middle line:84% The alternative is that, I take my lectures, I put them online, 01:02:25.210 --> 01:02:26.770 align:middle line:90% and we meet in Zoom once a week. 01:02:26.770 --> 01:02:28.520 align:middle line:84% And I still keep giving you your lectures. 01:02:28.520 --> 01:02:31.300 align:middle line:84% Basically every learner and institution in the world 01:02:31.300 --> 01:02:33.790 align:middle line:90% preferred the latter option. 01:02:33.790 --> 01:02:35.680 align:middle line:84% All throughout the pandemic, at all 01:02:35.680 --> 01:02:38.180 align:middle line:84% levels of the education around the world, we basically said, 01:02:38.180 --> 01:02:41.050 align:middle line:84% would you rather have this really high production value 01:02:41.050 --> 01:02:42.755 align:middle line:84% class kind of offered anonymously 01:02:42.755 --> 01:02:44.380 align:middle line:84% from some institution that maybe you've 01:02:44.380 --> 01:02:45.838 align:middle line:84% heard of but you have no connection 01:02:45.838 --> 01:02:49.308 align:middle line:84% or with your own harried professor, who's 01:02:49.308 --> 01:02:51.100 align:middle line:84% taking care of his kids while teaching you, 01:02:51.100 --> 01:02:53.100 align:middle line:84% who doesn't know anything about online learning? 01:02:53.100 --> 01:02:55.420 align:middle line:84% And people overwhelmingly picked the latter. 01:02:55.420 --> 01:02:56.602 align:middle line:90% I don't know why. 01:02:56.602 --> 01:02:58.060 align:middle line:84% I don't know why, we don't know why 01:02:58.060 --> 01:02:59.852 align:middle line:84% they picked the latter, other than the fact 01:02:59.852 --> 01:03:03.580 align:middle line:84% that education is a profoundly social enterprise. 01:03:03.580 --> 01:03:05.715 align:middle line:84% Even the comment you made about influencers having 01:03:05.715 --> 01:03:07.090 align:middle line:84% their own courses and things like 01:03:07.090 --> 01:03:09.280 align:middle line:84% that, you have a parasocial relationship 01:03:09.280 --> 01:03:12.370 align:middle line:84% with those influencers in a way that you wouldn't from Josh 01:03:12.370 --> 01:03:14.260 align:middle line:84% Angrist, who teaches the economics class here 01:03:14.260 --> 01:03:16.410 align:middle line:90% or something like that. 01:03:16.410 --> 01:03:16.910 align:middle line:90% All right. 01:03:16.910 --> 01:03:19.300 align:middle line:84% Let me try to say two more things before we head out. 01:03:19.300 --> 01:03:21.133 align:middle line:84% Oh, well, some of these things are on there. 01:03:21.133 --> 01:03:23.750 align:middle line:90% 01:03:23.750 --> 01:03:25.580 align:middle line:84% The main advantage that MOOCs have 01:03:25.580 --> 01:03:27.630 align:middle line:90% is they reduce faculty labor. 01:03:27.630 --> 01:03:29.820 align:middle line:84% They pull faculty out of the equation. 01:03:29.820 --> 01:03:33.200 align:middle line:84% They just make them television stars instead of teachers. 01:03:33.200 --> 01:03:37.880 align:middle line:84% And there's basically no way to do the economics of MOOCs, 01:03:37.880 --> 01:03:40.730 align:middle line:84% except to dramatically reduce human contact. 01:03:40.730 --> 01:03:45.020 align:middle line:84% But what helps people learn is human contact. 01:03:45.020 --> 01:03:47.645 align:middle line:84% The way that you persist in most of your learning environments, 01:03:47.645 --> 01:03:50.228 align:middle line:84% especially when you're learning something you don't care about 01:03:50.228 --> 01:03:51.870 align:middle line:84% and are not that interested in, is 01:03:51.870 --> 01:03:53.400 align:middle line:84% you care what your faculty thinks about it. 01:03:53.400 --> 01:03:54.650 align:middle line:90% You care where your peers are. 01:03:54.650 --> 01:03:58.190 align:middle line:84% You have all these other human things that keep you connected. 01:03:58.190 --> 01:04:01.040 align:middle line:84% And because of that, most people are not 01:04:01.040 --> 01:04:02.580 align:middle line:90% good at self-regulated learning. 01:04:02.580 --> 01:04:03.680 align:middle line:84% If the vast majority of people are not 01:04:03.680 --> 01:04:05.510 align:middle line:84% good at self-regulated learning, if the only way 01:04:05.510 --> 01:04:07.177 align:middle line:84% to get better at self-regulated learning 01:04:07.177 --> 01:04:09.900 align:middle line:84% is to go through the K-12 system and go to college, 01:04:09.900 --> 01:04:12.500 align:middle line:84% then this is not going to be a good system for helping folks 01:04:12.500 --> 01:04:16.730 align:middle line:84% further from opportunity get into college. 01:04:16.730 --> 01:04:19.280 align:middle line:84% The primary way that MOOCs were going to disrupt 01:04:19.280 --> 01:04:22.380 align:middle line:90% higher education is-- 01:04:22.380 --> 01:04:25.230 align:middle line:84% there was a very, for me, influential article that said, 01:04:25.230 --> 01:04:29.050 align:middle line:84% the actual innovation-- nothing was innovative in MOOCs. 01:04:29.050 --> 01:04:31.450 align:middle line:84% A learning management system is not new. edX was not new. 01:04:31.450 --> 01:04:32.410 align:middle line:90% Coursera was not new. 01:04:32.410 --> 01:04:33.760 align:middle line:90% Canvas was doing the same thing. 01:04:33.760 --> 01:04:34.720 align:middle line:90% Moodle was doing the same thing. 01:04:34.720 --> 01:04:36.220 align:middle line:84% Blackboard was doing the same thing. 01:04:36.220 --> 01:04:37.345 align:middle line:90% Autograders are not new. 01:04:37.345 --> 01:04:39.720 align:middle line:84% We had all kinds of autograders and all kinds of systems, 01:04:39.720 --> 01:04:41.580 align:middle line:84% that the actual innovation in MOOCs 01:04:41.580 --> 01:04:44.920 align:middle line:84% was putting a storefront on top of a learning management system, 01:04:44.920 --> 01:04:48.730 align:middle line:84% that the thing that was actually different was that, in the past, 01:04:48.730 --> 01:04:50.410 align:middle line:84% if you wanted to buy a Harvard course, 01:04:50.410 --> 01:04:53.760 align:middle line:84% you had to buy eight a year, and you had to rent an apartment, 01:04:53.760 --> 01:04:56.220 align:middle line:84% and you had to buy all your meals from the institution. 01:04:56.220 --> 01:04:59.250 align:middle line:84% You had to buy this giant bundle of stuff. 01:04:59.250 --> 01:05:02.550 align:middle line:84% And so the bet that Harvard, MIT, Stanford, other folks made 01:05:02.550 --> 01:05:05.460 align:middle line:84% is, what if we sold just one course at a time? 01:05:05.460 --> 01:05:07.620 align:middle line:84% What if you could come in and just 01:05:07.620 --> 01:05:11.200 align:middle line:84% take one Intro to CS class, one Introduction to Economics class. 01:05:11.200 --> 01:05:16.170 align:middle line:84% And millions of people were interested in that. 01:05:16.170 --> 01:05:17.820 align:middle line:84% I mean, the number of sign-ups that 01:05:17.820 --> 01:05:19.270 align:middle line:84% appeared in many of these courses 01:05:19.270 --> 01:05:21.760 align:middle line:84% were these astronomically large numbers. 01:05:21.760 --> 01:05:24.323 align:middle line:84% CS50 at Harvard had millions of people enrolled 01:05:24.323 --> 01:05:25.490 align:middle line:90% with just a couple of years. 01:05:25.490 --> 01:05:27.760 align:middle line:84% A tiny, tiny fraction of them finish. 01:05:27.760 --> 01:05:29.480 align:middle line:84% Something like 1% of them finish. 01:05:29.480 --> 01:05:33.225 align:middle line:84% On average, MOOCs, about 5% of people who register, about 10% 01:05:33.225 --> 01:05:34.600 align:middle line:84% of people who click into a course 01:05:34.600 --> 01:05:36.830 align:middle line:90% once will persist in a class. 01:05:36.830 --> 01:05:39.610 align:middle line:90% But pretty small numbers. 01:05:39.610 --> 01:05:44.950 align:middle line:84% However, it turned out that storefronts 01:05:44.950 --> 01:05:47.510 align:middle line:84% didn't solve all of the problems that universities stored. 01:05:47.510 --> 01:05:50.530 align:middle line:84% So people were excited to try out storefronts. 01:05:50.530 --> 01:05:53.530 align:middle line:84% But storefronts by themselves didn't confer legitimacy. 01:05:53.530 --> 01:05:56.620 align:middle line:84% Really traditional things like registrar offices 01:05:56.620 --> 01:05:58.360 align:middle line:90% confirmed legitimacy. 01:05:58.360 --> 01:06:02.260 align:middle line:84% Things that publish transcripts and degrees and diplomas 01:06:02.260 --> 01:06:04.270 align:middle line:90% confer legitimacy. 01:06:04.270 --> 01:06:09.130 align:middle line:84% And so this idea that MOOCs could move forward just 01:06:09.130 --> 01:06:13.100 align:middle line:84% by selling one course at a time proved untrue. 01:06:13.100 --> 01:06:15.310 align:middle line:84% So then MOOCs did something which 01:06:15.310 --> 01:06:17.780 align:middle line:84% was surprising to people, which they said, well, all right, 01:06:17.780 --> 01:06:20.380 align:middle line:84% why don't we start offering whole degree programs? 01:06:20.380 --> 01:06:23.360 align:middle line:84% How about instead of taking just one MOOC at a time, 01:06:23.360 --> 01:06:25.280 align:middle line:84% you'll take eight MOOCs and get an MBA? 01:06:25.280 --> 01:06:27.503 align:middle line:84% You'll take eight MOOCs and get a data science 01:06:27.503 --> 01:06:29.920 align:middle line:84% degree, a computer science degree, or something like that. 01:06:29.920 --> 01:06:32.900 align:middle line:84% There were companies that had been doing that for a long time. 01:06:32.900 --> 01:06:35.890 align:middle line:84% Those companies were called Online Program Managers. 01:06:35.890 --> 01:06:40.120 align:middle line:84% They are some of the most unreputable companies 01:06:40.120 --> 01:06:42.880 align:middle line:84% in the higher education space, for a few reasons. 01:06:42.880 --> 01:06:45.760 align:middle line:84% One is, they tended to be pretty secretive. 01:06:45.760 --> 01:06:48.640 align:middle line:84% So you would sign up for an online degree from 01:06:48.640 --> 01:06:51.400 align:middle line:84% an organization, and you think you're getting this degree from 01:06:51.400 --> 01:06:54.850 align:middle line:84% the University of Chicago, but you slowly realize that 01:06:54.850 --> 01:06:59.850 align:middle line:84% there's a company called 2U or MBANet or Pearson MBANet 01:06:59.850 --> 01:07:02.350 align:middle line:84% services or something like that, which is actually providing 01:07:02.350 --> 01:07:04.162 align:middle line:84% a bunch of your educational experiences. 01:07:04.162 --> 01:07:06.370 align:middle line:84% The Wall Street Journal, in the last couple of years, 01:07:06.370 --> 01:07:08.920 align:middle line:84% has had a bunch of exposés of boot camps 01:07:08.920 --> 01:07:10.848 align:middle line:84% that are branded by state universities, 01:07:10.848 --> 01:07:12.640 align:middle line:84% but they're actually just run by a company. 01:07:12.640 --> 01:07:15.980 align:middle line:84% It's called the Oregon Computer Science Boot Camp or whatever, 01:07:15.980 --> 01:07:18.710 align:middle line:84% Computer Programming Bootcamp, but nobody who teaches in it 01:07:18.710 --> 01:07:22.760 align:middle line:84% works for the University of Oregon, things like that. 01:07:22.760 --> 01:07:25.370 align:middle line:84% I mean, the other problem that universities faced 01:07:25.370 --> 01:07:29.630 align:middle line:84% was that a key precept of businesses 01:07:29.630 --> 01:07:33.385 align:middle line:84% is that you do not outsource your core competencies. 01:07:33.385 --> 01:07:34.760 align:middle line:84% So if you're a university and you 01:07:34.760 --> 01:07:37.233 align:middle line:84% outsource your janitorial services, that is a great idea. 01:07:37.233 --> 01:07:39.900 align:middle line:84% If you outsource your accounting services, that is a great idea. 01:07:39.900 --> 01:07:41.940 align:middle line:84% If you outsource your teaching and learning, 01:07:41.940 --> 01:07:43.137 align:middle line:90% that is a bad idea. 01:07:43.137 --> 01:07:44.720 align:middle line:84% The teaching and learning is the thing 01:07:44.720 --> 01:07:47.990 align:middle line:90% that you do as a university. 01:07:47.990 --> 01:07:50.490 align:middle line:90% So I don't know. 01:07:50.490 --> 01:07:53.720 align:middle line:84% I wrote this paper in 2019, which 01:07:53.720 --> 01:07:56.670 align:middle line:84% said something like, man, these massive open online courses, 01:07:56.670 --> 01:08:00.110 align:middle line:84% they sure seem to be behaving like Online Program Managers. 01:08:00.110 --> 01:08:02.242 align:middle line:84% They're partnering with these organizations. 01:08:02.242 --> 01:08:04.700 align:middle line:84% Part of the ways that they were compelling to organizations 01:08:04.700 --> 01:08:06.840 align:middle line:84% is that they would go to a university and say, 01:08:06.840 --> 01:08:10.640 align:middle line:84% we will pay up front for the development of these degree 01:08:10.640 --> 01:08:11.400 align:middle line:90% programs. 01:08:11.400 --> 01:08:13.790 align:middle line:84% But then, for the next 15 years, we'll 01:08:13.790 --> 01:08:19.948 align:middle line:84% take 40%, 50%, 60% of all the tuition revenue that comes in. 01:08:19.948 --> 01:08:22.343 align:middle line:84% In year one, that might seem like a good idea 01:08:22.343 --> 01:08:23.760 align:middle line:84% because you didn't spend any money 01:08:23.760 --> 01:08:25.479 align:middle line:84% and you got this new tuition revenue. 01:08:25.479 --> 01:08:29.880 align:middle line:84% But 10 years from now, you're like, why am I paying 2U, 01:08:29.880 --> 01:08:33.990 align:middle line:84% Coursera, edX, 60%, 50%, 40% of all the tuition revenue 01:08:33.990 --> 01:08:36.000 align:middle line:84% that's coming in when what they're doing is, 01:08:36.000 --> 01:08:37.470 align:middle line:90% in some respects, not that hard. 01:08:37.470 --> 01:08:38.970 align:middle line:84% They're running a learning management system 01:08:38.970 --> 01:08:40.262 align:middle line:90% and doing some online learning. 01:08:40.262 --> 01:08:42.090 align:middle line:84% We can figure out how to do those things. 01:08:42.090 --> 01:08:43.928 align:middle line:84% This was a bit of a controversial paper when 01:08:43.928 --> 01:08:45.720 align:middle line:84% I published it, and there was a whole bunch 01:08:45.720 --> 01:08:49.000 align:middle line:84% of people who said, come on, Justin, that's really not fair. 01:08:49.000 --> 01:08:52.370 align:middle line:84% You can't say that MOOCs are turning into Online Program 01:08:52.370 --> 01:08:52.870 align:middle line:90% Managers. 01:08:52.870 --> 01:08:56.100 align:middle line:84% But then two years later, 2U, who was the largest Online 01:08:56.100 --> 01:08:58.319 align:middle line:90% Program Manager, bought edX. 01:08:58.319 --> 01:09:04.830 align:middle line:84% And even that was a convergence that I wouldn't have projected. 01:09:04.830 --> 01:09:07.470 align:middle line:84% So I know some of you may know that all 01:09:07.470 --> 01:09:12.270 align:middle line:84% of the proceeds from the sale of edX 01:09:12.270 --> 01:09:15.040 align:middle line:84% went to this nonprofit controlled by Harvard and MIT, 01:09:15.040 --> 01:09:16.270 align:middle line:90% called Axim. 01:09:16.270 --> 01:09:20.859 align:middle line:90% So 2U paid $800 million for edX. 01:09:20.859 --> 01:09:24.939 align:middle line:84% And that $800 million went to start a new nonprofit. 01:09:24.939 --> 01:09:26.990 align:middle line:84% Headlines like this were kind of funny to me. 01:09:26.990 --> 01:09:29.620 align:middle line:84% "Harvard and MIT-led nonprofit to tackle longstanding 01:09:29.620 --> 01:09:31.420 align:middle line:90% inequities in education." 01:09:31.420 --> 01:09:34.644 align:middle line:90% But that's exactly what edX was. 01:09:34.644 --> 01:09:37.350 align:middle line:84% edX was a nonprofit designed to tackle inequities in education. 01:09:37.350 --> 01:09:38.870 align:middle line:84% It was like, well, that didn't work, 01:09:38.870 --> 01:09:41.520 align:middle line:90% so we're starting a new one. 01:09:41.520 --> 01:09:44.020 align:middle line:84% I just like how the big headline is like, this is this brand 01:09:44.020 --> 01:09:45.145 align:middle line:90% new thing that we're doing. 01:09:45.145 --> 01:09:48.160 align:middle line:84% And by the way, it's because we sold the last version of this 01:09:48.160 --> 01:09:49.090 align:middle line:90% that we just did. 01:09:49.090 --> 01:09:50.340 align:middle line:90% Tiny font at the bottom. 01:09:50.340 --> 01:09:52.960 align:middle line:90% 01:09:52.960 --> 01:09:56.500 align:middle line:84% This was last year's sales price for edX-- 01:09:56.500 --> 01:09:57.970 align:middle line:90% or for 2U. 01:09:57.970 --> 01:10:00.778 align:middle line:84% And yeah, their stock has declined 90% 01:10:00.778 --> 01:10:01.820 align:middle line:90% over the last five years. 01:10:01.820 --> 01:10:03.500 align:middle line:84% You can see the pandemic bubble here. 01:10:03.500 --> 01:10:05.420 align:middle line:84% People did a bunch of online learning. 01:10:05.420 --> 01:10:08.120 align:middle line:84% And then partly they were less enthusiastic about it. 01:10:08.120 --> 01:10:10.570 align:middle line:84% But also, as universities get better at offering online 01:10:10.570 --> 01:10:13.460 align:middle line:84% learning, they don't have to partner with groups like 2U 01:10:13.460 --> 01:10:14.630 align:middle line:90% anymore. 01:10:14.630 --> 01:10:19.460 align:middle line:84% This is actually today's stock quote, where it is down 99.36% 01:10:19.460 --> 01:10:21.890 align:middle line:90% over the last five years. 01:10:21.890 --> 01:10:27.260 align:middle line:84% So 2U paid for edX substantially more than the entire company 01:10:27.260 --> 01:10:29.240 align:middle line:90% of 2U is currently worth today. 01:10:29.240 --> 01:10:31.490 align:middle line:84% AUDIENCE: I guess I was just looking for clarification 01:10:31.490 --> 01:10:34.640 align:middle line:90% on edX and MITx and HarvardX. 01:10:34.640 --> 01:10:37.452 align:middle line:84% Is that all the same thing, like the MITx that we use now? 01:10:37.452 --> 01:10:38.160 align:middle line:90% JUSTIN REICH: No. 01:10:38.160 --> 01:10:38.827 align:middle line:90% Well, all right. 01:10:38.827 --> 01:10:43.460 align:middle line:84% So Harvard and MIT each put $30 million. 01:10:43.460 --> 01:10:47.330 align:middle line:84% They create units called HarvardX and MITx. 01:10:47.330 --> 01:10:50.520 align:middle line:90% Those are independent brands. 01:10:50.520 --> 01:10:53.000 align:middle line:84% The shared infrastructure that those things used 01:10:53.000 --> 01:10:54.680 align:middle line:90% was this thing called edX. 01:10:54.680 --> 01:10:55.920 align:middle line:90% edX was a company. 01:10:55.920 --> 01:10:58.940 align:middle line:84% It also created an open source learning management system 01:10:58.940 --> 01:11:00.020 align:middle line:90% called Open edX. 01:11:00.020 --> 01:11:06.390 align:middle line:84% So the nonprofit edX was sold to 2U to become part of a company. 01:11:06.390 --> 01:11:08.540 align:middle line:84% But there still is an open source project 01:11:08.540 --> 01:11:12.590 align:middle line:84% called edX, which is a learning management system software. 01:11:12.590 --> 01:11:14.330 align:middle line:84% MITx, and if there's still a thing 01:11:14.330 --> 01:11:16.590 align:middle line:84% called HarvardX, to some extent in their products, 01:11:16.590 --> 01:11:19.247 align:middle line:90% use this thing called Open edX. 01:11:19.247 --> 01:11:20.330 align:middle line:90% That's what they're doing. 01:11:20.330 --> 01:11:22.040 align:middle line:84% AUDIENCE: The infrastructure at MIT. 01:11:22.040 --> 01:11:25.460 align:middle line:84% The content that we look at when we go on MITx as students 01:11:25.460 --> 01:11:30.260 align:middle line:90% is not owned by this company. 01:11:30.260 --> 01:11:32.840 align:middle line:84% JUSTIN REICH: Yes, although it's slightly more complicated 01:11:32.840 --> 01:11:37.910 align:middle line:84% than that because I think they definitely bought the rights 01:11:37.910 --> 01:11:40.255 align:middle line:90% to display all or most of them. 01:11:40.255 --> 01:11:41.630 align:middle line:84% Maybe they could display anything 01:11:41.630 --> 01:11:43.172 align:middle line:84% that they had instructors' permission 01:11:43.172 --> 01:11:44.100 align:middle line:90% to be able to display. 01:11:44.100 --> 01:11:49.100 align:middle line:84% I took all of my courses off of edX when it was bought by 2U, 01:11:49.100 --> 01:11:51.810 align:middle line:84% because I don't want to support Online Program Managers. 01:11:51.810 --> 01:11:53.810 align:middle line:84% I don't think they're good for higher education. 01:11:53.810 --> 01:11:55.657 align:middle line:84% Some departments did that, things like that. 01:11:55.657 --> 01:11:57.990 align:middle line:84% They still might have some rights to display some of it, 01:11:57.990 --> 01:11:59.940 align:middle line:84% but they still basically did what they wanted. 01:11:59.940 --> 01:12:02.060 align:middle line:84% AUDIENCE: Because if you look at it, a lot of it-- 01:12:02.060 --> 01:12:04.320 align:middle line:84% and even OpenCourseWare is also from 2008. 01:12:04.320 --> 01:12:06.260 align:middle line:84% It hasn't been updated in many years, which 01:12:06.260 --> 01:12:07.260 align:middle line:90% I guess makes sense. 01:12:07.260 --> 01:12:08.052 align:middle line:90% JUSTIN REICH: Yeah. 01:12:08.052 --> 01:12:11.157 align:middle line:84% And in some ways, in theory, a lot of the argument 01:12:11.157 --> 01:12:12.490 align:middle line:90% is that it shouldn't have to be. 01:12:12.490 --> 01:12:18.550 align:middle line:84% How much did multivariable calculus change since 2008? 01:12:18.550 --> 01:12:20.130 align:middle line:90% Not that much. 01:12:20.130 --> 01:12:22.590 align:middle line:84% And part of the way that you gain the economy of scale 01:12:22.590 --> 01:12:24.040 align:middle line:84% is by leaving these things alone. 01:12:24.040 --> 01:12:24.310 align:middle line:90% All right. 01:12:24.310 --> 01:12:26.350 align:middle line:84% We'll come back, and I'll finish this in here, 01:12:26.350 --> 01:12:29.070 align:middle line:84% except to say that we tried a whole lot of stuff 01:12:29.070 --> 01:12:31.090 align:middle line:84% to develop a new science of learning. 01:12:31.090 --> 01:12:33.030 align:middle line:84% And probably the main thing that we learned 01:12:33.030 --> 01:12:35.130 align:middle line:84% is that people who click a lot into things 01:12:35.130 --> 01:12:36.630 align:middle line:84% do better than people who click less 01:12:36.630 --> 01:12:39.780 align:middle line:84% into things, which you probably didn't need a $60 million 01:12:39.780 --> 01:12:41.730 align:middle line:84% investment to be able to figure out. 01:12:41.730 --> 01:12:43.630 align:middle line:84% But I'll finish with this next time. 01:12:43.630 --> 01:12:45.190 align:middle line:84% You'll turn in your papers by email, 01:12:45.190 --> 01:12:48.000 align:middle line:84% and we'll talk a little bit more about intelligent tutors 01:12:48.000 --> 01:12:50.615 align:middle line:84% as well, both on Wednesday and the following Monday. 01:12:50.615 --> 01:12:52.990 align:middle line:84% If I can help you at all with paper one or anything else, 01:12:52.990 --> 01:12:53.490 align:middle line:90% let me know. 01:12:53.490 --> 01:12:56.390 align:middle line:84% Otherwise, have a wonderful rest of your week. 01:12:56.390 --> 01:13:08.000 align:middle line:90%