WEBVTT 00:00:00.060 --> 00:00:02.500 The following content is provided under a Creative 00:00:02.500 --> 00:00:04.019 Commons license. 00:00:04.019 --> 00:00:06.360 Your support will help MIT OpenCourseWare 00:00:06.360 --> 00:00:10.730 continue to offer high quality educational resources for free. 00:00:10.730 --> 00:00:13.330 To make a donation or view additional materials 00:00:13.330 --> 00:00:17.215 from 100 of MIT courses, visit MIT OpenCourseWare 00:00:17.215 --> 00:00:17.840 at ocw.mit.edu. 00:00:21.360 --> 00:00:22.594 RICHARD EBERHARDT: Hi, Pablo. 00:00:22.594 --> 00:00:23.350 PABLO SUAREZ: Good to be back. 00:00:23.350 --> 00:00:25.016 RICHARD EBERHARDT: Nice having you here. 00:00:25.016 --> 00:00:28.880 So we wanted to talk about a number 00:00:28.880 --> 00:00:32.940 of high-level postmortemy things for how the class went. 00:00:32.940 --> 00:00:36.110 So we've got some high-level things we wanted to talk about. 00:00:36.110 --> 00:00:39.160 We want to talk about the collaboration between MIT 00:00:39.160 --> 00:00:42.790 and the Red Cross/Red Crescent group and your folks-- 00:00:42.790 --> 00:00:45.960 how student preparedness went, in particular, comparing 00:00:45.960 --> 00:00:49.750 those teams that got a lot of support, like client support, 00:00:49.750 --> 00:00:50.687 like Snap, of course. 00:00:50.687 --> 00:00:53.020 But also, some of the other teams got a little bit more, 00:00:53.020 --> 00:00:54.690 and then the unsupported teams, which 00:00:54.690 --> 00:00:57.250 aren't exactly unsupported, but they got less, 00:00:57.250 --> 00:01:00.250 so that's largely the Heatwave Team, 00:01:00.250 --> 00:01:03.350 I think was the main one that we want to talk about there. 00:01:03.350 --> 00:01:07.770 So collaboration what do you all think about how 00:01:07.770 --> 00:01:08.934 we talked to each other? 00:01:08.934 --> 00:01:10.850 How we met with each other-- things like that? 00:01:13.755 --> 00:01:15.130 PHILIP TAN: I like the way how we 00:01:15.130 --> 00:01:17.150 started in the beginning of the semester. 00:01:17.150 --> 00:01:20.280 We had to talk-- and we had some Skype issues. 00:01:20.280 --> 00:01:23.470 But I think the idea of kicking off the entire class 00:01:23.470 --> 00:01:27.210 with the initial presentation of this 00:01:27.210 --> 00:01:28.960 is what your final project is going to be. 00:01:28.960 --> 00:01:30.751 It's going to take us a while to get there, 00:01:30.751 --> 00:01:32.620 but you come into class understanding 00:01:32.620 --> 00:01:33.870 this is what it's going to be. 00:01:33.870 --> 00:01:38.074 And Pablo being the person to describe 00:01:38.074 --> 00:01:39.490 why this is going to be important, 00:01:39.490 --> 00:01:42.690 I think, set lot of expectations well. 00:01:42.690 --> 00:01:51.310 So I think later on, when we actually got to that project, 00:01:51.310 --> 00:01:53.590 the degree to which each team actually 00:01:53.590 --> 00:01:55.770 could have been supported actually 00:01:55.770 --> 00:01:58.256 became more of an issue. 00:01:58.256 --> 00:02:00.130 The teams that you could directly talk with-- 00:02:00.130 --> 00:02:04.460 and you were going to be the primary user of the game-- 00:02:04.460 --> 00:02:07.456 clearly benefited. 00:02:07.456 --> 00:02:09.289 But you were right there from the beginning. 00:02:09.289 --> 00:02:13.050 These are the teams that, maybe, they did receive support, 00:02:13.050 --> 00:02:15.570 or some of them didn't receive support, 00:02:15.570 --> 00:02:17.420 but weren't talking with people who 00:02:17.420 --> 00:02:23.366 had been there right from the start, I think maybe 00:02:23.366 --> 00:02:24.990 had felt a little bit more disconnected 00:02:24.990 --> 00:02:28.480 from the main thrust of the class. 00:02:28.480 --> 00:02:31.190 At least when they got to that final assignment. 00:02:31.190 --> 00:02:35.190 So something I like to think about for future classes 00:02:35.190 --> 00:02:38.080 is the extent to which-- if we're 00:02:38.080 --> 00:02:41.030 going to repeat this format-- where the primary client comes 00:02:41.030 --> 00:02:44.870 in right at the beginning to set the tone for the entire class. 00:02:44.870 --> 00:02:49.120 How do we represent more of the variety of different things 00:02:49.120 --> 00:02:50.745 that you can do right at the beginning? 00:02:53.770 --> 00:02:55.760 So that when students think about what they're 00:02:55.760 --> 00:02:59.590 going to be doing throughout the entire semester, 00:02:59.590 --> 00:03:03.350 they are actually keeping full scope of the project in mind, 00:03:03.350 --> 00:03:06.400 not just the one example that might be presented to them 00:03:06.400 --> 00:03:07.544 on the first day of class. 00:03:07.544 --> 00:03:08.960 PABLO SUAREZ: And we also may want 00:03:08.960 --> 00:03:12.170 to have something like an engagement meter. 00:03:12.170 --> 00:03:14.330 Because I wasn't aware of the fact 00:03:14.330 --> 00:03:17.420 that some teams were feeling disengaged. 00:03:17.420 --> 00:03:19.430 I would show up, some people have questions, 00:03:19.430 --> 00:03:21.440 some other have less questions. 00:03:21.440 --> 00:03:25.550 And from my perspective, those who have questions I spent time 00:03:25.550 --> 00:03:29.820 and it was about the same chronometric time per group. 00:03:29.820 --> 00:03:31.540 I could see that some were more lost, 00:03:31.540 --> 00:03:33.262 but I wasn't aware that what was needed 00:03:33.262 --> 00:03:34.470 was more fire in their belly. 00:03:34.470 --> 00:03:35.240 So if we could-- 00:03:35.240 --> 00:03:35.960 RICHARD EBERHARDT: I couldn't see that either. 00:03:35.960 --> 00:03:36.350 PABLO SUAREZ: Right. 00:03:36.350 --> 00:03:38.683 RICHARD EBERHARDT: The students never reported it to us. 00:03:38.683 --> 00:03:40.480 SARA VERRILLI: Yeah, I was going to say, 00:03:40.480 --> 00:03:42.230 when we asked the students to do their postmortems, 00:03:42.230 --> 00:03:45.070 they were really clear about how this had been a big motivation, 00:03:45.070 --> 00:03:47.140 and interest had been a big problem for them. 00:03:47.140 --> 00:03:50.170 But they never mentioned it to us during check-in meetings 00:03:50.170 --> 00:03:53.150 when we were doing play testing, when we were working with them. 00:03:53.150 --> 00:03:55.420 So that's the sort of thing that we 00:03:55.420 --> 00:03:57.420 need to-- probably one of the things 00:03:57.420 --> 00:04:00.920 you do is probably to warn them about that. 00:04:00.920 --> 00:04:03.570 These are hard topics, and they can become boring topics 00:04:03.570 --> 00:04:05.850 very quickly. 00:04:05.850 --> 00:04:10.010 And they need to step up and let us know when they think 00:04:10.010 --> 00:04:12.020 they're facing a brick wall. 00:04:12.020 --> 00:04:14.960 RICHARD EBERHARDT: I think the clues were there. 00:04:14.960 --> 00:04:18.649 If we had means and a way to remind ourselves, 00:04:18.649 --> 00:04:21.160 like a checklist or something simple like that, just to say, 00:04:21.160 --> 00:04:22.210 are you engaged? 00:04:22.210 --> 00:04:23.604 What's your motivation? 00:04:23.604 --> 00:04:25.270 And, again, that's assuming that we even 00:04:25.270 --> 00:04:27.070 want that to be our job. 00:04:27.070 --> 00:04:28.540 I think for anything client based, 00:04:28.540 --> 00:04:30.037 it has be our, probably, our job. 00:04:30.037 --> 00:04:31.620 SARA VERRILLI: I think that when we're 00:04:31.620 --> 00:04:35.250 asking them to work on projects where they have relatively 00:04:35.250 --> 00:04:38.587 little control over the topic, I think we actually do need to. 00:04:38.587 --> 00:04:40.170 Because I think part of it was I don't 00:04:40.170 --> 00:04:42.050 think the teams knew how much freedom they 00:04:42.050 --> 00:04:46.622 had to deviate from the topic they had chosen 00:04:46.622 --> 00:04:47.330 and the specific. 00:04:47.330 --> 00:04:48.996 They thought they had chosen this thing, 00:04:48.996 --> 00:04:50.900 and now they were stuck with it. 00:04:50.900 --> 00:04:53.180 Even though one team was like, we 00:04:53.180 --> 00:04:55.060 chose a topic that just wasn't deep enough. 00:04:55.060 --> 00:04:57.490 There wasn't enough there. 00:04:57.490 --> 00:04:59.320 And if they had come and talked to us, 00:04:59.320 --> 00:05:00.820 we could have given them the freedom 00:05:00.820 --> 00:05:03.110 to well, go change to a new topic, or how can we go-- 00:05:03.110 --> 00:05:03.330 RICHARD EBERHARDT: Go deeper. 00:05:03.330 --> 00:05:04.600 SARA VERRILLI: --how can we go deeper in it? 00:05:04.600 --> 00:05:05.340 Right. 00:05:05.340 --> 00:05:08.010 But we were not sufficiently warned. 00:05:08.010 --> 00:05:09.840 PHILIP TAN: But that's where rapport 00:05:09.840 --> 00:05:13.450 with someone who knows a lot about the domain can help. 00:05:13.450 --> 00:05:16.720 And if they don't feel like they have that rapport or someone 00:05:16.720 --> 00:05:20.140 who they can readily contact to ask even seemingly 00:05:20.140 --> 00:05:22.522 silly questions, then we're not going 00:05:22.522 --> 00:05:24.105 to be able to get down to actually ask 00:05:24.105 --> 00:05:25.270 the important questions. 00:05:25.270 --> 00:05:26.686 Because sometimes they can't tell. 00:05:26.686 --> 00:05:30.190 They can't necessarily tell what looks like flagging motivation 00:05:30.190 --> 00:05:32.600 on a part of the team or what just 00:05:32.600 --> 00:05:35.911 seems like end-of-semester pressures on the students, 00:05:35.911 --> 00:05:36.410 right? 00:05:39.480 --> 00:05:41.460 They haven't been working on enough projects 00:05:41.460 --> 00:05:43.690 to be able to tell the difference. 00:05:43.690 --> 00:05:46.490 And so we do need to help them through that, I think. 00:05:46.490 --> 00:05:47.870 SARA VERRILLI: By that you mean the students, then? 00:05:47.870 --> 00:05:48.575 PHILIP TAN: The students. 00:05:48.575 --> 00:05:48.920 I mean the students. 00:05:48.920 --> 00:05:51.420 PABLO SUAREZ: And one of the things you may want to consider 00:05:51.420 --> 00:05:53.950 is we know that both in the real life-- in the future, 00:05:53.950 --> 00:05:56.800 in their future, as well as in future courses-- 00:05:56.800 --> 00:05:58.950 there will be limited ability of those 00:05:58.950 --> 00:06:01.600 who are experts to interact with the game design team. 00:06:01.600 --> 00:06:03.781 The game design team, you took the effort-- 00:06:03.781 --> 00:06:06.030 and I thought it was very well-balanced-- to have some 00:06:06.030 --> 00:06:08.640 who know programming, some who know narrative, some who 00:06:08.640 --> 00:06:11.340 know user interface, and so on. 00:06:11.340 --> 00:06:13.480 They will, by default, not have expertise 00:06:13.480 --> 00:06:17.780 on heatwaves or cholera or etc or climate science. 00:06:17.780 --> 00:06:21.740 They cannot assume that all they need to know-- 00:06:21.740 --> 00:06:23.650 including the appetite for engagement-- 00:06:23.650 --> 00:06:26.730 will be only in the client. 00:06:26.730 --> 00:06:32.530 MIT is a bubbling powerhouse of engaging stuff. 00:06:32.530 --> 00:06:35.825 And if they could, as part of what they are told-- they 00:06:35.825 --> 00:06:37.450 will have to do in the future, and what 00:06:37.450 --> 00:06:39.620 they have to do in this course, to reach out 00:06:39.620 --> 00:06:44.350 to engaging people, fellow students, and junior faculty 00:06:44.350 --> 00:06:48.430 or whoever may give them some time-- to just go and say, 00:06:48.430 --> 00:06:49.910 what is exciting about this? 00:06:49.910 --> 00:06:51.500 What are the cutting-edge areas? 00:06:51.500 --> 00:06:53.120 What are the institutional challenges. 00:06:53.120 --> 00:06:56.355 So that they also have to fuel their own appetite. 00:06:56.355 --> 00:06:57.980 RICHARD EBERHARDT: Having an assignment 00:06:57.980 --> 00:07:01.000 or having part of that final project being like here's 00:07:01.000 --> 00:07:03.650 the research component that we expect you to do, 00:07:03.650 --> 00:07:06.076 but we didn't put an underline over it. 00:07:06.076 --> 00:07:08.700 SARA VERRILLI: We didn't make it explicit that they needed to-- 00:07:08.700 --> 00:07:10.741 RICHARD EBERHARDT: We did when it was in trouble. 00:07:10.741 --> 00:07:14.080 Like for Saving Gora Gora, they were in big trouble. 00:07:14.080 --> 00:07:16.750 We told them a number-- you told them, we told them. 00:07:16.750 --> 00:07:19.220 I think you'd be surprised what they came up with. 00:07:19.220 --> 00:07:21.672 Because they actually, I think, they took the baton 00:07:21.672 --> 00:07:22.380 and went with it. 00:07:22.380 --> 00:07:23.190 SARA VERRILLI: They did that [INAUDIBLE]. 00:07:23.190 --> 00:07:23.490 PHILIP TAN: Nice. 00:07:23.490 --> 00:07:25.609 RICHARD EBERHARDT: The other cholera team did not, 00:07:25.609 --> 00:07:27.150 and they were given the same-- almost 00:07:27.150 --> 00:07:28.580 the exact same-- feedback. 00:07:28.580 --> 00:07:29.330 That is the thing. 00:07:29.330 --> 00:07:31.970 PHILIP TAN: I think there is a huge social barrier 00:07:31.970 --> 00:07:36.360 between the undergraduates and the rest of the institute. 00:07:36.360 --> 00:07:39.620 The noble laureates, and even the post-docs here, 00:07:39.620 --> 00:07:44.070 that's a lot of social pressure keeping students away 00:07:44.070 --> 00:07:45.650 from that kind of interaction. 00:07:45.650 --> 00:07:46.660 You're all right. 00:07:46.660 --> 00:07:50.086 All of these things do exist here at MIT, but some of this 00:07:50.086 --> 00:07:51.970 is reinforced by professors themselves-- 00:07:51.970 --> 00:07:53.553 professors who keep their doors closed 00:07:53.553 --> 00:07:56.120 when they in their office, things like that. 00:07:56.120 --> 00:08:01.680 And so I do think we have to do give them the license, 00:08:01.680 --> 00:08:04.014 and sometimes that's pointing them to specific people. 00:08:04.014 --> 00:08:06.430 RICHARD EBERHARDT: Yeah, giving them resources like here's 00:08:06.430 --> 00:08:07.220 a list. 00:08:07.220 --> 00:08:08.710 One of these might pan out for you. 00:08:08.710 --> 00:08:09.585 The others might not. 00:08:09.585 --> 00:08:13.950 And that's OK, just do what you can, kind of prime them for it. 00:08:13.950 --> 00:08:17.724 PHILIP TAN: It's hard to be a teenager in this space. 00:08:17.724 --> 00:08:19.390 RICHARD EBERHARDT: Actually, since we're 00:08:19.390 --> 00:08:20.973 talking about the student preparedness 00:08:20.973 --> 00:08:24.880 in that research like what we saw between what we consider 00:08:24.880 --> 00:08:28.152 the supported teams versus the unsupported teams, which 00:08:28.152 --> 00:08:30.110 may or may not be fair-- could be the motivated 00:08:30.110 --> 00:08:32.150 teams versus the unmotivated teams. 00:08:32.150 --> 00:08:33.826 Snap, in particular, really-- 00:08:33.826 --> 00:08:35.409 PABLO SUAREZ: [INAUDIBLE] correlation. 00:08:35.409 --> 00:08:37.809 RICHARD EBERHARDT: --really well-supported. 00:08:37.809 --> 00:08:41.977 They actually reached out to you all for help with the game. 00:08:41.977 --> 00:08:44.310 You actually even did testing for them that was actually 00:08:44.310 --> 00:08:46.780 coming from you and Jana, yeah, because you 00:08:46.780 --> 00:08:47.800 were running workshops. 00:08:47.800 --> 00:08:48.633 PABLO SUAREZ: Right. 00:08:48.633 --> 00:08:51.140 RICHARD EBERHARDT: How did those workshops run-- how did you 00:08:51.140 --> 00:08:52.980 interact with the team? 00:08:52.980 --> 00:08:54.570 PABLO SUAREZ: So, first of all, I 00:08:54.570 --> 00:08:58.070 think it's important to distinguish not only what you 00:08:58.070 --> 00:08:59.990 are calling supportive versus unsupportive, 00:08:59.990 --> 00:09:01.950 but also Snap versus everything else. 00:09:01.950 --> 00:09:04.660 Because Snap was a game concept that 00:09:04.660 --> 00:09:07.750 preexisted the course that needed refinement 00:09:07.750 --> 00:09:09.815 and digital interface. 00:09:09.815 --> 00:09:11.690 So from that perspective, it's not surprising 00:09:11.690 --> 00:09:13.150 that they have something clearly defined that they 00:09:13.150 --> 00:09:14.710 could run faster and deeper. 00:09:14.710 --> 00:09:17.040 And the group was a group that engaged and went 00:09:17.040 --> 00:09:19.890 deep and went fast. 00:09:19.890 --> 00:09:21.970 The interaction we had with them, 00:09:21.970 --> 00:09:25.450 also, very importantly, we have events 00:09:25.450 --> 00:09:27.250 that we could use the product. 00:09:27.250 --> 00:09:28.930 So it was much easier to give feedback 00:09:28.930 --> 00:09:30.445 saying when we do this, we expect 00:09:30.445 --> 00:09:33.070 people that would get bored, why don't we try this other thing. 00:09:33.070 --> 00:09:34.528 RICHARD EBERHARDT: It's milestones, 00:09:34.528 --> 00:09:35.950 which we give them milestones. 00:09:35.950 --> 00:09:37.630 But there is no weight behind them. 00:09:37.630 --> 00:09:40.620 These ones have an actual, if it doesn't work by then, 00:09:40.620 --> 00:09:42.670 someone's going to have egg on their face, right? 00:09:42.670 --> 00:09:45.000 PABLO SUAREZ: And from that perspective, we know this, 00:09:45.000 --> 00:09:49.820 but we can try to do more of giving students 00:09:49.820 --> 00:09:53.170 more proportion of work that we know-- that we, the clients 00:09:53.170 --> 00:09:54.800 know-- that we have milestones. 00:09:54.800 --> 00:09:57.570 And that is limited by how much of the real world 00:09:57.570 --> 00:09:59.630 gives us milestones to abide by. 00:09:59.630 --> 00:10:01.310 But we can try that. 00:10:01.310 --> 00:10:05.360 I do not feel comfortable with the label supportive versus 00:10:05.360 --> 00:10:10.150 unsupportive because it sounds like some got time and support 00:10:10.150 --> 00:10:11.660 and some others didn't. 00:10:11.660 --> 00:10:15.520 I think it's more about the form of the relationship 00:10:15.520 --> 00:10:18.640 and how the time was used or the opportunity to use the time 00:10:18.640 --> 00:10:19.510 was used. 00:10:19.510 --> 00:10:22.990 Because I have no doubt that you supported the team when 00:10:22.990 --> 00:10:25.110 you knew support was needed. 00:10:25.110 --> 00:10:28.280 And as far as I know, the same was true on our part. 00:10:28.280 --> 00:10:30.520 I think it's more about having sensors 00:10:30.520 --> 00:10:33.340 about when support is needed that is not being delivered. 00:10:33.340 --> 00:10:35.673 RICHARD EBERHARDT: I think when we're talking about them 00:10:35.673 --> 00:10:38.150 in this way, I'm using it based on how 00:10:38.150 --> 00:10:40.050 they describe their own relationship 00:10:40.050 --> 00:10:41.480 and their own experience. 00:10:41.480 --> 00:10:43.515 So that the teams who we are calling unsupported 00:10:43.515 --> 00:10:45.140 are the ones who, in their postmortems, 00:10:45.140 --> 00:10:46.598 called out and said, we didn't feel 00:10:46.598 --> 00:10:48.240 like we got client support. 00:10:48.240 --> 00:10:49.660 They got the same amount. 00:10:49.660 --> 00:10:52.000 It was a different way, different quality, 00:10:52.000 --> 00:10:53.110 different quantity. 00:10:53.110 --> 00:10:55.693 SARA VERRILLI: The way to put it is the same amount of support 00:10:55.693 --> 00:10:58.199 was there for them. 00:10:58.199 --> 00:10:59.990 One other way to put it is they didn't know 00:10:59.990 --> 00:11:02.590 how to reach out and get it. 00:11:02.590 --> 00:11:04.170 They didn't use the resources that 00:11:04.170 --> 00:11:06.810 were there because, at some level, 00:11:06.810 --> 00:11:08.470 we weren't pushing them at them. 00:11:08.470 --> 00:11:11.950 And a lot of MIT courses tend to encourage students to solve 00:11:11.950 --> 00:11:13.454 all their problems themselves. 00:11:13.454 --> 00:11:14.870 They don't want them reaching out. 00:11:14.870 --> 00:11:17.430 They want them-- figure this out. 00:11:17.430 --> 00:11:20.315 And so it may come as more of a second-- 00:11:20.315 --> 00:11:22.440 it may not come to them as first nature to go, hey, 00:11:22.440 --> 00:11:25.065 we need to go and reach out for more help and grab that-- 00:11:25.065 --> 00:11:27.440 RICHARD EBERHARDT: And so that's actually the same thing. 00:11:27.440 --> 00:11:29.020 So what we saw with the Snap team is 00:11:29.020 --> 00:11:31.360 we wanted-- the feedback they were getting from us 00:11:31.360 --> 00:11:33.615 was look at your front end, look at your UI. 00:11:33.615 --> 00:11:37.370 We wanted to see a lot of changes on the UI. 00:11:37.370 --> 00:11:39.365 Because they were working on these milestones, 00:11:39.365 --> 00:11:41.250 they were really worried about the back end 00:11:41.250 --> 00:11:43.291 so they were doing a lot of work in the back end. 00:11:46.272 --> 00:11:47.980 They were getting two different messages, 00:11:47.980 --> 00:11:50.970 and they chose client message rather than instructor message. 00:11:50.970 --> 00:11:52.275 Totally OK and totally viable. 00:11:54.997 --> 00:11:56.080 We took that into account. 00:11:56.080 --> 00:11:58.260 But I think the main issues we had 00:11:58.260 --> 00:12:00.150 with them is when they got our message, 00:12:00.150 --> 00:12:02.317 it was, they heard it early on. 00:12:02.317 --> 00:12:04.650 They didn't react to it until it was too late-- had they 00:12:04.650 --> 00:12:07.220 done a little bit extra in the beginning. 00:12:07.220 --> 00:12:09.770 So one thing I think we need to think about with the class 00:12:09.770 --> 00:12:13.410 is thinking about how are we helping them understand 00:12:13.410 --> 00:12:16.570 how to prioritize feedback? 00:12:16.570 --> 00:12:17.880 Right? 00:12:17.880 --> 00:12:20.500 What's more important? 00:12:20.500 --> 00:12:22.350 Could be because here's something that's 00:12:22.350 --> 00:12:23.474 actually going to get used. 00:12:23.474 --> 00:12:26.230 It could also be here's a lesson we want to make sure they get. 00:12:26.230 --> 00:12:27.396 I think they got the lesson. 00:12:27.396 --> 00:12:29.314 They just got it in a harder way. 00:12:29.314 --> 00:12:31.730 PHILIP TAN: Sometimes they get a lesson at a point of time 00:12:31.730 --> 00:12:33.880 that they can't actually execute on it. 00:12:33.880 --> 00:12:37.120 But hopefully they can do it for their next project after class. 00:12:39.650 --> 00:12:41.580 Something I want to suggest, actually, 00:12:41.580 --> 00:12:44.120 because we've been talking about how it's been difficult 00:12:44.120 --> 00:12:48.040 for us to figure out what's going on inside the teams. 00:12:48.040 --> 00:12:51.410 We can see the games and the testing, 00:12:51.410 --> 00:12:55.100 but the team dynamics, we don't find out about that 00:12:55.100 --> 00:12:57.950 until near the end of the class. 00:12:57.950 --> 00:13:02.010 And something that we used to do outside of this class, 00:13:02.010 --> 00:13:04.640 but in our research work, was to get all off 00:13:04.640 --> 00:13:07.540 the scrum masters together on a regular basis 00:13:07.540 --> 00:13:11.497 to just do an exchange of information with each other. 00:13:11.497 --> 00:13:13.830 They don't even have to be necessarily reporting to us-- 00:13:13.830 --> 00:13:16.020 although we can moderate those sessions. 00:13:16.020 --> 00:13:19.270 It's more about them sharing their experiences with each 00:13:19.270 --> 00:13:22.970 other, and then it may make it more obvious that there are 00:13:22.970 --> 00:13:25.849 resources out there that some teams are taking advantage 00:13:25.849 --> 00:13:26.890 of and some teams aren't. 00:13:30.094 --> 00:13:32.260 Of course, we can use those situations for teaching, 00:13:32.260 --> 00:13:37.640 but also for teaching each other. 00:13:37.640 --> 00:13:40.225 SARA VERRILLI: Creating some group, group support. 00:13:40.225 --> 00:13:41.850 RICHARD EBERHARDT: We talked about that 00:13:41.850 --> 00:13:44.590 before this semester started, but we couldn't figure out 00:13:44.590 --> 00:13:47.860 a way to do that fairly. 00:13:47.860 --> 00:13:49.900 I already feel producers bear the brunt 00:13:49.900 --> 00:13:52.419 of the work in the class. 00:13:52.419 --> 00:13:53.960 They're the ones we see all the time. 00:13:53.960 --> 00:13:56.480 We don't see the students working on programming 00:13:56.480 --> 00:13:57.530 or design as much. 00:13:57.530 --> 00:13:58.980 Design a little bit more, but programming, 00:13:58.980 --> 00:14:00.230 we just almost never see them. 00:14:00.230 --> 00:14:03.230 SARA VERRILLI: And we let the teams use the producers 00:14:03.230 --> 00:14:06.110 as their face-- so giving all the presentations, 00:14:06.110 --> 00:14:08.750 giving all the reports, answering questions when 00:14:08.750 --> 00:14:12.640 we ask questions in class. 00:14:12.640 --> 00:14:15.720 Think about how we convince the teams to share out that work 00:14:15.720 --> 00:14:17.370 a little bit more evenly. 00:14:17.370 --> 00:14:18.870 PHILIP TAN: Well, perhaps, maybe, we 00:14:18.870 --> 00:14:23.735 can reduce the extent that we require 00:14:23.735 --> 00:14:26.480 these producers to do presentations in class, 00:14:26.480 --> 00:14:30.666 and replace that with these more closed-off sessions 00:14:30.666 --> 00:14:32.790 where they don't feel like they have to be on stage 00:14:32.790 --> 00:14:34.510 and put on the best face for their team. 00:14:34.510 --> 00:14:36.080 They can actually just talk about this is the problem 00:14:36.080 --> 00:14:37.079 I'm having with my team. 00:14:38.597 --> 00:14:40.180 RICHARD EBERHARDT: So one thing I also 00:14:40.180 --> 00:14:41.596 wanted to make sure that we talked 00:14:41.596 --> 00:14:43.230 about what was it like for you? 00:14:43.230 --> 00:14:45.670 How much of your time did we demand? 00:14:45.670 --> 00:14:47.640 Was it a lot compared to other projects? 00:14:47.640 --> 00:14:48.620 Was it a little bit? 00:14:48.620 --> 00:14:50.625 Was it about moderate? 00:14:50.625 --> 00:14:53.000 PABLO SUAREZ: I want to start by saying one of the things 00:14:53.000 --> 00:14:55.760 that I really appreciated was the time we spent-- 00:14:55.760 --> 00:14:58.700 both face-to-face and emailing and so 00:14:58.700 --> 00:15:01.550 on-- before the beginning of the course, 00:15:01.550 --> 00:15:04.100 to have understanding of what you and your students 00:15:04.100 --> 00:15:07.320 can offer, what we need and can offer, 00:15:07.320 --> 00:15:09.160 and to manage expectations, as you said. 00:15:09.160 --> 00:15:10.990 I think that was very well done. 00:15:10.990 --> 00:15:12.590 And it's not always the case. 00:15:12.590 --> 00:15:16.620 So with the students and professors and Red Cross, 00:15:16.620 --> 00:15:19.480 in general, we [INAUDIBLE], so thank you. 00:15:19.480 --> 00:15:21.030 That was really appreciated. 00:15:21.030 --> 00:15:27.135 I think that amount of time that I personally had to give 00:15:27.135 --> 00:15:30.290 was about as much as I anticipated, about as much 00:15:30.290 --> 00:15:33.880 as I think is reasonable to expect for the future. 00:15:33.880 --> 00:15:36.770 We do have Jana, in particular, and a few other members 00:15:36.770 --> 00:15:39.120 of the team-- of the Red Cross team-- 00:15:39.120 --> 00:15:41.900 who were available to give a hand. 00:15:41.900 --> 00:15:44.460 And I think we needed to improve because I 00:15:44.460 --> 00:15:46.150 did make it very clear that I couldn't 00:15:46.150 --> 00:15:49.430 get five emails per day asking questions. 00:15:49.430 --> 00:15:53.460 But to have some kind of ability for the right contact when 00:15:53.460 --> 00:15:57.220 either engagement levels are too low or mega confusion 00:15:57.220 --> 00:16:01.380 levels are too high or sense of purpose is diluted and so on. 00:16:01.380 --> 00:16:04.570 So to refine that. 00:16:04.570 --> 00:16:09.730 I think it would be better to think of times 00:16:09.730 --> 00:16:15.050 when there can be slots of decided interaction 00:16:15.050 --> 00:16:18.110 between each team, and someone from the client organization 00:16:18.110 --> 00:16:22.050 that doesn't have to be during class. 00:16:22.050 --> 00:16:24.494 Because I would always interfere with your plan. 00:16:24.494 --> 00:16:25.910 You gave us one opportunity that I 00:16:25.910 --> 00:16:28.550 thought was very useful, but a little bit late 00:16:28.550 --> 00:16:32.346 in the development process and also very crammed. 00:16:32.346 --> 00:16:33.762 By the time we said, OK, these are 00:16:33.762 --> 00:16:35.760 the kind of things you can do, there was no time 00:16:35.760 --> 00:16:37.135 for them to think and reflect on, 00:16:37.135 --> 00:16:40.440 OK, we can do A or B or C. Show that to us, 00:16:40.440 --> 00:16:41.960 and then us providing feedback. 00:16:41.960 --> 00:16:44.940 So even if it were one more chance via Skype 00:16:44.940 --> 00:16:49.960 but scheduled, planned, and if-- it is in the nature of our work 00:16:49.960 --> 00:16:52.440 to have to travel, and it is in the nature of our work 00:16:52.440 --> 00:16:55.616 that travel timing changes outside of our control. 00:16:55.616 --> 00:16:57.740 So where there's a default date, and if things have 00:16:57.740 --> 00:16:59.020 to be changed, it is changed. 00:16:59.020 --> 00:17:02.420 So to know that there's going to be an instance of feedback that 00:17:02.420 --> 00:17:06.830 may be too little, too late, still but to anticipate that. 00:17:06.830 --> 00:17:09.079 RICHARD EBERHARDT: Yeah, part of that is the students, 00:17:09.079 --> 00:17:11.240 we don't always know the student's schedules. 00:17:11.240 --> 00:17:14.130 So it could possibly be planned in the beginning 00:17:14.130 --> 00:17:14.910 of the semester. 00:17:14.910 --> 00:17:16.500 But planning it before the semester 00:17:16.500 --> 00:17:17.910 would be really difficult. Unless we just say it-- 00:17:17.910 --> 00:17:19.810 this is the time that you're going to have. 00:17:19.810 --> 00:17:21.976 If you don't take advantage of it, you don't get it. 00:17:21.976 --> 00:17:24.099 PABLO SUAREZ: Right. 00:17:24.099 --> 00:17:26.260 It has to be we propose it's this week. 00:17:26.260 --> 00:17:27.540 This is the default time slot. 00:17:27.540 --> 00:17:31.036 If not, that time slot, agree on a different time slot. 00:17:31.036 --> 00:17:32.410 And then it's up to the students. 00:17:32.410 --> 00:17:33.060 RICHARD EBERHARDT: So you're actually 00:17:33.060 --> 00:17:35.450 saying you would have talked to them a little bit more. 00:17:35.450 --> 00:17:38.230 PABLO SUAREZ: If the problem was lack 00:17:38.230 --> 00:17:40.410 of engagement or perception of lack of support. 00:17:40.410 --> 00:17:40.720 RICHARD EBERHARDT: So it's really 00:17:40.720 --> 00:17:41.770 that planning and scheduling. 00:17:41.770 --> 00:17:43.310 So as long as that's in place, then the time 00:17:43.310 --> 00:17:44.810 is there, because it's been planned. 00:17:44.810 --> 00:17:45.460 OK. 00:17:45.460 --> 00:17:46.835 PABLO SUAREZ: And also I think it 00:17:46.835 --> 00:17:50.160 would have been nice for them to have access to, say, someone 00:17:50.160 --> 00:17:53.030 from the Ghana Team to look at the color, someone 00:17:53.030 --> 00:17:57.180 from the Heatwave Team in Buenos Aires-- to at least 00:17:57.180 --> 00:17:59.690 to get a flavor, there's another human being there that 00:17:59.690 --> 00:18:01.300 eventually can use their game. 00:18:01.300 --> 00:18:04.630 And I didn't do enough to help them envision 00:18:04.630 --> 00:18:06.730 the user-- either the Red Cross user 00:18:06.730 --> 00:18:11.190 or the community level or policy or government user. 00:18:11.190 --> 00:18:14.591 PHILIP TAN: I think we need to state that strongly 00:18:14.591 --> 00:18:17.410 that it's not just the information that you're 00:18:17.410 --> 00:18:18.000 providing. 00:18:18.000 --> 00:18:21.240 It's the fact that there is a real person who can really 00:18:21.240 --> 00:18:23.830 benefit from the work you're doing is much more 00:18:23.830 --> 00:18:25.855 powerful than some sort of abstracted audience. 00:18:25.855 --> 00:18:27.230 RICHARD EBERHARDT: So we provided 00:18:27.230 --> 00:18:30.990 plenty of documentation, but that's not 00:18:30.990 --> 00:18:32.130 the same as a human being. 00:18:32.130 --> 00:18:34.472 PHILIP TAN: That's not the emotional drive 00:18:34.472 --> 00:18:36.180 that you're going to get-- oh, someone is 00:18:36.180 --> 00:18:37.472 counting on me doing good work. 00:18:37.472 --> 00:18:39.180 RICHARD EBERHARDT: So maybe it might just 00:18:39.180 --> 00:18:42.200 be something for us to talk to the Terrascope folks 00:18:42.200 --> 00:18:43.362 to see how they do that. 00:18:43.362 --> 00:18:44.820 PHILIP TAN: Well, they specifically 00:18:44.820 --> 00:18:50.140 get people who can commit on regular meetings. 00:18:50.140 --> 00:18:53.350 And the idea is-- my understanding 00:18:53.350 --> 00:18:56.070 is that it's for the domain knowledge primarily. 00:18:56.070 --> 00:18:57.890 But you get that side benefit of here 00:18:57.890 --> 00:19:00.610 is a real human being who's excited about this problem, 00:19:00.610 --> 00:19:06.600 or maybe really concerned about this problem, 00:19:06.600 --> 00:19:09.266 and that can influence the motivation of a team. 00:19:09.266 --> 00:19:11.390 PABLO SUAREZ: And also, I don't know to what extent 00:19:11.390 --> 00:19:13.040 we can keep cramming what students have 00:19:13.040 --> 00:19:15.400 to do-- I know they already have to do way too much-- 00:19:15.400 --> 00:19:18.790 but to look at how that kind of field 00:19:18.790 --> 00:19:22.360 is currently being communicated in the game universe 00:19:22.360 --> 00:19:23.210 and outside of that. 00:19:23.210 --> 00:19:27.320 So PowerPoints or documents, they're, across the board, 00:19:27.320 --> 00:19:29.520 generally boring and unsuccessful-- 00:19:29.520 --> 00:19:31.530 even if they have the right content. 00:19:31.530 --> 00:19:34.030 So for them to get a sense that if someone really 00:19:34.030 --> 00:19:36.770 needs to create awareness on that topic, 00:19:36.770 --> 00:19:38.650 and this is the best they have in what 00:19:38.650 --> 00:19:42.500 exists in the real world, how about trying to give something 00:19:42.500 --> 00:19:46.860 that makes people want to do more through their game? 00:19:46.860 --> 00:19:50.740 PHILIP TAN: I think saying, this is how low the bar is, can we 00:19:50.740 --> 00:19:53.616 get it slightly better, is less motivating than saying, 00:19:53.616 --> 00:19:54.740 what's the best you can do? 00:19:58.995 --> 00:20:01.120 RICHARD EBERHARDT: But it's also choosing something 00:20:01.120 --> 00:20:02.620 that's going to get used. 00:20:02.620 --> 00:20:06.809 Why go in a direction that's-- if this is what those people 00:20:06.809 --> 00:20:09.350 are thinking about, if you give them something completely not 00:20:09.350 --> 00:20:11.724 like that at all, they're probably not going to know what 00:20:11.724 --> 00:20:12.930 to do with it. 00:20:12.930 --> 00:20:14.970 PHILIP TAN: But I mean, I can imagine things 00:20:14.970 --> 00:20:17.360 like evaluating current methods of teaching 00:20:17.360 --> 00:20:19.930 about a certain concept, to be part of that research process 00:20:19.930 --> 00:20:22.590 that we've been talking about-- it's understanding the problem. 00:20:22.590 --> 00:20:23.590 RICHARD EBERHARDT: Cool. 00:20:23.590 --> 00:20:26.760 All right, so I think it's about time to let you go. 00:20:26.760 --> 00:20:28.490 But thanks so much for coming in. 00:20:28.490 --> 00:20:30.690 PHILIP TAN: We thank you. 00:20:30.690 --> 00:20:32.240 Huge gratitude.