WEBVTT 00:00:00.050 --> 00:00:02.470 The following content is provided under a Creative 00:00:02.470 --> 00:00:03.880 Commons license. 00:00:03.880 --> 00:00:06.920 Your support will help MIT OpenCourseWare continue to 00:00:06.920 --> 00:00:10.570 offer high quality educational resources for free. 00:00:10.570 --> 00:00:13.470 To make a donation or view additional materials from 00:00:13.470 --> 00:00:19.290 hundreds of MIT courses, visit MIT OpenCourseWare at 00:00:19.290 --> 00:00:20.720 ocw.mit.edu. 00:00:20.720 --> 00:00:23.919 PROFESSOR: One example will be what happens when the battery 00:00:23.919 --> 00:00:26.100 runs up on the computer and what bubbles 00:00:26.100 --> 00:00:27.500 up when it's running. 00:00:27.500 --> 00:00:29.520 And there's certain computers that might [UNINTELLIGIBLE] 00:00:29.520 --> 00:00:30.050 hard drives. 00:00:30.050 --> 00:00:34.090 Some computers just keep it in RAM when you restart it. 00:00:34.090 --> 00:00:35.670 There's no big difference. 00:00:35.670 --> 00:00:40.188 What happens when an IM window pops up in the middle of the 00:00:40.188 --> 00:00:43.667 game and leaves the control screen? 00:00:43.667 --> 00:00:46.152 And all the keys start [UNINTELLIGIBLE] 00:00:46.152 --> 00:00:48.140 IM window. 00:00:48.140 --> 00:00:50.836 Ideally, they give you pop ups that doesn't work. 00:00:50.836 --> 00:00:54.730 But catching those is one of our favorite things that we do 00:00:54.730 --> 00:00:59.032 when we're making digital games here in the lab is like 00:00:59.032 --> 00:01:01.260 slamming all of our fingers on the keyboard at once and see 00:01:01.260 --> 00:01:02.860 what happens. 00:01:02.860 --> 00:01:07.250 And we're kind of looking for those things. 00:01:07.250 --> 00:01:11.650 We're also looking for subtler things like is the amount of 00:01:11.650 --> 00:01:13.342 RAM that those games are taking up in the computer 00:01:13.342 --> 00:01:17.240 slowly going up all the time? 00:01:17.240 --> 00:01:21.090 And of course, comparing it against a spec grade. 00:01:21.090 --> 00:01:25.470 So a character is supposed to be able to create a certain 00:01:25.470 --> 00:01:28.120 file hold style or something when you enter this key 00:01:28.120 --> 00:01:33.865 sequence, and 99 times out of 100 that works and that 1 out 00:01:33.865 --> 00:01:37.110 of 100 doesn't and we need to figure out why. 00:01:37.110 --> 00:01:37.670 That's our feat. 00:01:37.670 --> 00:01:40.920 So in a lot of those situations, what happens is 00:01:40.920 --> 00:01:46.850 that output comes back in the form of other parts and-- how 00:01:46.850 --> 00:01:50.092 many of you actually do programming? 00:01:50.092 --> 00:01:52.570 Oh, OK. 00:01:52.570 --> 00:01:55.935 How many of you have found a book report? 00:01:55.935 --> 00:01:57.753 OK, not that many. 00:02:00.770 --> 00:02:05.130 There are a couple of best practices where you follow a 00:02:05.130 --> 00:02:08.583 book report and this one translates a little bit into 00:02:08.583 --> 00:02:11.290 when you're really working in a team in a game. 00:02:11.290 --> 00:02:15.430 How do you communicate that something is wrong to someone 00:02:15.430 --> 00:02:18.850 else who is also responsible for the game, but may not 00:02:18.850 --> 00:02:22.500 necessarily have seen the problem? 00:02:22.500 --> 00:02:26.630 First of all, you describe what you were expecting to 00:02:26.630 --> 00:02:28.550 happen and what you were doing. 00:02:28.550 --> 00:02:32.795 I was expecting to go right when I push the right button. 00:02:32.795 --> 00:02:34.070 And then you write down what happened. 00:02:34.070 --> 00:02:38.410 You try to get as much detail as possible for the person to 00:02:38.410 --> 00:02:40.230 re-release this problem. 00:02:40.230 --> 00:02:45.660 Now, when it comes to war games, in many situations, the 00:02:45.660 --> 00:02:47.480 kind of detail that you need to provide largely comes up 00:02:47.480 --> 00:02:48.720 against it. 00:02:48.720 --> 00:02:53.160 So 'a' had this much money, 'b' had that much money, they 00:02:53.160 --> 00:02:57.010 try to do this game mechanic and no one knew what to do 00:02:57.010 --> 00:03:00.085 because the way the rules are written there's only one way 00:03:00.085 --> 00:03:02.450 to interpret it. 00:03:02.450 --> 00:03:06.590 And that gives your team enough information to be able 00:03:06.590 --> 00:03:08.920 to start acting on it. 00:03:08.920 --> 00:03:12.150 Because you are bargaining and your rules are kind of right 00:03:12.150 --> 00:03:15.330 out there in the open, there's usually less of a mystery to 00:03:15.330 --> 00:03:16.810 see where the problem occurs. 00:03:16.810 --> 00:03:19.470 But you know there's problem and it could end and you 00:03:19.470 --> 00:03:20.060 largely see where it lies. 00:03:20.060 --> 00:03:22.590 You may not know what to do to correct it, and that's 00:03:22.590 --> 00:03:25.320 actually where beta testing comes in. 00:03:25.320 --> 00:03:29.160 So but I'll get to the testing shortly. 00:03:29.160 --> 00:03:32.935 But technical testing, a lot of it again is comparing a 00:03:32.935 --> 00:03:33.950 given spec. 00:03:33.950 --> 00:03:35.382 Sometimes that spec is created from your team. 00:03:35.382 --> 00:03:36.330 This is what we want our game to do. 00:03:36.330 --> 00:03:40.340 This is what we expected to happen when we made this game. 00:03:40.340 --> 00:03:44.280 We expect that mid in the game, a certain game mechanic 00:03:44.280 --> 00:03:48.710 will start to take priority over other mechanics. 00:03:48.710 --> 00:03:51.900 Early on, everyone's buying property is a monopoly. 00:03:51.900 --> 00:03:56.990 Later on, everyone's going to be building houses and then 00:03:56.990 --> 00:03:58.910 even at the end, people start to mortgage stuff. 00:03:58.910 --> 00:04:00.920 That's what we kind of expected out of it. 00:04:00.920 --> 00:04:03.630 And then you play the game and you start realizing you 00:04:03.630 --> 00:04:06.400 mortgaging something way too early here, something's out of 00:04:06.400 --> 00:04:07.700 whack here. 00:04:07.700 --> 00:04:09.910 So that's the kind of technical 00:04:09.910 --> 00:04:10.780 testing you do in games. 00:04:10.780 --> 00:04:15.340 This is what we expect, why isn't it happening? 00:04:15.340 --> 00:04:19.940 The other side over here is kind of like being the 00:04:19.940 --> 00:04:23.750 defender of the player, being the player's advocate. 00:04:26.360 --> 00:04:30.130 How, can you be the person on the team that is actively 00:04:30.130 --> 00:04:34.516 looking for things in your game that are going to hurt 00:04:34.516 --> 00:04:36.648 your players and then mentioning it and saying, 00:04:36.648 --> 00:04:38.395 maybe we should do something about that. 00:04:38.395 --> 00:04:42.810 It's as simple as, here's the information that we want our 00:04:42.810 --> 00:04:44.760 players to have at a certain time. 00:04:44.760 --> 00:04:47.050 Here's the game state. 00:04:47.050 --> 00:04:49.790 Your characters have a certain state. 00:04:49.790 --> 00:04:52.790 They are this close to winning, but are this far away 00:04:52.790 --> 00:04:55.120 from winning to have these results here. 00:04:55.120 --> 00:04:57.445 Do the players understand that? 00:04:57.445 --> 00:05:00.730 Have you provided them with enough information to be able 00:05:00.730 --> 00:05:04.340 to make a good decision? 00:05:04.340 --> 00:05:06.145 If they know the game state, do they know what options they 00:05:06.145 --> 00:05:09.380 have in front of them? 00:05:09.380 --> 00:05:13.730 So as we play more and more board games this semester, you 00:05:13.730 --> 00:05:19.620 can actually look at the way how rules are presented 00:05:19.620 --> 00:05:19.870 redundantly. 00:05:19.870 --> 00:05:23.510 Sometimes they're written up in the rule sheet, but and of 00:05:23.510 --> 00:05:27.400 course you've got your index card-sized thing which goes 00:05:27.400 --> 00:05:29.270 through step one, two, three, four, these are the four 00:05:29.270 --> 00:05:31.238 things you've got to remember every single turn. 00:05:31.238 --> 00:05:33.581 You've got to get it, and everybody has a copy right in 00:05:33.581 --> 00:05:37.000 front of you because somebody on the team decided that that 00:05:37.000 --> 00:05:38.980 was a good idea. 00:05:38.980 --> 00:05:42.495 How did they come to that conclusion? 00:05:42.495 --> 00:05:47.170 Occasionally, they are doing it because they see there's a 00:05:47.170 --> 00:05:48.390 problem right away. 00:05:48.390 --> 00:05:52.060 There's too much information, and you just don't expect any 00:05:52.060 --> 00:05:54.690 player is ever going to be able to grasp all this 00:05:54.690 --> 00:05:55.485 information. 00:05:55.485 --> 00:06:00.423 So that's come out with a couple of ways to simplify it. 00:06:00.423 --> 00:06:03.734 Sometimes that involves taking up the mechanics, making 00:06:03.734 --> 00:06:07.750 something that was written up a text graphic report and some 00:06:07.750 --> 00:06:09.846 icons, reducing all the numbers by two so the math 00:06:09.846 --> 00:06:13.060 becomes easier. 00:06:13.060 --> 00:06:18.200 But sometimes, you do it by usability testing, and that's 00:06:18.200 --> 00:06:21.260 one kind of testing that often comes 00:06:21.260 --> 00:06:23.640 together with beta testing. 00:06:23.640 --> 00:06:26.850 I've invited Sara Verrilli, our development director, to 00:06:26.850 --> 00:06:30.709 come in here and talk about the kind of pre-testing that 00:06:30.709 --> 00:06:35.699 you can do, which is illustrated by this picture. 00:06:38.700 --> 00:06:42.500 And the book goes through several different kinds of 00:06:42.500 --> 00:06:45.410 environments where you do pre-testing that's sitting 00:06:45.410 --> 00:06:48.625 one-on-one with people and doing site 00:06:48.625 --> 00:06:51.320 testing for the group. 00:06:51.320 --> 00:06:54.422 Because this style is all about board games and card 00:06:54.422 --> 00:06:57.150 games, for the most part, we're designing games for 00:06:57.150 --> 00:06:58.050 three or four people to play at a time. 00:06:58.050 --> 00:07:01.440 The chances are, almost every single test that you're going 00:07:01.440 --> 00:07:03.016 to do [UNINTELLIGIBLE] at a single time. 00:07:03.016 --> 00:07:04.674 That's just the environment you're in 00:07:04.674 --> 00:07:05.570 that's meant to be played. 00:07:05.570 --> 00:07:09.390 That's why you end up testing it. 00:07:09.390 --> 00:07:11.056 Obviously, a single-play computer game, that's not 00:07:11.056 --> 00:07:13.800 necessarily the environment you're testing in. 00:07:13.800 --> 00:07:19.985 Actually, before I transition over, there is one thing I 00:07:19.985 --> 00:07:21.670 want to mention about technical testing. 00:07:21.670 --> 00:07:23.855 Sometimes the specs you're testing against are not the 00:07:23.855 --> 00:07:26.710 ones that came up that would necessarily come up 00:07:26.710 --> 00:07:28.640 with phi over t. 00:07:28.640 --> 00:07:30.520 If you're doing a computer game-- 00:07:30.520 --> 00:07:34.510 say for an Xbox or a PS3 or a Wii. 00:07:34.510 --> 00:07:37.001 There are these things called technical [? evaluation ?] 00:07:37.001 --> 00:07:42.280 requirements, which are basically huge lists of things 00:07:42.280 --> 00:07:46.391 that your game must do before Microsoft will sign your title 00:07:46.391 --> 00:07:50.720 or allow it to be released under that title. 00:07:50.720 --> 00:07:54.796 PC obviously doesn't have this, but if you release it on 00:07:54.796 --> 00:07:57.900 speed or something, then chances are, you'll have 00:07:57.900 --> 00:07:58.110 requirements. 00:07:58.110 --> 00:07:59.730 It could be something as simple as generally, it 00:07:59.730 --> 00:08:02.596 doesn't crash-- 00:08:02.596 --> 00:08:06.830 and games do crash, even on PS3 and Xbox-- 00:08:06.830 --> 00:08:12.330 you get things like what happens when you're playing a 00:08:12.330 --> 00:08:14.442 game on the PS3 or Xbox-- 00:08:14.442 --> 00:08:18.012 I guess more the PS2 and the first Xbox-- and someone yanks 00:08:18.012 --> 00:08:20.880 out the cable? 00:08:20.880 --> 00:08:23.510 All you're thinking is that you're-- and then the keyboard 00:08:23.510 --> 00:08:25.805 comes out, what happens? 00:08:25.805 --> 00:08:28.390 Does it continue? 00:08:28.390 --> 00:08:29.740 STUDENT: It buzzes. 00:08:29.740 --> 00:08:31.900 PROFESSOR: It goes into pause, right? 00:08:31.900 --> 00:08:35.289 And usually it tells you, your controller is disconnected. 00:08:35.289 --> 00:08:36.950 And that's a requirement. 00:08:36.950 --> 00:08:40.111 Someone had to write that code specifically to detect that 00:08:40.111 --> 00:08:44.610 situation and then throughout the suite across the game. 00:08:44.610 --> 00:08:47.590 And the reason everybody had to do that was because 00:08:47.590 --> 00:08:49.560 Microsoft and Sony and Nintendo-- 00:08:49.560 --> 00:08:50.730 well, not so much Nintendo nowadays 00:08:50.730 --> 00:08:53.230 because Nintendo is wireless-- 00:08:53.230 --> 00:08:57.880 but these are the kinds of requirements that they expect. 00:08:57.880 --> 00:09:01.248 If you lose your connection to Xbox live halfway through the 00:09:01.248 --> 00:09:03.650 game then something's going to happen. 00:09:03.650 --> 00:09:08.595 And so again, for that kind of technical testing, you're sort 00:09:08.595 --> 00:09:11.416 of comparing it against spec, but spec may not necessarily 00:09:11.416 --> 00:09:14.392 be something that you're keeping up with. 00:09:14.392 --> 00:09:18.860 It may be something that the publisher or your platform 00:09:18.860 --> 00:09:21.230 developer, platform manufacturer mandated. 00:09:21.230 --> 00:09:23.750 How does it affect the board games is that sometimes, board 00:09:23.750 --> 00:09:27.225 games are actually going to require half the 00:09:27.225 --> 00:09:28.480 requirements to that. 00:09:28.480 --> 00:09:31.543 If you're responsible for the design of the box, for 00:09:31.543 --> 00:09:33.930 instance, there are things they're going to require on 00:09:33.930 --> 00:09:38.046 the outside of the box, like what ages are the game 00:09:38.046 --> 00:09:40.820 applicable for, how long is it meant to be played, how many 00:09:40.820 --> 00:09:43.680 people are going to be playing your game. 00:09:43.680 --> 00:09:49.800 I think Days of Wonder, which is a publisher, actually 00:09:49.800 --> 00:09:54.370 requires a character to be on the box. 00:09:54.370 --> 00:09:58.120 Even if your game has no characters, there must be a 00:09:58.120 --> 00:10:01.750 human-like presence on the box or your players are on the box 00:10:01.750 --> 00:10:05.010 because they think that it increases sales, and they 00:10:05.010 --> 00:10:06.720 probably have the numbers to back it up. 00:10:06.720 --> 00:10:10.180 That's why if you look at things like Ticket for Ride, 00:10:10.180 --> 00:10:12.168 for instance, has people. 00:10:12.168 --> 00:10:16.450 If you play Ticket to Ride, it has no people in it, but it 00:10:16.450 --> 00:10:19.940 has people on the box because they want people to think 00:10:19.940 --> 00:10:21.335 that's what users are. 00:10:25.070 --> 00:10:31.397 So that's just an example of things that a board game 00:10:31.397 --> 00:10:32.790 publisher may require you to do. 00:10:32.790 --> 00:10:37.180 Very common that you'll have to do translations in a range 00:10:37.180 --> 00:10:40.920 of different translations for the different countries it's 00:10:40.920 --> 00:10:43.016 going to be released in. 00:10:43.016 --> 00:10:45.435 It may not be your job to do it, the publishers may end up 00:10:45.435 --> 00:10:47.027 being the ones helping you with the translation, getting 00:10:47.027 --> 00:10:49.260 all the publishing data formatted decently, but 00:10:49.260 --> 00:10:52.560 someone along the stage is making sure that happens 00:10:52.560 --> 00:10:55.274 before the game hits the shelf. 00:10:58.340 --> 00:11:00.618 OK, that's technical testing. 00:11:00.618 --> 00:11:05.956 That's a little bit about becoming a player and so let's 00:11:05.956 --> 00:11:09.372 talk about data center a bit. 00:11:09.372 --> 00:11:13.764 SARA VERRILLI: Let me bump you a little here, 00:11:13.764 --> 00:11:14.460 if you don't care. 00:11:14.460 --> 00:11:15.710 PROFESSOR: Oh, right. 00:11:37.466 --> 00:11:38.954 SARA VERRILLI: OK. 00:11:38.954 --> 00:11:39.946 Let's see. 00:11:39.946 --> 00:11:41.930 Let's actually go ahead. 00:11:45.898 --> 00:11:47.882 So focus testing. 00:11:47.882 --> 00:11:50.767 This is a very similar track for the summer program folks. 00:11:50.767 --> 00:11:53.840 I have about 10 minutes to go back and relook at my notes 00:11:53.840 --> 00:11:57.980 and relook at the lecture, so if it's a little stumbly in 00:11:57.980 --> 00:12:01.209 parts, it's because I haven't given it for a while and 00:12:01.209 --> 00:12:01.460 haven't thought about it. 00:12:01.460 --> 00:12:02.766 It's much more geared, the original talk is much more 00:12:02.766 --> 00:12:05.140 geared towards computer testing, and testing 00:12:05.140 --> 00:12:08.890 specifically single-player computer games than it's 00:12:08.890 --> 00:12:12.770 testing board games, which are almost a multi-user. 00:12:12.770 --> 00:12:16.598 So first of all, focus testing is actually dissimilar. 00:12:16.598 --> 00:12:20.370 Focus testing is only a specific case of user testing, 00:12:20.370 --> 00:12:22.720 which is to say, you're testing your game using actual 00:12:22.720 --> 00:12:25.910 users, not your development team, not people who play 00:12:25.910 --> 00:12:28.484 board games 12 hours a day because that's what they love 00:12:28.484 --> 00:12:31.100 to do all the time, but more the sort of average person who 00:12:31.100 --> 00:12:33.645 might want to pick up your game off the shelf, go buy it 00:12:33.645 --> 00:12:37.300 and then go play it with their family and then go out to 00:12:37.300 --> 00:12:40.198 dinner at Chuck E. Cheese or something. 00:12:40.198 --> 00:12:44.245 Another type of user tester is usability testing, which is 00:12:44.245 --> 00:12:46.770 fairly similar to focus testing in as much as you're 00:12:46.770 --> 00:12:48.360 still using users for it. 00:12:48.360 --> 00:12:50.860 Focus testing is usually much more geared towards are people 00:12:50.860 --> 00:12:52.073 enjoying your game? 00:12:52.073 --> 00:12:55.370 Do people want to play your game? 00:12:55.370 --> 00:12:59.060 Are people going through the sort of play experiences you 00:12:59.060 --> 00:13:01.290 want them to have as they play their game? 00:13:01.290 --> 00:13:02.920 Because usually when you create a game, you're 00:13:02.920 --> 00:13:05.410 thinking, oh, like for Settlers of Catan, you expect 00:13:05.410 --> 00:13:06.950 people to spend a lot of time talking with each other 00:13:06.950 --> 00:13:09.326 trading things back and forth. 00:13:09.326 --> 00:13:12.960 When you're making Diplomacy, you expect people to spend a 00:13:12.960 --> 00:13:16.200 lot of time arguing with each other over well, if you invade 00:13:16.200 --> 00:13:19.060 that country, I'll invade that country and we'll gang up on 00:13:19.060 --> 00:13:21.280 him and we'll all come out of the game and then we'll just 00:13:21.280 --> 00:13:25.098 divide four between the two of us and win while you're making 00:13:25.098 --> 00:13:25.618 secret underhanded arrangements with the other 00:13:25.618 --> 00:13:28.000 guys, but OK, so he's going to attack you on this ground, we 00:13:28.000 --> 00:13:32.330 both know that, that sort of thing, as opposed to, say, 00:13:32.330 --> 00:13:35.192 Transamerica which is just more putting pieces down and 00:13:35.192 --> 00:13:38.420 it's pretty much independent play in many ways. 00:13:41.420 --> 00:13:45.612 Usability testing is much more about how you easy is it for 00:13:45.612 --> 00:13:46.920 people to play your game. 00:13:46.920 --> 00:13:49.150 Do they understand your rules? 00:13:49.150 --> 00:13:50.490 Are the mechanics clear? 00:13:50.490 --> 00:13:53.100 When they sit down and start sorting out the tokens, can 00:13:53.100 --> 00:13:57.055 they tell which tokens are the player pieces and which cards 00:13:57.055 --> 00:13:58.056 are which and how you use those? 00:13:58.056 --> 00:14:01.220 Do the icons on your game make sense to people? 00:14:01.220 --> 00:14:02.570 Do they understand them? 00:14:02.570 --> 00:14:03.020 Things like that. 00:14:03.020 --> 00:14:06.162 It's not really about whether or not people like your game 00:14:06.162 --> 00:14:07.400 or are enjoying your game. 00:14:07.400 --> 00:14:11.056 It's about can people understand your game and start 00:14:11.056 --> 00:14:12.306 playing it. 00:14:14.500 --> 00:14:19.545 So here we go. 00:14:19.545 --> 00:14:23.703 One big problem we run into when people sit down and we 00:14:23.703 --> 00:14:25.385 have people playing your game and giving lots of comments, 00:14:25.385 --> 00:14:27.670 this was really fun or this really sucked, or I hate this 00:14:27.670 --> 00:14:29.080 part where I have to spend time talking to the other 00:14:29.080 --> 00:14:33.120 players, I want to be able to stare at just my cards and 00:14:33.120 --> 00:14:36.350 board completely without interacting with anyone else. 00:14:36.350 --> 00:14:37.710 Focus testers will give you lots and lots of feedback, 00:14:37.710 --> 00:14:41.140 some of it is useful, some of it is not. 00:14:41.140 --> 00:14:45.986 And you need to remember that focus testers are not actually 00:14:45.986 --> 00:14:46.890 people building the game. 00:14:46.890 --> 00:14:47.740 Focus testers are the people who you are trying to get 00:14:47.740 --> 00:14:52.020 information about your game from and one problem that we 00:14:52.020 --> 00:14:56.890 often see in experienced game developers and people making 00:14:56.890 --> 00:15:01.560 games doing is, oh, the focus tester said we have to make 00:15:01.560 --> 00:15:06.840 the whole game red, quick, let's make everything red. 00:15:06.840 --> 00:15:11.180 So and that's usually, that's not a very 00:15:11.180 --> 00:15:11.957 good solution, right? 00:15:11.957 --> 00:15:13.780 Who knows what the problem with this actually was, but 00:15:13.780 --> 00:15:16.970 what you need to do is find out why did the players want 00:15:16.970 --> 00:15:18.010 to make everything red? 00:15:18.010 --> 00:15:19.550 Is it because they don't like the artistic 00:15:19.550 --> 00:15:21.858 design of the game? 00:15:21.858 --> 00:15:22.340 Is it because someone in the game is colorblind? 00:15:22.340 --> 00:15:26.196 Someone playing the game currently is colorblind, is 00:15:26.196 --> 00:15:29.088 having a hard time telling things apart so they'd like 00:15:29.088 --> 00:15:31.304 everything to be the same color so everyone's having the 00:15:31.304 --> 00:15:32.275 same problem they are. 00:15:32.275 --> 00:15:33.250 I mean, who knows? 00:15:33.250 --> 00:15:37.720 It's hard to tell what people are thinking. 00:15:37.720 --> 00:15:40.320 When they give you advice, and when they give you these 00:15:40.320 --> 00:15:41.560 comments, you have to have some way to filter those 00:15:41.560 --> 00:15:44.260 comments, some way to think about them and some way to 00:15:44.260 --> 00:15:48.500 decide what actually is the right reaction to take in view 00:15:48.500 --> 00:15:52.525 of what you want your game to do and how you want people to 00:15:52.525 --> 00:15:52.660 play the game. 00:15:52.660 --> 00:15:54.887 Since you're the game designers and developers, you 00:15:54.887 --> 00:15:57.100 have to make those decisions. 00:15:57.100 --> 00:16:00.014 Focus testers can only give you data. 00:16:00.014 --> 00:16:02.882 You shouldn't ask them to design your game for you. 00:16:02.882 --> 00:16:05.750 Maybe you actually have good ideas that you want to use, 00:16:05.750 --> 00:16:08.316 but you want to make sure that they're good ideas that you do 00:16:08.316 --> 00:16:10.800 want to use. 00:16:10.800 --> 00:16:16.866 So you may notice, these are both my lectures, and these 00:16:16.866 --> 00:16:20.170 are pictures that Mike [? Dropho ?] drew for a 00:16:20.170 --> 00:16:21.720 poster, and we love him so much. 00:16:21.720 --> 00:16:25.150 So in the ideal focus testing situation, you're really 00:16:25.150 --> 00:16:29.090 trying to get as much data as you can about your player 00:16:29.090 --> 00:16:32.765 playing the game without spoiling the information. 00:16:32.765 --> 00:16:36.060 What you'd really like to recreate in a lot of ways is, 00:16:36.060 --> 00:16:37.826 so you just bought the game and have 00:16:37.826 --> 00:16:38.610 gotten home, what happens? 00:16:38.610 --> 00:16:43.100 And that's what this picture is all about, there's somebody 00:16:43.100 --> 00:16:47.240 luring somebody to play the game without any information. 00:16:53.117 --> 00:17:03.995 So things to think about when you're planning your focus 00:17:03.995 --> 00:17:09.520 testing is, what are you testing? 00:17:09.520 --> 00:17:11.900 What questions do you want to ask? 00:17:11.900 --> 00:17:13.328 What information do you want to get out of it? 00:17:13.328 --> 00:17:15.940 How are you going to test it? 00:17:15.940 --> 00:17:18.490 How many players do you need? 00:17:18.490 --> 00:17:21.170 What kind of interactions do you want to see? 00:17:21.170 --> 00:17:24.554 On the backhand, who are you testing with? 00:17:24.554 --> 00:17:26.582 What kind of people are going to make your game? 00:17:26.582 --> 00:17:30.912 Are you making a game for children age 9 to 15? 00:17:30.912 --> 00:17:33.407 Are you making it for college-age students? 00:17:33.407 --> 00:17:37.648 Are you making a game that you think that older players will 00:17:37.648 --> 00:17:38.896 want to play? 00:17:38.896 --> 00:17:41.391 Because if you're making a game that you think is going 00:17:41.391 --> 00:17:43.387 to be popular with the senior citizens market, testing it 00:17:43.387 --> 00:17:45.882 with a bunch of elementary school kids probably isn't 00:17:45.882 --> 00:17:48.876 going to go very will. 00:17:48.876 --> 00:17:52.312 Trivial Pursuit Classic Edition, a game that came out 00:17:52.312 --> 00:17:56.130 about 20-odd years ago, I suspect that if you sat down a 00:17:56.130 --> 00:18:00.082 group of high school students with it and had them play it, 00:18:00.082 --> 00:18:00.786 about three quarters of the questions they'd just stare at 00:18:00.786 --> 00:18:05.022 you and be like, I've never heard of that. 00:18:05.022 --> 00:18:09.468 So, that's certainly where the audience comes in. 00:18:09.468 --> 00:18:14.408 So, the what, planning out the questions that you think 00:18:14.408 --> 00:18:15.396 you're going to get information for. 00:18:15.396 --> 00:18:19.350 When you know what information for focus testing, it's a lot 00:18:19.350 --> 00:18:22.630 easier to know what to keep track of, how to take notes, 00:18:22.630 --> 00:18:25.981 what reactions you're most interested in as people are 00:18:25.981 --> 00:18:27.238 playing the game. 00:18:27.238 --> 00:18:30.136 So, what questions are you going to ask? 00:18:30.136 --> 00:18:32.080 What data it takes to answer your question? 00:18:32.080 --> 00:18:33.930 And how are you going to get that data? 00:18:33.930 --> 00:18:35.220 Are you going to go all through 00:18:35.220 --> 00:18:36.780 observation taking notes? 00:18:36.780 --> 00:18:38.620 Are you going to just step in and out of questions? 00:18:38.620 --> 00:18:42.249 So why did you move that character from there, just out 00:18:42.249 --> 00:18:43.873 of curiosity I'm wondering why you made that move. 00:18:43.873 --> 00:18:46.308 Will you tell me about it? 00:18:46.308 --> 00:18:49.230 Or, you can ask them to fill out a questionnaire 00:18:49.230 --> 00:18:49.717 at the end of it. 00:18:49.717 --> 00:18:53.126 Depending on what data you want, different 00:18:53.126 --> 00:18:54.610 methods will work best. 00:18:54.610 --> 00:19:01.684 In general, usually the best way to do it is observation. 00:19:01.684 --> 00:19:05.636 As soon as you ask someone questions, you sort of perturb 00:19:05.636 --> 00:19:08.366 the system and when you ask them to fill out a survey, 00:19:08.366 --> 00:19:09.930 it's after they've done it. 00:19:09.930 --> 00:19:12.030 They've kind of forgotten what they were doing. 00:19:12.030 --> 00:19:14.520 They may not exactly remember, and to be perfectly honest, 00:19:14.520 --> 00:19:15.530 people would really rather play your game 00:19:15.530 --> 00:19:17.140 than fill out a form. 00:19:17.140 --> 00:19:21.650 So they'll be more engaged more interested and more 00:19:21.650 --> 00:19:24.606 interactive when they're playing a game and that's when 00:19:24.606 --> 00:19:25.770 you're going to get your best data. 00:19:25.770 --> 00:19:28.190 But there are times when you want to ask someone something 00:19:28.190 --> 00:19:29.920 sort of analytical and you don't want someone to be 00:19:29.920 --> 00:19:32.910 thinking analytically while they're playing the game in a 00:19:32.910 --> 00:19:35.614 lot of ways , and so there are times when that reflection 00:19:35.614 --> 00:19:39.125 comes out so much better and it's better to have that 00:19:39.125 --> 00:19:40.320 written down and get it in your own words. 00:19:40.320 --> 00:19:44.505 STUDENT: You would hope it plays better than a survey. 00:19:44.505 --> 00:19:46.920 SARA VERRILLI: If your game is better than a survey, you 00:19:46.920 --> 00:19:51.358 probably already observed a problem right there and you're 00:19:51.358 --> 00:19:53.310 probably glad you found it. 00:19:57.230 --> 00:19:58.390 So how? 00:19:58.390 --> 00:20:01.030 In order to get all that data, you're going to have to 00:20:01.030 --> 00:20:03.250 actually interact with the user, someone who's never seen 00:20:03.250 --> 00:20:06.300 your game before, someone who doesn't actually care whether 00:20:06.300 --> 00:20:08.550 or not you get good information out of 00:20:08.550 --> 00:20:09.250 playing this game. 00:20:09.250 --> 00:20:11.330 They came here to sit down and play the game, and while they 00:20:11.330 --> 00:20:14.260 would like to help you by giving you information, that's 00:20:14.260 --> 00:20:17.030 not their primary goal in a lot of ways. 00:20:17.030 --> 00:20:19.074 Usually, their primary goal is to sit down, and play the 00:20:19.074 --> 00:20:19.190 game, and have a good time. 00:20:19.190 --> 00:20:21.610 So you need to be able to structure the interaction with 00:20:21.610 --> 00:20:25.210 them so that getting the data to you is as 00:20:25.210 --> 00:20:27.310 easy for them as possible. 00:20:27.310 --> 00:20:30.115 So we need to think about what this player needs before he or 00:20:30.115 --> 00:20:30.740 she sits down to play. 00:20:30.740 --> 00:20:32.464 Usually, in a board game or a card game, 00:20:32.464 --> 00:20:33.990 that's the set of rules. 00:20:33.990 --> 00:20:36.520 In a computer, it's always trickier. 00:20:36.520 --> 00:20:39.700 Usually when we're bringing in people to test the first 00:20:39.700 --> 00:20:42.690 prototypes, we have a game that works, but we don't have 00:20:42.690 --> 00:20:43.490 any instructions in game. 00:20:43.490 --> 00:20:45.390 If we're going to have a tutorial or anything like 00:20:45.390 --> 00:20:48.080 that, that's usually one of the last things that goes in. 00:20:48.080 --> 00:20:50.620 So when you're testing computer games early in the 00:20:50.620 --> 00:20:53.170 process, usually you end up sitting down and writing out, 00:20:53.170 --> 00:20:54.820 well, this is what the tutorial would look like if we 00:20:54.820 --> 00:20:57.120 had it, so read this and then sit down and play it. 00:20:57.120 --> 00:20:59.005 With board games and card games, it's actually a little 00:20:59.005 --> 00:21:01.610 bit easier, because you can say, a ha, this is the rules 00:21:01.610 --> 00:21:03.530 that are going to package with the games. 00:21:03.530 --> 00:21:05.810 Give you the rules, step back and see what happens. 00:21:05.810 --> 00:21:07.920 So, it's actually an easier aspect for 00:21:07.920 --> 00:21:10.330 testing board games. 00:21:10.330 --> 00:21:14.636 And finally, treat the players politely. 00:21:14.636 --> 00:21:16.995 When they find a misspelling in your rules or when they 00:21:16.995 --> 00:21:20.050 discover that you've left out six of the key cards to play 00:21:20.050 --> 00:21:23.005 the game, oh, yeah, you're right, thank you for pointing 00:21:23.005 --> 00:21:25.868 that out, I wouldn't have noticed that if you hadn't 00:21:25.868 --> 00:21:26.260 shown it to me. 00:21:26.260 --> 00:21:28.255 Whether you would have or not doesn't really matter. 00:21:28.255 --> 00:21:30.490 They're doing you a favor by playing your game that isn't 00:21:30.490 --> 00:21:33.460 quite working well, so you want them to know that their 00:21:33.460 --> 00:21:37.557 contribution is invaluable and you appreciate the time and 00:21:37.557 --> 00:21:38.807 effort they've put into it. 00:21:42.850 --> 00:21:46.110 A few things, the more they talk about your game and the 00:21:46.110 --> 00:21:48.390 mechanics they find that break or that do or don't work, 00:21:48.390 --> 00:21:51.482 that's all data right there, That's all great. 00:21:51.482 --> 00:21:54.077 Another great advantage of testing board games is you 00:21:54.077 --> 00:21:56.240 usually get groups of players, like two or three or four 00:21:56.240 --> 00:21:59.520 players, and when you have multiple players, they'll talk 00:21:59.520 --> 00:22:00.526 to each other about it. 00:22:00.526 --> 00:22:01.080 Oh, I noticed this. 00:22:01.080 --> 00:22:03.390 Did you notice this strategy? 00:22:03.390 --> 00:22:05.115 Yeah, I did, but I noticed that if you add that strategy 00:22:05.115 --> 00:22:07.780 that you can do this thing and this other thing and that's 00:22:07.780 --> 00:22:09.240 all really neat. 00:22:09.240 --> 00:22:10.926 When you're testing single-player computer games 00:22:10.926 --> 00:22:13.940 like we usually do in the summer program, you've got a 00:22:13.940 --> 00:22:15.990 single user that is sitting there between 00:22:15.990 --> 00:22:17.020 them and the computer. 00:22:17.020 --> 00:22:19.735 You have to watch and figure out what they're thinking. 00:22:19.735 --> 00:22:22.776 When you have a group, it's not usually that hard to get 00:22:22.776 --> 00:22:24.553 them talking about what they're doing and what they're 00:22:24.553 --> 00:22:27.347 thinking, and so that's a really good way to try 00:22:27.347 --> 00:22:29.073 and get more data. 00:22:34.020 --> 00:22:38.215 So yeah, here's sort of the end summary. 00:22:38.215 --> 00:22:40.100 Have a specific question and questions in mind. 00:22:40.100 --> 00:22:43.380 Know what you're doing in focus testing, know why you're 00:22:43.380 --> 00:22:44.330 doing usability testing. 00:22:44.330 --> 00:22:48.150 If you don't know what you're trying to test for, then your 00:22:48.150 --> 00:22:50.370 data is going to be a lot harder to figure 00:22:50.370 --> 00:22:51.220 out what to do with. 00:22:51.220 --> 00:22:53.810 So you always want to we have that question in mind. 00:22:53.810 --> 00:22:55.250 That's actually a really big one. 00:22:55.250 --> 00:22:59.974 What data you need to answer that question so you can 00:22:59.974 --> 00:23:03.040 collect it and know how to collect it, and make sure that 00:23:03.040 --> 00:23:05.750 you can give the information to the players, a standardized 00:23:05.750 --> 00:23:08.925 set of information to the players and step away so 00:23:08.925 --> 00:23:11.200 you're not sort of coloring the information and giving 00:23:11.200 --> 00:23:13.770 them the things you're talking about by, oh I forgot to 00:23:13.770 --> 00:23:17.290 mention, oh and did I say, so you've actually got sort of a 00:23:17.290 --> 00:23:23.635 standardized standard size play group scenario set down. 00:23:23.635 --> 00:23:26.605 OK, I think that's actually what [? I came for. ?] 00:23:29.580 --> 00:23:33.496 PROFESSOR: So a couple of words about when to be 00:23:33.496 --> 00:23:37.104 testing, which is really as soon as possible. 00:23:37.104 --> 00:23:40.540 Usually, before you think it's possible, you should be 00:23:40.540 --> 00:23:44.780 already starting to plan your fist test because at that 00:23:44.780 --> 00:23:47.228 point, you're probably going for the lowest-hanging fruit, 00:23:47.228 --> 00:23:50.216 which will be probably somebody else in this class 00:23:50.216 --> 00:23:53.900 because they're going to give you feedback anyways, you can 00:23:53.900 --> 00:23:56.340 use that favor in exchange. 00:23:56.340 --> 00:23:59.480 The whole point of making testing as simple as possible 00:23:59.480 --> 00:24:02.830 is you want to be able to get information while you still 00:24:02.830 --> 00:24:08.550 have time to fix it, to do something with that data. 00:24:08.550 --> 00:24:12.050 What has happened a lot of times in this class is we have 00:24:12.050 --> 00:24:15.650 a project team where they come up with a game requires say, 00:24:15.650 --> 00:24:20.100 generating so many cards or that they're going to spend 00:24:20.100 --> 00:24:23.150 most of the project deadline actually creating those cards. 00:24:23.150 --> 00:24:26.710 They'll test it, but they'll test it on themselves because 00:24:26.710 --> 00:24:28.830 the team already gets it and they're not 00:24:28.830 --> 00:24:29.720 looking for other people. 00:24:29.720 --> 00:24:34.696 They have a plan to ask their friends to come in for a 00:24:34.696 --> 00:24:35.374 specific time, they've set aside an hour or 00:24:35.374 --> 00:24:36.330 something like that. 00:24:36.330 --> 00:24:39.200 So what happens is that the last week comes around and 00:24:39.200 --> 00:24:41.270 they start playing it with people who haven't seen the 00:24:41.270 --> 00:24:44.020 game before and then discovered problems. 00:24:44.020 --> 00:24:45.970 Unfortunately, the game has 100 cards that all 00:24:45.970 --> 00:24:48.350 now need to be fixed. 00:24:48.350 --> 00:24:50.540 This is a problem because you just discovered 00:24:50.540 --> 00:24:52.826 that problem too late. 00:24:52.826 --> 00:24:54.730 A couple of ways to handle that, though, one, don't 00:24:54.730 --> 00:24:59.236 decide on a game where making changes across the entire game 00:24:59.236 --> 00:25:02.140 is going to take you more time than to test it because that's 00:25:02.140 --> 00:25:04.820 going to make it impossible to fix anything. 00:25:04.820 --> 00:25:06.910 But the other one is just testing early. 00:25:06.910 --> 00:25:10.335 As soon as you've got a couple of key mechanics down, sure 00:25:10.335 --> 00:25:12.580 you are planning to play this game with 100 00:25:12.580 --> 00:25:15.360 cards, start with 20. 00:25:15.360 --> 00:25:19.150 Start with a small of group of cards, hence your core can see 00:25:19.150 --> 00:25:20.540 people and understand that they're going to discover 00:25:20.540 --> 00:25:24.190 things in the production of those 20 cards and in the 00:25:24.190 --> 00:25:27.160 testing of those 20 cards that maybe you don't want to be 00:25:27.160 --> 00:25:31.790 doing for the next test and you can redo those 20 cards 00:25:31.790 --> 00:25:33.930 pretty quickly the way that you can't redo 100 cards. 00:25:33.930 --> 00:25:35.280 Just an example. 00:25:35.280 --> 00:25:38.530 I'm really not encouraging people to do a 100-card game, 00:25:38.530 --> 00:25:39.805 by the way. 00:25:39.805 --> 00:25:41.986 A regular poker deck, 52 cards, that's all you need for 00:25:41.986 --> 00:25:43.236 a really interesting game. 00:25:45.600 --> 00:25:47.760 Let's see. 00:25:47.760 --> 00:25:50.140 Some other things. 00:25:50.140 --> 00:25:53.520 Say you're doing a test, but it's not with 00:25:53.520 --> 00:25:54.880 your entire team present. 00:25:54.880 --> 00:25:59.620 Say it's just you and your teammates are all living in 00:25:59.620 --> 00:26:02.700 different dorms and you decided to play with 00:26:02.700 --> 00:26:03.120 those in your dorm. 00:26:03.120 --> 00:26:05.370 That's great, because it's a cool game and you've got your 00:26:05.370 --> 00:26:08.060 feedback, how do you communicate that 00:26:08.060 --> 00:26:09.020 back to your team? 00:26:09.020 --> 00:26:11.890 I talked a little bit about writing up a little report, 00:26:11.890 --> 00:26:14.976 but here's a couple of other tips because one thing you 00:26:14.976 --> 00:26:17.520 have to remember is that chances are, every single idea 00:26:17.520 --> 00:26:19.400 in the game, somebody in your team is probably really 00:26:19.400 --> 00:26:22.970 passionate about it and what you're basically trying to do 00:26:22.970 --> 00:26:25.733 is try to convince someone in your team that maybe this idea 00:26:25.733 --> 00:26:27.550 should be changed. 00:26:27.550 --> 00:26:30.370 And your game will probably be better for it, but there are 00:26:30.370 --> 00:26:33.470 some tactful was of telling that. 00:26:33.470 --> 00:26:36.900 So let's see. 00:26:36.900 --> 00:26:39.680 Keep it short, this also applies to book reports. 00:26:39.680 --> 00:26:42.580 I talked about you giving people enough information to 00:26:42.580 --> 00:26:46.710 reproduce what the problem was so that they can understand 00:26:46.710 --> 00:26:49.940 the state of the game and why the problem is occurring. 00:26:49.940 --> 00:26:53.380 And the flip side of that is not too much information, not 00:26:53.380 --> 00:26:56.320 more information than is necessary to keep it short. 00:26:56.320 --> 00:26:59.870 Little [? farm ?] codes, give them something that's easy for 00:26:59.870 --> 00:27:01.220 people to scan. 00:27:01.220 --> 00:27:04.149 Don't make them read and essay for them to figure out what 00:27:04.149 --> 00:27:06.250 comments will be solved. 00:27:06.250 --> 00:27:08.610 Now actually, one thing to keep in mind and if you keep 00:27:08.610 --> 00:27:12.060 in mind and think about what Sara said earlier about the 00:27:12.060 --> 00:27:14.470 data that you decide to collect, one problem with 00:27:14.470 --> 00:27:16.850 board games is with all the interesting conversations 00:27:16.850 --> 00:27:19.960 happening around the table and multiple players suggesting 00:27:19.960 --> 00:27:22.992 strategies to each other, you can come up with reams of 00:27:22.992 --> 00:27:24.390 notes in a single play test session. 00:27:24.390 --> 00:27:29.180 How do you decide to digest that is really important. 00:27:29.180 --> 00:27:33.735 And you're the only person there and you're the only 00:27:33.735 --> 00:27:35.611 person on your team, you need to try to let the rest 00:27:35.611 --> 00:27:36.840 of your team know. 00:27:36.840 --> 00:27:39.050 You need to put in the work of digesting all of that 00:27:39.050 --> 00:27:43.860 information before necessarily giving it over to your team. 00:27:43.860 --> 00:27:47.780 SARA VERRILLI: One possible solution also is you can get 00:27:47.780 --> 00:27:52.680 permission from play testers for a tape recorder. 00:27:52.680 --> 00:27:56.476 Just get a digital recorder and record all of the 00:27:56.476 --> 00:27:59.060 conversation as it goes on. 00:27:59.060 --> 00:28:01.596 PROFESSOR: But you still want to pick out the keys, you 00:28:01.596 --> 00:28:04.090 don't want to revisit the entire conversation. 00:28:04.090 --> 00:28:07.873 Because if the game took an hour, and every single one of 00:28:07.873 --> 00:28:11.240 your team members visits, by now you've lost three hours. 00:28:11.240 --> 00:28:14.350 So you want to find that, do that little bit of work to 00:28:14.350 --> 00:28:16.580 save yourself the detail. 00:28:16.580 --> 00:28:19.570 But that's the way because you can focus, you can take down 00:28:19.570 --> 00:28:23.110 notes on the main points and just let the tape recorder 00:28:23.110 --> 00:28:26.370 record all the details so that you don't have to worry about 00:28:26.370 --> 00:28:29.295 recording and really, the main points are the things that you 00:28:29.295 --> 00:28:30.130 want to make sure you get across to your team. 00:28:30.130 --> 00:28:32.526 The tape recorder is just evidence. 00:28:32.526 --> 00:28:37.460 You could use video depending on who you're recording with. 00:28:37.460 --> 00:28:40.604 Sometimes people get kind of antsy when they 00:28:40.604 --> 00:28:41.880 know there's a camera. 00:28:41.880 --> 00:28:43.109 That being said-- 00:28:43.109 --> 00:28:48.099 SARA VERRILLI: And also, just put the video tilted at the 00:28:48.099 --> 00:28:48.941 board so you're not recording the players, you're 00:28:48.941 --> 00:28:49.097 recording the game. 00:28:49.097 --> 00:28:50.842 PROFESSOR: Actually webcams are really good for 00:28:50.842 --> 00:28:51.093 that sort of thing. 00:28:51.093 --> 00:28:51.592 SARA VERRILLI: Yeah. 00:28:51.592 --> 00:28:55.782 PROFESSOR: You don't run out of tape, just record straight 00:28:55.782 --> 00:28:56.083 to hard drive. 00:28:56.083 --> 00:28:56.582 SARA VERRILLI: Easy to fast forward through, also. 00:28:56.582 --> 00:29:00.740 PROFESSOR: Yeah, with your editing tools. 00:29:00.740 --> 00:29:02.560 Let's see. 00:29:02.560 --> 00:29:05.820 So when you're reporting to other people, think like 00:29:05.820 --> 00:29:08.640 bullet points for instance. 00:29:08.640 --> 00:29:13.410 Huge long paragraphs of prose, not so much. 00:29:13.410 --> 00:29:14.490 Let's see. 00:29:14.490 --> 00:29:16.930 Try to avoid using the following words, broken, 00:29:16.930 --> 00:29:25.000 problem, unusable, confusing, kind of odd, you can get more 00:29:25.000 --> 00:29:27.680 specific than that, but these are words that kind of like 00:29:27.680 --> 00:29:31.150 imply not only is there a problem, someone made a 00:29:31.150 --> 00:29:34.900 mistake in his design of the game. 00:29:34.900 --> 00:29:39.589 And truth be told, most of the time it wasn't a mistake, you 00:29:39.589 --> 00:29:41.923 just didn't have the complete play testing, you probably 00:29:41.923 --> 00:29:42.740 just had the designer. 00:29:42.740 --> 00:29:44.575 But you don't want to make someone feel like they've made 00:29:44.575 --> 00:29:49.060 a mistake because that makes them defensive and then 00:29:49.060 --> 00:29:51.814 they're going to try to defend how what they wrote in the 00:29:51.814 --> 00:29:55.230 first place is actually correct, which is actually not 00:29:55.230 --> 00:29:56.480 correct either. 00:29:58.850 --> 00:30:01.934 You can talk about features that need attention. 00:30:01.934 --> 00:30:06.774 With attention, this feature could be confused. 00:30:06.774 --> 00:30:10.270 I guess that's actually on the edge there. 00:30:10.270 --> 00:30:18.722 So but just generally, pick your words carefully and one 00:30:18.722 --> 00:30:23.024 thing that Sara mentioned earlier was that testers are 00:30:23.024 --> 00:30:26.080 not designers and key designers are actually not a 00:30:26.080 --> 00:30:27.397 good testing team. 00:30:27.397 --> 00:30:31.728 So for the most part, you are immediately going to exclude 00:30:31.728 --> 00:30:34.737 yourself from your target audience because you know too 00:30:34.737 --> 00:30:35.010 much of the game. 00:30:35.010 --> 00:30:36.590 This is also covered in your book. 00:30:36.590 --> 00:30:39.210 As soon as you've played through your game once or 00:30:39.210 --> 00:30:40.290 twice, that's it. 00:30:40.290 --> 00:30:41.800 Your kind of work is finished. 00:30:41.800 --> 00:30:46.626 You can still be a good technical tester, I guess. 00:30:46.626 --> 00:30:49.914 You can still check whether some mechanics are working the 00:30:49.914 --> 00:30:53.240 way that you want to work, but you can't tell whether the 00:30:53.240 --> 00:30:55.232 dynamics in your settings are working anymore. 00:30:55.232 --> 00:30:58.718 You have to get your focus testing done. 00:30:58.718 --> 00:31:02.204 SARA VERRILLI: You also really can't tell, you can never 00:31:02.204 --> 00:31:05.690 tell-- you can never share your usability tester. 00:31:05.690 --> 00:31:07.184 It will always make sense to you. 00:31:07.184 --> 00:31:10.504 It's amazing that what makes sense to you is completely 00:31:10.504 --> 00:31:14.156 incomprehensible to the very intelligent person you give 00:31:14.156 --> 00:31:15.152 the game to. 00:31:15.152 --> 00:31:16.397 Really, truly, I've been thinking 00:31:16.397 --> 00:31:17.144 about this one myself. 00:31:17.144 --> 00:31:22.124 PROFESSOR: Usability testing, by the way, is one of those 00:31:22.124 --> 00:31:24.655 things that even if you only have the chance to do a pop-up 00:31:24.655 --> 00:31:29.170 partial test like, all I have is this one card, the game's 00:31:29.170 --> 00:31:34.130 not complete, but grab someone walking down the corridor on 00:31:34.130 --> 00:31:37.939 the way to the washroom and say, come in here and take a 00:31:37.939 --> 00:31:38.980 look at this thing I'm working on. 00:31:38.980 --> 00:31:39.950 Even that is better information. 00:31:39.950 --> 00:31:42.990 STUDENT: Hey kid, come in here. 00:31:42.990 --> 00:31:46.120 Come in here, I have something to show you? 00:31:46.120 --> 00:31:46.590 I have a surprise for you. 00:31:46.590 --> 00:31:47.530 [LAUGHTER] 00:31:47.530 --> 00:31:51.220 SARA VERRILLI: Actually, you can get going testing for 00:31:51.220 --> 00:31:52.090 [? serious. ?] 00:31:52.090 --> 00:31:54.175 STUDENT: Free candy. 00:31:54.175 --> 00:31:56.676 PROFESSOR: [? Mona ?] actually calls it employee kidnapping. 00:31:56.676 --> 00:31:59.870 [LAUGHTER] 00:31:59.870 --> 00:32:07.400 Of course, he's the creative director, so he can do that. 00:32:07.400 --> 00:32:10.786 SARA VERRILLI: It works best if you're accosting people 00:32:10.786 --> 00:32:13.370 whom you have power over. 00:32:13.370 --> 00:32:14.610 PROFESSOR: The audio thing, by the way, is one of those 00:32:14.610 --> 00:32:17.730 things that could be really useful if people are difficult 00:32:17.730 --> 00:32:18.580 [? pundits ?]. 00:32:18.580 --> 00:32:22.415 And so it might be one of those things that you want to 00:32:22.415 --> 00:32:25.880 keep in reserve, it's like, OK, there is a problem here 00:32:25.880 --> 00:32:28.005 and the team goes, I don't think there's a problem here. 00:32:28.005 --> 00:32:30.280 I don't understand why anyone's having problems here. 00:32:30.280 --> 00:32:35.522 Then you quit, because it's kind of a bludgeon 00:32:35.522 --> 00:32:37.560 actually on the team. 00:32:37.560 --> 00:32:40.410 On one hand, if you have a good agent team, they're going 00:32:40.410 --> 00:32:43.400 hear it and go, oh my god, we have a problem and immediately 00:32:43.400 --> 00:32:44.417 try to fix it. 00:32:44.417 --> 00:32:45.770 But it's awesome. 00:32:45.770 --> 00:32:50.868 On the other hand, it's like, well, do you need to do a 00:32:50.868 --> 00:32:54.510 recording for every single offer that we have? 00:32:54.510 --> 00:32:55.730 That's something to be known about, so 00:32:55.730 --> 00:32:57.456 keep that in reserve. 00:33:01.210 --> 00:33:03.010 I think that's pretty much it. 00:33:09.310 --> 00:33:12.610 I have an exercise, but I think [UNINTELLIGIBLE] 00:33:12.610 --> 00:33:14.410 so I can turn that off. 00:33:18.010 --> 00:33:19.810 Do you have any questions while I 00:33:19.810 --> 00:33:21.310 graph the data segments? 00:33:21.310 --> 00:33:21.610 No? 00:33:21.610 --> 00:33:22.860 OK.