WEBVTT 00:00:00.030 --> 00:00:02.430 The following content is provided under a Creative 00:00:02.430 --> 00:00:03.990 Commons license. 00:00:03.990 --> 00:00:06.860 Your support will help MIT OpenCourseWare continue to 00:00:06.860 --> 00:00:10.550 offer high quality educational resources for free. 00:00:10.550 --> 00:00:13.410 To make a donation or view additional materials from 00:00:13.410 --> 00:00:17.510 hundreds of MIT courses, visit MIT OpenCourseWare at 00:00:17.510 --> 00:00:19.950 ocw.mit.edu. 00:00:19.950 --> 00:00:26.100 PROFESSOR: So, yesterday in the recitation we talked a 00:00:26.100 --> 00:00:29.260 little bit about how to debug programs on Cell. 00:00:29.260 --> 00:00:32.420 Today I'm going to talk a little more about debugging 00:00:32.420 --> 00:00:36.120 parallel programs in general and give you some common tips 00:00:36.120 --> 00:00:39.430 that might be helpful in helping you track down 00:00:39.430 --> 00:00:41.290 problems that you run into. 00:00:45.190 --> 00:00:48.170 As you might have gotten a feel for yesterday, debugging 00:00:48.170 --> 00:00:51.650 parallel programs is harder than normal sequential 00:00:51.650 --> 00:00:54.090 programs. So in normal sequential programs, you have 00:00:54.090 --> 00:00:55.830 your traditional set of bugs which 00:00:55.830 --> 00:00:57.220 parallel programs inherit. 00:00:57.220 --> 00:01:00.890 So that doesn't get much harder, but then you add on 00:01:00.890 --> 00:01:03.490 new things I can go wrong because of parallelization. 00:01:03.490 --> 00:01:05.540 So things like synchronization, things like 00:01:05.540 --> 00:01:07.790 deadlocks, things like data races. 00:01:07.790 --> 00:01:09.220 So you have to get those right. 00:01:09.220 --> 00:01:11.310 Now you have to debug your program and figure 00:01:11.310 --> 00:01:12.680 out how to do that. 00:01:12.680 --> 00:01:15.530 One of the things you'll see here is a lot of tools might 00:01:15.530 --> 00:01:19.440 not be as good as you'd like them to in terms of providing 00:01:19.440 --> 00:01:22.840 you the functionality for debugging. 00:01:22.840 --> 00:01:26.490 Add to that that bugs in parallel programs often just 00:01:26.490 --> 00:01:32.300 go away if you change one statement in your code -- you 00:01:32.300 --> 00:01:34.640 reorder things and all of a sudden the bug is gone. 00:01:34.640 --> 00:01:37.360 It's kind of like those pointer problems in C, where 00:01:37.360 --> 00:01:40.940 you might add a word, add a new variable somewhere and 00:01:40.940 --> 00:01:42.505 problem's gone, or you add a print-off and 00:01:42.505 --> 00:01:44.130 the problem is gone. 00:01:44.130 --> 00:01:47.060 So here it gets harder because those things can get rid of 00:01:47.060 --> 00:01:50.530 deadlocks, so it makes it really hard to have an 00:01:50.530 --> 00:01:53.210 experiment that you can repeat and come down to where the 00:01:53.210 --> 00:01:55.630 problem is. 00:01:55.630 --> 00:01:57.910 So what might you want in a debugger. 00:01:57.910 --> 00:02:01.190 So this is a list that I've come up with, and if you have 00:02:01.190 --> 00:02:03.760 some ideas we'll want to throw them out. 00:02:03.760 --> 00:02:05.970 I'm thinking in terms of debugging parallel program, 00:02:05.970 --> 00:02:09.190 what I want is a visual debugging system that really 00:02:09.190 --> 00:02:11.490 let's me see all the processors that I have in my 00:02:11.490 --> 00:02:14.790 network in my multi-processor system. 00:02:14.790 --> 00:02:19.060 That includes actual computing and the actual network that 00:02:19.060 --> 00:02:21.055 they're interconnecting all the processors that are going 00:02:21.055 --> 00:02:23.220 to be communicating with each other. 00:02:23.220 --> 00:02:25.160 So I'd like to be able to see what code is 00:02:25.160 --> 00:02:26.610 running on each processor. 00:02:26.610 --> 00:02:29.110 I'd like to see which edges are being used to send 00:02:29.110 --> 00:02:30.300 messages around. 00:02:30.300 --> 00:02:33.010 I might want to know which processors are blocked -- that 00:02:33.010 --> 00:02:37.250 might help me identify deadlock problems. 00:02:37.250 --> 00:02:40.110 For these kinds of scenarios it might be tricky to define 00:02:40.110 --> 00:02:42.940 step, because there's no global clock, you can't force 00:02:42.940 --> 00:02:45.150 everybody to proceed through one step. 00:02:45.150 --> 00:02:47.590 What's one step on one processor might be different 00:02:47.590 --> 00:02:49.270 on another, especially if they're not all 00:02:49.270 --> 00:02:50.460 running the same code. 00:02:50.460 --> 00:02:55.170 So how do you actually do that without a global clock. 00:02:55.170 --> 00:02:58.150 So that can get a little bit tricky. 00:02:58.150 --> 00:03:01.230 It likely won't help with data races, because I'm looking at 00:03:01.230 --> 00:03:03.660 global communication problems, I'm looking at trying to 00:03:03.660 --> 00:03:05.680 identify what's deadlocked and what's not. 00:03:05.680 --> 00:03:08.040 So if there are data races, this kind of tool may or may 00:03:08.040 --> 00:03:09.940 not help with that. 00:03:09.940 --> 00:03:11.380 In general, this is the tool that I 00:03:11.380 --> 00:03:14.060 would build for debugging. 00:03:14.060 --> 00:03:16.430 I looked around on the web to see what's out there for 00:03:16.430 --> 00:03:17.640 debugging parallel programs, and I 00:03:17.640 --> 00:03:19.820 found this called TotalView. 00:03:19.820 --> 00:03:22.260 This is actually something you have to buy, it's not free. 00:03:22.260 --> 00:03:24.790 I don't know if they have evaluation licenses or 00:03:24.790 --> 00:03:28.070 licences for academic purposes. 00:03:28.070 --> 00:03:29.940 It kind of gets close to some of the things 00:03:29.940 --> 00:03:32.330 I was talking about. 00:03:32.330 --> 00:03:34.540 You have processors that shows your communication between 00:03:34.540 --> 00:03:39.040 those processors, how much data is being sent through. 00:03:39.040 --> 00:03:42.730 This particular version uses NPI, which we talked about in 00:03:42.730 --> 00:03:43.780 previous lectures. 00:03:43.780 --> 00:03:46.570 So it's sort of helpful in being able to see the 00:03:46.570 --> 00:03:49.180 computation, looking at the communication, and 00:03:49.180 --> 00:03:51.130 tracking down bugs. 00:03:51.130 --> 00:03:53.930 But it doesn't get much better from there. 00:03:53.930 --> 00:03:58.600 You know, how many people have used printouts for debugging? 00:03:58.600 --> 00:04:01.600 It's the most popular way of debugging, and even I still 00:04:01.600 --> 00:04:03.500 use it for debugging some of the Cell 00:04:03.500 --> 00:04:06.680 programs we've been writing. 00:04:06.680 --> 00:04:09.490 I know the TAs actually use this is well. 00:04:09.490 --> 00:04:14.100 Yesterday you got a hands-on experience with GDB, and GDB 00:04:14.100 --> 00:04:17.440 is a nice debugger, but it lacks a lot of things that you 00:04:17.440 --> 00:04:20.830 might want, especially for debugging parallel programs. 00:04:20.830 --> 00:04:22.860 You saw, for example, that you have multiple threads, you 00:04:22.860 --> 00:04:24.980 need to be able switch between the threads, getting the 00:04:24.980 --> 00:04:28.190 context right, being able to name the variables is tricky. 00:04:28.190 --> 00:04:32.820 So there's a lot of things that could be improved. 00:04:32.820 --> 00:04:35.970 There are some research debuggers, like something 00:04:35.970 --> 00:04:37.720 we've built as part of the streaming 00:04:37.720 --> 00:04:39.160 projects, StreamIt debugger. 00:04:39.160 --> 00:04:40.870 I'll show you some screenshots of this so you can 00:04:40.870 --> 00:04:42.290 see what we can do. 00:04:42.290 --> 00:04:45.930 So in the StreamIt debugger, remember we have -- so this is 00:04:45.930 --> 00:04:48.720 actually built in Eclipse and you can download this off the 00:04:48.720 --> 00:04:49.900 web as well. 00:04:49.900 --> 00:04:52.230 You can look at your stream graph. 00:04:52.230 --> 00:04:55.410 Unfortunately, I couldn't get a split join in there, much to 00:04:55.410 --> 00:04:56.990 Bill's dismay. 00:04:56.990 --> 00:04:59.180 So you can't see, for example, the split join in all the 00:04:59.180 --> 00:05:01.740 communication. 00:05:01.740 --> 00:05:04.850 Each one of these is a filter, and if you recall the filter 00:05:04.850 --> 00:05:08.710 is the computational element in your stream graph and 00:05:08.710 --> 00:05:10.300 interconnected by channels. 00:05:10.300 --> 00:05:12.210 So channels communicate data. 00:05:12.210 --> 00:05:14.940 So what you see here -- well, you might not be able to quite 00:05:14.940 --> 00:05:17.010 see it -- actually see the data that's being passed 00:05:17.010 --> 00:05:18.890 through from one filter to the other. 00:05:18.890 --> 00:05:20.890 You can actually go in there and change the value if you 00:05:20.890 --> 00:05:23.780 wanted to or highlight particular value and see how 00:05:23.780 --> 00:05:25.870 it flows down through the graph. 00:05:25.870 --> 00:05:28.520 If you had a split join, then you might be able to -- in 00:05:28.520 --> 00:05:29.420 fact, you can do this. 00:05:29.420 --> 00:05:33.400 You can look at each path of the split join independently 00:05:33.400 --> 00:05:35.490 and you can look at it in sequence. 00:05:35.490 --> 00:05:38.300 Because the split join has nice semantics, it's actually 00:05:38.300 --> 00:05:42.475 you can replicate the behavior because of the static nature 00:05:42.475 --> 00:05:44.800 is everything is deterministic. 00:05:44.800 --> 00:05:46.050 So this is very helpful. 00:05:46.050 --> 00:05:50.220 We we did a user study two years ago almost with 00:05:50.220 --> 00:05:52.770 something like 30 MIT students who use the C bugger and gave 00:05:52.770 --> 00:05:56.120 us feedback on it. 00:05:56.120 --> 00:06:01.730 So we gave them like 10 problems, 10 code snippets, 00:06:01.730 --> 00:06:04.000 each of them had a bug in it, we asked them to find it. 00:06:04.000 --> 00:06:06.090 So a lot of them found to debugger to be helpful in 00:06:06.090 --> 00:06:09.010 being able to track the flow of data and being able to see 00:06:09.010 --> 00:06:09.810 what goes wrong. 00:06:09.810 --> 00:06:13.460 So if you had, for example, a division that resulted in NaN 00:06:13.460 --> 00:06:15.750 and not a number, floating point division you can 00:06:15.750 --> 00:06:17.750 immediately see on a screen, so you know exactly where to 00:06:17.750 --> 00:06:18.880 go look for it. 00:06:18.880 --> 00:06:22.110 Doing that would print-offs might not be as easy. 00:06:22.110 --> 00:06:24.930 So sometimes visual debugging can be very nice. 00:06:24.930 --> 00:06:27.300 Unfortunately, visual debugging for the Cell isn't 00:06:27.300 --> 00:06:28.110 that great. 00:06:28.110 --> 00:06:32.240 So this is the Cell plug-in in Eclipse. 00:06:32.240 --> 00:06:35.180 I've mentioned just to some of you if you want to run it you 00:06:35.180 --> 00:06:39.050 can run it from a Playstation 3, but if more than one of you 00:06:39.050 --> 00:06:41.920 is running it then it becomes unusable 00:06:41.920 --> 00:06:44.360 because of memory issues. 00:06:44.360 --> 00:06:50.050 You can install this on other Linux machines and remotely 00:06:50.050 --> 00:06:51.750 debug on the Playstation 3 hardware. 00:06:51.750 --> 00:06:55.410 So the two remote machines can talk through GDB ports. 00:06:55.410 --> 00:06:57.770 I can talk to you about how to set that up if you want to, 00:06:57.770 --> 00:06:59.260 but it doesn't really add anything 00:06:59.260 --> 00:07:00.830 over E-Max, for example. 00:07:00.830 --> 00:07:03.950 It just might look fancier than an E-Max window or a GDB 00:07:03.950 --> 00:07:05.530 on the command line prompt. 00:07:05.530 --> 00:07:07.200 So this is the code from yesterday. 00:07:07.200 --> 00:07:09.610 These are the exercises we asked you to do. 00:07:09.610 --> 00:07:11.130 You can look at the different threads. 00:07:11.130 --> 00:07:13.190 If you have debug Java programs in Eclipse this 00:07:13.190 --> 00:07:14.860 should look very familiar. 00:07:14.860 --> 00:07:18.030 You can look at the different variables. 00:07:18.030 --> 00:07:19.760 You still have the naming problem. 00:07:19.760 --> 00:07:22.800 So yesterday, remember you had to qualify which control box 00:07:22.800 --> 00:07:23.700 you were looking at? 00:07:23.700 --> 00:07:25.750 Still the same kind of issue -- have to do some trickery to 00:07:25.750 --> 00:07:27.010 find it here. 00:07:27.010 --> 00:07:29.840 It doesn't have the nice visual aspect of showing you 00:07:29.840 --> 00:07:32.980 which code is running on which SPE, and you might not be able 00:07:32.980 --> 00:07:37.510 to find mailbox synchronization problems. 00:07:37.510 --> 00:07:39.700 Maybe those things will come in the future, and the fact, 00:07:39.700 --> 00:07:41.210 they likely will. 00:07:41.210 --> 00:07:42.950 But a lot of that is sort of still lacking. 00:07:42.950 --> 00:07:45.530 So what do you do in the meantime, In the next two 00:07:45.530 --> 00:07:47.650 weeks as you're writing your programs? 00:07:47.650 --> 00:07:51.560 So I've looked around for some tips or some talks and 00:07:51.560 --> 00:07:56.350 lectures and what people have done in terms of improving the 00:07:56.350 --> 00:07:59.280 process for debugging parallel codes. 00:07:59.280 --> 00:08:01.860 Probably the best thing I've found is this talk that was 00:08:01.860 --> 00:08:05.190 given at University of Maryland on defect patterns. 00:08:05.190 --> 00:08:09.260 So the rest of these slides are largely 00:08:09.260 --> 00:08:11.370 drawn from that talk. 00:08:11.370 --> 00:08:13.680 I'm going to identify just a few of them to give you some 00:08:13.680 --> 00:08:16.576 examples so you can understand what to look for and what are 00:08:16.576 --> 00:08:19.040 some common symptoms, what are some common prevention 00:08:19.040 --> 00:08:20.770 techniques. 00:08:20.770 --> 00:08:24.660 So defect patterns, just like the programming patterns we 00:08:24.660 --> 00:08:27.520 talked about, are meant to help you conjure up to write 00:08:27.520 --> 00:08:28.600 contextual information. 00:08:28.600 --> 00:08:31.240 You had what are things you should look for if you're 00:08:31.240 --> 00:08:33.190 communicating with somebody else. 00:08:33.190 --> 00:08:35.470 What kind of terminology do you use so that you don't have 00:08:35.470 --> 00:08:40.370 to explain things down to every last detail. 00:08:40.370 --> 00:08:42.930 At the end of this course, one thing I'd like to do is maybe 00:08:42.930 --> 00:08:45.220 get some feedback from each of you as to what are some of the 00:08:45.220 --> 00:08:48.920 problems that you ran into in writing your programs, and how 00:08:48.920 --> 00:08:51.110 you actually went about debugging them, and maybe we 00:08:51.110 --> 00:08:55.050 can come up with Cell defect patterns, and maybe Cell 00:08:55.050 --> 00:08:58.430 defect recipes for resolving those defect patterns. 00:09:02.050 --> 00:09:05.920 So, probably the worst one of all, and the easiest one to 00:09:05.920 --> 00:09:09.130 fix is that you have new language features or new 00:09:09.130 --> 00:09:12.050 language extensions that are not well understood. 00:09:12.050 --> 00:09:15.290 This is especially true when you take a class of students 00:09:15.290 --> 00:09:18.340 and they don't really know the language, they don't know all 00:09:18.340 --> 00:09:23.150 the tools, and you ask them to do a project in four weeks and 00:09:23.150 --> 00:09:24.120 you expect things to work. 00:09:24.120 --> 00:09:28.410 So there's a lot for everybody to pick up and understand. 00:09:28.410 --> 00:09:31.620 So you might have inconsistent types that you use in terms of 00:09:31.620 --> 00:09:33.240 calling a function. 00:09:33.240 --> 00:09:35.430 There might be alignment issues, which some of 00:09:35.430 --> 00:09:36.900 you have run into. 00:09:36.900 --> 00:09:38.830 You might use the wrong functions. 00:09:38.830 --> 00:09:41.190 You know the functionality you want but you just don't know 00:09:41.190 --> 00:09:44.510 how to name it and so you might use the wrong function. 00:09:44.510 --> 00:09:47.300 Some of these are easy to fix because you might get a 00:09:47.300 --> 00:09:48.470 compile time error. 00:09:48.470 --> 00:09:51.150 If you have mismatch in function parameters then you 00:09:51.150 --> 00:09:52.840 can fix that very easily. 00:09:52.840 --> 00:09:55.780 Some defects -- you know, very natural parallel programs 00:09:55.780 --> 00:09:58.730 might not come up until run time, so you might end up with 00:09:58.730 --> 00:10:02.220 crashes or just erroneous behavior. 00:10:02.220 --> 00:10:03.840 I really think this is probably the easiest one to 00:10:03.840 --> 00:10:07.140 fix, and the prevention technique that I would 00:10:07.140 --> 00:10:10.400 recommend is if there's something you're unfamiliar 00:10:10.400 --> 00:10:13.830 about or you're not sure about how to use something, ask. 00:10:13.830 --> 00:10:16.930 But also, you don't need to know all the functions that 00:10:16.930 --> 00:10:20.600 are available in something like the Cell language 00:10:20.600 --> 00:10:22.440 extensions for C. 00:10:22.440 --> 00:10:23.760 Yes, there are a lot of functions -- you know, the 00:10:23.760 --> 00:10:26.890 manuals, hundreds of pages, and you can't possibly go 00:10:26.890 --> 00:10:28.600 through it all and nobody becomes an expert in 00:10:28.600 --> 00:10:29.280 everything. 00:10:29.280 --> 00:10:32.500 But understand just a few basic concepts and features. 00:10:32.500 --> 00:10:36.730 So, David identified a bunch that he found useful for 00:10:36.730 --> 00:10:41.850 writing the programs, and some of the ones that are up on the 00:10:41.850 --> 00:10:47.210 web page under the recipes for this course list a few more. 00:10:47.210 --> 00:10:50.760 And so this might help you in just understanding how these 00:10:50.760 --> 00:10:53.070 functions work, understanding basic mechanisms that they 00:10:53.070 --> 00:10:54.810 give you, and that's good enough because it'll 00:10:54.810 --> 00:10:55.790 help you get by. 00:10:55.790 --> 00:10:59.250 Certainly for doing the project under short time 00:10:59.250 --> 00:11:01.570 constraints, you don't need to know all the advanced features 00:11:01.570 --> 00:11:04.560 that Cell might have. Or you can probably just pick them up 00:11:04.560 --> 00:11:07.550 on the fly as you need them. 00:11:07.550 --> 00:11:09.010 So what are some more interesting 00:11:09.010 --> 00:11:10.980 problems that come up. 00:11:10.980 --> 00:11:14.560 I think one that is probably not too unfamiliar is this 00:11:14.560 --> 00:11:17.820 space decomposition problems. So, if you remember, space 00:11:17.820 --> 00:11:20.230 decomposition is really data distribution. 00:11:20.230 --> 00:11:23.320 You have a serial program that you want to parallelize. 00:11:23.320 --> 00:11:25.450 And what that means if you have to actually send data 00:11:25.450 --> 00:11:28.450 around to different processors so that each one knows how to 00:11:28.450 --> 00:11:31.030 compute locally. 00:11:31.030 --> 00:11:34.440 Here you might get things like segmentation faults, alignment 00:11:34.440 --> 00:11:39.650 problems, you might have index out of range errors. 00:11:39.650 --> 00:11:42.900 What this comes of is forgetting to change things or 00:11:42.900 --> 00:11:45.420 overlooking some simple things that don't carryover from the 00:11:45.420 --> 00:11:47.670 sequential case that are parallel case. 00:11:47.670 --> 00:11:50.200 So what you might want to do is validate your distributions 00:11:50.200 --> 00:11:51.980 and your memory partitions correctly. 00:11:51.980 --> 00:11:54.970 So what's an example? 00:11:54.970 --> 00:11:59.610 So suppose you had an array or a list of cells, 00:11:59.610 --> 00:12:01.120 each cell has a number. 00:12:01.120 --> 00:12:04.420 What you want to do is at each step of the computation for 00:12:04.420 --> 00:12:07.550 any given cell, you want add the value to the left of it 00:12:07.550 --> 00:12:09.210 and the value to the right of it. 00:12:09.210 --> 00:12:11.660 So here we have cell zero has the value 2. 00:12:11.660 --> 00:12:13.730 We'll assume that the n disconnectors are first, so 00:12:13.730 --> 00:12:17.910 this is like a circular list, a circular buffer. 00:12:17.910 --> 00:12:22.060 So adding the left and right neighbor would get me, in this 00:12:22.060 --> 00:12:24.600 case, 3 plus 1, 4. 00:12:24.600 --> 00:12:26.570 And so on and so forth. 00:12:26.570 --> 00:12:28.710 You want to repeat this computation for n steps. 00:12:28.710 --> 00:12:31.100 So this might be very common in computations where you're 00:12:31.100 --> 00:12:34.190 doing unit [? book ?] communication. 00:12:34.190 --> 00:12:37.560 So what's a straightforward sequential implementation? 00:12:37.560 --> 00:12:39.850 Well, you can use two buffers -- 00:12:39.850 --> 00:12:42.220 one for the current time step, and you do all the 00:12:42.220 --> 00:12:43.330 calculations in that. 00:12:43.330 --> 00:12:47.970 Then you use another buffer for next time step. 00:12:47.970 --> 00:12:50.530 Then you swap the two. 00:12:50.530 --> 00:12:52.830 So the code might look something like this. 00:12:52.830 --> 00:12:56.470 Sequential c code, my two buffers, here's my loop. 00:12:56.470 --> 00:12:58.960 I write into one buffer and then I switch 00:12:58.960 --> 00:13:00.380 the two buffers around. 00:13:00.380 --> 00:13:03.450 Any questions so far? 00:13:03.450 --> 00:13:07.035 So now, what are some things that can go wrong when you try 00:13:07.035 --> 00:13:11.060 to parallelize this? 00:13:11.060 --> 00:13:13.580 So how would you actually parallelize this code? 00:13:13.580 --> 00:13:15.990 Well, we saw in some of your labs, for example, that you 00:13:15.990 --> 00:13:18.620 can take a big array, split it up in smaller chunks and 00:13:18.620 --> 00:13:21.450 assign each chunk to one particular processor. 00:13:21.450 --> 00:13:23.610 So we can use that technique here. 00:13:23.610 --> 00:13:28.540 So each processor, we have n of them, rather size of them, 00:13:28.540 --> 00:13:32.000 and it's going to get some number of elements. 00:13:32.000 --> 00:13:35.060 So each time step, we have to compute locally all the 00:13:35.060 --> 00:13:37.870 communications, but then there's some special cases 00:13:37.870 --> 00:13:40.270 that we need to treat at the boundaries, right. 00:13:40.270 --> 00:13:42.430 So if I have this chunk and I need to do my new neighbor 00:13:42.430 --> 00:13:44.800 communication, I don't have this particular cells. 00:13:44.800 --> 00:13:47.230 I have to go out there and request it. 00:13:47.230 --> 00:13:48.890 Similarly, somebody has to send me this 00:13:48.890 --> 00:13:49.960 particular data item. 00:13:49.960 --> 00:13:54.890 So there's some data exchange that has to happen. 00:13:54.890 --> 00:13:59.300 So in the decomposition, you write your parallel code. 00:13:59.300 --> 00:14:01.930 Here, each buffer is a different size. 00:14:01.930 --> 00:14:05.550 What you do is you have some local, which says this is how 00:14:05.550 --> 00:14:08.510 much of the data I'm getting, and has a total number of 00:14:08.510 --> 00:14:11.490 elements, besides the number of processors. 00:14:11.490 --> 00:14:14.540 Local essentially tells me the size of my chunk. 00:14:14.540 --> 00:14:17.900 I'm iterating through from zero and local and I'm doing 00:14:17.900 --> 00:14:19.710 essentially the same computation. 00:14:19.710 --> 00:14:21.400 So what's a bug in here? 00:14:21.400 --> 00:14:22.260 Anybody see it? 00:14:22.260 --> 00:14:24.780 Sort of giving you some hints of things highlighted in red 00:14:24.780 --> 00:14:26.170 or there's something wrong with the things 00:14:26.170 --> 00:14:27.420 highlighted in red. 00:14:36.790 --> 00:14:38.090 There's another hint. 00:14:38.090 --> 00:14:40.330 So this is essentially the computations going on at every 00:14:40.330 --> 00:14:44.230 processor, this is my buffer, and at every step I have to do 00:14:44.230 --> 00:14:46.320 the calculations, taking care of the boundary edges. 00:14:51.840 --> 00:14:55.100 Anybody want to take a stab? 00:14:55.100 --> 00:14:55.450 Mark? 00:14:55.450 --> 00:15:00.490 AUDIENCE: Is it that the nextbuffer zero needs to look 00:15:00.490 --> 00:15:01.740 at data from 1? 00:15:04.720 --> 00:15:06.820 PROFESSOR: Next buffer zero, right. 00:15:06.820 --> 00:15:08.780 So what might be a fix to that? 00:15:12.910 --> 00:15:14.810 So the next buffer is zero. 00:15:14.810 --> 00:15:18.850 So if this is zero then buffer of x minus 1 plus 00:15:18.850 --> 00:15:20.100 this points to what? 00:15:23.160 --> 00:15:24.640 AUDIENCE: So you need to start at 1 and iterate. 00:15:24.640 --> 00:15:26.840 PROFESSOR: Right, exactly. 00:15:26.840 --> 00:15:30.780 It's a local plus 1, if you were going to do these. 00:15:30.780 --> 00:15:32.150 So that's one bug. 00:15:32.150 --> 00:15:34.810 The other thing is the assumption that your data 00:15:34.810 --> 00:15:37.800 elements might be divisible by the number of processors that 00:15:37.800 --> 00:15:40.260 you have. So you pick the decomposition that might not 00:15:40.260 --> 00:15:42.110 be symmetric across all processors. 00:15:42.110 --> 00:15:46.320 So it's more subtle, I think, to keep in mind. 00:15:46.320 --> 00:15:48.640 So that's one particular kind of problem that might come up 00:15:48.640 --> 00:15:51.350 on your decomposing data and replicating among different 00:15:51.350 --> 00:15:52.610 processors. 00:15:52.610 --> 00:15:56.380 So you have to be careful about what are your boundary 00:15:56.380 --> 00:16:00.650 cases going to be and how are you going to deal with those. 00:16:00.650 --> 00:16:03.870 The more difficult one is synchronization. 00:16:03.870 --> 00:16:06.630 So synchronization is when you're sending data from one 00:16:06.630 --> 00:16:09.530 processor to the other and you might end up with deadlock, 00:16:09.530 --> 00:16:11.920 because one is trying to send, the other's trying to send, 00:16:11.920 --> 00:16:15.020 and neither can make progress until the other's received. 00:16:15.020 --> 00:16:19.390 So your program hangs, you get non-deterministic behavior or 00:16:19.390 --> 00:16:22.110 output, every time you run your program you get a 00:16:22.110 --> 00:16:24.910 different result -- that can drive you crazy. 00:16:24.910 --> 00:16:27.910 So some of the defects can be very subtle. 00:16:27.910 --> 00:16:29.750 This is probably where you'll spend most of your time trying 00:16:29.750 --> 00:16:32.400 to figure it out. 00:16:32.400 --> 00:16:35.100 So one of the ways to prevent this is to actually look at 00:16:35.100 --> 00:16:37.680 how you're orchestrating your communication and doing it 00:16:37.680 --> 00:16:38.700 very carefully. 00:16:38.700 --> 00:16:42.660 So look at, for example, what's going on here. 00:16:42.660 --> 00:16:45.620 So this is the same problem, and what I'm doing is now this 00:16:45.620 --> 00:16:48.780 is the parallel version and I'm sending the boundary 00:16:48.780 --> 00:16:52.870 cases, the boundary cells to the different processors. 00:16:52.870 --> 00:16:54.290 This is an SPMD program. 00:16:54.290 --> 00:16:58.240 So an SPMD program has every processor running essentially 00:16:58.240 --> 00:16:59.380 the same code. 00:16:59.380 --> 00:17:01.640 So this code is replicated over n processors and 00:17:01.640 --> 00:17:03.290 everybody's trying to do this same thing. 00:17:03.290 --> 00:17:06.500 So what's the problem with this code? 00:17:06.500 --> 00:17:11.060 We're doing send of next buffer zero. 00:17:11.060 --> 00:17:13.320 Here, rank essentially just says each 00:17:13.320 --> 00:17:14.880 processor has a rank. 00:17:14.880 --> 00:17:17.360 So this is a way of identifying things. 00:17:17.360 --> 00:17:20.280 So I'm trying to send it to my previous guy, I'm trying to 00:17:20.280 --> 00:17:23.740 send it to the next guy, and here I'm sending the value at 00:17:23.740 --> 00:17:27.150 the far extreme of the buffer to the next processor and then 00:17:27.150 --> 00:17:28.560 to the previous processor. 00:17:28.560 --> 00:17:29.630 Anybody see what's wrong here? 00:17:29.630 --> 00:17:33.610 AUDIENCE: So are these blocking things? 00:17:33.610 --> 00:17:35.160 PROFESSOR: Yeah, imagine they're blocking things. 00:17:35.160 --> 00:17:37.149 AUDIENCE: Why will that deadlock? 00:17:40.630 --> 00:17:42.100 PROFESSOR: Right. 00:17:42.100 --> 00:17:45.380 So this will deadlock, right. 00:17:45.380 --> 00:17:48.110 So this will deadlock because this processor is trying to 00:17:48.110 --> 00:17:50.590 send here, this processor is trying to send here, but 00:17:50.590 --> 00:17:53.410 neither is receiving yet, so neither makes progress. 00:17:53.410 --> 00:17:55.940 So how would you fix it at this point? 00:17:55.940 --> 00:17:59.950 You might not want to use a blocking send all the time. 00:17:59.950 --> 00:18:03.600 So if your architecture allows you to have different flavors 00:18:03.600 --> 00:18:05.960 of communication, so synchronous versus an 00:18:05.960 --> 00:18:08.680 asynchronous, a blocking versus non-blocking, you'll 00:18:08.680 --> 00:18:11.790 want to avoid using constructs that can lead you to deadlock 00:18:11.790 --> 00:18:14.020 if you don't need to. 00:18:14.020 --> 00:18:17.430 The other mechanism -- this was pointed out briefly in the 00:18:17.430 --> 00:18:19.980 talk on parallel programming -- you want to order your 00:18:19.980 --> 00:18:21.440 sends and receives properly. 00:18:21.440 --> 00:18:22.330 So alternate them. 00:18:22.330 --> 00:18:24.160 So you have a send in one processor, a 00:18:24.160 --> 00:18:25.970 receive in the other. 00:18:25.970 --> 00:18:28.430 You can use that to prevent deadlock and get the 00:18:28.430 --> 00:18:30.890 communication patterns right. 00:18:30.890 --> 00:18:32.640 There could be more interesting cases that come up 00:18:32.640 --> 00:18:35.230 if you're communicating over a network where you might end up 00:18:35.230 --> 00:18:40.470 with cyclic patterns leading to loops, and that also can 00:18:40.470 --> 00:18:41.720 create some problems for you. 00:18:44.770 --> 00:18:48.780 The last two I'll talk about aren't really bugs in that 00:18:48.780 --> 00:18:51.090 they might not cause your program to break or compute 00:18:51.090 --> 00:18:52.050 incorrectly. 00:18:52.050 --> 00:18:53.890 Things might work properly, but you might not get the 00:18:53.890 --> 00:18:56.460 actual performance that you're expecting. 00:18:56.460 --> 00:19:01.660 So these are performance bucks or performance defects. 00:19:01.660 --> 00:19:04.940 So side effects of parallelization is often case 00:19:04.940 --> 00:19:07.560 that you're focusing on your parallel code and you might 00:19:07.560 --> 00:19:09.820 ignore things that are going on in your sequential code, 00:19:09.820 --> 00:19:12.060 and that might lead you to, essentially you've spent all 00:19:12.060 --> 00:19:14.720 this time trying to parallelize your code, but 00:19:14.720 --> 00:19:16.760 your end result is not getting the performance that you 00:19:16.760 --> 00:19:19.030 expect because things look sequential. 00:19:19.030 --> 00:19:21.120 So what's wrong here? 00:19:21.120 --> 00:19:26.240 So as an example, imagine that we're doing instead of reading 00:19:26.240 --> 00:19:30.160 data from a -- 00:19:30.160 --> 00:19:32.020 so, in the previous case I didn't show you how we were 00:19:32.020 --> 00:19:34.720 reading data into the different buffers, but suppose 00:19:34.720 --> 00:19:37.240 we were getting it from some files, so input buffer. 00:19:37.240 --> 00:19:39.750 So now we have an SPMD program again, everybody's trying to 00:19:39.750 --> 00:19:41.010 read from this buffer. 00:19:41.010 --> 00:19:42.930 What could go wrong here? 00:19:42.930 --> 00:19:45.150 Anybody have an idea? 00:19:45.150 --> 00:19:48.050 So every processor is opening the file and then it's going 00:19:48.050 --> 00:19:50.650 to figure out how much to skip and it'll start reading from 00:19:50.650 --> 00:19:51.245 that location. 00:19:51.245 --> 00:19:53.830 So everybody's reading from a file, so that's OK, nobody's 00:19:53.830 --> 00:19:54.510 modifying it. 00:19:54.510 --> 00:19:55.950 But what can go wrong here? 00:19:55.950 --> 00:19:59.887 AUDIENCE: [INAUDIBLE PHRASE]. 00:20:07.270 --> 00:20:09.760 PROFESSOR: Right. 00:20:09.760 --> 00:20:13.010 So essentially, sequentialize your execution because reading 00:20:13.010 --> 00:20:16.770 from the file system becomes the bottleneck. 00:20:16.770 --> 00:20:19.550 So you'll want to schedule input and output carefully. 00:20:19.550 --> 00:20:21.680 You might find that not everybody needs to do the 00:20:21.680 --> 00:20:22.860 input and output. 00:20:22.860 --> 00:20:26.520 Only one processor has to do the input and then it can 00:20:26.520 --> 00:20:28.370 distribute it to all the different processors. 00:20:28.370 --> 00:20:32.390 So, in the Master/Slave model, which a lot of you are using 00:20:32.390 --> 00:20:36.550 for the Cell programming, the Master can just read the data 00:20:36.550 --> 00:20:37.810 from the input files and distribute it 00:20:37.810 --> 00:20:38.633 to everybody else. 00:20:38.633 --> 00:20:39.900 So this avoids some of the problems 00:20:39.900 --> 00:20:41.740 with input and output. 00:20:41.740 --> 00:20:45.150 You can have similar kinds of problems if you're reading 00:20:45.150 --> 00:20:46.130 from other devices. 00:20:46.130 --> 00:20:48.740 It doesn't have to be the file system. 00:20:48.740 --> 00:20:52.520 So here's another one, a little more subtle. 00:20:52.520 --> 00:20:55.810 So you're generating data--. 00:20:55.810 --> 00:20:57.196 Hey, Allen, what's up? 00:20:57.196 --> 00:20:59.965 AUDIENCE: I somehow missed the distinction between when 00:20:59.965 --> 00:21:02.231 you're waiting for the master to read all the dat aand 00:21:02.231 --> 00:21:04.748 distribute it, and waiting for the other [? processes ?] to 00:21:04.748 --> 00:21:07.600 get through so I can read my private data, isn't it going 00:21:07.600 --> 00:21:10.910 to be about the same time on this? 00:21:10.910 --> 00:21:11.090 PROFESSOR: No. 00:21:11.090 --> 00:21:15.130 So here, just essentially, the Master reads the file as part 00:21:15.130 --> 00:21:17.410 of the initialization. 00:21:17.410 --> 00:21:18.050 Then you distribute it. 00:21:18.050 --> 00:21:19.770 So distribution can happen at run time. 00:21:19.770 --> 00:21:23.250 So, the initialization you don't care about because 00:21:23.250 --> 00:21:25.080 hopefully that's a small part of the code. 00:21:28.680 --> 00:21:31.550 So this code is guarded by rank equals Master, so only it 00:21:31.550 --> 00:21:32.270 does this code. 00:21:32.270 --> 00:21:35.200 Then here you might have the command that says wait until 00:21:35.200 --> 00:21:38.680 I've received it and then execute, or on the cells, then 00:21:38.680 --> 00:21:41.270 these might be the SPE create threads that happen after 00:21:41.270 --> 00:21:42.340 you've read the data. 00:21:42.340 --> 00:21:45.200 So hopefully, initialization time is not something you have 00:21:45.200 --> 00:21:46.450 concern about too much. 00:21:48.830 --> 00:21:52.930 So if you're generating data on the fly or dynamically, so 00:21:52.930 --> 00:21:56.250 here we might use the Srand function to sort of start with 00:21:56.250 --> 00:21:58.830 a random seeing and then fill in the buffer 00:21:58.830 --> 00:22:00.330 with some random data. 00:22:00.330 --> 00:22:01.580 So what could go wrong here? 00:22:04.250 --> 00:22:10.070 So in Srand, when you're using a random function -- sorry, 00:22:10.070 --> 00:22:12.340 this is the same function. 00:22:12.340 --> 00:22:14.640 When you're using a random, a pseudo random number 00:22:14.640 --> 00:22:17.390 generator, you have to give it a seed, then if everybody 00:22:17.390 --> 00:22:20.060 starts off with the same seed, then you might end up with the 00:22:20.060 --> 00:22:22.640 same random number sequence. 00:22:22.640 --> 00:22:25.960 If that's something you're using to parallelize your 00:22:25.960 --> 00:22:28.460 computation, you might, in effect, end up with the same 00:22:28.460 --> 00:22:31.820 kind of sequence on each processor and you lose all 00:22:31.820 --> 00:22:34.250 kinds of parallelization. 00:22:34.250 --> 00:22:36.860 So there's some hidden serialization issues in some 00:22:36.860 --> 00:22:38.940 of the functions that you might use that you 00:22:38.940 --> 00:22:40.190 should be aware of. 00:22:42.350 --> 00:22:44.430 The last one I'll talk about is 00:22:44.430 --> 00:22:46.570 performance scalability defect. 00:22:46.570 --> 00:22:50.170 So here you parallelize your code, things look good, but 00:22:50.170 --> 00:22:51.300 you're still not getting -- 00:22:51.300 --> 00:22:54.030 you've taken care of all your IO issue, you're still not 00:22:54.030 --> 00:22:55.370 getting the performance you want. 00:22:55.370 --> 00:22:57.710 So, why is that? 00:22:57.710 --> 00:23:01.430 You might have -- remember your Amdahl's law, and what 00:23:01.430 --> 00:23:03.580 you want is an efficiency that's linear. 00:23:03.580 --> 00:23:08.500 Every time you add one processor you want a straight 00:23:08.500 --> 00:23:11.410 line curve between the number of processors and speed up. 00:23:11.410 --> 00:23:13.490 This should be a linear relationship. 00:23:13.490 --> 00:23:16.440 So you might see sublinear speed ups, and you want to 00:23:16.440 --> 00:23:17.980 figure out why that is. 00:23:17.980 --> 00:23:20.680 Some of the common causes here, and this will be the end 00:23:20.680 --> 00:23:23.280 up focus of the next talk is, unbalanced amount of 00:23:23.280 --> 00:23:24.100 computation. 00:23:24.100 --> 00:23:25.420 Remember, dynamic load balancing 00:23:25.420 --> 00:23:27.050 versus static load balancing. 00:23:27.050 --> 00:23:29.210 Your work estimation might be wrong and so you might end up 00:23:29.210 --> 00:23:32.660 with some processors idling, other processors 00:23:32.660 --> 00:23:34.940 doing too much work. 00:23:34.940 --> 00:23:37.280 So the way to prevent this is to actually look at the work 00:23:37.280 --> 00:23:40.370 that's being done and figure out whether it's actually 00:23:40.370 --> 00:23:42.380 roughly the same amount of work everywhere. 00:23:42.380 --> 00:23:45.040 Here you might need profiling tools to help, and so I'm 00:23:45.040 --> 00:23:46.250 going to talk about this in a lot more 00:23:46.250 --> 00:23:49.930 detail in the next lecture. 00:23:49.930 --> 00:23:53.800 So in summary, there are lots of different bugs that you 00:23:53.800 --> 00:23:56.030 might come up with. 00:23:56.030 --> 00:23:59.490 There's a few that I've identified here, some common 00:23:59.490 --> 00:24:01.070 things you should look out for. 00:24:01.070 --> 00:24:03.520 So the erroneous use of language features understand 00:24:03.520 --> 00:24:06.790 only a few basic concepts of the entire language extension 00:24:06.790 --> 00:24:10.270 set that you have. Space decomposition, side effects 00:24:10.270 --> 00:24:11.540 from parallelization. 00:24:11.540 --> 00:24:14.330 Don't ignore sequential code. 00:24:14.330 --> 00:24:16.430 Last one is trying to understand your performance 00:24:16.430 --> 00:24:17.390 scalability. 00:24:17.390 --> 00:24:18.760 But there are other kinds of bugs, like 00:24:18.760 --> 00:24:19.990 data races, for example. 00:24:19.990 --> 00:24:22.200 So what can you do with those? 00:24:22.200 --> 00:24:24.800 So remember, data races you have different concurrent 00:24:24.800 --> 00:24:25.990 threads and they're trying to update 00:24:25.990 --> 00:24:28.010 the same memory location. 00:24:28.010 --> 00:24:30.400 So depending on who gets to write first and when you 00:24:30.400 --> 00:24:34.540 actually do your read, you might get a different result. 00:24:34.540 --> 00:24:37.880 So with data race detection, these things are actually 00:24:37.880 --> 00:24:38.840 getting better. 00:24:38.840 --> 00:24:41.140 There are tools out there that will essentially generate 00:24:41.140 --> 00:24:43.530 traces as your program is running. 00:24:43.530 --> 00:24:46.140 So for each thread you instrumented and you look at 00:24:46.140 --> 00:24:48.120 every load stored at executes. 00:24:48.120 --> 00:24:51.040 Then what you do is you look at the load in stores between 00:24:51.040 --> 00:24:53.410 the difference threads and see if there's any intersections, 00:24:53.410 --> 00:24:57.600 any orderings that might give you erroneous behavior. 00:24:57.600 --> 00:25:00.770 So this is getting better, it's getting more automated. 00:25:00.770 --> 00:25:02.610 Intel Threadchecker is one example. 00:25:02.610 --> 00:25:05.310 There are others. 00:25:05.310 --> 00:25:07.670 I really think the trend in debugging will be towards 00:25:07.670 --> 00:25:11.720 trace-based systems. You'll have things like 00:25:11.720 --> 00:25:12.430 checkpointing. 00:25:12.430 --> 00:25:15.550 So as your program is running you can take a snapshot of 00:25:15.550 --> 00:25:17.830 where it is in the execution, and then you can use that 00:25:17.830 --> 00:25:21.390 snapshot later on to inspect it and see what went wrong. 00:25:21.390 --> 00:25:23.380 I think you might even have features like replay. 00:25:23.380 --> 00:25:26.350 In fact, some people are working on this in research 00:25:26.350 --> 00:25:28.120 and in industry. 00:25:28.120 --> 00:25:31.000 So you might be able to say uh-oh, something went wrong. 00:25:31.000 --> 00:25:33.840 Here's my list of checkpoints, can you replay the execution 00:25:33.840 --> 00:25:36.360 from this particular stage in the computation. 00:25:36.360 --> 00:25:40.540 So it helps you focus down in the entire lifetime of 00:25:40.540 --> 00:25:42.320 execution on a particular chunk where 00:25:42.320 --> 00:25:45.710 things have gone wrong. 00:25:45.710 --> 00:25:48.900 This is sort of a personal dream. 00:25:48.900 --> 00:25:51.300 I think one day we'll have the equivalent of a TiVo for your 00:25:51.300 --> 00:25:54.350 programs, and you can use it for debugging. 00:25:54.350 --> 00:25:56.470 So my program is running, something goes wrong, I can 00:25:56.470 --> 00:25:58.970 rewind it, I can inspect things, do my traditional 00:25:58.970 --> 00:26:02.700 debugging, change things maybe even, and then start replaying 00:26:02.700 --> 00:26:07.580 things and letting the program execute. 00:26:07.580 --> 00:26:10.320 In fact, we're working on things like this here at MIT 00:26:10.320 --> 00:26:12.140 and with collaborators elsewhere. 00:26:12.140 --> 00:26:15.890 So, this was a short lecture. 00:26:15.890 --> 00:26:16.530 We'll take a break. 00:26:16.530 --> 00:26:18.240 You can do the quizzes. 00:26:18.240 --> 00:26:19.410 Note on the quizzes, there are two 00:26:19.410 --> 00:26:20.720 different kinds of questions. 00:26:20.720 --> 00:26:24.570 They're very similar, just one word is different, and so 00:26:24.570 --> 00:26:26.720 you'll want to just keep that in mind when you're discussing 00:26:26.720 --> 00:26:28.310 it with others. 00:26:28.310 --> 00:26:31.320 Then about 5, 10 minutes and then we'll continue with the 00:26:31.320 --> 00:26:33.710 rest of the talk, lecture 2. 00:26:33.710 --> 00:26:34.960 Thanks.