WEBVTT 00:00:00.000 --> 00:00:02.430 The following content is provided under a Creative 00:00:02.430 --> 00:00:03.880 Commons license. 00:00:03.880 --> 00:00:06.950 Your support will help MIT OpenCourseWare continue to 00:00:06.950 --> 00:00:10.550 offer high-quality educational resources for free. 00:00:10.550 --> 00:00:13.690 To make a donation or view additional materials from 00:00:13.690 --> 00:00:17.920 hundreds of MIT courses, visit MIT OpenCourseWare at ocw. 00:00:17.920 --> 00:00:19.170 mit.edu. 00:00:22.890 --> 00:00:28.070 PROFESSOR: So last week, last few lectures, you heard about 00:00:28.070 --> 00:00:32.760 parallel architectures and started with lecture four on 00:00:32.760 --> 00:00:34.290 discussions of concurrency. 00:00:34.290 --> 00:00:37.730 How do you take applications or independent actors that 00:00:37.730 --> 00:00:40.260 want to operate on the same data and make 00:00:40.260 --> 00:00:42.790 them run safely together? 00:00:42.790 --> 00:00:46.930 And so just recapping the last two lectures, you saw really 00:00:46.930 --> 00:00:49.330 two primary classes of architectures. 00:00:49.330 --> 00:00:51.800 Although Saman talked about a few more. 00:00:51.800 --> 00:00:55.630 There was the class of shared memory processors, you know, 00:00:55.630 --> 00:00:59.500 the multicores that Intel, AMD, and PowerPC, for example, 00:00:59.500 --> 00:01:03.970 have today, where you have one copy of the data. 00:01:03.970 --> 00:01:06.660 And that's really shared among all the different processors 00:01:06.660 --> 00:01:09.770 because they essentially share the same memory. 00:01:09.770 --> 00:01:14.460 And you need things like atomicity and synchronization 00:01:14.460 --> 00:01:17.770 to be able to make sure that the sharing is done properly 00:01:17.770 --> 00:01:22.230 so that you don't get into data race situations where 00:01:22.230 --> 00:01:24.790 multiple processors try to update the same data element 00:01:24.790 --> 00:01:28.260 and you end up with erroneous results. 00:01:28.260 --> 00:01:30.530 You also heard about distributed memory processors. 00:01:30.530 --> 00:01:35.780 So an example of that might be the Cell, loosely said, where 00:01:35.780 --> 00:01:39.710 you have cores that primarily access their own local memory. 00:01:39.710 --> 00:01:43.940 And while you can have a single global memory address 00:01:43.940 --> 00:01:46.410 space, to get data from memory you essentially have to 00:01:46.410 --> 00:01:49.720 communicate with the different processors to explicitly fetch 00:01:49.720 --> 00:01:51.340 data in and out. 00:01:51.340 --> 00:01:54.640 So things like data distribution, where the data 00:01:54.640 --> 00:01:56.730 is, and what your communication pattern is like 00:01:56.730 --> 00:01:57.980 affect your performance. 00:02:01.270 --> 00:02:05.340 So what I'm going to talk about in today's lecture is 00:02:05.340 --> 00:02:06.940 programming these two different kinds of 00:02:06.940 --> 00:02:09.260 architectures, shared memory processors and distributed 00:02:09.260 --> 00:02:14.570 memory processors, and present you with some concepts for 00:02:14.570 --> 00:02:16.890 commonly programming these machines. 00:02:16.890 --> 00:02:19.620 So in shared memory processors, you have, say, n 00:02:19.620 --> 00:02:21.070 processors, 1 to n. 00:02:21.070 --> 00:02:23.890 And they're connected to a single memory. 00:02:23.890 --> 00:02:27.810 And if one processor asks for the value stored at address X, 00:02:27.810 --> 00:02:29.270 everybody knows where it'll go look. 00:02:29.270 --> 00:02:33.480 Because there's only one address X. And so different 00:02:33.480 --> 00:02:36.000 processors can communicate through shared variables. 00:02:36.000 --> 00:02:38.620 And you need things like locking, as I mentioned, to 00:02:38.620 --> 00:02:43.900 avoid race conditions or erroneous computation. 00:02:43.900 --> 00:02:45.860 So as an example of parallelization, you know, 00:02:45.860 --> 00:02:48.220 straightforward parallelization in a shared 00:02:48.220 --> 00:02:52.000 memory machine, would be you have the simple loop that's 00:02:52.000 --> 00:02:54.610 just running through an array. 00:02:54.610 --> 00:02:58.140 And you're adding elements of array A to elements of array 00:02:58.140 --> 00:03:02.480 B. And you're going to write them to some new array, C. 00:03:02.480 --> 00:03:04.880 Well, if I gave you this loop you can probably recognize 00:03:04.880 --> 00:03:07.370 that there's really no data dependencies here. 00:03:07.370 --> 00:03:09.860 I can split up this loop into three chunks -- let's say I 00:03:09.860 --> 00:03:13.010 have three processors -- where one processor does all the 00:03:13.010 --> 00:03:16.310 computations for iterations zero through three, so the 00:03:16.310 --> 00:03:17.470 first four iterations. 00:03:17.470 --> 00:03:19.530 Second processor does the next four iterations. 00:03:19.530 --> 00:03:21.600 And the third processor does the last four iterations. 00:03:21.600 --> 00:03:26.473 And so that's shown with the little -- should have brought 00:03:26.473 --> 00:03:27.680 a laser pointer. 00:03:27.680 --> 00:03:29.500 So that's showing here. 00:03:29.500 --> 00:03:31.980 And what you might need to do is some mechanism to 00:03:31.980 --> 00:03:34.840 essentially tell the different processors, here's the code 00:03:34.840 --> 00:03:38.110 that you need to run and maybe where to start. 00:03:38.110 --> 00:03:41.070 And then you may need some way of sort of synchronizing these 00:03:41.070 --> 00:03:43.765 different processors that say, I'm done, I can move on to the 00:03:43.765 --> 00:03:46.490 next computation steps. 00:03:46.490 --> 00:03:48.950 So this is an example of a data parallel computation. 00:03:48.950 --> 00:03:51.720 The loop has no real dependencies and, you know, 00:03:51.720 --> 00:03:55.840 each processor can operate on different data sets. 00:03:55.840 --> 00:03:58.540 And what you could do is you can have a process -- 00:03:58.540 --> 00:04:03.360 this is a single application that forks off or creates what 00:04:03.360 --> 00:04:04.840 are commonly called the threads. 00:04:04.840 --> 00:04:08.230 And each thread goes on and executes in this case the same 00:04:08.230 --> 00:04:09.860 computation. 00:04:09.860 --> 00:04:12.970 So a single process can create multiple concurrent threads. 00:04:12.970 --> 00:04:17.290 And really each thread is just a mechanism for encapsulating 00:04:17.290 --> 00:04:20.060 some trace of execution, some execution path. 00:04:20.060 --> 00:04:22.980 So in this case you're essentially encapsulating this 00:04:22.980 --> 00:04:24.860 particular loop here. 00:04:24.860 --> 00:04:28.670 And maybe you parameterize your start index and your 00:04:28.670 --> 00:04:32.990 ending index or maybe your loop bounds. 00:04:32.990 --> 00:04:35.420 And in a shared memory processor, since you're 00:04:35.420 --> 00:04:37.470 communicating -- 00:04:37.470 --> 00:04:40.760 since there's only a single memory, really you don't need 00:04:40.760 --> 00:04:43.900 to do anything special about the data in this particular 00:04:43.900 --> 00:04:46.270 example, because everybody knows where to go look for it. 00:04:46.270 --> 00:04:47.240 Everybody can access it. 00:04:47.240 --> 00:04:48.330 Everything's independent. 00:04:48.330 --> 00:04:52.850 There's no real issues with races or deadlocks. 00:04:52.850 --> 00:04:56.930 So I just wrote down some actual code for that loop that 00:04:56.930 --> 00:04:59.315 parallelize it using Pthreads, a commonly 00:04:59.315 --> 00:05:01.530 used threading mechanism. 00:05:01.530 --> 00:05:04.200 Just to give you a little bit of flavor for, you know, the 00:05:04.200 --> 00:05:07.580 complexity of -- the simple loop that we had expands to a 00:05:07.580 --> 00:05:09.840 lot more code in this case. 00:05:09.840 --> 00:05:11.300 So you have your array. 00:05:11.300 --> 00:05:13.050 It has 12 elements. 00:05:13.050 --> 00:05:16.930 A, B, and C. And you have the basic functions. 00:05:16.930 --> 00:05:18.710 So this is the actual code or computation that we 00:05:18.710 --> 00:05:20.010 want to carry out. 00:05:20.010 --> 00:05:22.190 And what I've done here is I've parameterized where 00:05:22.190 --> 00:05:24.790 you're essentially starting in the array. 00:05:24.790 --> 00:05:26.090 So you get this parameter. 00:05:26.090 --> 00:05:29.210 And then you calculate four iterations' worth. 00:05:29.210 --> 00:05:31.312 And this is essentially the computation that we're 00:05:31.312 --> 00:05:31.870 carrying out. 00:05:31.870 --> 00:05:35.500 And now in my main program or in my main function, rather, 00:05:35.500 --> 00:05:39.320 what I do is I have this concept of threads that I'm 00:05:39.320 --> 00:05:40.140 going to create. 00:05:40.140 --> 00:05:44.140 In this case I'm going to create three of them. 00:05:44.140 --> 00:05:46.040 There are some parameters that I have to pass in, so some 00:05:46.040 --> 00:05:48.670 attributes which are now going to get into here. 00:05:48.670 --> 00:05:50.470 But then I pass in the function pointer. 00:05:50.470 --> 00:05:53.220 This is essentially a mechanism that says once I've 00:05:53.220 --> 00:05:56.030 created this thread, I go to this function and execute this 00:05:56.030 --> 00:05:57.920 particular code. 00:05:57.920 --> 00:05:59.320 And then some arguments that are functions. 00:05:59.320 --> 00:06:01.850 So here I'm just passing in an index at which each loop 00:06:01.850 --> 00:06:03.400 switch starts with. 00:06:03.400 --> 00:06:06.790 And after I've created each thread here, implicitly in the 00:06:06.790 --> 00:06:08.140 thread creation, the code can just 00:06:08.140 --> 00:06:10.210 immediately start running. 00:06:10.210 --> 00:06:12.800 And then once all the threads have started running, I can 00:06:12.800 --> 00:06:15.470 essentially just exit the program 00:06:15.470 --> 00:06:16.720 because I've completed. 00:06:19.070 --> 00:06:22.100 So what I've shown you with that first example was the 00:06:22.100 --> 00:06:24.670 concept of, or example of, data parallelism. 00:06:24.670 --> 00:06:27.790 So you're performing the same computation, but instead of 00:06:27.790 --> 00:06:31.140 operating on one big chunk of data, I've partitioned the 00:06:31.140 --> 00:06:35.020 data into smaller chunks and I've replicated the 00:06:35.020 --> 00:06:38.190 computation so that I can get that kind of parallelism. 00:06:38.190 --> 00:06:40.700 But there's another form of parallelism called control 00:06:40.700 --> 00:06:44.630 parallelism, which essentially uses the same model of 00:06:44.630 --> 00:06:48.330 threading but doesn't necessarily have to run the 00:06:48.330 --> 00:06:51.380 same function or run the same computation each thread. 00:06:51.380 --> 00:06:53.400 So I've sort of illustrated that in the illustration 00:06:53.400 --> 00:06:58.050 there, where these are your data parallel computations and 00:06:58.050 --> 00:07:06.390 these are some other computations in your code. 00:07:06.390 --> 00:07:09.370 So there is sort of a programming model that allows 00:07:09.370 --> 00:07:13.580 you to do this kind of parallelism and tries to sort 00:07:13.580 --> 00:07:17.840 of help the programmer by taking their sequential code 00:07:17.840 --> 00:07:20.770 and then adding annotations that say, this loop is data 00:07:20.770 --> 00:07:27.500 parallel or this set of code is has this kind of control 00:07:27.500 --> 00:07:29.000 parallelism in it. 00:07:29.000 --> 00:07:31.760 So you start with your parallel code. 00:07:31.760 --> 00:07:34.900 This is the same program, multiple data kind of 00:07:34.900 --> 00:07:35.870 parallelization. 00:07:35.870 --> 00:07:38.020 So you might have seen in the previous talk and the previous 00:07:38.020 --> 00:07:40.390 lecture, it was SIMD, single instruction or same 00:07:40.390 --> 00:07:43.130 instruction, multiple data, which allowed you to execute 00:07:43.130 --> 00:07:45.950 the same operation, you know, and add over 00:07:45.950 --> 00:07:47.490 multiple data elements. 00:07:47.490 --> 00:07:50.630 So here it's a similar kind of terminology. 00:07:50.630 --> 00:07:54.510 There's same program, multiple data, and multiple program, 00:07:54.510 --> 00:07:54.920 multiple data. 00:07:54.920 --> 00:07:59.600 This talk is largely focused on the SPMD model, where you 00:07:59.600 --> 00:08:02.860 essentially have one central decision maker or you're 00:08:02.860 --> 00:08:05.670 trying to solve one central computation. 00:08:05.670 --> 00:08:07.190 And you're trying to parallelize that over your 00:08:07.190 --> 00:08:09.470 architecture to get the best performance. 00:08:09.470 --> 00:08:12.990 So you start off with your program and then you annotate 00:08:12.990 --> 00:08:16.200 the code with what's parallel and what's not parallel. 00:08:16.200 --> 00:08:18.990 And you might add in some synchronization directives so 00:08:18.990 --> 00:08:21.450 that if you do in fact have sharing, you might want to use 00:08:21.450 --> 00:08:25.320 the right locking mechanism to guarantee safety. 00:08:25.320 --> 00:08:28.990 Now, in OpenMP, there are some limitations as 00:08:28.990 --> 00:08:29.840 to what it can do. 00:08:29.840 --> 00:08:32.130 So it in fact assumes that the programmer 00:08:32.130 --> 00:08:33.260 knows what he's doing. 00:08:33.260 --> 00:08:35.670 And the programmer is largely responsible for getting the 00:08:35.670 --> 00:08:38.680 synchronization right, or that if they're sharing that they 00:08:38.680 --> 00:08:42.900 get those dependencies protected correctly. 00:08:42.900 --> 00:08:45.850 So you can take your program, insert these annotations, and 00:08:45.850 --> 00:08:49.950 then you go on and test and debug. 00:08:49.950 --> 00:08:54.730 So a simple OpenMP example, again using the simple loop -- 00:08:54.730 --> 00:08:56.930 now, I've thrown away some of the extra code -- you're 00:08:56.930 --> 00:09:00.190 adding these two extra pragmas in this case. 00:09:00.190 --> 00:09:05.370 The first one, your parallel pragma, I call the data 00:09:05.370 --> 00:09:08.450 parallel pragma, really says that you can execute as many 00:09:08.450 --> 00:09:13.120 of the following code block as there are processors or as 00:09:13.120 --> 00:09:15.250 many as you have thread contexts. 00:09:15.250 --> 00:09:18.260 So in this case I implicitly made the assumption that I 00:09:18.260 --> 00:09:20.840 have three processors, so I can automatically partition my 00:09:20.840 --> 00:09:23.060 code into three sets. 00:09:23.060 --> 00:09:25.380 And this transformation can sort of be done automatically 00:09:25.380 --> 00:09:27.810 by the compiler. 00:09:27.810 --> 00:09:31.750 And then there's a for pragma that says this loop is 00:09:31.750 --> 00:09:36.430 parallel and you can divide up the work in the mechanism 00:09:36.430 --> 00:09:37.230 that's work sharing. 00:09:37.230 --> 00:09:39.430 So multiple threads can collaborate to solve the same 00:09:39.430 --> 00:09:42.540 computation, but each one does a smaller amount of work. 00:09:42.540 --> 00:09:49.760 So this is in contrast to what I'm going to focus on a lot 00:09:49.760 --> 00:09:52.190 more in the rest of our talk, which is distributed memory 00:09:52.190 --> 00:09:55.080 processors and programming for distributed memories. 00:09:55.080 --> 00:09:58.760 And this will feel a lot more like programming for the Cell 00:09:58.760 --> 00:10:00.780 as you get more and more involved in that and your 00:10:00.780 --> 00:10:03.930 projects get more intense. 00:10:03.930 --> 00:10:08.220 So in distributed memory processors, to recap the 00:10:08.220 --> 00:10:11.020 previous lectures, you have n processors. 00:10:11.020 --> 00:10:13.070 Each processor has its own memory. 00:10:13.070 --> 00:10:16.180 And they essentially share the interconnection network. 00:10:19.030 --> 00:10:21.670 Each processor has its own address, X. So when a 00:10:21.670 --> 00:10:25.440 processor, P1, asks for X it knows where to go look. 00:10:25.440 --> 00:10:27.810 It's going to look in its own local memory. 00:10:27.810 --> 00:10:31.490 So if all processors are asking for the same value as 00:10:31.490 --> 00:10:33.670 sort of address X, then each one goes and looks in a 00:10:33.670 --> 00:10:35.280 different place. 00:10:35.280 --> 00:10:37.620 So there are n places to look, really. 00:10:37.620 --> 00:10:39.830 And what's stored in those addresses will vary because 00:10:39.830 --> 00:10:42.200 it's everybody's local memory. 00:10:42.200 --> 00:10:46.150 So if one processor, say P1, wants to look at the value 00:10:46.150 --> 00:10:49.480 stored in processor two's address, it actually has to 00:10:49.480 --> 00:10:51.120 explicitly request it. 00:10:51.120 --> 00:10:53.320 The processor two has to send it data. 00:10:53.320 --> 00:10:55.880 And processor one has to figure out, you know, what to 00:10:55.880 --> 00:10:56.720 do with that copy. 00:10:56.720 --> 00:10:57.970 So it has to store it somewhere. 00:11:00.720 --> 00:11:06.400 So this message passing really exposes explicit communication 00:11:06.400 --> 00:11:08.300 to exchange data. 00:11:08.300 --> 00:11:11.620 And you'll see that there are different kinds of data 00:11:11.620 --> 00:11:12.800 communications. 00:11:12.800 --> 00:11:16.240 But really the concept of what you exchange has three 00:11:16.240 --> 00:11:19.030 different -- or four different, rather -- 00:11:19.030 --> 00:11:20.300 things you need to address. 00:11:20.300 --> 00:11:22.760 One is how is the data described and 00:11:22.760 --> 00:11:24.530 what does it describe? 00:11:24.530 --> 00:11:25.970 How are the processes identified? 00:11:25.970 --> 00:11:27.870 So how do I identify that processor one is 00:11:27.870 --> 00:11:28.990 sending me this data? 00:11:28.990 --> 00:11:30.760 And if I'm receiving data how do I know who I'm 00:11:30.760 --> 00:11:32.010 receiving it from? 00:11:34.210 --> 00:11:35.570 Are all messages the same? 00:11:35.570 --> 00:11:38.100 Well, you know, if I send a message to somebody, do I have 00:11:38.100 --> 00:11:40.770 any guarantee that it's received or not? 00:11:40.770 --> 00:11:43.450 And what does it mean for a send operation or a receive 00:11:43.450 --> 00:11:44.720 operation to be completed? 00:11:44.720 --> 00:11:47.580 You know, is there some sort of acknowledgment process? 00:11:50.440 --> 00:11:53.380 So an example of a message passing program -- and if 00:11:53.380 --> 00:11:55.580 you've started to look at the lab you'll see that this is 00:11:55.580 --> 00:11:59.770 essentially where the lab came from. 00:11:59.770 --> 00:12:00.920 It's the same idea. 00:12:00.920 --> 00:12:05.050 I've created -- here I have some two-dimensional space. 00:12:05.050 --> 00:12:07.380 And I have points in this two-dimensional space. 00:12:07.380 --> 00:12:10.920 I have points B, which are these blue circles, and I have 00:12:10.920 --> 00:12:14.160 points A which I've represented as these yellow or 00:12:14.160 --> 00:12:17.040 golden squares. 00:12:17.040 --> 00:12:19.740 And what I want to do is for every point in A I want to 00:12:19.740 --> 00:12:22.820 calculate the distance to all of the points B. So there's 00:12:22.820 --> 00:12:24.190 sort of a pair wise interaction 00:12:24.190 --> 00:12:26.880 between the two arrays. 00:12:26.880 --> 00:12:29.710 So a simple loop that essentially does this -- 00:12:29.710 --> 00:12:34.980 and there are n squared interactions, you have, you 00:12:34.980 --> 00:12:37.410 know, a loop that loops over all the A elements, a loop 00:12:37.410 --> 00:12:39.920 that loops over all the B elements. 00:12:39.920 --> 00:12:42.350 And you essentially calculate in this case Euclidean 00:12:42.350 --> 00:12:43.500 distance which I'm not showing. 00:12:43.500 --> 00:12:44.990 And you store it into some new array. 00:12:47.890 --> 00:12:51.800 So if I give you two processors to do this work, 00:12:51.800 --> 00:12:54.180 processor one and processor two, and I give you some 00:12:54.180 --> 00:12:56.780 mechanism to share between the two -- 00:12:56.780 --> 00:12:58.270 so here's my CPU. 00:12:58.270 --> 00:12:59.940 Each processor has local memory. 00:12:59.940 --> 00:13:01.300 What would be some approach for actually 00:13:01.300 --> 00:13:02.360 parallelizing this? 00:13:02.360 --> 00:13:05.240 Anybody look at the lab yet? 00:13:05.240 --> 00:13:08.860 OK, so what would you do with two processors? 00:13:14.734 --> 00:13:19.630 AUDIENCE: One has half memory [INAUDIBLE] 00:13:19.630 --> 00:13:21.120 PROFESSOR: Right. 00:13:21.120 --> 00:13:24.360 So what was said was that you split one of the arrays in two 00:13:24.360 --> 00:13:26.590 and you can actually get that kind of concurrency. 00:13:26.590 --> 00:13:29.080 So, you know, let's say processor one 00:13:29.080 --> 00:13:30.470 already has the data. 00:13:30.470 --> 00:13:33.320 And it has some place that it's already allocated where 00:13:33.320 --> 00:13:37.270 it's going to write C, the results of the computation, 00:13:37.270 --> 00:13:39.860 then I can break up the work just like it was suggested. 00:13:39.860 --> 00:13:43.550 So what P1 has to do is send data to P2. 00:13:43.550 --> 00:13:44.650 It says here's the data. 00:13:44.650 --> 00:13:46.190 Here's the computation. 00:13:46.190 --> 00:13:48.250 Go ahead and help me out. 00:13:48.250 --> 00:13:53.340 So I send the first array elements, and then I send half 00:13:53.340 --> 00:13:55.580 of the other elements that I want the 00:13:55.580 --> 00:13:57.630 calculations done for. 00:13:57.630 --> 00:14:00.760 And then P1 and P2 can now sort of start 00:14:00.760 --> 00:14:01.670 computing in parallel. 00:14:01.670 --> 00:14:06.650 But notice that P2 has its own array that it's going to store 00:14:06.650 --> 00:14:08.210 results in. 00:14:08.210 --> 00:14:12.210 And so as these compute they actually fill in different 00:14:12.210 --> 00:14:16.570 logical places or logical parts of the overall matrix. 00:14:16.570 --> 00:14:19.720 So what has to be done is at the end for P1 to have all the 00:14:19.720 --> 00:14:23.880 results, P2 has to send it sort of the rest of the matrix 00:14:23.880 --> 00:14:24.400 to complete it. 00:14:24.400 --> 00:14:27.200 And so now P1 has all the results. 00:14:27.200 --> 00:14:29.480 The computation is done and you can move on. 00:14:29.480 --> 00:14:31.690 Does that make sense? 00:14:31.690 --> 00:14:31.870 OK. 00:14:31.870 --> 00:14:36.490 So you'll get to actually do this as part of your labs. 00:14:36.490 --> 00:14:39.600 So in this example messaging program, you have started out 00:14:39.600 --> 00:14:41.170 with a sequential code. 00:14:41.170 --> 00:14:43.330 And we had two processors. 00:14:43.330 --> 00:14:45.570 So processor one actually sends the code. 00:14:45.570 --> 00:14:47.410 So it is essentially a template for the code you'll 00:14:47.410 --> 00:14:49.350 end up writing. 00:14:49.350 --> 00:14:52.330 And it does have to work in the outer loop. 00:14:52.330 --> 00:14:57.480 So this n array over which it is iterating the A array, is 00:14:57.480 --> 00:14:58.840 it's only doing half as many. 00:15:01.910 --> 00:15:06.170 And processor two has to actually receive the data. 00:15:06.170 --> 00:15:09.140 And it specifies where to receive the data into. 00:15:09.140 --> 00:15:13.110 So I've omitted some things, for example, extra information 00:15:13.110 --> 00:15:15.260 sort of hidden in these parameters. 00:15:15.260 --> 00:15:18.650 So here you're sending all of A, all of B. Whereas, you 00:15:18.650 --> 00:15:21.600 know, you could have specified extra parameters that says, 00:15:21.600 --> 00:15:24.740 you know, I'm sending you A. Here's n elements to read from 00:15:24.740 --> 00:15:28.110 A. Here's B. Here's n by two elements to read 00:15:28.110 --> 00:15:31.040 from B. And so on. 00:15:31.040 --> 00:15:33.650 But the computation is essentially the same except 00:15:33.650 --> 00:15:38.140 for the index at which you start, in this case changed 00:15:38.140 --> 00:15:40.370 for processor two. 00:15:40.370 --> 00:15:43.610 And now, when the computation is done, this guy essentially 00:15:43.610 --> 00:15:47.180 waits until the data is received. 00:15:47.180 --> 00:15:49.410 Processor two eventually sends it that data and now 00:15:49.410 --> 00:15:50.050 you can move on. 00:15:50.050 --> 00:15:52.450 AUDIENCE: I have a question. 00:15:52.450 --> 00:15:53.215 PROFESSOR: Yeah? 00:15:53.215 --> 00:15:57.583 AUDIENCE: So would processor two have to wait for the data 00:15:57.583 --> 00:15:58.554 from processor one? 00:15:58.554 --> 00:15:59.790 PROFESSOR: Yeah, so there's a -- 00:15:59.790 --> 00:16:01.760 I'll get into that later. 00:16:01.760 --> 00:16:03.980 So what does it mean to receive? 00:16:06.660 --> 00:16:09.670 To do this computation, I actually need this instruction 00:16:09.670 --> 00:16:10.420 to complete. 00:16:10.420 --> 00:16:13.250 So what does it need for that instruction to complete? 00:16:13.250 --> 00:16:15.370 I do have to get the data because otherwise I don't know 00:16:15.370 --> 00:16:16.360 what to compute on. 00:16:16.360 --> 00:16:19.140 So there is some implicit synchronization 00:16:19.140 --> 00:16:20.100 that you have to do. 00:16:20.100 --> 00:16:21.490 And in some cases it's explicit. 00:16:21.490 --> 00:16:24.770 So I'll get into that a little bit later. 00:16:24.770 --> 00:16:27.400 Does that sort of hint at the answer? 00:16:27.400 --> 00:16:28.140 Are you still confused? 00:16:28.140 --> 00:16:33.040 AUDIENCE: So processor one doesn't do the computation but 00:16:33.040 --> 00:16:34.330 it still sends the data -- 00:16:34.330 --> 00:16:41.470 PROFESSOR: So in terms of tracing, processor one sends 00:16:41.470 --> 00:16:43.940 the data and then can immediately start executing 00:16:43.940 --> 00:16:45.380 its code, right? 00:16:45.380 --> 00:16:48.960 Processor two, in this particular example, has to 00:16:48.960 --> 00:16:50.500 wait until it receives the data. 00:16:50.500 --> 00:16:52.810 So once this receive completes, then you can 00:16:52.810 --> 00:16:54.770 actually go and start executing 00:16:54.770 --> 00:16:56.070 the rest of the code. 00:16:56.070 --> 00:16:58.930 So imagine that it essentially says, wait until I have data. 00:16:58.930 --> 00:16:59.950 Wait until I have something to do. 00:16:59.950 --> 00:17:01.720 Does that help? 00:17:01.720 --> 00:17:04.670 AUDIENCE: Can the main processor 00:17:04.670 --> 00:17:06.080 [UNINTELLIGIBLE PHRASE] 00:17:06.080 --> 00:17:06.920 PROFESSOR: Can the main processor -- 00:17:06.920 --> 00:17:11.196 AUDIENCE: I mean, in Cell, everybody is not peers. 00:17:11.196 --> 00:17:13.360 There is a master there. 00:17:13.360 --> 00:17:17.760 And what master can do instead of doing computation, master 00:17:17.760 --> 00:17:21.713 can be basically the quarterback, sending data, 00:17:21.713 --> 00:17:22.061 receiving data. 00:17:22.061 --> 00:17:25.534 And SPEs can be basically waiting for data, get the 00:17:25.534 --> 00:17:26.750 computation, send it back. 00:17:26.750 --> 00:17:30.728 So in some sense in Cell you probably don't want to do the 00:17:30.728 --> 00:17:32.498 computation on the master. 00:17:32.498 --> 00:17:34.140 Because that means the master slows down. 00:17:34.140 --> 00:17:36.608 The master will do only the data management. 00:17:36.608 --> 00:17:41.730 So that might be one symmetrical [UNINTELLIGIBLE] 00:17:41.730 --> 00:17:43.280 PROFESSOR: And you'll see that in the example. 00:17:43.280 --> 00:17:46.500 Because the PPE in that case has to send the data to two 00:17:46.500 --> 00:17:49.390 different SPEs. 00:17:49.390 --> 00:17:49.580 Yup? 00:17:49.580 --> 00:17:51.136 AUDIENCE: In some sense [UNINTELLIGIBLE PHRASE] 00:17:54.250 --> 00:17:57.005 at points seems to be [UNINTELLIGIBLE] sense that if 00:17:57.005 --> 00:18:00.368 -- so have a huge array and you want to 00:18:00.368 --> 00:18:03.090 [UNINTELLIGIBLE PHRASE] the data to receive the whole 00:18:03.090 --> 00:18:05.940 array, then you have to [UNINTELLIGIBLE] 00:18:05.940 --> 00:18:08.600 PROFESSOR: Yeah, we'll get into that later. 00:18:08.600 --> 00:18:09.840 Yeah, I mean, that's a good point. 00:18:09.840 --> 00:18:11.450 You know, communication is not cheap. 00:18:11.450 --> 00:18:15.140 And if you sort of don't take that into consideration, you 00:18:15.140 --> 00:18:16.920 end up paying a lot for overhead for 00:18:16.920 --> 00:18:17.640 parallelizing things. 00:18:17.640 --> 00:18:20.120 AUDIENCE: [INAUDIBLE] 00:18:20.120 --> 00:18:22.250 PROFESSOR: Well, you can do things in software as well. 00:18:22.250 --> 00:18:24.210 We'll get into that. 00:18:24.210 --> 00:18:27.850 OK, so some crude performance analysis. 00:18:27.850 --> 00:18:29.960 So I have to calculate this distance. 00:18:29.960 --> 00:18:31.610 And given two processors, I can 00:18:31.610 --> 00:18:34.020 effectively get a 2x speedup. 00:18:34.020 --> 00:18:37.620 By dividing up the work I can get done in half the time. 00:18:37.620 --> 00:18:40.250 Well, if you gave me four processors, I can maybe get 00:18:40.250 --> 00:18:44.980 done four times as fast. And in my communication model 00:18:44.980 --> 00:18:48.690 here, I have one copy of one array that's essentially 00:18:48.690 --> 00:18:50.540 sending to every processor. 00:18:50.540 --> 00:18:54.060 And there's some subset of A. So I'm partitioning my other 00:18:54.060 --> 00:18:55.580 array into smaller subsets. 00:18:55.580 --> 00:18:58.320 And I'm sending those to each of the different processors. 00:18:58.320 --> 00:19:01.610 So we'll get into terminology for how to actually name these 00:19:01.610 --> 00:19:03.820 communications later. 00:19:03.820 --> 00:19:05.930 But really the thing to take away here is that this 00:19:05.930 --> 00:19:09.000 granularity -- how I'm partitioning A -- affects my 00:19:09.000 --> 00:19:12.580 performance and communication almost directly. 00:19:12.580 --> 00:19:14.690 And, you know, the comment that was just made is that, 00:19:14.690 --> 00:19:16.090 you know, what do you do about communication? 00:19:16.090 --> 00:19:17.040 It's not free. 00:19:17.040 --> 00:19:19.270 So all of those will be addressed. 00:19:19.270 --> 00:19:22.150 So to understand performance, we sort of summarize three 00:19:22.150 --> 00:19:23.950 main concepts that you essentially need to 00:19:23.950 --> 00:19:24.890 understand. 00:19:24.890 --> 00:19:28.945 One is coverage, or in other words, how much parallelism do 00:19:28.945 --> 00:19:32.880 I actually have in my application? 00:19:32.880 --> 00:19:35.980 And this can actually affect, you know, how much work is it 00:19:35.980 --> 00:19:38.730 worth spending on this particular application? 00:19:38.730 --> 00:19:39.910 Granularity -- 00:19:39.910 --> 00:19:41.860 you know, how do you partition your data among your different 00:19:41.860 --> 00:19:43.980 processors so that you can keep communication down, so 00:19:43.980 --> 00:19:46.970 you can keep synchronization down, and so on. 00:19:46.970 --> 00:19:48.130 Locality -- 00:19:48.130 --> 00:19:50.990 so while not shown in the particular example, if two 00:19:50.990 --> 00:19:54.840 processors are communicating, if they are close in space or 00:19:54.840 --> 00:19:57.570 far in space, or if the communication between two 00:19:57.570 --> 00:20:01.400 processors is far cheaper than two other processors, can I 00:20:01.400 --> 00:20:02.540 exploit that in some way? 00:20:02.540 --> 00:20:07.370 And so we'll talk about that as well. 00:20:07.370 --> 00:20:11.170 So an example of sort of parallelism in an application, 00:20:11.170 --> 00:20:14.030 there are two essentially projects that are doing ray 00:20:14.030 --> 00:20:17.030 tracing, so I thought I'd have this slide here. 00:20:17.030 --> 00:20:20.890 You know, how much parallelism do you have in 00:20:20.890 --> 00:20:23.490 a ray tracing program. 00:20:23.490 --> 00:20:26.170 In ray tracing what you do is you essentially have some 00:20:26.170 --> 00:20:30.080 camera source, some observer. 00:20:30.080 --> 00:20:33.060 And you're trying to figure out, you know, how to color or 00:20:33.060 --> 00:20:35.180 how to shade different pixels in your screen. 00:20:35.180 --> 00:20:38.160 So what you do is you shoot rays from a particular source 00:20:38.160 --> 00:20:38.850 through your plane. 00:20:38.850 --> 00:20:41.270 And then you see how the rays bounce off of other objects. 00:20:41.270 --> 00:20:44.310 And that allows you to render scenes in various ways. 00:20:44.310 --> 00:20:47.580 So you have different kinds of parallelisms. You have your 00:20:47.580 --> 00:20:50.190 primary ray that's shot in. 00:20:50.190 --> 00:20:52.750 And if you're shooting into something like water or some 00:20:52.750 --> 00:20:57.440 very reflective surface, or some surface that can actually 00:20:57.440 --> 00:21:03.410 reflect, transmit, you can essentially end up with a lot 00:21:03.410 --> 00:21:06.980 more rays that are created at run time. 00:21:06.980 --> 00:21:10.040 So there's dynamic parallelism in this particular example. 00:21:10.040 --> 00:21:11.800 And you can shoot a lot of rays from here. 00:21:11.800 --> 00:21:14.190 So there's different kinds of parallelism you can exploit. 00:21:17.500 --> 00:21:21.080 Not all prior programs have this kind of, sort of, a lot 00:21:21.080 --> 00:21:23.450 of parallelism, or embarrassingly parallel 00:21:23.450 --> 00:21:25.180 computation. 00:21:25.180 --> 00:21:27.490 You know, you saw some basic code 00:21:27.490 --> 00:21:29.090 sequences in earlier lectures. 00:21:29.090 --> 00:21:31.020 So there's a sequential part. 00:21:31.020 --> 00:21:32.710 And the reason this is sequential is because there 00:21:32.710 --> 00:21:35.020 are data flow dependencies between each of the different 00:21:35.020 --> 00:21:36.090 computations. 00:21:36.090 --> 00:21:38.600 So here I calculate a, but I need the result of a to do 00:21:38.600 --> 00:21:39.660 this instruction. 00:21:39.660 --> 00:21:43.300 I calculate d here and I need that result to calculate e. 00:21:43.300 --> 00:21:45.722 But then this loop really here is just assigning or it's 00:21:45.722 --> 00:21:46.860 initializing some big array. 00:21:46.860 --> 00:21:49.940 And I can really do that in parallel. 00:21:49.940 --> 00:21:52.520 So I have sequential parts and parallel parts. 00:21:52.520 --> 00:21:56.590 So how does that affect my overall speedups? 00:21:56.590 --> 00:22:00.460 And so there's this law which is really a demonstration of 00:22:00.460 --> 00:22:03.550 diminishing returns, Amdahl's Law. 00:22:03.550 --> 00:22:07.850 It says that if, you know, you have a really fast car, it's 00:22:07.850 --> 00:22:10.310 only as good to you as fast as you can drive it. 00:22:10.310 --> 00:22:12.880 So if there's a lot of congestion on your road or, 00:22:12.880 --> 00:22:15.320 you know, there are posted speed limits or some other 00:22:15.320 --> 00:22:17.700 mechanism, you really can't exploit all the 00:22:17.700 --> 00:22:18.670 speed of your car. 00:22:18.670 --> 00:22:22.440 Or in other words, you're only as fast as the fastest 00:22:22.440 --> 00:22:25.940 mechanisms of the computation that you can have. 00:22:25.940 --> 00:22:31.530 So to look at this in more detail, your potential speedup 00:22:31.530 --> 00:22:35.210 is really proportional to the fraction of the code that can 00:22:35.210 --> 00:22:36.420 be parallelized. 00:22:36.420 --> 00:22:38.730 So if I have some computation -- let's say it has three 00:22:38.730 --> 00:22:42.355 parts: a sequential part that takes 25 seconds, a parallel 00:22:42.355 --> 00:22:45.120 part that takes 50 seconds, and a sequential part that 00:22:45.120 --> 00:22:47.250 runs in 25 seconds. 00:22:47.250 --> 00:22:49.680 So the total execution time is 100 seconds. 00:22:49.680 --> 00:22:53.390 And if I have one processor, that's really all I can do. 00:22:53.390 --> 00:22:55.350 And if she gave me more than one processor -- so let's say 00:22:55.350 --> 00:22:57.010 I have five processors. 00:22:57.010 --> 00:22:59.390 Well, I can't do anything about the sequential work. 00:22:59.390 --> 00:23:01.920 So that's still going to take 25 seconds. 00:23:01.920 --> 00:23:04.120 And I can't do anything about this sequential work either. 00:23:04.120 --> 00:23:05.850 That still takes 25 seconds. 00:23:05.850 --> 00:23:09.080 But this parallel part I can essentially break up among the 00:23:09.080 --> 00:23:10.060 different processors. 00:23:10.060 --> 00:23:11.470 So five in this case. 00:23:11.470 --> 00:23:13.680 And that gets me, you know, five-way parallelism. 00:23:13.680 --> 00:23:16.970 So the 50 seconds now is reduced to 10 seconds. 00:23:16.970 --> 00:23:18.990 Is that clear so far? 00:23:18.990 --> 00:23:22.330 So the overall running time in that case is 60 seconds. 00:23:22.330 --> 00:23:24.690 So what would be my speedup? 00:23:24.690 --> 00:23:29.580 Well, you calculate speedup, old running time divided by 00:23:29.580 --> 00:23:30.790 the new running time. 00:23:30.790 --> 00:23:32.820 So 100 seconds divided by 60 seconds. 00:23:32.820 --> 00:23:37.020 Or my parallel version is 1.67 times faster. 00:23:37.020 --> 00:23:38.660 So this is great. 00:23:38.660 --> 00:23:40.690 If I increase the number of processors, then I should be 00:23:40.690 --> 00:23:41.980 able to get more and more parallelism. 00:23:41.980 --> 00:23:47.530 But it also means that there's sort of an upper bound on how 00:23:47.530 --> 00:23:49.160 much speedup you can get. 00:23:49.160 --> 00:23:52.450 So if you look at the fraction of work in your application 00:23:52.450 --> 00:23:54.840 that's parallel, that's p. 00:23:54.840 --> 00:23:58.530 And your number of processors, well, your speedup is -- 00:23:58.530 --> 00:24:01.270 let's say the old running time is just one unit of work. 00:24:01.270 --> 00:24:04.340 If the time it takes for the sequential work -- so that's 1 00:24:04.340 --> 00:24:08.720 minus p, since p is the fraction of the parallel work. 00:24:08.720 --> 00:24:11.120 And it's the time to do the parallel work. 00:24:11.120 --> 00:24:14.480 And since I can parallelize that fraction over n 00:24:14.480 --> 00:24:18.500 processors, I can sort of reduce that to really small 00:24:18.500 --> 00:24:19.640 amounts in the limit. 00:24:19.640 --> 00:24:20.890 Does that make sense so far? 00:24:23.290 --> 00:24:26.260 So the speedup can tend to 1 over 1 minus p in the limit. 00:24:26.260 --> 00:24:29.560 If I increase the number of processors or that gets really 00:24:29.560 --> 00:24:33.820 large, that's essentially my upper bound on how fast 00:24:33.820 --> 00:24:36.110 programs can work. 00:24:36.110 --> 00:24:39.550 You know, how much can I exploit out of my program? 00:24:39.550 --> 00:24:40.370 So this is great. 00:24:40.370 --> 00:24:43.540 What this law says -- the implication here is if your 00:24:43.540 --> 00:24:45.790 program has a lot of inherent parallelism, you 00:24:45.790 --> 00:24:47.010 can do really well. 00:24:47.010 --> 00:24:49.090 But if your program doesn't have any parallelism, well, 00:24:49.090 --> 00:24:50.450 there's really nothing you can do. 00:24:50.450 --> 00:24:52.680 So parallel architectures won't really help you. 00:24:52.680 --> 00:24:54.660 And there's some interesting trade-offs, for example, that 00:24:54.660 --> 00:24:58.850 you might consider if you're designing a chip or if you're 00:24:58.850 --> 00:25:01.670 looking at an application or domain of applications, 00:25:01.670 --> 00:25:06.080 figuring out what is the best architecture to run them on. 00:25:06.080 --> 00:25:10.510 So in terms of performance scalability, as I increase the 00:25:10.510 --> 00:25:12.890 number of processors, I have speedup. 00:25:12.890 --> 00:25:15.230 You can define, sort of, an efficiency to 00:25:15.230 --> 00:25:16.570 be linear at 100%. 00:25:16.570 --> 00:25:20.590 But typically you end up in sort of the sublinear domain. 00:25:20.590 --> 00:25:25.800 That's because communication is not often free. 00:25:25.800 --> 00:25:28.350 But you can get super linear speedups ups on real 00:25:28.350 --> 00:25:31.450 architectures because of secondary and tertiary effects 00:25:31.450 --> 00:25:34.770 that come from register allocation or caching effects. 00:25:34.770 --> 00:25:38.120 So they can hide a lot of latency or you can take 00:25:38.120 --> 00:25:41.090 advantage of a lot of pipelining mechanisms in the 00:25:41.090 --> 00:25:43.330 architecture to get super linear speedups. 00:25:43.330 --> 00:25:46.620 So you can end up in two different domains. 00:25:49.860 --> 00:25:55.000 So a small, you know, overview of the extent of parallelism 00:25:55.000 --> 00:25:56.440 in your program and how that affects 00:25:56.440 --> 00:25:59.280 your overall execution. 00:25:59.280 --> 00:26:01.620 And the other concept is granularity. 00:26:01.620 --> 00:26:03.520 So given that I have this much parallelism, how 00:26:03.520 --> 00:26:04.590 do I exploit it? 00:26:04.590 --> 00:26:06.810 There are different ways of exploiting it, and that comes 00:26:06.810 --> 00:26:09.440 down to, well, how do I subdivide my problem? 00:26:09.440 --> 00:26:12.410 What is the granularity of the sub-problems I'm going to 00:26:12.410 --> 00:26:15.320 calculate on? 00:26:15.320 --> 00:26:19.130 And, really, granularity from my perspective, is just a 00:26:19.130 --> 00:26:22.760 qualitative measure of what is the ratio of your computation 00:26:22.760 --> 00:26:24.950 to your communication? 00:26:24.950 --> 00:26:27.130 So if you're doing a lot of computation, very little 00:26:27.130 --> 00:26:29.400 communication, you could be doing really 00:26:29.400 --> 00:26:30.580 well or vice versa. 00:26:30.580 --> 00:26:33.240 Then you could be computation limited, and so you need a lot 00:26:33.240 --> 00:26:35.450 of bandwidth for example in your architecture. 00:26:35.450 --> 00:26:37.258 AUDIENCE: Like before, you really didn't have to give 00:26:37.258 --> 00:26:40.880 every single processor an entire copy of B. 00:26:40.880 --> 00:26:41.850 PROFESSOR: Right. 00:26:41.850 --> 00:26:42.010 Yeah. 00:26:42.010 --> 00:26:45.070 Good point. 00:26:48.190 --> 00:26:50.960 And as you saw in the previous slides, you have -- 00:26:50.960 --> 00:26:53.380 computation stages are separated by 00:26:53.380 --> 00:26:54.790 communication stages. 00:26:54.790 --> 00:26:57.070 And your communication in a lot of cases essentially 00:26:57.070 --> 00:26:59.060 serves as synchronization. 00:26:59.060 --> 00:27:01.560 I need everybody to get to the same point before I can move 00:27:01.560 --> 00:27:04.780 on logically in my computation. 00:27:04.780 --> 00:27:07.830 So there are two kinds of sort of classes of granularity. 00:27:07.830 --> 00:27:10.840 There's fine grain and, as you'll see, coarse grain. 00:27:10.840 --> 00:27:14.010 So in fine-grain parallelism, you have low computation to 00:27:14.010 --> 00:27:15.460 communication ratio. 00:27:15.460 --> 00:27:18.150 And that has good properties in that you have a small 00:27:18.150 --> 00:27:21.010 amount of work done between communication stages. 00:27:24.410 --> 00:27:27.910 And it has bad properties in that it gives you less 00:27:27.910 --> 00:27:29.530 performance opportunity. 00:27:32.190 --> 00:27:33.850 It should be more, right? 00:27:33.850 --> 00:27:35.100 More opportunity for -- 00:27:35.100 --> 00:27:35.600 AUDIENCE: No. 00:27:35.600 --> 00:27:36.100 Less. 00:27:36.100 --> 00:27:36.480 PROFESSOR: Sorry. 00:27:36.480 --> 00:27:37.120 Yeah, yeah, sorry. 00:27:37.120 --> 00:27:38.370 I didn't get enough sleep. 00:27:41.670 --> 00:27:44.470 So less opportunities for performance enhancement, but 00:27:44.470 --> 00:27:48.870 you have high communication ratio because essentially 00:27:48.870 --> 00:27:50.800 you're communicating very often. 00:27:50.800 --> 00:27:55.120 So these are the computations here and these yellow bars are 00:27:55.120 --> 00:27:56.850 the synchronization points. 00:27:56.850 --> 00:27:59.640 So I have to distribute data or communicate. 00:27:59.640 --> 00:28:02.000 I do computations but, you know, computation 00:28:02.000 --> 00:28:02.990 doesn't last very long. 00:28:02.990 --> 00:28:06.410 And I do more communication or more synchronization, and I 00:28:06.410 --> 00:28:07.750 repeat the process. 00:28:07.750 --> 00:28:11.370 So naturally you can adjust this granularity to sort of 00:28:11.370 --> 00:28:12.530 reduce the communication overhead. 00:28:12.530 --> 00:28:14.510 AUDIENCE: [UNINTELLIGIBLE PHRASE] 00:28:14.510 --> 00:28:17.010 two things in that overhead part. 00:28:17.010 --> 00:28:18.120 One is the volume. 00:28:18.120 --> 00:28:21.380 So one, communication. 00:28:21.380 --> 00:28:23.484 Also there's a large part of synchronization cost. 00:28:23.484 --> 00:28:26.210 Basically you get a communication goal and you 00:28:26.210 --> 00:28:29.022 have to go start the messages and wait until 00:28:29.022 --> 00:28:29.210 everybody is done. 00:28:29.210 --> 00:28:32.250 So that overhead also can go. 00:28:32.250 --> 00:28:35.642 Even if you don't send that much data, just the fact that 00:28:35.642 --> 00:28:38.260 you are communicating, that means you have to do a lot of 00:28:38.260 --> 00:28:43.076 this additional bookkeeping stuff, that especially in the 00:28:43.076 --> 00:28:43.679 distributed [? memory ?] 00:28:43.679 --> 00:28:44.954 [? machine is ?] pretty expensive. 00:28:44.954 --> 00:28:45.910 PROFESSOR: Yeah. 00:28:45.910 --> 00:28:48.780 Thanks. 00:28:48.780 --> 00:28:52.490 So in coarse-grain parallelism, you sort of make 00:28:52.490 --> 00:28:55.070 the work chunks more and more so that you do the 00:28:55.070 --> 00:28:57.440 communication synchronization less and less. 00:28:57.440 --> 00:28:58.440 And so that's shown here. 00:28:58.440 --> 00:29:00.750 You do longer pieces of work and have fewer 00:29:00.750 --> 00:29:04.370 synchronization stages. 00:29:04.370 --> 00:29:09.690 So in that regime, you can have more opportunities for 00:29:09.690 --> 00:29:12.590 performance improvements, but the tricky thing that you get 00:29:12.590 --> 00:29:15.730 into is what's called load balancing. 00:29:15.730 --> 00:29:19.090 So if each of these different computations takes differing 00:29:19.090 --> 00:29:22.130 amounts of time to complete, then what you might end up 00:29:22.130 --> 00:29:25.380 doing is a lot of people might end up idle as they wait until 00:29:25.380 --> 00:29:27.690 everybody's essentially reached their finish line. 00:29:27.690 --> 00:29:27.960 Yep? 00:29:27.960 --> 00:29:30.000 AUDIENCE: If you don't have to acknowledge that something's 00:29:30.000 --> 00:29:31.593 done can't you just say, [? 00:29:31.593 --> 00:29:32.040 OK, I'm done with your salt ?]. 00:29:32.040 --> 00:29:33.570 Hand it to the initial processor 00:29:33.570 --> 00:29:35.100 and keep doing whatever? 00:29:35.100 --> 00:29:37.470 PROFESSOR: So, you can do that in cases where that 00:29:37.470 --> 00:29:40.430 essentially there is a mechanism -- or the 00:29:40.430 --> 00:29:42.340 application allows for it. 00:29:42.340 --> 00:29:44.473 But as I'll show -- well, you won't see until the next 00:29:44.473 --> 00:29:46.340 lecture -- there are dependencies, for example, 00:29:46.340 --> 00:29:48.420 that might preclude you from doing that. 00:29:48.420 --> 00:29:51.170 If everybody needs to reach the same point because you're 00:29:51.170 --> 00:29:53.550 updating a large data structure before you can go 00:29:53.550 --> 00:29:55.290 on, then you might not be able to do that. 00:29:55.290 --> 00:29:58.810 So think of doing molecular dynamics simulations. 00:29:58.810 --> 00:30:00.970 You need everybody to calculate a new position 00:30:00.970 --> 00:30:02.895 before you can go on and calculate new kinds of coarse 00:30:02.895 --> 00:30:03.030 interactions. 00:30:03.030 --> 00:30:04.700 AUDIENCE: [UNINTELLIGIBLE] nothing else to calculate yet. 00:30:04.700 --> 00:30:05.700 PROFESSOR: Right. 00:30:05.700 --> 00:30:06.410 AUDIENCE: But also there is pipelining. 00:30:06.410 --> 00:30:08.400 So what do you talk about [UNINTELLIGIBLE] because you 00:30:08.400 --> 00:30:10.785 might want to get the next data while you're computing 00:30:10.785 --> 00:30:13.830 now so that when I'm done I can start sending. 00:30:13.830 --> 00:30:16.660 [UNINTELLIGIBLE PHRASE] you can all have some of that. 00:30:16.660 --> 00:30:17.970 PROFESSOR: Yep. 00:30:17.970 --> 00:30:21.935 Yeah, because communication is such an intensive part, there 00:30:21.935 --> 00:30:22.920 are different ways of dealing with it. 00:30:22.920 --> 00:30:26.000 And that will be right after load balancing. 00:30:26.000 --> 00:30:28.880 So the load balancing problem is just an illustration. 00:30:28.880 --> 00:30:33.610 And things that appear in sort of this lightish pink will 00:30:33.610 --> 00:30:34.900 serve as sort of visual cues. 00:30:34.900 --> 00:30:36.985 This is the same color coding scheme that David's using in 00:30:36.985 --> 00:30:37.780 the recitations. 00:30:37.780 --> 00:30:39.620 So this is PPU code. 00:30:39.620 --> 00:30:42.180 Things that appear in yellow will be SPU code. 00:30:42.180 --> 00:30:44.490 And these are just meant to essentially show you how you 00:30:44.490 --> 00:30:47.780 might do things like this on Cell, just to help you along 00:30:47.780 --> 00:30:50.930 in picking up more of the syntax and functionality you 00:30:50.930 --> 00:30:53.190 need for your programs. 00:30:53.190 --> 00:30:58.090 So in the load balancing problem, you essentially have, 00:30:58.090 --> 00:31:03.930 let's say, three different threads of computation. 00:31:03.930 --> 00:31:06.880 And so that's shown here: red, blue, and orange. 00:31:06.880 --> 00:31:08.650 And you've reached some communication stage. 00:31:08.650 --> 00:31:13.560 So the PPU program in this case is saying, send a message 00:31:13.560 --> 00:31:16.340 to each of my SPEs, to each of my different processors, that 00:31:16.340 --> 00:31:18.210 you're ready to start. 00:31:18.210 --> 00:31:21.150 And so now once every processor gets that message, 00:31:21.150 --> 00:31:22.370 they can start computing. 00:31:22.370 --> 00:31:25.970 And let's assume they have data and so on ready to go. 00:31:25.970 --> 00:31:29.880 And what's going to happen is each processor is going to run 00:31:29.880 --> 00:31:31.710 through the computation at different rates. 00:31:31.710 --> 00:31:33.980 Now this could be because one processor 00:31:33.980 --> 00:31:35.100 is faster than another. 00:31:35.100 --> 00:31:36.960 Or it could be because one processor is 00:31:36.960 --> 00:31:37.930 more loaded than another. 00:31:37.930 --> 00:31:40.550 Or it could be just because each processor is assigned 00:31:40.550 --> 00:31:43.090 sort of differing amounts of work. 00:31:43.090 --> 00:31:44.090 So one has a short loop. 00:31:44.090 --> 00:31:46.240 One has a longer loop. 00:31:46.240 --> 00:31:49.400 And so as the animation shows, sort of, execution proceeds 00:31:49.400 --> 00:31:52.930 and everybody's waiting until the orange guy has completed. 00:31:52.930 --> 00:31:56.200 But nobody could have made progress until everybody's 00:31:56.200 --> 00:32:00.130 reached synchronization point because, you know, there's a 00:32:00.130 --> 00:32:03.160 strict dependence that's being enforced here that says, I'm 00:32:03.160 --> 00:32:04.820 going to wait until everybody's told me they're 00:32:04.820 --> 00:32:08.130 done before I go on to the next step of computation. 00:32:08.130 --> 00:32:10.500 And so you know, in Cell you do that using 00:32:10.500 --> 00:32:12.360 mailboxes in this case. 00:32:12.360 --> 00:32:13.610 That clear so far? 00:32:16.380 --> 00:32:19.160 So how do you get around this load balancing problem? 00:32:19.160 --> 00:32:20.800 Well, there are two different ways. 00:32:20.800 --> 00:32:22.430 There's static load balancing. 00:32:22.430 --> 00:32:24.540 I know my application really, really well. 00:32:24.540 --> 00:32:27.280 And I understand sort of different computations. 00:32:27.280 --> 00:32:29.920 So what I can do is I can divide up the work and have a 00:32:29.920 --> 00:32:33.320 static mapping of the work to my processors. 00:32:33.320 --> 00:32:36.890 And static mapping just means, you know, in this particular 00:32:36.890 --> 00:32:40.680 example, that I'm going to assign the work to different 00:32:40.680 --> 00:32:43.820 processors and that's what the processors will do. 00:32:43.820 --> 00:32:47.410 Work can't shift around between processors. 00:32:47.410 --> 00:32:49.330 And so in this case I have a work queue. 00:32:49.330 --> 00:32:51.750 Each of those bars is some computation. 00:32:51.750 --> 00:32:55.590 You know, I can assign some chunk to P1, processor one, 00:32:55.590 --> 00:32:57.400 some chunk to processor two. 00:32:57.400 --> 00:32:59.060 And then computation can go on. 00:32:59.060 --> 00:33:03.320 Those allocations don't change. 00:33:03.320 --> 00:33:05.430 So this works well if I understand the application, 00:33:05.430 --> 00:33:08.760 well and I know the computation, and my cores are 00:33:08.760 --> 00:33:11.780 relatively homogeneous and, you know, there's not a lot of 00:33:11.780 --> 00:33:13.540 contention for them. 00:33:13.540 --> 00:33:16.690 So if all the cores are the same, each core has an equal 00:33:16.690 --> 00:33:18.710 amount of work -- the total amount of work -- this works 00:33:18.710 --> 00:33:21.470 really well because nobody is sitting too idle. 00:33:21.470 --> 00:33:24.140 It doesn't work so well for heterogeneous architectures or 00:33:24.140 --> 00:33:24.590 multicores. 00:33:24.590 --> 00:33:26.690 Because one might be faster than the other. 00:33:26.690 --> 00:33:29.470 It increases the complexity of the allocation I need to do. 00:33:29.470 --> 00:33:33.170 If there's a lot of contention for some resources, then that 00:33:33.170 --> 00:33:36.470 can affect the static load balancing. 00:33:36.470 --> 00:33:41.180 So work distribution might end up being uneven. 00:33:41.180 --> 00:33:43.740 So the alternative is dynamic load balancing. 00:33:43.740 --> 00:33:46.040 And you certainly could do sort of a hybrid load 00:33:46.040 --> 00:33:48.860 balancing, static plus dynamic mechanism. 00:33:48.860 --> 00:33:51.360 Although I don't have that in the slides. 00:33:51.360 --> 00:33:57.020 So in the dynamic load balancing scheme, two 00:33:57.020 --> 00:33:59.860 different mechanisms I'm going to illustrate. 00:33:59.860 --> 00:34:01.620 So in the first scheme, you start with something like the 00:34:01.620 --> 00:34:03.610 static mechanism. 00:34:03.610 --> 00:34:07.090 So I have some work going to processor one. 00:34:07.090 --> 00:34:10.080 And I have some work going to processor two. 00:34:10.080 --> 00:34:14.840 But then as processor two executes and completes faster 00:34:14.840 --> 00:34:17.610 than processor one, it takes on some of the additional work 00:34:17.610 --> 00:34:18.520 from processor one. 00:34:18.520 --> 00:34:20.650 So the work that was here is now shifted. 00:34:20.650 --> 00:34:24.160 And so you can keep helping out, you know, your other 00:34:24.160 --> 00:34:27.040 processors to compute things faster. 00:34:27.040 --> 00:34:29.630 In the other scheme, you have a work queue where you 00:34:29.630 --> 00:34:32.780 essentially are distributing work on the fly. 00:34:32.780 --> 00:34:35.740 So as things complete, you're just sending them 00:34:35.740 --> 00:34:37.240 more work to do. 00:34:37.240 --> 00:34:39.610 So in this animation here, I start off. 00:34:39.610 --> 00:34:41.820 I send work to two different processors. 00:34:41.820 --> 00:34:44.780 P2 is really fast so it's just zipping through things. 00:34:44.780 --> 00:34:48.360 And then P1 eventually finishes and new work is 00:34:48.360 --> 00:34:50.570 allocated to the two different schemes. 00:34:50.570 --> 00:34:54.120 So dynamic load balancing is intended to sort of give equal 00:34:54.120 --> 00:34:57.250 amounts of work in a different scheme for processors. 00:34:57.250 --> 00:34:59.880 So it really increased utilization and spent less and 00:34:59.880 --> 00:35:03.310 less time being idle. 00:35:03.310 --> 00:35:03.500 OK. 00:35:03.500 --> 00:35:08.480 So load balancing was one part of sort of how granularity can 00:35:08.480 --> 00:35:10.270 have a performance trade-off. 00:35:10.270 --> 00:35:11.660 The other is synchronization. 00:35:11.660 --> 00:35:14.520 So there were already some good questions as to, well, 00:35:14.520 --> 00:35:16.350 you know, how does this play into overall execution? 00:35:16.350 --> 00:35:17.000 When can I wait? 00:35:17.000 --> 00:35:18.810 When can't I wait? 00:35:18.810 --> 00:35:21.930 So I'm going to illustrate it with just a simple data 00:35:21.930 --> 00:35:22.960 dependence graph. 00:35:22.960 --> 00:35:25.600 Although you can imagine that in each one of these circles 00:35:25.600 --> 00:35:27.550 there's some really heavy load computation. 00:35:27.550 --> 00:35:29.340 And you'll see that in the next lecture, in fact. 00:35:29.340 --> 00:35:32.540 So if I have some simple computation here -- 00:35:32.540 --> 00:35:33.930 I have some operands. 00:35:33.930 --> 00:35:35.800 I'm doing an addition. 00:35:35.800 --> 00:35:36.910 Here I do another addition. 00:35:36.910 --> 00:35:38.690 I need both of these results before I can do this 00:35:38.690 --> 00:35:40.140 multiplication. 00:35:40.140 --> 00:35:43.070 Here I have, you know, some loop that's adding through 00:35:43.070 --> 00:35:44.200 some array elements. 00:35:44.200 --> 00:35:46.870 I need all those results before I do final substraction 00:35:46.870 --> 00:35:49.580 and produce my final result. 00:35:49.580 --> 00:35:52.330 So what are some synchronization points here? 00:35:52.330 --> 00:35:55.630 Well, it really depends on how I allocate the different 00:35:55.630 --> 00:35:59.090 instructions to processors. 00:35:59.090 --> 00:36:02.580 So if I have an allocation that just says, well, let's 00:36:02.580 --> 00:36:06.240 put all these chains on one processor, put these two 00:36:06.240 --> 00:36:08.470 chains on two different processors, well, where are my 00:36:08.470 --> 00:36:09.890 synchronization points? 00:36:09.890 --> 00:36:13.120 Well, it depends on where this guy is and where this guy is. 00:36:13.120 --> 00:36:15.880 Because for this instruction to execute, it needs to 00:36:15.880 --> 00:36:17.910 receive data from P1 and P2. 00:36:17.910 --> 00:36:23.260 So if P1 and P2 are different from what's in that box, 00:36:23.260 --> 00:36:24.100 somebody has to wait. 00:36:24.100 --> 00:36:24.460 And so there's a 00:36:24.460 --> 00:36:26.550 synchronization that has to happen. 00:36:32.200 --> 00:36:34.520 So essentially at all join points there's potential for 00:36:34.520 --> 00:36:35.510 synchronization. 00:36:35.510 --> 00:36:37.810 But I can adjust the granularity so that I can 00:36:37.810 --> 00:36:40.330 remove more and more synchronization points. 00:36:46.920 --> 00:36:50.950 So if I had assigned all this entire sub-graph to the same 00:36:50.950 --> 00:36:53.580 processor, I really get rid of the synchronization because it 00:36:53.580 --> 00:36:56.910 is essentially local to that particular processor. 00:36:56.910 --> 00:36:58.820 And there's no extra messaging that would have to happen 00:36:58.820 --> 00:37:01.415 across processors that says, I'm ready, or I'm ready to 00:37:01.415 --> 00:37:04.330 send you data, or you can move on to the next step. 00:37:04.330 --> 00:37:06.650 And so in this case the last synchronization point would be 00:37:06.650 --> 00:37:07.890 at this join point. 00:37:07.890 --> 00:37:12.470 Let's say if it's allocated on P1 or on some other processor. 00:37:12.470 --> 00:37:14.420 So how would I get rid of this synchronization point? 00:37:14.420 --> 00:37:19.080 AUDIENCE: Do the whole thing. 00:37:19.080 --> 00:37:19.390 PROFESSOR: Right. 00:37:19.390 --> 00:37:22.160 You put the entire thing on a single processor. 00:37:22.160 --> 00:37:23.940 But you get no parallelism in this case. 00:37:23.940 --> 00:37:26.360 So the coarse-grain, fine-grain grain parallelism 00:37:26.360 --> 00:37:30.410 granularity issue comes to play. 00:37:30.410 --> 00:37:33.410 So the last sort of thing I'm going to talk about in terms 00:37:33.410 --> 00:37:37.440 of how granularity impacts performance -- and this was 00:37:37.440 --> 00:37:39.570 already touched on -- is that communication is really not 00:37:39.570 --> 00:37:42.760 cheap and can be quite overwhelming on a lot of 00:37:42.760 --> 00:37:43.820 architectures. 00:37:43.820 --> 00:37:46.600 And what's interesting about multicores is that they're 00:37:46.600 --> 00:37:48.840 essentially putting a lot more resources closer 00:37:48.840 --> 00:37:50.470 together on a chip. 00:37:50.470 --> 00:37:55.010 So it essentially is changing the factors for communication. 00:37:55.010 --> 00:37:57.910 So rather than having, you know, your parallel cluster 00:37:57.910 --> 00:38:00.700 now which is connected, say, by ethernet or some other 00:38:00.700 --> 00:38:03.690 high-speed link, now you essentially have large 00:38:03.690 --> 00:38:05.920 clusters or will have large clusters on a chip. 00:38:05.920 --> 00:38:09.250 So communication factors really change. 00:38:09.250 --> 00:38:15.310 But the cost model is relatively captured by these 00:38:15.310 --> 00:38:16.560 different parameters. 00:38:19.730 --> 00:38:22.450 So what is the cost of my communication? 00:38:22.450 --> 00:38:26.915 Well, it's equal to, well, how many messages am I sending and 00:38:26.915 --> 00:38:30.130 what is the frequency with which I'm sending them? 00:38:30.130 --> 00:38:32.130 There's some overhead for message. 00:38:32.130 --> 00:38:34.140 So I have to actually package data together. 00:38:34.140 --> 00:38:37.920 I have to stick in a control header and then send it out. 00:38:37.920 --> 00:38:40.360 So that takes me some work on the receiver side. 00:38:40.360 --> 00:38:41.450 I have to take the message. 00:38:41.450 --> 00:38:46.390 I maybe have to decode the header, figure out where to 00:38:46.390 --> 00:38:48.470 store the data that's coming in on the message. 00:38:48.470 --> 00:38:51.700 So there's some overhead associated with that as well. 00:38:51.700 --> 00:38:55.420 There's a network delay for sending a message, so putting 00:38:55.420 --> 00:38:58.375 a message on the network so that it can be transmitted, or 00:38:58.375 --> 00:38:59.880 picking things up off the network. 00:38:59.880 --> 00:39:04.050 So there's a latency also associated with how long does 00:39:04.050 --> 00:39:07.765 it take for a message to get from point A to point B. What 00:39:07.765 --> 00:39:11.020 is the bandwidth that I have across a link? 00:39:11.020 --> 00:39:13.350 So if I have a lot of bandwidth then that can really 00:39:13.350 --> 00:39:16.670 lower my communication cost. But if I have little bandwidth 00:39:16.670 --> 00:39:19.450 then that can really create contention. 00:39:19.450 --> 00:39:20.940 How much data am I sending? 00:39:20.940 --> 00:39:22.780 And, you know, number of messages. 00:39:22.780 --> 00:39:25.730 So this numerator here is really an average of the data 00:39:25.730 --> 00:39:29.590 that you're sending per communication. 00:39:29.590 --> 00:39:32.100 There's a cost induced per contention. 00:39:32.100 --> 00:39:33.880 And then finally there's -- so all of 00:39:33.880 --> 00:39:35.580 these are added factors. 00:39:35.580 --> 00:39:37.830 The higher they are, except for bandwidth, because it's in 00:39:37.830 --> 00:39:39.020 the denominator here, the worse your 00:39:39.020 --> 00:39:40.800 communication cost becomes. 00:39:40.800 --> 00:39:45.010 So you can try to reduce the communication cost by 00:39:45.010 --> 00:39:46.220 communicating less. 00:39:46.220 --> 00:39:47.100 So you adjust your granularity. 00:39:47.100 --> 00:39:50.460 And that can impact your synchronization or what kind 00:39:50.460 --> 00:39:52.960 of data you're shipping around. 00:39:52.960 --> 00:39:55.450 You can do some architectural tweaks or maybe some software 00:39:55.450 --> 00:39:58.800 tweaks to really get the network latency down and the 00:39:58.800 --> 00:40:00.190 overhead per message down. 00:40:00.190 --> 00:40:03.880 So on something like raw architecture, which we saw in 00:40:03.880 --> 00:40:06.230 Saman's lecture, there's a really fast mechanism to 00:40:06.230 --> 00:40:08.900 communicate your nearest neighbor in three cycles. 00:40:08.900 --> 00:40:12.580 So one processor can send a single operand to another 00:40:12.580 --> 00:40:16.310 reasonably fast. You know, you can improve the bandwidth 00:40:16.310 --> 00:40:19.010 again in architectural mechanism. 00:40:19.010 --> 00:40:21.720 You can do some tricks as to how you package your data in 00:40:21.720 --> 00:40:24.490 each message. 00:40:24.490 --> 00:40:27.380 And lastly, what I'm going to talk about in a couple of 00:40:27.380 --> 00:40:30.320 slides is, well, I can also improve it using some 00:40:30.320 --> 00:40:31.965 mechanisms that try to increase the 00:40:31.965 --> 00:40:33.390 overlap between messages. 00:40:33.390 --> 00:40:35.160 And what does this really mean? 00:40:35.160 --> 00:40:37.070 What am I overlapping it with? 00:40:37.070 --> 00:40:40.100 And it's really the communication and computation 00:40:40.100 --> 00:40:43.590 stages are going to somehow get aligned. 00:40:43.590 --> 00:40:46.300 So before I actually show you that, I just want to point out 00:40:46.300 --> 00:40:48.220 that there are two kinds of messages. 00:40:48.220 --> 00:40:51.020 There's data messages, and these are, for example, the 00:40:51.020 --> 00:40:54.240 arrays that I'm sending around to different processors for 00:40:54.240 --> 00:40:57.760 the distance calculations between points in space. 00:40:57.760 --> 00:40:59.650 But there are also control messages. 00:40:59.650 --> 00:41:02.460 So control messages essentially say, I'm done, or 00:41:02.460 --> 00:41:06.900 I'm ready to go, or is there any work for me to do? 00:41:06.900 --> 00:41:09.700 So on Cell, control messages, you know, you can think of 00:41:09.700 --> 00:41:12.560 using Mailboxes for those and the DMAs for doing the data 00:41:12.560 --> 00:41:13.590 communication. 00:41:13.590 --> 00:41:16.170 So data messages are relatively much larger -- 00:41:16.170 --> 00:41:19.150 you're sending a lot of data -- versus control messages 00:41:19.150 --> 00:41:22.190 that are really much shorter, just essentially just sending 00:41:22.190 --> 00:41:23.930 you very brief information. 00:41:27.640 --> 00:41:30.980 So in order to get that overlap, what you can do is 00:41:30.980 --> 00:41:33.610 essentially use this concept of pipelining. 00:41:33.610 --> 00:41:35.250 So you've seen pipelining in superscalar. 00:41:35.250 --> 00:41:37.620 Someone talked about that. 00:41:37.620 --> 00:41:40.130 And what you are essentially trying to do is break up the 00:41:40.130 --> 00:41:43.950 communication and computation into different stages and then 00:41:43.950 --> 00:41:45.860 figure out a way to overlap them so that you can 00:41:45.860 --> 00:41:47.970 essentially hide the latency for the 00:41:47.970 --> 00:41:51.090 sends and the receives. 00:41:51.090 --> 00:41:54.830 So let's say you have some work that you're doing, and it 00:41:54.830 --> 00:41:57.570 really requires you to send the data -- 00:41:57.570 --> 00:42:00.200 somebody has to send you the data or you essentially have 00:42:00.200 --> 00:42:02.440 to wait until you get it. 00:42:02.440 --> 00:42:04.850 And then after you've waited and the data is there, you can 00:42:04.850 --> 00:42:06.470 actually go on and do your work. 00:42:06.470 --> 00:42:07.670 So these are color coded. 00:42:07.670 --> 00:42:11.540 So this is essentially one iteration of the work. 00:42:11.540 --> 00:42:15.220 And so you could overlap them by breaking up the work into 00:42:15.220 --> 00:42:21.050 send, wait, work stages, where each iteration trying to send 00:42:21.050 --> 00:42:24.340 or request the data for the next iteration, I wait on the 00:42:24.340 --> 00:42:27.420 data from a previous iteration and then I do my work. 00:42:27.420 --> 00:42:29.910 So depending on how I partition, I can really get 00:42:29.910 --> 00:42:32.280 really good overlap. 00:42:32.280 --> 00:42:35.320 And so what you want to get to is the concept of the steady 00:42:35.320 --> 00:42:40.130 state, where in your main loop body, all you're doing is 00:42:40.130 --> 00:42:43.590 essentially pre-fetching or requesting data that's going 00:42:43.590 --> 00:42:46.030 to be used in future iterations for future work. 00:42:46.030 --> 00:42:49.890 And then you're waiting on -- 00:42:49.890 --> 00:42:51.490 yeah. 00:42:51.490 --> 00:42:54.100 I think my color coding is a little bogus. 00:42:54.100 --> 00:42:55.860 That's good. 00:42:55.860 --> 00:42:58.360 So here's an example of how you might do this kind of 00:42:58.360 --> 00:43:01.700 buffer pipelining in Cell. 00:43:01.700 --> 00:43:05.710 So I have some main loop that's going to do some work, 00:43:05.710 --> 00:43:07.670 that's encapsulating this process data. 00:43:07.670 --> 00:43:09.780 And what I'm going to use is two buffers. 00:43:09.780 --> 00:43:12.750 So the scheme is also called double buffering. 00:43:12.750 --> 00:43:15.200 I'm going to use this ID to represent which buffer I'm 00:43:15.200 --> 00:43:15.700 going to use. 00:43:15.700 --> 00:43:18.910 So it's either buffer zero or buffer one. 00:43:18.910 --> 00:43:21.230 And this instruction here essentially flips the bit. 00:43:21.230 --> 00:43:23.700 So it's either zero or one. 00:43:23.700 --> 00:43:27.620 So I fetch data into buffer zero and then I enter my loop. 00:43:27.620 --> 00:43:30.680 So this is essentially the first send, which is trying to 00:43:30.680 --> 00:43:33.330 get me one iteration ahead. 00:43:33.330 --> 00:43:37.760 So I enter this mail loop and I do some calculation to 00:43:37.760 --> 00:43:40.380 figure out where to write the next data. 00:43:40.380 --> 00:43:43.735 And then I do another request for the next data item that 00:43:43.735 --> 00:43:47.800 I'm going to -- sorry, there's an m missing here -- 00:43:47.800 --> 00:43:50.900 I'm going to fetch data into a different buffer, right. 00:43:50.900 --> 00:43:54.360 This is ID where I've already flipped the bit once. 00:43:54.360 --> 00:43:58.160 So this get is going to write data into buffer zero. 00:43:58.160 --> 00:44:01.730 And this get is going to write data into buffer one. 00:44:01.730 --> 00:44:02.770 I flip the bit again. 00:44:02.770 --> 00:44:08.720 So now I'm going to issue a wait instruction that says is 00:44:08.720 --> 00:44:10.380 the data from buffer zero ready? 00:44:10.380 --> 00:44:13.260 And if it is then I can go on and actually do my work. 00:44:13.260 --> 00:44:15.590 Does that make sense? 00:44:15.590 --> 00:44:16.150 People are confused? 00:44:16.150 --> 00:44:17.400 Should I go over it again? 00:44:19.772 --> 00:44:21.720 AUDIENCE: [INAUDIBLE] 00:44:21.720 --> 00:44:24.260 PROFESSOR: So this is an [? x or. ?] 00:44:24.260 --> 00:44:27.710 So I could have just said buffer equals zero or buffer 00:44:27.710 --> 00:44:28.960 equals one. 00:44:33.220 --> 00:44:34.410 Oh, sorry. 00:44:34.410 --> 00:44:34.620 This is one. 00:44:34.620 --> 00:44:34.690 Yeah. 00:44:34.690 --> 00:44:36.980 Yeah. 00:44:36.980 --> 00:44:40.950 So this is a one here. 00:44:40.950 --> 00:44:41.440 Last-minute editing. 00:44:41.440 --> 00:44:43.430 It's right there. 00:44:43.430 --> 00:44:44.672 Did that confuse you? 00:44:44.672 --> 00:44:45.195 AUDIENCE: No. 00:44:45.195 --> 00:44:47.810 But, like, I don't see [INAUDIBLE] 00:44:47.810 --> 00:44:48.140 PROFESSOR: Oh. 00:44:48.140 --> 00:44:48.410 OK. 00:44:48.410 --> 00:44:50.190 So I'll go over it again. 00:44:50.190 --> 00:44:53.600 So this get here is going to write into ID zero. 00:44:53.600 --> 00:44:56.500 So that's buffer zero. 00:44:56.500 --> 00:44:57.930 And then I'm going to change the ID. 00:44:57.930 --> 00:44:59.400 So imagine there's a one here. 00:44:59.400 --> 00:45:04.190 So now the next time I use ID, which is here, I'm trying to 00:45:04.190 --> 00:45:04.950 get the data. 00:45:04.950 --> 00:45:07.920 And I'm going to write it to buffer one. 00:45:07.920 --> 00:45:11.270 The DMA on the Cell processor essentially says I can send 00:45:11.270 --> 00:45:14.450 this request off and I can check later to see when that 00:45:14.450 --> 00:45:15.710 data is available. 00:45:15.710 --> 00:45:17.940 But that data is going to go into a different buffer, 00:45:17.940 --> 00:45:19.420 essentially B1. 00:45:19.420 --> 00:45:22.450 Whereas I'm going to work on buffer zero. 00:45:22.450 --> 00:45:25.920 Because I changed the ID back here. 00:45:25.920 --> 00:45:27.610 Now you get it? 00:45:27.610 --> 00:45:30.790 So I fetch data into buffer zero initially 00:45:30.790 --> 00:45:31.540 before I start to loop. 00:45:31.540 --> 00:45:34.110 And then I start working. 00:45:34.110 --> 00:45:36.380 I probably should have had an animation in here. 00:45:36.380 --> 00:45:39.430 So then you go into your main loop. 00:45:39.430 --> 00:45:42.880 You try to start fetching into buffet one and then you try to 00:45:42.880 --> 00:45:44.280 compute out of buffer zero. 00:45:44.280 --> 00:45:46.130 But before you can start computing out of buffer zero, 00:45:46.130 --> 00:45:48.250 you just have to make sure that your data is there. 00:45:48.250 --> 00:45:52.790 And so that's what the synchronization is doing here. 00:45:52.790 --> 00:45:55.180 Hope that was clear. 00:45:55.180 --> 00:45:58.710 OK, so this kind of computation and communication 00:45:58.710 --> 00:46:01.680 overlap really helps in hiding the latency. 00:46:01.680 --> 00:46:04.990 And it can be real useful in terms of improving 00:46:04.990 --> 00:46:06.240 performance. 00:46:09.720 --> 00:46:13.450 And there are different kinds of communication patterns. 00:46:13.450 --> 00:46:14.720 So there's point to point. 00:46:14.720 --> 00:46:18.080 And you can use these both for data communication or control 00:46:18.080 --> 00:46:19.060 communication. 00:46:19.060 --> 00:46:20.880 And it just means that, you know, one processor can 00:46:20.880 --> 00:46:23.580 explicitly send a message to another processor. 00:46:23.580 --> 00:46:26.345 There's also broadcast that says, hey, I have some data 00:46:26.345 --> 00:46:28.603 that everybody's interested in, so I can just broadcast it 00:46:28.603 --> 00:46:31.000 to everybody on the network. 00:46:31.000 --> 00:46:32.930 Or a reduce, which is the opposite. 00:46:32.930 --> 00:46:36.350 It says everybody on the network has data that I need 00:46:36.350 --> 00:46:39.840 to compute, so everybody send me their data. 00:46:39.840 --> 00:46:42.790 There's an all to all, which says all processors should 00:46:42.790 --> 00:46:46.410 just do a global exchange of data that they have. And then 00:46:46.410 --> 00:46:48.100 there's a scatter and a gather. 00:46:48.100 --> 00:46:50.810 So a scatter and a gather are really different types of 00:46:50.810 --> 00:46:54.770 broadcast. So it's one to several or one to many. 00:46:54.770 --> 00:46:57.340 And gather, which is many to one. 00:46:57.340 --> 00:47:00.370 So this is useful when you're doing a computation that 00:47:00.370 --> 00:47:04.670 really is trying to pull data in together but only from a 00:47:04.670 --> 00:47:06.260 subset of all processors. 00:47:06.260 --> 00:47:12.250 So it depends on how you've partitioned your problems. 00:47:12.250 --> 00:47:16.430 So there's a well-known sort of message passing library 00:47:16.430 --> 00:47:24.950 specification called MPI that tries to specify all of these 00:47:24.950 --> 00:47:29.120 different communications in order to sort of facilitate 00:47:29.120 --> 00:47:30.310 parallel programming. 00:47:30.310 --> 00:47:34.590 Its full featured actually has more types of communications 00:47:34.590 --> 00:47:36.660 and more kinds of functionality than I showed on 00:47:36.660 --> 00:47:38.870 the previous slides. 00:47:38.870 --> 00:47:41.360 But it's not a language or a compiler specification. 00:47:41.360 --> 00:47:43.470 It's really just a library that you can implement in 00:47:43.470 --> 00:47:45.750 various ways on different architectures. 00:47:45.750 --> 00:47:50.720 Again, it's same program, multiple data, or supports the 00:47:50.720 --> 00:47:53.080 SPMD model. 00:47:53.080 --> 00:47:55.990 And it works reasonably well for parallel architectures for 00:47:55.990 --> 00:47:59.760 clusters, heterogeneous multicores, homogeneous 00:47:59.760 --> 00:48:01.830 multicores. 00:48:01.830 --> 00:48:05.240 Because really all it's doing is just abstracting out -- 00:48:05.240 --> 00:48:08.130 it's giving you a mechanism to abstract out all the 00:48:08.130 --> 00:48:11.540 communication that you would need in your computation. 00:48:11.540 --> 00:48:15.280 So you can have additional things like precise buffer 00:48:15.280 --> 00:48:16.370 management. 00:48:16.370 --> 00:48:19.250 You can have some collective operations. 00:48:19.250 --> 00:48:22.985 I'll show an example of for doing things things in a 00:48:22.985 --> 00:48:27.370 scalable manner when a lot of things need to communicate 00:48:27.370 --> 00:48:29.140 with each other. 00:48:29.140 --> 00:48:32.840 So just a brief history of where MPI came from. 00:48:32.840 --> 00:48:35.270 And, you know, very early when, you know, parallel 00:48:35.270 --> 00:48:38.260 computers started becoming more and more widespread and 00:48:38.260 --> 00:48:40.720 there were these networks and people had problems porting 00:48:40.720 --> 00:48:43.840 their applications or writing applications for these 00:48:43.840 --> 00:48:45.860 [? came, ?] just because it was difficult, as you might be 00:48:45.860 --> 00:48:48.200 finding in terms of programming things with the 00:48:48.200 --> 00:48:50.540 Cell processor. 00:48:50.540 --> 00:48:52.800 You know, there needed to be ways to sort of address the 00:48:52.800 --> 00:48:54.860 spectrum of communication. 00:48:54.860 --> 00:48:58.330 And it often helps to have a standard because if everybody 00:48:58.330 --> 00:49:01.680 implements the same standard specification, that allows 00:49:01.680 --> 00:49:03.370 your code to be ported around from one 00:49:03.370 --> 00:49:04.710 architecture to the other. 00:49:04.710 --> 00:49:08.090 And so MPI came around. 00:49:08.090 --> 00:49:11.130 The forum was organized in 1992. 00:49:11.130 --> 00:49:13.980 And that had a lot of people participating in it from 00:49:13.980 --> 00:49:17.000 vendors, you know, people like IBM, a company like IBM, 00:49:17.000 --> 00:49:23.030 Intel, and people who had expertise in writing 00:49:23.030 --> 00:49:27.550 libraries, users who were interested in using these 00:49:27.550 --> 00:49:31.910 kinds of specifications to do their computations, so 00:49:31.910 --> 00:49:36.010 scientific people who were in the scientific domain. 00:49:36.010 --> 00:49:38.050 And it was finished in about 18 months. 00:49:38.050 --> 00:49:40.582 I don't know if that's a reasonably long time or a 00:49:40.582 --> 00:49:40.950 short time. 00:49:40.950 --> 00:49:44.340 But considering, you know, I think the MPEG-4 standard took 00:49:44.340 --> 00:49:48.170 a bit longer to do, as a comparison point. 00:49:48.170 --> 00:49:49.880 I don't have the actual data. 00:49:49.880 --> 00:49:53.270 So point-to-point communication -- 00:49:53.270 --> 00:49:56.590 and again, a reminder, this is how you would do it on Cell. 00:49:56.590 --> 00:50:00.300 These are SPE sends and receives. 00:50:00.300 --> 00:50:02.490 You have one processor that's sending 00:50:02.490 --> 00:50:04.660 it to another processor. 00:50:04.660 --> 00:50:06.260 Or you have some network in between. 00:50:06.260 --> 00:50:08.430 And processor A can essentially send the data 00:50:08.430 --> 00:50:11.910 explicitly to processor two. 00:50:11.910 --> 00:50:14.880 And the message in this case would include how the data is 00:50:14.880 --> 00:50:17.240 packaged, some other information such as the length 00:50:17.240 --> 00:50:20.570 of the data, destination, possibly some tag so you can 00:50:20.570 --> 00:50:22.650 identify the actual communication. 00:50:22.650 --> 00:50:26.560 And, you know, there's an actual mapping for the actual 00:50:26.560 --> 00:50:29.760 functions on Cell. 00:50:29.760 --> 00:50:32.950 And there's a get for the send and a put for the receive. 00:50:39.230 --> 00:50:41.760 So there's a question of, well, how do I know if my data 00:50:41.760 --> 00:50:42.880 actually got sent? 00:50:42.880 --> 00:50:45.410 How do I know if it was received? 00:50:45.410 --> 00:50:49.190 And there's, you know, you can think of a synchronous send 00:50:49.190 --> 00:50:52.200 and a synchronous receive, or asynchronous communication. 00:50:52.200 --> 00:50:53.940 So in the synchronous communication, you actually 00:50:53.940 --> 00:50:55.390 wait for notification. 00:50:55.390 --> 00:50:57.250 So this is kind of like your fax machine. 00:50:57.250 --> 00:50:58.900 You put something into your fax. 00:50:58.900 --> 00:51:01.190 It goes out and you eventually get a beep that says your 00:51:01.190 --> 00:51:02.260 transmission was OK. 00:51:02.260 --> 00:51:04.900 Or if it wasn't OK then, you know, you get a message that 00:51:04.900 --> 00:51:06.630 says, you know, something went wrong. 00:51:06.630 --> 00:51:09.520 And you can redo your communication. 00:51:09.520 --> 00:51:11.730 An asynchronous send is kind of like your -- 00:51:11.730 --> 00:51:13.055 AUDIENCE: Most [UNINTELLIGIBLE] you could get 00:51:13.055 --> 00:51:13.320 a reply too. 00:51:13.320 --> 00:51:15.340 PROFESSOR: Yeah, you can get a reply. 00:51:15.340 --> 00:51:16.420 Thanks. 00:51:16.420 --> 00:51:18.360 An asynchronous send, it's like you write a letter, you 00:51:18.360 --> 00:51:20.680 go put it in the mailbox, and you don't know whether it 00:51:20.680 --> 00:51:26.425 actually made it into the actual postman's bag and it 00:51:26.425 --> 00:51:29.660 was delivered to your destination or if it was 00:51:29.660 --> 00:51:30.940 actually delivered. 00:51:30.940 --> 00:51:34.200 So you only know that the message was sent. 00:51:34.200 --> 00:51:36.250 You know, you put it in the mailbox. 00:51:36.250 --> 00:51:37.940 But you don't know anything else about what happened to 00:51:37.940 --> 00:51:41.100 the message along the way. 00:51:41.100 --> 00:51:43.530 There's also the concept of a blocking versus a 00:51:43.530 --> 00:51:44.940 non-blocking message. 00:51:44.940 --> 00:51:48.110 So this is orthogonal really to synchronous versus 00:51:48.110 --> 00:51:49.930 asynchronous. 00:51:49.930 --> 00:51:54.940 So in blocking messages, a sender waits until there's 00:51:54.940 --> 00:51:58.440 some signal that says the message has been transmitted. 00:51:58.440 --> 00:52:03.180 So this is, for example if I'm writing data into a buffer, 00:52:03.180 --> 00:52:05.350 and the buffer essentially gets transmitted to somebody 00:52:05.350 --> 00:52:10.590 else, we wait until the buffer is empty. 00:52:10.590 --> 00:52:13.070 And what that means is that somebody has read it on the 00:52:13.070 --> 00:52:15.350 other end or somebody has drained that buffer from 00:52:15.350 --> 00:52:16.670 somewhere else. 00:52:16.670 --> 00:52:19.820 The receiver, if he's waiting on data, well, he just waits. 00:52:19.820 --> 00:52:21.850 He essentially blocks until somebody has put 00:52:21.850 --> 00:52:24.110 data into the buffer. 00:52:24.110 --> 00:52:26.470 And you can get into potential deadlock situations. 00:52:26.470 --> 00:52:30.100 So you saw deadlock with locks in the concurrency talk. 00:52:30.100 --> 00:52:30.920 I'm going to show you a different 00:52:30.920 --> 00:52:32.590 kind of deadlock example. 00:52:35.200 --> 00:52:40.630 An example of a blocking send on Cell -- 00:52:40.630 --> 00:52:43.250 allows you to use mailboxes. 00:52:43.250 --> 00:52:47.260 Or you can sort of use mailboxes for that. 00:52:47.260 --> 00:52:50.050 Mailboxes again are just for communicating short messages, 00:52:50.050 --> 00:52:53.910 really, not necessarily for communicating data messages. 00:52:53.910 --> 00:52:58.340 So an SPE does some work, and then it writes out a message, 00:52:58.340 --> 00:53:02.750 in this case to notify the PPU that, let's say, it's done. 00:53:02.750 --> 00:53:04.660 And then it goes on and does more work. 00:53:04.660 --> 00:53:08.230 And then it wants to notify the PPU of something else. 00:53:08.230 --> 00:53:12.190 So in this case this particular send will block 00:53:12.190 --> 00:53:14.930 because, let's say, the PPU hasn't drained its mailbox. 00:53:14.930 --> 00:53:16.220 It hasn't read the mailbox. 00:53:16.220 --> 00:53:20.960 So you essentially stop and wait until the PPU has, you 00:53:20.960 --> 00:53:21.810 know, caught up. 00:53:21.810 --> 00:53:26.860 AUDIENCE: So all mailbox sends are blocking? 00:53:26.860 --> 00:53:27.990 PROFESSOR: Yes. 00:53:27.990 --> 00:53:29.240 David says yes. 00:53:36.680 --> 00:53:39.730 A non-blocking send is something that essentially 00:53:39.730 --> 00:53:44.400 allows you to send a message out and just continue on. 00:53:44.400 --> 00:53:48.650 You don't care exactly about what's happened to the message 00:53:48.650 --> 00:53:51.910 or what's going on with the receiver. 00:53:51.910 --> 00:53:54.530 So you write the data into the buffer and you 00:53:54.530 --> 00:53:56.040 just continue executing. 00:53:56.040 --> 00:53:58.740 And this really helps you in terms of avoiding idle times 00:53:58.740 --> 00:54:00.560 and deadlocks, but it might not always be the 00:54:00.560 --> 00:54:01.840 thing that you want. 00:54:01.840 --> 00:54:05.970 So an example of sort of a non-blocking send and wait on 00:54:05.970 --> 00:54:09.170 Cell is using the DMAs to ship data out. 00:54:09.170 --> 00:54:11.580 You know, you can put something, put in a request to 00:54:11.580 --> 00:54:13.910 send data out on the DMA. 00:54:13.910 --> 00:54:19.190 And you could wait for it if you want in terms of reading 00:54:19.190 --> 00:54:22.680 the status bits to make sure it's completed. 00:54:22.680 --> 00:54:27.530 OK, so what is a source of deadlock in the blocking case? 00:54:27.530 --> 00:54:30.130 And it really comes about if you don't really have enough 00:54:30.130 --> 00:54:33.140 buffering in your communication network. 00:54:33.140 --> 00:54:36.300 And often you can resolve that by having additional storage. 00:54:36.300 --> 00:54:38.700 So let's say I have processor one and processor two and 00:54:38.700 --> 00:54:41.000 they're trying to send messages to each other. 00:54:41.000 --> 00:54:43.710 So processor one sends a message at the same time 00:54:43.710 --> 00:54:45.170 processor two sends a message. 00:54:45.170 --> 00:54:46.990 And these are going to go, let's say, 00:54:46.990 --> 00:54:49.200 into the same buffer. 00:54:49.200 --> 00:54:53.815 Well, neither can make progress because somebody has 00:54:53.815 --> 00:54:55.930 to essentially drain that buffer before these receives 00:54:55.930 --> 00:54:57.180 can execute. 00:55:00.350 --> 00:55:02.990 So what happens with that code is it really depends on how 00:55:02.990 --> 00:55:04.830 much buffering you have between the two. 00:55:04.830 --> 00:55:06.370 If you have a lot of buffering, then you may never 00:55:06.370 --> 00:55:08.000 see the deadlock. 00:55:08.000 --> 00:55:13.930 But if you have a really tiny buffer, then you do a send. 00:55:13.930 --> 00:55:18.180 The other person can't do the send because the buffer hasn't 00:55:18.180 --> 00:55:18.970 been drained. 00:55:18.970 --> 00:55:21.220 And so you end up with a deadlock. 00:55:21.220 --> 00:55:23.600 And so a potential solution is, well, you actually 00:55:23.600 --> 00:55:24.620 increase your buffer length. 00:55:24.620 --> 00:55:26.170 But that doesn't always work because you can 00:55:26.170 --> 00:55:27.480 still get into trouble. 00:55:27.480 --> 00:55:30.040 So what you might need to do is essentially be more 00:55:30.040 --> 00:55:33.990 diligent about how you order your sends and receives. 00:55:33.990 --> 00:55:36.740 So if you have processor one doing a send, make sure it's 00:55:36.740 --> 00:55:39.400 matched up with a receive on the other end. 00:55:39.400 --> 00:55:42.050 And similarly, if you're doing a receive here, make sure 00:55:42.050 --> 00:55:44.100 there's sort of a matching send on the other end. 00:55:44.100 --> 00:55:46.870 And that helps you in sort of making sure that things are 00:55:46.870 --> 00:55:51.160 operating reasonably in lock step at, you know, partially 00:55:51.160 --> 00:55:52.410 ordered times. 00:55:59.750 --> 00:56:03.600 That was really examples of point-to-point communication. 00:56:03.600 --> 00:56:05.990 A broadcast mechanism is slightly different. 00:56:05.990 --> 00:56:09.190 It says, I have data that I want to send to everybody. 00:56:09.190 --> 00:56:11.750 It could be really efficient for sending short control 00:56:11.750 --> 00:56:15.570 messages, maybe even efficient for sending data messages. 00:56:15.570 --> 00:56:18.950 So as an example, if you remember our calculation of 00:56:18.950 --> 00:56:22.380 distances between all points, the parallelization strategy 00:56:22.380 --> 00:56:24.900 said, well, I'm going to send one copy of 00:56:24.900 --> 00:56:28.370 the array A to everybody. 00:56:28.370 --> 00:56:29.820 In the two processor case that was easy. 00:56:29.820 --> 00:56:32.730 But if I have n processors, then rather than sending 00:56:32.730 --> 00:56:35.570 point-to-point communication from A to everybody else, what 00:56:35.570 --> 00:56:38.300 I could do is just, say, broadcast A to everybody and 00:56:38.300 --> 00:56:41.420 they can grab it off the network. 00:56:41.420 --> 00:56:45.210 So in MPI there's this function, MPI 00:56:45.210 --> 00:56:46.560 broadcast, that does that. 00:56:46.560 --> 00:56:51.700 I'm using sort of generic abstract sends, receives and 00:56:51.700 --> 00:56:53.350 broadcasts in my examples. 00:56:53.350 --> 00:56:55.600 So you can broadcast A to everybody. 00:56:55.600 --> 00:56:59.550 And then if I have n processors, then what I might 00:56:59.550 --> 00:57:03.310 do is distribute the m's in a round robin manner to each of 00:57:03.310 --> 00:57:04.230 the different processes. 00:57:04.230 --> 00:57:05.710 So you pointed this out. 00:57:05.710 --> 00:57:07.180 I don't have to send B to everybody. 00:57:07.180 --> 00:57:09.390 I can just send, you know, in this case, 00:57:09.390 --> 00:57:10.420 one particular element. 00:57:10.420 --> 00:57:12.840 Is that clear? 00:57:16.838 --> 00:57:18.670 AUDIENCE: There's no broadcast on Cell? 00:57:18.670 --> 00:57:21.210 PROFESSOR: There is no broadcast on Cell. 00:57:21.210 --> 00:57:25.680 There is no mechanism for reduction either. 00:57:25.680 --> 00:57:30.340 And you can't quite do scatters and gathers. 00:57:30.340 --> 00:57:32.430 I don't think. 00:57:32.430 --> 00:57:35.380 OK, so an example of a reduction, you know, I said 00:57:35.380 --> 00:57:37.650 it's the opposite of a broadcast. Everybody has data 00:57:37.650 --> 00:57:39.860 that needs to essentially get to the same point. 00:57:39.860 --> 00:57:45.610 So as an example, if everybody in this room had a value, 00:57:45.610 --> 00:57:47.770 including myself, and I wanted to know what is the collective 00:57:47.770 --> 00:57:49.670 value of everybody in the room, you all have to 00:57:49.670 --> 00:57:50.840 send me your data. 00:57:50.840 --> 00:57:53.800 Now, this is important because if -- you know, in this case 00:57:53.800 --> 00:57:54.700 we're doing an addition. 00:57:54.700 --> 00:57:56.290 It's an associative operation. 00:57:56.290 --> 00:57:58.050 So what we can do is we can be smart about 00:57:58.050 --> 00:57:59.350 how the data is sent. 00:57:59.350 --> 00:58:02.160 So, you know, guys that are close together can essentially 00:58:02.160 --> 00:58:03.420 add up their numbers and forward me. 00:58:03.420 --> 00:58:05.120 So instead of getting n messages I 00:58:05.120 --> 00:58:06.520 can get log n messages. 00:58:06.520 --> 00:58:08.450 And so if every pair of you added your numbers and 00:58:08.450 --> 00:58:11.040 forwarded me that, that cuts down communication by half. 00:58:11.040 --> 00:58:13.640 And so you can, you know -- starting from the back of 00:58:13.640 --> 00:58:16.420 room, by the time you get to me, I only get two messages 00:58:16.420 --> 00:58:18.800 instead of n messages. 00:58:18.800 --> 00:58:21.680 So a reduction combines data from all processors. 00:58:21.680 --> 00:58:24.030 In MPI, you know, there's this function MPI 00:58:24.030 --> 00:58:26.300 reduce for doing that. 00:58:26.300 --> 00:58:29.180 And the collective operations are things that are 00:58:29.180 --> 00:58:29.920 associative. 00:58:29.920 --> 00:58:32.730 And subtract -- 00:58:32.730 --> 00:58:33.660 sorry. 00:58:33.660 --> 00:58:39.500 And or and -- you can read them on the slide. 00:58:39.500 --> 00:58:42.540 There is a semantic caveat here that no processor can 00:58:42.540 --> 00:58:45.760 finish the reduction before all processors have at least 00:58:45.760 --> 00:58:49.730 sent it one data or have contributed, rather, a 00:58:49.730 --> 00:58:51.790 particular value. 00:58:51.790 --> 00:58:54.740 So in many numerical algorithms, you can actually 00:58:54.740 --> 00:59:00.200 use the broadcast and send to broadcast and reduce in place 00:59:00.200 --> 00:59:04.430 of sends and receives because it really improves the 00:59:04.430 --> 00:59:06.970 simplicity of your computation. 00:59:06.970 --> 00:59:09.260 You don't have to do n sends to communicate there. 00:59:09.260 --> 00:59:11.350 You can just broadcast. It gives you a mechanism for 00:59:11.350 --> 00:59:13.970 essentially having a shared memory abstraction on 00:59:13.970 --> 00:59:16.150 distributed memory architecture. 00:59:16.150 --> 00:59:18.392 There are things like all to all communication which would 00:59:18.392 --> 00:59:19.730 also help you in that sense. 00:59:19.730 --> 00:59:24.960 Although I don't talk about all to all communication here. 00:59:24.960 --> 00:59:27.360 So I'm going to show you an example of sort of a more 00:59:27.360 --> 00:59:29.810 detailed MPI. 00:59:29.810 --> 00:59:32.690 But I also want to contrast this to the OpenMP programming 00:59:32.690 --> 00:59:36.000 on shared memory processors because one might look simpler 00:59:36.000 --> 00:59:38.470 than the other. 00:59:38.470 --> 00:59:40.540 So suppose that you have a numerical integration method 00:59:40.540 --> 00:59:44.990 that essentially you're going to use to calculate pi. 00:59:44.990 --> 00:59:48.200 So as you get finer and finer, you can get more accurate -- 00:59:48.200 --> 00:59:50.030 as you shrink these intervals you can get 00:59:50.030 --> 00:59:53.900 better values for pi. 00:59:53.900 --> 00:59:58.690 And the code for doing that is some C code. 00:59:58.690 --> 01:00:00.270 You have some variables. 01:00:00.270 --> 01:00:02.800 And then you have a step that essentially tells you how many 01:00:02.800 --> 01:00:04.730 times you're going to do this computation. 01:00:04.730 --> 01:00:07.440 And for each time step you calculate this particular 01:00:07.440 --> 01:00:08.540 function here. 01:00:08.540 --> 01:00:11.040 And you add it all up and in the end you can sort of print 01:00:11.040 --> 01:00:13.640 out what is the value of pi that you calculated. 01:00:13.640 --> 01:00:16.600 So clearly as, you know, as you shrink your intervals, you 01:00:16.600 --> 01:00:20.340 can get more and more accurate measures of pi. 01:00:20.340 --> 01:00:22.740 So that translates to increasing the number of steps 01:00:22.740 --> 01:00:29.160 in that particular C code. 01:00:29.160 --> 01:00:33.700 So you can use that numerical integration to calculate pi 01:00:33.700 --> 01:00:34.900 with OpenMP. 01:00:34.900 --> 01:00:37.356 And what that translates to is -- sorry, there should have 01:00:37.356 --> 01:00:41.370 been an animation here to ask you what I should add in. 01:00:41.370 --> 01:00:43.220 You have this particular loop. 01:00:43.220 --> 01:00:46.330 And this is computation that you want to parallelize. 01:00:46.330 --> 01:00:48.890 And there is really four questions that you essentially 01:00:48.890 --> 01:00:50.540 have to go through. 01:00:50.540 --> 01:00:52.220 Are there variables that are shared? 01:00:52.220 --> 01:00:54.420 Because you have to get the process right. 01:00:54.420 --> 01:00:56.200 If there are variables that are shared, you have to 01:00:56.200 --> 01:01:01.830 explicitly synchronize them and use locks to protect them. 01:01:01.830 --> 01:01:02.880 What values are private? 01:01:02.880 --> 01:01:08.690 So in OpenMP, things that are private are data on the stack, 01:01:08.690 --> 01:01:12.000 things that are defined lexically within the scope of 01:01:12.000 --> 01:01:16.030 the computation that you encapsulate by an OpenMP 01:01:16.030 --> 01:01:18.900 pragma, and what variables you might want 01:01:18.900 --> 01:01:20.340 to use for a reduction. 01:01:20.340 --> 01:01:23.490 So in this case I'm doing a summation, and this is the 01:01:23.490 --> 01:01:25.400 computation that I can parallelize. 01:01:25.400 --> 01:01:28.770 Then I essentially want to do a reduction for the plus 01:01:28.770 --> 01:01:32.980 operator since I'm doing an addition on this variable. 01:01:32.980 --> 01:01:34.760 This loop here is parallel. 01:01:34.760 --> 01:01:36.040 It's data parallel. 01:01:36.040 --> 01:01:38.900 I can split it up. 01:01:38.900 --> 01:01:41.190 The for loop is also -- 01:01:41.190 --> 01:01:43.810 I can do this work sharing on it. 01:01:43.810 --> 01:01:45.960 So I use the parallel for pragma. 01:01:45.960 --> 01:01:49.360 And the variable x here is private. 01:01:49.360 --> 01:01:52.010 It's defined here but I can essentially give a directive 01:01:52.010 --> 01:01:53.530 that says, this is private. 01:01:53.530 --> 01:01:56.290 You can essentially rename it on each processor. 01:01:56.290 --> 01:01:58.890 Its value won't have any effect on the overall 01:01:58.890 --> 01:02:01.270 computation because each computation will have its own 01:02:01.270 --> 01:02:03.910 local copy. 01:02:03.910 --> 01:02:06.360 That clear so far? 01:02:06.360 --> 01:02:09.950 So computing pi with integration using MPI takes up 01:02:09.950 --> 01:02:12.110 two slides. 01:02:12.110 --> 01:02:13.980 You know, I could fit it on one slide but you couldn't see 01:02:13.980 --> 01:02:15.170 it in the back. 01:02:15.170 --> 01:02:16.990 So there's some initialization. 01:02:16.990 --> 01:02:20.130 In fact, I think there's only six basic MPI commands that 01:02:20.130 --> 01:02:22.120 you need for computing. 01:02:22.120 --> 01:02:26.030 Three of them are here and you'll see the others are MPI 01:02:26.030 --> 01:02:27.680 send and MPI receive. 01:02:27.680 --> 01:02:31.550 And there's one more that you'll see on the next slide. 01:02:31.550 --> 01:02:33.170 So there's some loop that says while I'm 01:02:33.170 --> 01:02:36.790 not done keep computing. 01:02:36.790 --> 01:02:39.120 And what you do is you broadcast n to all the 01:02:39.120 --> 01:02:39.990 different processors. 01:02:39.990 --> 01:02:42.900 N is really your time step. 01:02:42.900 --> 01:02:47.660 How many small intervals of execution are you going to do? 01:02:47.660 --> 01:02:49.930 And you can go through, do your computation. 01:02:49.930 --> 01:02:52.510 So now this -- the MPI essentially encapsulates the 01:02:52.510 --> 01:02:55.290 computation over n processors. 01:02:55.290 --> 01:02:58.250 And then you get to an MPI reduce command at some point 01:02:58.250 --> 01:03:01.810 that says, OK, what values did everybody compute? 01:03:01.810 --> 01:03:03.430 Do the reduction on that. 01:03:03.430 --> 01:03:06.410 Write that value into my MPI. 01:03:06.410 --> 01:03:10.700 Now what happens here is there's processor ID zero 01:03:10.700 --> 01:03:12.140 which I'm going to consider the master. 01:03:12.140 --> 01:03:15.030 So he's the one who's going to actually print out the value. 01:03:15.030 --> 01:03:18.780 So the reduction essentially synchronizes until everybody's 01:03:18.780 --> 01:03:22.370 communicated a value to processor zero. 01:03:22.370 --> 01:03:23.990 And then it can print out the pi. 01:03:23.990 --> 01:03:27.660 And then you can finalize, which actually makes sure the 01:03:27.660 --> 01:03:28.890 computation can exit. 01:03:28.890 --> 01:03:30.360 And you can go on and terminate. 01:03:35.710 --> 01:03:39.750 So the last concept in terms of understanding performance 01:03:39.750 --> 01:03:43.010 for parallelism is this notion of locality. 01:03:43.010 --> 01:03:46.240 And there's locality in your communication and locality in 01:03:46.240 --> 01:03:47.930 your computation. 01:03:47.930 --> 01:03:50.700 So what do I mean by that? 01:03:50.700 --> 01:03:55.690 So in terms of communication, you know, if I have two 01:03:55.690 --> 01:03:58.830 operations and let's say -- this is a picture or schematic 01:03:58.830 --> 01:04:02.130 of what the MIT raw chip looks like. 01:04:02.130 --> 01:04:03.570 Each one of these is a core. 01:04:03.570 --> 01:04:06.620 There's some network, some basic computation elements. 01:04:06.620 --> 01:04:09.270 And if I have, you know, an addition that feeds into a 01:04:09.270 --> 01:04:12.910 shift, well, I can put the addition here and the shift 01:04:12.910 --> 01:04:15.720 there, but that means I have a really long path that I need 01:04:15.720 --> 01:04:17.700 to go to in terms of communicating 01:04:17.700 --> 01:04:19.190 that data value around. 01:04:19.190 --> 01:04:22.590 So the computation naturally should just be closer together 01:04:22.590 --> 01:04:25.950 because that decreases the latency that I need to 01:04:25.950 --> 01:04:27.940 communicate. 01:04:27.940 --> 01:04:30.300 So rather than doing net mapping, what I might want to 01:04:30.300 --> 01:04:32.900 do is just go to somebody who is close to me and available. 01:04:32.900 --> 01:04:35.130 AUDIENCE: Also there are volume issues. 01:04:35.130 --> 01:04:37.140 So assume more than that. 01:04:37.140 --> 01:04:40.309 A lot of other people also want to communicate. 01:04:40.309 --> 01:04:43.296 So if [UNINTELLIGIBLE] randomly distributed, you can 01:04:43.296 --> 01:04:44.292 assume there's a lot more communication 01:04:44.292 --> 01:04:47.880 going into the channel. 01:04:47.880 --> 01:04:52.710 Whereas if you put locality in there then you can scale 01:04:52.710 --> 01:04:56.210 communication much better than scaling the network. 01:05:00.380 --> 01:05:02.400 PROFESSOR: There's also a notion of locality in terms of 01:05:02.400 --> 01:05:03.040 memory accesses. 01:05:03.040 --> 01:05:07.880 And these are potentially also very important or more 01:05:07.880 --> 01:05:10.010 important, rather, because of the latencies 01:05:10.010 --> 01:05:12.310 for accessing memory. 01:05:12.310 --> 01:05:15.860 So if I have, you know, this loop that's doing some 01:05:15.860 --> 01:05:19.270 addition or some computation on an array and I distribute 01:05:19.270 --> 01:05:21.900 it, say, over four processors -- 01:05:21.900 --> 01:05:24.880 this is, again, let's assume a data parallel loop. 01:05:24.880 --> 01:05:27.460 So what I can do is have a work sharing mechanism that 01:05:27.460 --> 01:05:29.970 says, this thread here will operate on 01:05:29.970 --> 01:05:31.570 the first four indices. 01:05:31.570 --> 01:05:34.135 This thread here will operate on the next four indices and 01:05:34.135 --> 01:05:36.120 the next four and the next four. 01:05:36.120 --> 01:05:39.530 And then you essentially get to join barrier and then you 01:05:39.530 --> 01:05:40.950 can continue on. 01:05:40.950 --> 01:05:44.840 And if we consider how the access patterns are going to 01:05:44.840 --> 01:05:50.730 be generated for this particular loop, well, in the 01:05:50.730 --> 01:05:52.560 sequential case I'm essentially 01:05:52.560 --> 01:05:54.200 generating them in sequence. 01:05:54.200 --> 01:05:56.620 So that allows me to exploit, for example, on traditional 01:05:56.620 --> 01:05:59.780 [? CAT ?] architecture, a notion of spatial locality. 01:05:59.780 --> 01:06:03.620 If I look at how things are organized in memory, in the 01:06:03.620 --> 01:06:06.710 sequential case I can perhaps fetch an 01:06:06.710 --> 01:06:07.870 entire block at a time. 01:06:07.870 --> 01:06:11.340 So I can fetch all the elements of A0 01:06:11.340 --> 01:06:12.680 to A3 in one shot. 01:06:12.680 --> 01:06:16.820 I can fetch all the elements of A4 to A7 in one shot. 01:06:16.820 --> 01:06:19.520 And that allows me to essentially improve 01:06:19.520 --> 01:06:22.080 performance because I overlap communication. 01:06:22.080 --> 01:06:25.890 I'm predicting that once I see a reference, I'm going to use 01:06:25.890 --> 01:06:29.140 data that's adjacent to it in space. 01:06:29.140 --> 01:06:31.070 There's also a notion of temporal locality that says 01:06:31.070 --> 01:06:33.990 that if I use some particular data element, I'm going to 01:06:33.990 --> 01:06:35.430 reuse it later on. 01:06:35.430 --> 01:06:37.970 I'm not showing that here. 01:06:37.970 --> 01:06:41.190 But in the parallel case what could happen is if each one of 01:06:41.190 --> 01:06:43.920 these threads is requesting a different data element -- and 01:06:43.920 --> 01:06:49.470 let's say execution essentially proceeds -- you 01:06:49.470 --> 01:06:51.230 know, all the threads are requesting their 01:06:51.230 --> 01:06:53.970 data at the same time. 01:06:53.970 --> 01:06:56.240 Then all these requests are going to end up going to the 01:06:56.240 --> 01:06:59.130 same memory bank. 01:06:59.130 --> 01:07:02.420 The first thread is requesting ace of zero. 01:07:02.420 --> 01:07:05.770 The next thread is requesting ace of four, the next thread 01:07:05.770 --> 01:07:08.750 ace of eight, next thread ace of 12. 01:07:08.750 --> 01:07:11.010 And all of these happen to be in the same memory bank. 01:07:11.010 --> 01:07:13.040 So what that means is, you know, there's a lot of 01:07:13.040 --> 01:07:14.940 contention for that one memory bank. 01:07:14.940 --> 01:07:17.650 And in effect I've serialized the computation. 01:07:17.650 --> 01:07:17.850 Right? 01:07:17.850 --> 01:07:20.620 Everybody see that? 01:07:20.620 --> 01:07:23.090 And, you know, this can be a problem in that you can 01:07:23.090 --> 01:07:26.920 essentially fully serialize the computation in that, you 01:07:26.920 --> 01:07:29.720 know, there's contention on the first bank, contention on 01:07:29.720 --> 01:07:33.620 the second bank, and then contention on the third bank, 01:07:33.620 --> 01:07:35.040 and then contention on the fourth bank. 01:07:35.040 --> 01:07:38.000 And so I've done absolutely nothing other than pay 01:07:38.000 --> 01:07:39.720 overhead for parallelization. 01:07:39.720 --> 01:07:42.590 I've made extra work for myself [? concreting ?] 01:07:42.590 --> 01:07:44.250 the threads. 01:07:44.250 --> 01:07:46.810 Maybe I've done some extra work in terms of 01:07:46.810 --> 01:07:48.840 synchronization. 01:07:48.840 --> 01:07:50.460 So I'm fully serial. 01:07:52.980 --> 01:07:55.810 So what you want to do is actually reorganize the way 01:07:55.810 --> 01:07:59.840 data is laid out in memory so that you can effectively get 01:07:59.840 --> 01:08:01.620 the benefit of parallelization. 01:08:01.620 --> 01:08:07.420 So if you have the data organized as is there, you can 01:08:07.420 --> 01:08:09.670 shuffle things around. 01:08:09.670 --> 01:08:13.000 And then you end up with fully parallel or a layout that's 01:08:13.000 --> 01:08:16.320 more amenable to full parallelism because now each 01:08:16.320 --> 01:08:17.930 thread is going to a different bank. 01:08:17.930 --> 01:08:20.650 And that essentially gives you a four-way parallelism. 01:08:20.650 --> 01:08:22.750 And so you get the performance benefits. 01:08:26.480 --> 01:08:30.623 So there are different kinds of sort of considerations you 01:08:30.623 --> 01:08:34.400 need to take into account for shared memory architectures in 01:08:34.400 --> 01:08:37.400 terms of how the design affects the memory latency. 01:08:37.400 --> 01:08:42.190 So in a uniform memory access architecture, every processor 01:08:42.190 --> 01:08:44.240 is either, you can think of it as being 01:08:44.240 --> 01:08:45.200 equidistant from memory. 01:08:45.200 --> 01:08:48.295 Or another way, it has the same access latency for 01:08:48.295 --> 01:08:50.100 getting data from memory. 01:08:50.100 --> 01:08:54.480 Most shared memory architectures are non-uniform, 01:08:54.480 --> 01:08:56.710 also known as NUMA architecture. 01:08:56.710 --> 01:08:59.460 So you have physically partitioned memories. 01:08:59.460 --> 01:09:03.020 And the processors can have the same address space, but 01:09:03.020 --> 01:09:05.930 the placement of data affects the performance because going 01:09:05.930 --> 01:09:10.860 to one bank versus another can be faster or slower. 01:09:10.860 --> 01:09:12.560 So what kind of architecture is Cell? 01:09:12.560 --> 01:09:19.100 Yeah. 01:09:19.100 --> 01:09:19.710 No guesses? 01:09:19.710 --> 01:09:22.910 AUDIENCE: It's not a shared memory. 01:09:22.910 --> 01:09:23.150 PROFESSOR: Right. 01:09:23.150 --> 01:09:24.720 It's not a shared memory architecture. 01:09:27.770 --> 01:09:30.500 So a summary of parallel performance factors. 01:09:30.500 --> 01:09:32.390 So there's three things I tried to cover. 01:09:34.970 --> 01:09:36.510 Coverage or the extent of parallelism in the 01:09:36.510 --> 01:09:37.420 application. 01:09:37.420 --> 01:09:40.480 So you saw Amdahl's Law and it actually gave you a sort of a 01:09:40.480 --> 01:09:43.990 model that said when is parallelizing your application 01:09:43.990 --> 01:09:45.000 going to be worthwhile? 01:09:45.000 --> 01:09:46.750 And it really boils down to how much parallelism you 01:09:46.750 --> 01:09:48.990 actually have in your particular algorithm. 01:09:48.990 --> 01:09:50.870 If your algorithm is sequential, then there's 01:09:50.870 --> 01:09:56.110 really nothing you can do for programming for performance 01:09:56.110 --> 01:09:57.820 using parallel architectures. 01:09:57.820 --> 01:10:02.500 I talked about granularity of the data partitioning and the 01:10:02.500 --> 01:10:04.360 granularity of the work distribution. 01:10:04.360 --> 01:10:06.080 You know, if you had really fine-grain things versus 01:10:06.080 --> 01:10:08.200 really coarse-grain things, how does that translate to 01:10:08.200 --> 01:10:10.980 different communication costs? 01:10:10.980 --> 01:10:13.770 And then last thing I shared was locality. 01:10:13.770 --> 01:10:16.620 So if you have near neighbors talking, that may be different 01:10:16.620 --> 01:10:19.230 than two things that are further apart in space 01:10:19.230 --> 01:10:20.730 communicating. 01:10:20.730 --> 01:10:23.690 And there are some issues in terms of the memory latency 01:10:23.690 --> 01:10:28.310 and how you actually can take advantage of that. 01:10:28.310 --> 01:10:33.400 So this really is an overview of sort of the parallel 01:10:33.400 --> 01:10:36.670 programming concepts and the performance implications. 01:10:36.670 --> 01:10:39.530 So the next lecture will be, you know, how do I actually 01:10:39.530 --> 01:10:40.560 parallelize my program? 01:10:40.560 --> 01:10:42.480 And we'll talk about that.