1 00:00:00,000 --> 00:00:02,520 The following content is provided under a Creative 2 00:00:02,520 --> 00:00:03,970 Commons license. 3 00:00:03,970 --> 00:00:06,360 Your support will help MIT OpenCourseWare 4 00:00:06,360 --> 00:00:10,660 continue to offer high quality educational resources for free. 5 00:00:10,660 --> 00:00:13,350 To make a donation or view additional materials 6 00:00:13,350 --> 00:00:17,190 from hundreds of MIT courses, visit MIT OpenCourseWare 7 00:00:17,190 --> 00:00:20,579 at NCAA ocw.mit.edu. 8 00:00:26,648 --> 00:00:28,190 GEORGE VERGHESE: I wanted to give you 9 00:00:28,190 --> 00:00:31,370 an overview of what the system is that we're talking about, 10 00:00:31,370 --> 00:00:34,490 the communication network. 11 00:00:34,490 --> 00:00:38,480 We have a source that's trying to communicate to a receiver. 12 00:00:38,480 --> 00:00:41,160 We've talked about converting-- 13 00:00:41,160 --> 00:00:43,130 well, we'll talk some more, actually-- 14 00:00:43,130 --> 00:00:48,363 about converting the source information to binary digits. 15 00:00:48,363 --> 00:00:50,030 And then where we've spent a lot of time 16 00:00:50,030 --> 00:00:52,220 is talking about source coding, which 17 00:00:52,220 --> 00:00:54,920 is trying to extract the redundancy in the message 18 00:00:54,920 --> 00:00:58,640 that you want to send so that basically every binary digit 19 00:00:58,640 --> 00:01:01,940 you then put on the channel carries as much information as 20 00:01:01,940 --> 00:01:02,520 possible. 21 00:01:02,520 --> 00:01:05,390 So now I'm going to really stop making a distinction 22 00:01:05,390 --> 00:01:07,010 between bits and binary digits. 23 00:01:07,010 --> 00:01:09,740 I'll just say bits when I might mean binary digits. 24 00:01:09,740 --> 00:01:11,390 But once you're into the system here 25 00:01:11,390 --> 00:01:13,580 and you've done your source coding, 26 00:01:13,580 --> 00:01:16,100 a binary digit carries a bit of information, 27 00:01:16,100 --> 00:01:18,050 in general, if you've done a good job 28 00:01:18,050 --> 00:01:19,472 of extracting the redundancy. 29 00:01:19,472 --> 00:01:20,930 So we're talking about a bit stream 30 00:01:20,930 --> 00:01:24,020 here that you're trying to get across to the other end. 31 00:01:24,020 --> 00:01:26,540 At the other end, the bitstream is received. 32 00:01:26,540 --> 00:01:29,120 There's the decoding step, which is 33 00:01:29,120 --> 00:01:32,450 what we've seen with Huffman or LZW, the decoding end. 34 00:01:32,450 --> 00:01:34,190 And then you do whatever it is you're 35 00:01:34,190 --> 00:01:37,080 going to do in the application. 36 00:01:37,080 --> 00:01:39,110 So we've really said all we're going 37 00:01:39,110 --> 00:01:41,720 to say about the source coding and decoding, 38 00:01:41,720 --> 00:01:45,770 and the rest of what we're going to do is focus on what happens 39 00:01:45,770 --> 00:01:46,490 inside here. 40 00:01:50,590 --> 00:01:53,800 Now what happens inside there at some stage 41 00:01:53,800 --> 00:01:58,670 involves a physical communication link. 42 00:01:58,670 --> 00:02:00,937 So you might be talking bitstreams at either end, 43 00:02:00,937 --> 00:02:02,770 but somehow you've got to deal with the fact 44 00:02:02,770 --> 00:02:05,008 that most of these channels, most 45 00:02:05,008 --> 00:02:07,300 of the channels of interest, they're physical channels, 46 00:02:07,300 --> 00:02:10,810 they work with continuous valued, continuous time 47 00:02:10,810 --> 00:02:11,980 quantities. 48 00:02:11,980 --> 00:02:14,230 For instance, the voltage on a cable 49 00:02:14,230 --> 00:02:16,210 might be used to transmit information, 50 00:02:16,210 --> 00:02:19,030 light on the fiber, electromagnetic waves 51 00:02:19,030 --> 00:02:21,990 through space, acoustic waves in air or water-- 52 00:02:21,990 --> 00:02:23,740 in fact acoustic waves in air is something 53 00:02:23,740 --> 00:02:27,010 you see a lot of when you come to the later labs-- 54 00:02:27,010 --> 00:02:30,520 indentations on vinyl or plastic. 55 00:02:30,520 --> 00:02:35,380 Let's see, that's records or CD's. 56 00:02:35,380 --> 00:02:36,880 And that actually brings up a point. 57 00:02:36,880 --> 00:02:39,430 We don't often think of storage as being communication, 58 00:02:39,430 --> 00:02:42,280 but storage is really communication 59 00:02:42,280 --> 00:02:45,430 with potentially a very long time delay in the channel. 60 00:02:45,430 --> 00:02:47,980 You put something on the storage medium, 61 00:02:47,980 --> 00:02:51,660 and then weeks or months or years or centuries later, 62 00:02:51,660 --> 00:02:53,410 you're trying to extract that information. 63 00:02:53,410 --> 00:02:57,520 So we can still think of all of this as a communication 64 00:02:57,520 --> 00:02:59,320 channel, and indeed, decoding ideas 65 00:02:59,320 --> 00:03:03,190 are essential to making CDS work, to having 66 00:03:03,190 --> 00:03:06,520 a CD resistant to scratches and thumbprints, and everything 67 00:03:06,520 --> 00:03:08,470 else that, all the other indignities 68 00:03:08,470 --> 00:03:10,300 that they're subject to-- 69 00:03:10,300 --> 00:03:11,330 magnetization. 70 00:03:11,330 --> 00:03:13,810 So all of these physical modalities 71 00:03:13,810 --> 00:03:17,960 that are used to translate information. 72 00:03:17,960 --> 00:03:20,120 Here's one you may not have thought about, 73 00:03:20,120 --> 00:03:22,620 mud pulse telemetry. 74 00:03:22,620 --> 00:03:27,740 So when you're drilling for oil, you'd 75 00:03:27,740 --> 00:03:31,490 like to get information from the drill bit at the bottom. 76 00:03:31,490 --> 00:03:33,590 And normal electronics doesn't work 77 00:03:33,590 --> 00:03:36,110 too well, because the temperatures are fiercely hot 78 00:03:36,110 --> 00:03:37,400 down there. 79 00:03:37,400 --> 00:03:39,890 You need that information to help you steer the drill bit, 80 00:03:39,890 --> 00:03:41,932 to get information about what sort of rock you're 81 00:03:41,932 --> 00:03:43,710 going through, and all of that. 82 00:03:43,710 --> 00:03:48,110 So they actually seriously do use pressure pulses 83 00:03:48,110 --> 00:03:52,130 and in the slurry that's cooling the drill bit to try and convey 84 00:03:52,130 --> 00:03:54,180 information back to the top. 85 00:03:54,180 --> 00:03:57,833 So they'll modulate the pressure down 86 00:03:57,833 --> 00:03:59,750 at the end of the drill bit, and hope that you 87 00:03:59,750 --> 00:04:02,600 can detect it at the top. 88 00:04:02,600 --> 00:04:05,360 One word over there that stands out 89 00:04:05,360 --> 00:04:07,920 is they talk about digital information. 90 00:04:07,920 --> 00:04:11,330 So even in the context of communicating through mud, 91 00:04:11,330 --> 00:04:12,920 they're thinking about how to actually 92 00:04:12,920 --> 00:04:16,273 have bits that they communicate on this analog channel. 93 00:04:16,273 --> 00:04:17,690 So this is very much in the flavor 94 00:04:17,690 --> 00:04:19,410 of what we're trying to do. 95 00:04:19,410 --> 00:04:22,130 We're trying to communicate digital information, 96 00:04:22,130 --> 00:04:25,760 this is sequences of numbers or sequences of signs or symbols, 97 00:04:25,760 --> 00:04:28,010 but we're trying to do it over a physical channel that 98 00:04:28,010 --> 00:04:32,270 takes continuous valued, continuous time, waveforms. 99 00:04:32,270 --> 00:04:36,650 So that's really some of what we'll be talking about. 100 00:04:36,650 --> 00:04:44,020 So the kind of link that we have starts off with bits. 101 00:04:44,020 --> 00:04:47,140 But in the middle, has to deal with the physical link 102 00:04:47,140 --> 00:04:48,340 on which you have signals. 103 00:04:48,340 --> 00:04:50,542 We'll refer to these continuous time waveforms. 104 00:04:50,542 --> 00:04:52,000 They're not always continuous time. 105 00:04:52,000 --> 00:04:54,380 You could sample them and get discrete time waveforms, 106 00:04:54,380 --> 00:04:54,880 as well. 107 00:04:54,880 --> 00:04:57,310 But the signals that you see in the physical medium 108 00:04:57,310 --> 00:04:58,300 will refer to-- 109 00:04:58,300 --> 00:05:00,175 the quantities you see in the physical medium 110 00:05:00,175 --> 00:05:01,420 will refer to as signals. 111 00:05:01,420 --> 00:05:04,492 You need some way to map the bits to the signals. 112 00:05:04,492 --> 00:05:05,950 You've got a bit sequence, you need 113 00:05:05,950 --> 00:05:09,100 to convert it to a continuous time waveform in some fashion. 114 00:05:09,100 --> 00:05:12,470 And then at the other end, to recover the bitstream. 115 00:05:12,470 --> 00:05:15,070 And you might do that by some sampling and processing, 116 00:05:15,070 --> 00:05:20,812 and then a translation process back. 117 00:05:20,812 --> 00:05:22,270 The little lightning there is meant 118 00:05:22,270 --> 00:05:25,165 to suggest that you're subject to all sorts of noise 119 00:05:25,165 --> 00:05:27,290 and disturbances when you're on that physical link. 120 00:05:27,290 --> 00:05:30,220 So that's something that's critical to the design 121 00:05:30,220 --> 00:05:30,890 of the system. 122 00:05:30,890 --> 00:05:32,480 You have to design your overall system 123 00:05:32,480 --> 00:05:37,660 so that your robust to perturbations in that middle 124 00:05:37,660 --> 00:05:40,240 section. 125 00:05:40,240 --> 00:05:46,450 So the particular application is dealing with your specifics 126 00:05:46,450 --> 00:05:49,450 and with producing a bitstream that's 127 00:05:49,450 --> 00:05:52,060 got the redundancy mapped out of it. 128 00:05:52,060 --> 00:05:54,850 At this point, it doesn't matter to me 129 00:05:54,850 --> 00:05:57,980 for communication across the channel what that bitstream is 130 00:05:57,980 --> 00:05:59,138 or where it came from. 131 00:05:59,138 --> 00:06:01,180 I'm just trying to do a good job of delivering it 132 00:06:01,180 --> 00:06:05,890 to the other end where the user can extract that. 133 00:06:05,890 --> 00:06:07,930 Now here's the funny thing. 134 00:06:07,930 --> 00:06:11,740 We've just done a lot of work to extract redundancy 135 00:06:11,740 --> 00:06:14,420 from the messages here. 136 00:06:14,420 --> 00:06:16,990 We're going to put redundancy back in. 137 00:06:16,990 --> 00:06:19,960 Because the way you guard against disturbances 138 00:06:19,960 --> 00:06:22,970 in the channel is by introducing redundancy. 139 00:06:22,970 --> 00:06:24,970 You need to give yourself a little room 140 00:06:24,970 --> 00:06:29,233 to recognize that something bad has happened to your signal, 141 00:06:29,233 --> 00:06:31,900 or something bad has happened to the data you're sending across, 142 00:06:31,900 --> 00:06:33,490 and then to recover from it. 143 00:06:33,490 --> 00:06:35,290 So we will actually be talking about how 144 00:06:35,290 --> 00:06:39,280 to reintroduce redundancy, but this is introducing it now 145 00:06:39,280 --> 00:06:42,190 in a bitstream where the binary digits are essentially equally 146 00:06:42,190 --> 00:06:43,570 likely. 147 00:06:43,570 --> 00:06:46,870 You pulled out all the application-specific knowledge, 148 00:06:46,870 --> 00:06:48,820 and used it to do the source coding. 149 00:06:48,820 --> 00:06:51,670 Now you've just got a stream of zeros and ones, 150 00:06:51,670 --> 00:06:55,180 each one carrying a bit of information, presumably. 151 00:06:55,180 --> 00:06:57,122 And now you want to protect it for transport 152 00:06:57,122 --> 00:06:59,080 across the channel, and you're going to do that 153 00:06:59,080 --> 00:07:01,630 by putting in additional bits. 154 00:07:01,630 --> 00:07:04,780 So you're going to introduce bits in a structured way 155 00:07:04,780 --> 00:07:06,110 to provide some redundancy. 156 00:07:06,110 --> 00:07:09,920 So we'll be talking about that in more detail. 157 00:07:09,920 --> 00:07:13,880 So this is the single link picture. 158 00:07:13,880 --> 00:07:17,310 When you've got a network, it's a little different. 159 00:07:17,310 --> 00:07:19,910 So I haven't shown all the other links here, 160 00:07:19,910 --> 00:07:23,840 but you should imagine a network with many possible paths 161 00:07:23,840 --> 00:07:26,870 from a source to a destination. 162 00:07:26,870 --> 00:07:28,370 Some of these links might come down, 163 00:07:28,370 --> 00:07:30,403 and so you'll want to find an alternative path. 164 00:07:30,403 --> 00:07:31,820 Some of these links might be busy, 165 00:07:31,820 --> 00:07:33,720 and you want to find an alternative path. 166 00:07:33,720 --> 00:07:36,880 So there's a whole network of links in here. 167 00:07:36,880 --> 00:07:41,590 In this setting, it turns out that the good way to do this 168 00:07:41,590 --> 00:07:45,790 is to break up your bitstream into what are called packets. 169 00:07:45,790 --> 00:07:50,950 These are maybe 1,000 bits or 4,000 bits, 170 00:07:50,950 --> 00:07:53,530 or whatever your protocol is. 171 00:07:53,530 --> 00:07:58,840 They're broken up into packets of chunks of bits, 172 00:07:58,840 --> 00:08:02,100 and then treated as packets for transport along the network. 173 00:08:02,100 --> 00:08:04,833 So the point is that one packet to a given user 174 00:08:04,833 --> 00:08:06,250 might travel one particular route, 175 00:08:06,250 --> 00:08:08,260 but another packet might travel another route. 176 00:08:08,260 --> 00:08:12,080 And then these get reassembled at the destination. 177 00:08:12,080 --> 00:08:13,720 So you think in terms of packets when 178 00:08:13,720 --> 00:08:16,220 you think in terms of rooting on the network. 179 00:08:16,220 --> 00:08:19,130 This idea of packet communication, by the way, 180 00:08:19,130 --> 00:08:20,630 there's a name associated with that, 181 00:08:20,630 --> 00:08:23,580 which is Kleinrock, again, a PhD student at MIT 182 00:08:23,580 --> 00:08:28,270 in the same golden years that I keep referring back to. 183 00:08:28,270 --> 00:08:31,190 But it's a very broad area. 184 00:08:31,190 --> 00:08:33,100 So you've got the packets. 185 00:08:33,100 --> 00:08:36,610 Those arrive at the links. 186 00:08:36,610 --> 00:08:38,302 They're actually switches here that 187 00:08:38,302 --> 00:08:40,510 decide which of the links emanating from the switches 188 00:08:40,510 --> 00:08:42,070 you want to send the packet on. 189 00:08:42,070 --> 00:08:43,630 So, again, imagine all the links. 190 00:08:43,630 --> 00:08:45,230 I haven't drawn them in. 191 00:08:45,230 --> 00:08:49,390 So the packet gets treated as a unit for shipping on the links. 192 00:08:49,390 --> 00:08:52,600 But once you've committed to a particular link, 193 00:08:52,600 --> 00:08:55,900 it's like transmitting on a single link, again. 194 00:08:55,900 --> 00:09:00,780 So you've got to go through your packets 195 00:09:00,780 --> 00:09:06,060 to bits to signal to bits to packets transformation. 196 00:09:06,060 --> 00:09:08,400 So it's not that there's a particular transformation 197 00:09:08,400 --> 00:09:09,660 in going from packets to bits, you're 198 00:09:09,660 --> 00:09:10,860 just viewing it differently. 199 00:09:10,860 --> 00:09:14,320 You're not treating it as a packet, 200 00:09:14,320 --> 00:09:15,570 you're looking in on each bit. 201 00:09:15,570 --> 00:09:19,020 You're looking to code each bit on an analog signal, 202 00:09:19,020 --> 00:09:20,490 get it across the physical medium. 203 00:09:23,420 --> 00:09:25,730 So that's really the key to this. 204 00:09:25,730 --> 00:09:29,300 What we end up doing is coding, or mapping, 205 00:09:29,300 --> 00:09:32,150 or the word modulating is used, and we'll see more of that. 206 00:09:32,150 --> 00:09:35,220 We modulate the desired sequence onto a continuous time 207 00:09:35,220 --> 00:09:35,970 waveform. 208 00:09:35,970 --> 00:09:43,100 So what you might imagine is, you could have a sequence 209 00:09:43,100 --> 00:09:47,720 01101, and what you're going to do with your mapping is try 210 00:09:47,720 --> 00:09:50,180 and generate a continuous time waveform, 211 00:09:50,180 --> 00:09:52,850 which in some fashion codes that sequence. 212 00:09:52,850 --> 00:09:54,030 And it could be very simple. 213 00:09:54,030 --> 00:09:57,230 It could be a voltage level of 0, 214 00:09:57,230 --> 00:10:01,040 held for some interval of time to represent the 1-- 215 00:10:01,040 --> 00:10:05,510 sorry, to represent the 0, held for some time 216 00:10:05,510 --> 00:10:12,140 again to represent the second symbol, which is, again, a 1. 217 00:10:12,140 --> 00:10:16,160 And then you come back down to 0, back to 1 again. 218 00:10:16,160 --> 00:10:17,570 So it could be as simple as this. 219 00:10:20,860 --> 00:10:22,510 OK, so this is now-- 220 00:10:22,510 --> 00:10:25,240 you can think of it as some voltage. 221 00:10:25,240 --> 00:10:27,310 We use the word voltage a lot, but we just 222 00:10:27,310 --> 00:10:28,420 mean an analog signal. 223 00:10:28,420 --> 00:10:29,470 We'll use the word voltage, we're 224 00:10:29,470 --> 00:10:31,053 thinking of voltage on a cable, but it 225 00:10:31,053 --> 00:10:32,780 could be any analog signal. 226 00:10:32,780 --> 00:10:36,940 So this is really the digital signaling end of it. 227 00:10:36,940 --> 00:10:40,660 So you take the bit sequence represented now on, 228 00:10:40,660 --> 00:10:43,030 coded onto a continuous time waveform, 229 00:10:43,030 --> 00:10:45,640 or modulated onto a continuous time waveform. 230 00:10:45,640 --> 00:10:50,530 We'll see richer uses of that word as we progress. 231 00:10:50,530 --> 00:10:52,150 The particular scheme I've shown here 232 00:10:52,150 --> 00:10:55,750 is what you'd call a bi-level signaling scheme, the two 233 00:10:55,750 --> 00:10:57,920 voltage levels that you use. 234 00:10:57,920 --> 00:11:01,180 We refer to that also as bipolar signaling, 235 00:11:01,180 --> 00:11:03,250 although sometimes that's restricted 236 00:11:03,250 --> 00:11:06,880 to the case where the two voltage levels are opposite 237 00:11:06,880 --> 00:11:07,870 in sign. 238 00:11:07,870 --> 00:11:10,360 So you can imagine a signaling scheme that 239 00:11:10,360 --> 00:11:12,970 uses this for 0, this for 1. 240 00:11:16,100 --> 00:11:17,990 So this could be a bipolar scheme. 241 00:11:23,130 --> 00:11:24,960 And then this continuous time waveform 242 00:11:24,960 --> 00:11:27,270 gets put on the physical channel, presumably 243 00:11:27,270 --> 00:11:32,590 gets distorted, it gets some noise added to it. 244 00:11:32,590 --> 00:11:35,370 So at the other end, you get some approximation to this. 245 00:11:35,370 --> 00:11:38,190 And then you've got to reconstruct the bit sequence. 246 00:11:38,190 --> 00:11:40,920 You might do that by sampling this and processing 247 00:11:40,920 --> 00:11:44,490 the samples, and then taking some measure of the waveform. 248 00:11:44,490 --> 00:11:47,580 We'll see more of that later, you'll do that in lab 2, 249 00:11:47,580 --> 00:11:50,680 in labs, as well. 250 00:11:50,680 --> 00:11:53,400 So in some fashion, you recover from this 251 00:11:53,400 --> 00:11:55,002 your estimate of the bit sequence. 252 00:11:55,002 --> 00:11:57,210 But you can imagine, the amount of waveform like this 253 00:11:57,210 --> 00:12:00,450 goes through a physical channel with distortion, noise, 254 00:12:00,450 --> 00:12:02,110 and then re-sampling and processing 255 00:12:02,110 --> 00:12:05,370 and so on, you are likely to get errors back 256 00:12:05,370 --> 00:12:07,860 at the receiving end. 257 00:12:07,860 --> 00:12:10,920 Ideally, you would get exactly this waveform at the receiving 258 00:12:10,920 --> 00:12:13,050 end, you'd have no trouble distinguishing 259 00:12:13,050 --> 00:12:15,113 between the samples of the two levels, 260 00:12:15,113 --> 00:12:16,905 and you could reconstruct the bit sequence. 261 00:12:25,190 --> 00:12:28,580 Now we are going to, in the middle section of this course, 262 00:12:28,580 --> 00:12:32,040 say a lot more about the signals aspect of it. 263 00:12:32,040 --> 00:12:34,610 But for us now, it's actually helpful to stick 264 00:12:34,610 --> 00:12:36,200 to thinking in terms of bits. 265 00:12:45,030 --> 00:12:48,620 So we've got bits in, bits out. 266 00:12:48,620 --> 00:12:50,510 Somewhere in here, we've got signals 267 00:12:50,510 --> 00:12:55,760 in the physical channel, signals, noise, 268 00:12:55,760 --> 00:12:56,720 physical channel. 269 00:13:01,540 --> 00:13:03,040 And then we've got whatever it is 270 00:13:03,040 --> 00:13:07,390 that does the transformation from bits to-- 271 00:13:07,390 --> 00:13:09,340 so this is some kind of a transformer, 272 00:13:09,340 --> 00:13:12,460 let's say, from bits to signals and from signals to bits. 273 00:13:15,620 --> 00:13:18,810 But let's look at an abstraction that's end to end here, 274 00:13:18,810 --> 00:13:19,910 bits to bits. 275 00:13:19,910 --> 00:13:23,120 OK what's coming in as a bitstream, what you're 276 00:13:23,120 --> 00:13:26,600 receiving is a bitstream. 277 00:13:26,600 --> 00:13:29,870 There's an idealization of this channel that's used a lot, 278 00:13:29,870 --> 00:13:32,510 and that's referred to as the binary symmetric channel. 279 00:13:32,510 --> 00:13:37,530 And what that says is, you got a 1 coming in, 280 00:13:37,530 --> 00:13:40,560 that's most likely going to be received as 1, 281 00:13:40,560 --> 00:13:45,600 but there's some chance it's going to be received as a 0. 282 00:13:45,600 --> 00:13:48,210 And let's put probabilities here. 283 00:13:48,210 --> 00:13:49,560 I think the notes use epsilon. 284 00:13:49,560 --> 00:13:53,943 I seem to have used p, little p, on my slide, 285 00:13:53,943 --> 00:13:55,110 so let me stick to little p. 286 00:13:57,780 --> 00:14:03,810 So a 1 coming in is transformed in error 287 00:14:03,810 --> 00:14:05,850 to a 0 with some small probability, 288 00:14:05,850 --> 00:14:09,060 presumably small, with 1 minus p it's intact, 289 00:14:09,060 --> 00:14:11,520 and then the same thing on the side. 290 00:14:11,520 --> 00:14:18,030 A 0 comes in with some probability, comes out as a 0, 291 00:14:18,030 --> 00:14:24,190 but with some probability it actually gets flipped to 1. 292 00:14:24,190 --> 00:14:25,750 Now, we use the word symmetric here 293 00:14:25,750 --> 00:14:28,810 to say that we're assuming identical probabilities 294 00:14:28,810 --> 00:14:30,640 of going 1 to 0, 0 to 1. 295 00:14:30,640 --> 00:14:32,708 You can easily imagine an unsymmetrical channel 296 00:14:32,708 --> 00:14:34,750 where you have different probabilities in the two 297 00:14:34,750 --> 00:14:37,900 directions, but we'll stick to the symmetric case. 298 00:14:37,900 --> 00:14:40,990 Binary, of course, because we're dealing with binary sequences 299 00:14:40,990 --> 00:14:44,020 of the two ends. 300 00:14:44,020 --> 00:14:46,870 And what we're imagining is that this is a memory-less channel. 301 00:14:46,870 --> 00:14:48,910 In other words, I can look at this transmission 302 00:14:48,910 --> 00:14:49,700 by transmission. 303 00:14:49,700 --> 00:14:53,512 So a bit comes in, this is how the output is determined. 304 00:14:53,512 --> 00:14:55,720 And then the next bit comes in, and it knows nothing. 305 00:14:55,720 --> 00:14:57,190 There's no memory in the system, it 306 00:14:57,190 --> 00:14:58,565 knows nothing about what decision 307 00:14:58,565 --> 00:15:00,045 was made in the previous case. 308 00:15:00,045 --> 00:15:02,170 Now you can imagine more complicated channel models 309 00:15:02,170 --> 00:15:05,675 with memory, but this is a good starting point. 310 00:15:05,675 --> 00:15:07,300 So that's the binary symmetric channel. 311 00:15:13,570 --> 00:15:21,130 So question now, if we wanted to get a bitstream over reliably, 312 00:15:21,130 --> 00:15:23,560 any ideas on how we can counteract 313 00:15:23,560 --> 00:15:27,890 the effect of this p, probability of flipping? 314 00:15:27,890 --> 00:15:28,693 Yeah? 315 00:15:28,693 --> 00:15:34,130 AUDIENCE: You could have like a range [INAUDIBLE].. 316 00:15:34,130 --> 00:15:35,630 GEORGE VERGHESE: Well, at this point 317 00:15:35,630 --> 00:15:37,820 I'm back to the ones and zeros. 318 00:15:37,820 --> 00:15:38,660 There's no signals. 319 00:15:38,660 --> 00:15:39,990 The signals are in here. 320 00:15:39,990 --> 00:15:42,110 So what you're thinking of maybe is, 321 00:15:42,110 --> 00:15:44,590 how do I reliably map bits to signals? 322 00:15:44,590 --> 00:15:48,170 And what you're saying is, you can design your signaling here 323 00:15:48,170 --> 00:15:51,650 in a way that reduces the p. 324 00:15:51,650 --> 00:15:53,450 The p that I'm thinking of here is the end 325 00:15:53,450 --> 00:15:55,310 to end error probability. 326 00:15:55,310 --> 00:15:59,360 If I designed the inner part better, I might lower the p. 327 00:15:59,360 --> 00:16:01,100 But for a given p, is there something 328 00:16:01,100 --> 00:16:04,310 that I could be doing to improve my chances of getting 329 00:16:04,310 --> 00:16:05,510 the bit across correctly? 330 00:16:05,510 --> 00:16:06,290 Yeah? 331 00:16:06,290 --> 00:16:09,987 AUDIENCE: [INAUDIBLE] 332 00:16:09,987 --> 00:16:11,570 GEORGE VERGHESE: OK, so the suggestion 333 00:16:11,570 --> 00:16:14,810 is that we introduce redundancy by just repeating it. 334 00:16:14,810 --> 00:16:16,820 So send the 1, repeat the 1. 335 00:16:16,820 --> 00:16:19,320 Repeat it many times if you need to. 336 00:16:19,320 --> 00:16:21,590 And so what would you suggest that the receiver should 337 00:16:21,590 --> 00:16:24,750 do if you do a repetition like this? 338 00:16:24,750 --> 00:16:27,860 How should the receiver decide? 339 00:16:27,860 --> 00:16:30,080 If I send five ones in a row-- yeah? 340 00:16:30,080 --> 00:16:33,146 AUDIENCE: [INAUDIBLE] 341 00:16:33,146 --> 00:16:35,322 GEORGE VERGHESE: I'm sorry, say that again? 342 00:16:35,322 --> 00:16:39,741 AUDIENCE: [INAUDIBLE] 343 00:16:42,405 --> 00:16:44,530 GEORGE VERGHESE: You're talked about requesting a-- 344 00:16:44,530 --> 00:16:45,822 I'm not hearing well what you-- 345 00:16:45,822 --> 00:16:49,626 AUDIENCE: [INAUDIBLE] 346 00:16:55,520 --> 00:16:57,770 GEORGE VERGHESE: OK, so you're using the word request. 347 00:16:57,770 --> 00:16:59,180 You're talking about the receiver 348 00:16:59,180 --> 00:17:02,450 sending something back to the sender. 349 00:17:02,450 --> 00:17:05,510 But if we're with this channel and the sender 350 00:17:05,510 --> 00:17:07,310 has to make its own decisions about how 351 00:17:07,310 --> 00:17:10,394 to get things across without a possibility of feedback. 352 00:17:10,394 --> 00:17:14,186 AUDIENCE: [INAUDIBLE] 353 00:17:18,768 --> 00:17:21,060 GEORGE VERGHESE: OK, so I think I understand a bit now. 354 00:17:21,060 --> 00:17:24,710 So if you're repeating this, the chance of more than half 355 00:17:24,710 --> 00:17:26,720 of them being wrong is very small. 356 00:17:26,720 --> 00:17:28,860 I think that was the idea that you had, as well. 357 00:17:28,860 --> 00:17:31,897 So repetition is likely to reduce the chances 358 00:17:31,897 --> 00:17:33,980 that you go wrong here if you use a majority rule. 359 00:17:33,980 --> 00:17:36,990 Majority rule would be a simple rule. 360 00:17:36,990 --> 00:17:43,250 Send five repetitions, and if only two are flipped, well, 361 00:17:43,250 --> 00:17:45,110 you just decide in favor of the majority. 362 00:17:45,110 --> 00:17:48,320 Because it's more likely that none or one or two are 363 00:17:48,320 --> 00:17:51,818 flipped than that three or four or five are flipped. 364 00:17:51,818 --> 00:17:52,610 So that's the idea. 365 00:17:52,610 --> 00:17:55,850 So this is what's called a replication code, 366 00:17:55,850 --> 00:17:58,450 and actually, it can work very well. 367 00:18:01,120 --> 00:18:05,310 So what you see on the horizontal axis 368 00:18:05,310 --> 00:18:07,020 is the replication factor. 369 00:18:07,020 --> 00:18:10,848 You can replicate it 5 times, 10 times, 15 times, 370 00:18:10,848 --> 00:18:12,390 and here is the probability of error. 371 00:18:12,390 --> 00:18:14,098 And it actually goes down. 372 00:18:14,098 --> 00:18:15,390 And you can do the computation. 373 00:18:15,390 --> 00:18:19,380 This is actually a fairly simple computation, 374 00:18:19,380 --> 00:18:21,000 you're basically doing coin tosses 375 00:18:21,000 --> 00:18:23,208 and seeing what's the probability that more than half 376 00:18:23,208 --> 00:18:28,180 of the coins I flip come up one way. 377 00:18:28,180 --> 00:18:30,550 And you're counting that to decide 378 00:18:30,550 --> 00:18:32,450 how the majority rule works. 379 00:18:32,450 --> 00:18:36,620 I'm sorry, for the epsilon here, it's supposed to be the p. 380 00:18:36,620 --> 00:18:39,290 OK, so is this good? 381 00:18:39,290 --> 00:18:42,030 Good enough? 382 00:18:42,030 --> 00:18:43,378 Yeah? 383 00:18:43,378 --> 00:18:47,210 AUDIENCE: [INAUDIBLE] 384 00:18:52,172 --> 00:18:53,630 GEORGE VERGHESE: The more you send, 385 00:18:53,630 --> 00:18:57,440 the more you're wasting time on that one bit. 386 00:18:57,440 --> 00:18:59,960 So this is the point, that you can do the replication 387 00:18:59,960 --> 00:19:01,700 and reduce your probability of error, 388 00:19:01,700 --> 00:19:03,742 but what's the information you're getting across? 389 00:19:03,742 --> 00:19:07,460 You're doing all of this to get that one bit across reliably, 390 00:19:07,460 --> 00:19:09,680 but you've got a lot of bits backing up, 391 00:19:09,680 --> 00:19:10,640 waiting to get across. 392 00:19:10,640 --> 00:19:16,280 So the code rate, the rate at which information gets across 393 00:19:16,280 --> 00:19:19,550 is a 1 over n if you're doing n replications. 394 00:19:19,550 --> 00:19:22,380 If you're doing n replications, it's only 1 over n. 395 00:19:24,920 --> 00:19:28,040 The rate at which you're getting information across in terms 396 00:19:28,040 --> 00:19:30,437 of bits is 1 over n. 397 00:19:30,437 --> 00:19:32,270 So you're dropping the probability of error, 398 00:19:32,270 --> 00:19:35,230 but you're also dropping the transmission rate. 399 00:19:35,230 --> 00:19:37,820 So this is really unacceptable. 400 00:19:37,820 --> 00:19:41,030 It turns out, though, that we can do a lot better. 401 00:19:41,030 --> 00:19:44,490 We can do a lot better. 402 00:19:44,490 --> 00:19:46,850 What I'm going to do now is say a little bit 403 00:19:46,850 --> 00:19:49,430 about what Shannon had to say about it. 404 00:19:49,430 --> 00:19:53,340 I hope you'll allow me to teach you about something that we're 405 00:19:53,340 --> 00:19:55,357 not going to test you on just so you can learn 406 00:19:55,357 --> 00:19:57,690 a little bit about this, and then I'll get back to stuff 407 00:19:57,690 --> 00:19:58,770 that we will test you on. 408 00:19:58,770 --> 00:20:00,810 OK, is that all right? 409 00:20:00,810 --> 00:20:06,490 I know you didn't pay for this, but we'll do it anyway. 410 00:20:06,490 --> 00:20:09,130 So here's Shannon, defining something 411 00:20:09,130 --> 00:20:11,800 that the thermodynamics people and so on 412 00:20:11,800 --> 00:20:12,915 didn't really think to do. 413 00:20:12,915 --> 00:20:14,290 They may have done it indirectly, 414 00:20:14,290 --> 00:20:16,960 but it didn't arise in where they were working 415 00:20:16,960 --> 00:20:18,520 with entropy, and all of that. 416 00:20:18,520 --> 00:20:22,360 Shannon defined something called a mutual information, 417 00:20:22,360 --> 00:20:24,760 given by this symbol. 418 00:20:24,760 --> 00:20:28,480 X and Y are random variables, random quantities. 419 00:20:28,480 --> 00:20:31,640 What we know about H of x is the entropy in X, 420 00:20:31,640 --> 00:20:33,940 so it's our uncertainty about X. It's 421 00:20:33,940 --> 00:20:39,130 the expected information when you're told something about X. 422 00:20:39,130 --> 00:20:40,810 This symbol denotes the uncertainty 423 00:20:40,810 --> 00:20:46,180 in X, given information about Y. So it's the conditional entropy 424 00:20:46,180 --> 00:20:47,240 here. 425 00:20:47,240 --> 00:20:49,090 And so what this is asking is, how much is 426 00:20:49,090 --> 00:20:53,578 our uncertainty about X reduced by having information, having 427 00:20:53,578 --> 00:20:54,370 a measurement of Y? 428 00:20:57,060 --> 00:20:59,970 That's very relevant to a channel where the input is X 429 00:20:59,970 --> 00:21:01,830 and the output is Y. We're saying, 430 00:21:01,830 --> 00:21:03,450 we see the output of the channel, 431 00:21:03,450 --> 00:21:06,270 we want to infer what happened to the input, what 432 00:21:06,270 --> 00:21:07,915 the input sent. 433 00:21:07,915 --> 00:21:10,290 The mutual information between these two random variables 434 00:21:10,290 --> 00:21:11,560 surely has to be important. 435 00:21:11,560 --> 00:21:14,040 So what's the reduction and uncertainty 436 00:21:14,040 --> 00:21:17,280 that results from having a measurement of Y? 437 00:21:17,280 --> 00:21:20,010 That's a question of interest not just 438 00:21:20,010 --> 00:21:22,290 in communications as such, but in all sorts 439 00:21:22,290 --> 00:21:24,090 of inference questions. 440 00:21:24,090 --> 00:21:27,098 OK, I'm going to have a slide of equations. 441 00:21:27,098 --> 00:21:28,890 They might look scary, but actually they're 442 00:21:28,890 --> 00:21:31,740 very simple, given what you already know how to do. 443 00:21:31,740 --> 00:21:33,240 First, I have to define for you what 444 00:21:33,240 --> 00:21:35,770 I mean by conditional entropy. 445 00:21:35,770 --> 00:21:39,100 So I'm saying it's the entropy of X conditioned 446 00:21:39,100 --> 00:21:41,200 on having a particular measurement of Y. 447 00:21:41,200 --> 00:21:46,010 So suppose you know that Y takes the value of little y sub j. 448 00:21:46,010 --> 00:21:48,470 You use the same formula that we've used for entropy, 449 00:21:48,470 --> 00:21:50,810 except your probabilities are all now 450 00:21:50,810 --> 00:21:53,430 probabilities conditioned on that information. 451 00:21:53,430 --> 00:21:57,580 So instead of just p of xi, you have p of xi given yj. 452 00:21:57,580 --> 00:21:59,120 So it's the same formula. 453 00:21:59,120 --> 00:22:00,920 But if you've been given information, 454 00:22:00,920 --> 00:22:03,440 then you have to condition on it. 455 00:22:03,440 --> 00:22:05,750 So that's the definition. 456 00:22:05,750 --> 00:22:09,840 This is the conditional entropy given a specific value for y. 457 00:22:09,840 --> 00:22:12,110 But if all I tell you is I'm going to be giving you 458 00:22:12,110 --> 00:22:15,320 a value for Y and I haven't told you the value yet, 459 00:22:15,320 --> 00:22:16,700 what's the conditional entropy? 460 00:22:16,700 --> 00:22:19,250 Then what you want to do is average over all possible Y's 461 00:22:19,250 --> 00:22:20,520 that you might get. 462 00:22:20,520 --> 00:22:22,520 So you're going to take this conditional entropy 463 00:22:22,520 --> 00:22:25,250 for the given Y, and then take the weighted average 464 00:22:25,250 --> 00:22:26,673 of the probabilities. 465 00:22:26,673 --> 00:22:28,340 So that's how you compute this quantity, 466 00:22:28,340 --> 00:22:29,788 and it's quite straightforward. 467 00:22:29,788 --> 00:22:31,580 It's not very different from what you have. 468 00:22:34,310 --> 00:22:37,340 And then if you put in what you know about how joint 469 00:22:37,340 --> 00:22:39,600 probabilities and conditional probabilities worked-- 470 00:22:39,600 --> 00:22:42,440 this was the definition of conditional probability that we 471 00:22:42,440 --> 00:22:45,380 had in, I think, the first lecture-- 472 00:22:45,380 --> 00:22:51,260 you discover that actually the joint entropy of these two 473 00:22:51,260 --> 00:22:53,750 random variables can be factored in two particular ways. 474 00:22:53,750 --> 00:22:57,440 And that allows you to deduce that the mutual information is 475 00:22:57,440 --> 00:22:57,950 symmetric. 476 00:22:57,950 --> 00:23:01,100 In other words, the mutual information between X and Y 477 00:23:01,100 --> 00:23:03,740 is the same as the mutual information between Y an X. 478 00:23:03,740 --> 00:23:05,875 So there's no difference in that. 479 00:23:05,875 --> 00:23:07,250 That might be a little surprising 480 00:23:07,250 --> 00:23:10,400 given that we were thinking of X as the input to the channel 481 00:23:10,400 --> 00:23:11,880 and Y is the output of the channel, 482 00:23:11,880 --> 00:23:13,463 but it turns out that that's the case. 483 00:23:17,363 --> 00:23:19,780 So let's actually compute it for the channel that we know. 484 00:23:19,780 --> 00:23:22,210 This is the binary symmetric channel. 485 00:23:22,210 --> 00:23:25,640 Let's compute the mutual information 486 00:23:25,640 --> 00:23:29,520 between the input and output for the binary symmetric channel. 487 00:23:29,520 --> 00:23:32,645 So here is the definition IXY. 488 00:23:32,645 --> 00:23:35,270 We've just shown that it doesn't matter which order you take it 489 00:23:35,270 --> 00:23:37,370 in, and it turns out the computation is easier 490 00:23:37,370 --> 00:23:38,360 if you flip the order. 491 00:23:38,360 --> 00:23:40,730 So I'm going to write this as uncertainty 492 00:23:40,730 --> 00:23:44,540 in Y minus the uncertainty in Y given the measurement of H-- 493 00:23:44,540 --> 00:23:47,600 of X, sorry. 494 00:23:47,600 --> 00:23:51,200 So I'm going to compute it in that fashion. 495 00:23:51,200 --> 00:23:52,850 What's the uncertainty in Y? 496 00:24:05,400 --> 00:24:07,740 Actually, I should probably have said here 497 00:24:07,740 --> 00:24:11,460 that, let's assume X takes 0 and 1-- 498 00:24:11,460 --> 00:24:13,380 I might be wrong in saying that this 499 00:24:13,380 --> 00:24:16,170 doesn't depend on the distribution of the input. 500 00:24:16,170 --> 00:24:21,750 Let's assume 0 and 1 are equally likely at the input. 501 00:24:21,750 --> 00:24:24,360 If the 0 and 1 are equally likely at the input, 502 00:24:24,360 --> 00:24:27,170 what's the uncertainty in Y? 503 00:24:31,630 --> 00:24:33,450 There's a little bit of uncertainty. 504 00:24:33,450 --> 00:24:36,030 It's equally likely to be a 0 or 1. 505 00:24:36,030 --> 00:24:38,610 I had actually written that assumption in, 506 00:24:38,610 --> 00:24:42,000 and then I took it out, but I think I'm wrong in saying that. 507 00:24:42,000 --> 00:24:44,700 So here we have the 1 for the uncertainty in Y. 508 00:24:44,700 --> 00:24:48,060 What about the uncertainty in Y given X? 509 00:24:48,060 --> 00:24:50,220 So I give you a particular value for X. Let's 510 00:24:50,220 --> 00:24:52,820 say X is equal to 1. 511 00:24:52,820 --> 00:24:55,430 So I give you a particular value for X. 512 00:24:55,430 --> 00:24:59,220 What's the uncertainty in Y? 513 00:24:59,220 --> 00:25:03,060 Well, Y is 0 with probability little p, 514 00:25:03,060 --> 00:25:06,720 and it's 1 with probability 1 minus p. 515 00:25:06,720 --> 00:25:08,790 And that's really the binary entropy function 516 00:25:08,790 --> 00:25:10,540 that we had drawn last time. 517 00:25:10,540 --> 00:25:13,900 So you can actually work out all these pieces 518 00:25:13,900 --> 00:25:17,020 and discover-- let's see, here is the binary entropy function. 519 00:25:17,020 --> 00:25:19,840 Just to remind you, this is for a coin toss. 520 00:25:19,840 --> 00:25:22,330 If something can be 1 with probability p, 0 521 00:25:22,330 --> 00:25:25,120 with probability 1 minus p or the other way around, 522 00:25:25,120 --> 00:25:30,430 the entropy associated with that is H of p. 523 00:25:30,430 --> 00:25:34,030 And we have 1 minus H of p for the mutual information 524 00:25:34,030 --> 00:25:36,370 between the input and output of the binary channel. 525 00:25:36,370 --> 00:25:40,420 So here's what 1 minus H of p looks like. 526 00:25:40,420 --> 00:25:42,820 All right, so what's a low-noise channel? 527 00:25:42,820 --> 00:25:46,480 A low-noise channel is one with a very small value of p. 528 00:25:46,480 --> 00:25:48,790 And what this says is that the mutual information 529 00:25:48,790 --> 00:25:53,140 between the input and output is on the order of one bit. 530 00:25:53,140 --> 00:25:55,570 So if you're told what Y is, you've 531 00:25:55,570 --> 00:25:57,230 got a very good idea what X is. 532 00:25:57,230 --> 00:26:00,040 That makes sense, because it's a low-noise channel. 533 00:26:00,040 --> 00:26:01,930 But if you get to a channel that has around 534 00:26:01,930 --> 00:26:05,080 the 0.5 probability of flipping the bit, 535 00:26:05,080 --> 00:26:09,600 then the mutual information is very small. 536 00:26:09,600 --> 00:26:14,240 So it doesn't reflect what you'd like to see. 537 00:26:14,240 --> 00:26:16,100 Here's another notion, which is entirely 538 00:26:16,100 --> 00:26:20,720 Shannon, which is the idea of channel capacity. 539 00:26:20,720 --> 00:26:22,760 And what he's saying now is in order 540 00:26:22,760 --> 00:26:24,590 to characterize the channel, rather than 541 00:26:24,590 --> 00:26:26,540 the input or the output, let's ask 542 00:26:26,540 --> 00:26:28,430 what the maximum mutual information 543 00:26:28,430 --> 00:26:31,058 is over all possible distributions 544 00:26:31,058 --> 00:26:32,600 that you might have for X. So I'm not 545 00:26:32,600 --> 00:26:36,410 going to specify X being 0 and 1 with equal probability. 546 00:26:39,200 --> 00:26:40,790 If you go through that computation, 547 00:26:40,790 --> 00:26:45,080 you find that it's exactly the shape that we had before. 548 00:26:45,080 --> 00:26:47,580 It's exactly, for the binary symmetric channel, that happens 549 00:26:47,580 --> 00:26:50,430 to be exactly this curve. 550 00:26:50,430 --> 00:26:55,313 So the channel capacity for the binary channel is exactly-- 551 00:26:55,313 --> 00:26:56,730 for the binary symmetric channel-- 552 00:26:56,730 --> 00:26:59,460 is exactly this curve. 553 00:26:59,460 --> 00:27:02,100 So that gives us an idea of the maximum information 554 00:27:02,100 --> 00:27:06,770 that you could be transmitting across the channel. 555 00:27:06,770 --> 00:27:09,500 Now that's just the definition, but it turns out 556 00:27:09,500 --> 00:27:13,940 to have some very practical implications for how fast 557 00:27:13,940 --> 00:27:16,490 and how accurately you can transmit data on a channel. 558 00:27:16,490 --> 00:27:19,790 And here's Shannon's result. What he says 559 00:27:19,790 --> 00:27:24,090 is that you can theoretically transmit information 560 00:27:24,090 --> 00:27:29,340 at an average rate below the channel capacity 561 00:27:29,340 --> 00:27:32,310 with arbitrarily low error. 562 00:27:32,310 --> 00:27:34,860 So that's the shocking thing, that as long 563 00:27:34,860 --> 00:27:36,810 as you stay below channel capacity, 564 00:27:36,810 --> 00:27:40,200 you can transmit with arbitrary low error. 565 00:27:40,200 --> 00:27:45,990 If you try and get to rates above that, 566 00:27:45,990 --> 00:27:48,420 you're going to run into trouble. 567 00:27:48,420 --> 00:27:52,230 You can't get that probability of error to vanish. 568 00:27:52,230 --> 00:27:54,160 Now, how do you do this? 569 00:27:54,160 --> 00:27:57,510 Well, the prescription is take long strings of that input 570 00:27:57,510 --> 00:28:01,380 message stream, take k bits of that input message stream, 571 00:28:01,380 --> 00:28:05,820 code it onto it n larger than k code words, 572 00:28:05,820 --> 00:28:09,240 send that through the channel. 573 00:28:09,240 --> 00:28:12,520 If n is very large and k is very large, 574 00:28:12,520 --> 00:28:15,390 the rate at which you're transmitting is k over n, 575 00:28:15,390 --> 00:28:17,310 you can transmit at a rate k over n 576 00:28:17,310 --> 00:28:22,085 that lives below C with as low an error as you want. 577 00:28:22,085 --> 00:28:23,460 The way to make the error smaller 578 00:28:23,460 --> 00:28:25,020 is to take longer and longer blocks. 579 00:28:27,910 --> 00:28:30,490 This was kind of an existence proof. 580 00:28:30,490 --> 00:28:34,310 He didn't actually show you specific instructions, 581 00:28:34,310 --> 00:28:38,590 necessarily, in that proof for how to introduce the redundancy 582 00:28:38,590 --> 00:28:39,620 to make this happen. 583 00:28:39,620 --> 00:28:42,370 But it was actually a result that said, 584 00:28:42,370 --> 00:28:44,500 you can't be satisfied with the replication code. 585 00:28:44,500 --> 00:28:47,170 You can do a lot better, and how much better you can do 586 00:28:47,170 --> 00:28:49,540 is indicated by that channel capacity. 587 00:28:49,540 --> 00:28:53,420 OK, let's come back to testable stuff. 588 00:28:53,420 --> 00:28:56,950 We're going to actually design ways to introduce redundancy, 589 00:28:56,950 --> 00:29:00,190 motivated by this [? Shannon ?] result, 590 00:29:00,190 --> 00:29:02,590 for very practical settings. 591 00:29:02,590 --> 00:29:05,050 And a key notion we're going to use 592 00:29:05,050 --> 00:29:09,370 is that of the Hamming distance between two strings. 593 00:29:09,370 --> 00:29:11,950 So the Hamming distance-- 594 00:29:11,950 --> 00:29:14,080 you've seen this in some recitations-- basically, 595 00:29:14,080 --> 00:29:24,490 the Hamming distance between two strings 596 00:29:24,490 --> 00:29:28,030 is just a number of positions in which the two strings differ 597 00:29:28,030 --> 00:29:29,080 from each other. 598 00:29:29,080 --> 00:29:33,340 so the Hamming distance here between these two strings, 599 00:29:33,340 --> 00:29:50,830 let's say string 1, string 2, is what, 1, 2, 3. 600 00:29:50,830 --> 00:29:52,780 These strings differ in three positions. 601 00:29:52,780 --> 00:29:54,910 Another way to think of it is, how many bits 602 00:29:54,910 --> 00:29:58,750 do I have to flip in one to get the other one? 603 00:29:58,750 --> 00:30:01,510 So how many hops does it take, in some sense, 604 00:30:01,510 --> 00:30:05,030 to get from one to the other? 605 00:30:05,030 --> 00:30:16,960 All right, so here's how the notion of adding redundancy 606 00:30:16,960 --> 00:30:18,840 comes in. 607 00:30:18,840 --> 00:30:22,500 Suppose we have a 0 or a 1 to send across. 608 00:30:22,500 --> 00:30:25,085 This is our bit for that transmission. 609 00:30:25,085 --> 00:30:27,210 What we're going to do is actually code it not as 0 610 00:30:27,210 --> 00:30:29,640 and 1, but as 00 and 11. 611 00:30:32,440 --> 00:30:35,170 If we've got just a single bit corrupted, 612 00:30:35,170 --> 00:30:38,110 we go to something that's not a code word. 613 00:30:38,110 --> 00:30:43,720 We go from 11 01 or to 10, or we go from 00 to 01 or 10. 614 00:30:43,720 --> 00:30:46,360 We receive something that's not a code word. 615 00:30:46,360 --> 00:30:50,490 That allows us to detect that an error was made. 616 00:30:50,490 --> 00:30:55,020 So what we've essentially done is, in Hamming distance, 617 00:30:55,020 --> 00:30:59,310 we've introduced some distance between the code words 618 00:30:59,310 --> 00:31:02,100 that we're using to transmit on the channel. 619 00:31:02,100 --> 00:31:06,270 It takes two hops to get from one code word to the other. 620 00:31:06,270 --> 00:31:08,265 There's a Hamming distance of two. 621 00:31:08,265 --> 00:31:10,140 And, therefore, if you only have a single bit 622 00:31:10,140 --> 00:31:12,270 error when you transmit on the channel, 623 00:31:12,270 --> 00:31:14,757 you're not going to get all the way to another code word. 624 00:31:14,757 --> 00:31:16,590 You won't be at any code word you recognize, 625 00:31:16,590 --> 00:31:18,510 and you'll know that you made an error. 626 00:31:18,510 --> 00:31:23,130 So this is an example of how you start to introduce redundancy 627 00:31:23,130 --> 00:31:26,820 in the stream so that you can detect and perhaps even correct 628 00:31:26,820 --> 00:31:27,900 errors. 629 00:31:27,900 --> 00:31:29,460 Here's another example. 630 00:31:29,460 --> 00:31:33,670 Now this is-- these, by the way, are still looking replication 631 00:31:33,670 --> 00:31:37,810 codes because to send a 0, we're repeating 0 three times, 632 00:31:37,810 --> 00:31:40,432 and to send a 1, we're repeating 1 one time. 633 00:31:40,432 --> 00:31:42,640 But we're going to do more elaborate versions of this 634 00:31:42,640 --> 00:31:44,660 that are not replication codes. 635 00:31:44,660 --> 00:31:49,180 But imagine now I've drawn the corners of a cube. 636 00:31:49,180 --> 00:31:53,890 Each circle here across an edge is Hamming distance 1 637 00:31:53,890 --> 00:31:55,810 from the adjacent one. 638 00:31:55,810 --> 00:31:59,830 Right So to go from the sequence that I'm 639 00:31:59,830 --> 00:32:01,990 using to represent to 0 to the sequence 640 00:32:01,990 --> 00:32:07,060 that I use to represent a 1, I've got to do 1, 2, 3 hops, 641 00:32:07,060 --> 00:32:10,380 there's a Hamming distance of 3 there. 642 00:32:10,380 --> 00:32:14,160 So if I had only a single bit error, 643 00:32:14,160 --> 00:32:16,570 is it possible for me to correct? 644 00:32:16,570 --> 00:32:19,650 If I know that my errors are limited to single bit errors, 645 00:32:19,650 --> 00:32:21,930 can I correct when I receive an incorrect string? 646 00:32:24,650 --> 00:32:27,410 If I start with 000 and I have a single bit error, 647 00:32:27,410 --> 00:32:30,770 I can only go to these adjacent vertices. 648 00:32:30,770 --> 00:32:33,860 And those are not going to be confused with vertices that are 649 00:32:33,860 --> 00:32:37,658 one step adjacent to the 111. 650 00:32:37,658 --> 00:32:39,200 So I'll know that I've made an error, 651 00:32:39,200 --> 00:32:41,960 and I'll know to correct it back to 000, 652 00:32:41,960 --> 00:32:45,290 provided I know that only one error has been made. 653 00:32:45,290 --> 00:32:47,850 Now, I might just assume that only one error has been made. 654 00:32:47,850 --> 00:32:50,300 And so once in a while, I'll think I'm correcting, 655 00:32:50,300 --> 00:32:53,360 but I'll be getting something wrong. 656 00:32:53,360 --> 00:32:55,730 And that has to be contended with and calculated, 657 00:32:55,730 --> 00:32:57,040 but this is the basic idea. 658 00:33:02,760 --> 00:33:04,620 More generally what we're thinking about 659 00:33:04,620 --> 00:33:07,620 is taking key, message bits. 660 00:33:14,030 --> 00:33:16,850 So this corresponds to 2 to the k possible messages. 661 00:33:20,390 --> 00:33:33,770 We're going to embed this in ended code words where 662 00:33:33,770 --> 00:33:36,320 n is greater than k. 663 00:33:36,320 --> 00:33:38,750 So if you imagine a generalization 664 00:33:38,750 --> 00:33:43,610 of this picture, what we've got is a hypercube with 2 665 00:33:43,610 --> 00:33:47,750 to the n possible nodes, corresponding to all 666 00:33:47,750 --> 00:33:49,280 the possible combinations here. 667 00:33:56,990 --> 00:34:01,840 We're going to assign 2 to the k of those nodes to code words, 668 00:34:01,840 --> 00:34:03,730 and the rest will be left free to just leave 669 00:34:03,730 --> 00:34:05,470 some space between the code words. 670 00:34:09,510 --> 00:34:11,639 the rate of the code then, the rate of the code 671 00:34:11,639 --> 00:34:14,880 is, how many message bits are you're getting across 672 00:34:14,880 --> 00:34:17,520 on average per transmission? 673 00:34:17,520 --> 00:34:21,570 And so the rate is going to be k over n. 674 00:34:21,570 --> 00:34:23,969 Because for every n bits that you send, 675 00:34:23,969 --> 00:34:25,860 you're getting across k bits of information. 676 00:34:25,860 --> 00:34:28,560 This k message bits and every n transmitted bits. 677 00:34:34,170 --> 00:34:44,989 So here is the general statement in terms of Hamming distance 678 00:34:44,989 --> 00:34:46,650 and what you can do with the code. 679 00:34:46,650 --> 00:34:49,940 So first of all, I've got a set of code words. 680 00:34:49,940 --> 00:34:55,580 What's really important is what's the minimum Hamming 681 00:34:55,580 --> 00:35:00,170 distance between my code words. 682 00:35:04,605 --> 00:35:06,730 Because that's the point of greatest vulnerability. 683 00:35:06,730 --> 00:35:09,610 That's where I'm most likely to get confused. 684 00:35:09,610 --> 00:35:11,450 So if you give me a set of code words, 685 00:35:11,450 --> 00:35:12,610 I can look at the Hamming distance 686 00:35:12,610 --> 00:35:13,840 between any two of them. 687 00:35:13,840 --> 00:35:15,345 I've got to search all these pairs 688 00:35:15,345 --> 00:35:17,470 and find out which is the minimum Hamming distance. 689 00:35:17,470 --> 00:35:20,560 That's the point of maximum vulnerability, 690 00:35:20,560 --> 00:35:23,358 and this is what we're calling d, little d. 691 00:35:27,000 --> 00:35:29,550 So the picture I like to think about 692 00:35:29,550 --> 00:35:32,460 is I've got some valid code word here. 693 00:35:36,710 --> 00:35:38,810 I've got some other valid code word here. 694 00:35:45,690 --> 00:35:48,840 And if I told you that over all the code 695 00:35:48,840 --> 00:35:52,530 words in my set, the minimum Hamming distance 696 00:35:52,530 --> 00:35:56,430 I find is 3, what that means is I've got to do three 697 00:35:56,430 --> 00:36:02,680 hops to get to the other code word. 698 00:36:02,680 --> 00:36:05,620 This hop means I've changed 1 bit in the valid code word 699 00:36:05,620 --> 00:36:09,070 to get to some other sequence, not a code word. 700 00:36:09,070 --> 00:36:11,980 And then 1 bit to get to this one, 701 00:36:11,980 --> 00:36:13,820 and 1 more bit to get to this one, 702 00:36:13,820 --> 00:36:15,910 and this is now another valid code word. 703 00:36:15,910 --> 00:36:17,950 OK, so if I say the minimum Hamming distance is 704 00:36:17,950 --> 00:36:20,710 3, that means that you will find a pair of words 705 00:36:20,710 --> 00:36:23,860 with these three hops to get you from one to the other. 706 00:36:33,610 --> 00:36:36,310 How many errors can you detect with a code like this? 707 00:36:38,930 --> 00:36:42,220 So you send a valid code word across the channel, 708 00:36:42,220 --> 00:36:45,700 bits get flipped, up to how many errors could 709 00:36:45,700 --> 00:36:48,880 you detect without being fooled? 710 00:36:48,880 --> 00:36:53,470 If I tell you this is less than or equal to e errors, 711 00:36:53,470 --> 00:36:57,010 how large can e be to guarantee that you 712 00:36:57,010 --> 00:37:02,320 won't get a transmission of one valid code word that ends up 713 00:37:02,320 --> 00:37:03,550 as another valid code word? 714 00:37:03,550 --> 00:37:04,265 Yeah? 715 00:37:04,265 --> 00:37:05,140 AUDIENCE: [INAUDIBLE] 716 00:37:05,140 --> 00:37:06,098 GEORGE VERGHESE: Sorry? 717 00:37:06,098 --> 00:37:08,930 AUDIENCE: [INAUDIBLE] 718 00:37:08,930 --> 00:37:11,420 GEORGE VERGHESE: I'm talking not about correction 719 00:37:11,420 --> 00:37:14,532 but about detection of an error. 720 00:37:14,532 --> 00:37:16,990 How many errors could you detect in this kind of a picture? 721 00:37:20,240 --> 00:37:23,063 So I can afford to have one error, two errors, 722 00:37:23,063 --> 00:37:25,480 and the third hour will bring me to about valid code word, 723 00:37:25,480 --> 00:37:28,360 and I won't know that I've made an error. 724 00:37:28,360 --> 00:37:30,970 So if you want to look at how many errors you can detect, 725 00:37:30,970 --> 00:37:35,430 it's what's given by the upper one there, d minus 1. 726 00:37:35,430 --> 00:37:37,290 So if the minimum Hamming distance is d, 727 00:37:37,290 --> 00:37:40,510 you can detect up to d minus 1 hours. 728 00:37:40,510 --> 00:37:41,940 What about correction? 729 00:37:45,920 --> 00:37:47,760 How many errors could you correct here? 730 00:37:47,760 --> 00:37:50,380 Can you correct any errors here? 731 00:37:50,380 --> 00:37:52,315 Up to one, right? 732 00:37:52,315 --> 00:37:54,190 And then you can look at this more generally, 733 00:37:54,190 --> 00:37:57,760 and you see the general formula is d minus 1 over 2, 734 00:37:57,760 --> 00:38:00,010 the floor of that. 735 00:38:00,010 --> 00:38:02,660 So that tells you how many error you can correct. 736 00:38:02,660 --> 00:38:07,160 So the minimum Hamming distance is actually a key thing. 737 00:38:10,230 --> 00:38:12,030 Now, how do you build codes which 738 00:38:12,030 --> 00:38:13,840 have desired characteristics? 739 00:38:13,840 --> 00:38:16,350 For instance, suppose you want to send 4 bit messages. 740 00:38:16,350 --> 00:38:18,210 So k equals 4. 741 00:38:18,210 --> 00:38:21,125 You want to have single error correction. 742 00:38:21,125 --> 00:38:23,250 So that means you want this kind of a picture here. 743 00:38:23,250 --> 00:38:26,350 You need Hamming distance 3, at least. 744 00:38:26,350 --> 00:38:27,830 How will you produce a code? 745 00:38:27,830 --> 00:38:32,220 All right, so this is not an obvious thing at all. 746 00:38:32,220 --> 00:38:35,550 Here's an example of one that satisfies the construction. 747 00:38:35,550 --> 00:38:39,960 You need to actually expand to sending 7 bits, 748 00:38:39,960 --> 00:38:42,640 so n is equal to 7. 749 00:38:42,640 --> 00:38:43,990 How many messages do we have? 750 00:38:43,990 --> 00:38:48,320 We have 16 messages, so that corresponds to a k of 4, 751 00:38:48,320 --> 00:38:49,780 2 to the 4 is 16. 752 00:38:49,780 --> 00:38:52,050 So we've got 16 different messages. 753 00:38:52,050 --> 00:38:54,690 We could have counted those messages with 4 bits, 754 00:38:54,690 --> 00:38:56,490 but we're going to add in redundancy 755 00:38:56,490 --> 00:39:00,660 to get 7 bits per message, resulting in these code words. 756 00:39:00,660 --> 00:39:02,520 These code words, this set of code words 757 00:39:02,520 --> 00:39:06,513 has the property that the minimum Hamming distance is 3. 758 00:39:06,513 --> 00:39:08,430 So you can correct up to a single error, here. 759 00:39:11,700 --> 00:39:14,340 But it takes-- in principle, it takes a search, 760 00:39:14,340 --> 00:39:18,160 and it's not necessarily easy to do. 761 00:39:18,160 --> 00:39:21,840 But we'll see how to do that efficiently. 762 00:39:21,840 --> 00:39:22,690 All right. 763 00:39:22,690 --> 00:39:24,690 Let me show you how-- and this is something that 764 00:39:24,690 --> 00:39:28,080 you're probably quite familiar with-- 765 00:39:28,080 --> 00:39:31,020 how by making n equals k plus 1, you can already-- 766 00:39:35,260 --> 00:39:38,440 suppose I choose n equals k plus 1, 767 00:39:38,440 --> 00:39:42,670 which means I'm taking the message bit and adding 1 bit. 768 00:39:42,670 --> 00:39:46,970 We're going to add what's called a parity bit. 769 00:39:46,970 --> 00:39:50,990 And you can do this in different ways, but we're going to do-- 770 00:39:50,990 --> 00:39:56,080 let's see-- what I'm going to do with this is guarantee 771 00:39:56,080 --> 00:40:00,250 that the minimum distance between valid code words 772 00:40:00,250 --> 00:40:01,210 is at least 2. 773 00:40:08,320 --> 00:40:09,890 So let's see how to do that. 774 00:40:09,890 --> 00:40:12,610 And there'll be some computation in not just the parity 775 00:40:12,610 --> 00:40:14,200 calculations, but other stuff we'll do 776 00:40:14,200 --> 00:40:17,860 that builds on computations of zeros and ones. 777 00:40:17,860 --> 00:40:21,130 The computations are what you've probably seen elsewhere 778 00:40:21,130 --> 00:40:23,500 with Boolean algebra. 779 00:40:23,500 --> 00:40:26,440 This is what's called computation in Galois field 2. 780 00:40:26,440 --> 00:40:28,920 So GF2 is another symbol you'll see. 781 00:40:28,920 --> 00:40:33,970 0 plus 0 is a 0, 1 plus 0 or 0 plus 1 is 1, 1 plus 1 is 0. 782 00:40:33,970 --> 00:40:38,200 So this is like an exclusive or addition, and multiplication 783 00:40:38,200 --> 00:40:40,910 works in the usual fashion. 784 00:40:40,910 --> 00:40:44,187 So all our computations are with zeros and ones, 785 00:40:44,187 --> 00:40:46,520 and you want to keep that in mind as we go through this. 786 00:40:46,520 --> 00:40:54,420 So here's what we do for a simple way to add redundancy. 787 00:40:54,420 --> 00:40:57,870 We'll take the message and not a single bit 788 00:40:57,870 --> 00:41:02,580 to make the total number of ones in the resulting code word 789 00:41:02,580 --> 00:41:03,578 even. 790 00:41:03,578 --> 00:41:05,120 So this is what's called even parity. 791 00:41:05,120 --> 00:41:08,490 You can have the opposite choice of odd parity. 792 00:41:08,490 --> 00:41:11,972 So if you now receive a code word 793 00:41:11,972 --> 00:41:14,430 with an odd number of ones, you know you've made a mistake. 794 00:41:17,157 --> 00:41:18,740 How do I know that the minimum Hamming 795 00:41:18,740 --> 00:41:20,510 distance is 2 in this case? 796 00:41:27,080 --> 00:41:30,640 I have to be able to produce for you some other code word 797 00:41:30,640 --> 00:41:32,800 that I've had with two hops. 798 00:41:36,990 --> 00:41:38,880 Any ideas there? 799 00:41:38,880 --> 00:41:41,880 So I give you a code word, which is the original message 800 00:41:41,880 --> 00:41:44,580 word with a parity bit. 801 00:41:44,580 --> 00:41:52,470 Can I make two bit flips in that and get a new code word, 802 00:41:52,470 --> 00:41:54,270 I mean, a valid code word? 803 00:41:54,270 --> 00:41:59,403 Because then I'd have Hamming distance 2, right? 804 00:41:59,403 --> 00:42:00,570 Can you think of what to do? 805 00:42:00,570 --> 00:42:01,143 Yeah? 806 00:42:01,143 --> 00:42:05,374 AUDIENCE: If you flip a 1 to 0 or 0 to 1, 807 00:42:05,374 --> 00:42:09,402 then n equals [INAUDIBLE]. 808 00:42:09,402 --> 00:42:10,860 GEORGE VERGHESE: But that's not yet 809 00:42:10,860 --> 00:42:13,728 given me-- so the suggestion was flip one of the bits. 810 00:42:13,728 --> 00:42:15,270 AUDIENCE: So then for the second one, 811 00:42:15,270 --> 00:42:18,450 you'll either still have an odd number or [? even. ?] 812 00:42:18,450 --> 00:42:21,060 GEORGE VERGHESE: So if you flip, for instance an easy way 813 00:42:21,060 --> 00:42:22,953 to see this is flip one of the message bits 814 00:42:22,953 --> 00:42:24,495 and flip the party bit, for instance. 815 00:42:27,630 --> 00:42:31,200 No that doesn't do it, does it? 816 00:42:31,200 --> 00:42:35,760 Because then you've changed two 0's to 2 or two 1's to a 0, 817 00:42:35,760 --> 00:42:39,117 and your parity is wrong, then. 818 00:42:39,117 --> 00:42:40,950 So you can have a two bit error that ends up 819 00:42:40,950 --> 00:42:43,080 not being detected, but all single bit errors 820 00:42:43,080 --> 00:42:45,270 will get detected. 821 00:42:45,270 --> 00:42:46,920 That, again, correlates with the fact 822 00:42:46,920 --> 00:42:49,530 that we set the minimum Hamming distance at 2. 823 00:42:49,530 --> 00:42:52,560 The number of errors you can detect is d minus 1. 824 00:42:52,560 --> 00:42:53,250 That's 1. 825 00:42:57,450 --> 00:42:59,280 And the number of errors you can correct, 826 00:42:59,280 --> 00:43:02,765 well, it's d minus 1 divided by 2, the floor of that, 827 00:43:02,765 --> 00:43:04,140 and you can't correct any errors. 828 00:43:13,070 --> 00:43:16,810 All right, now we're going to be building more elaborate codes 829 00:43:16,810 --> 00:43:20,260 than parity or replication. 830 00:43:20,260 --> 00:43:22,555 These are going to be called linear block codes. 831 00:43:42,570 --> 00:43:45,390 And there are different ways to set this up. 832 00:43:45,390 --> 00:43:46,830 Here's one way to think of it. 833 00:43:46,830 --> 00:43:50,600 If you're comfortable with the matrix multiplication, 834 00:43:50,600 --> 00:43:51,850 here's one way to think of it. 835 00:43:51,850 --> 00:43:53,830 And we're going to be using this actually. 836 00:43:53,830 --> 00:43:56,320 So if you aren't already comfortable with matrix 837 00:43:56,320 --> 00:44:00,670 multiplication, maybe you should get comfortable soon. 838 00:44:00,670 --> 00:44:07,200 What we have is a vector here, which has our message in it. 839 00:44:07,200 --> 00:44:10,530 So we stick our message in there. 840 00:44:10,530 --> 00:44:13,460 This is just a bunch of zeros and ones. 841 00:44:13,460 --> 00:44:17,960 I've got a matrix here, which I call a generator matrix, 842 00:44:17,960 --> 00:44:18,890 generator matrix. 843 00:44:24,410 --> 00:44:40,870 And this is a matrix of zeros and ones, as well, and so on. 844 00:44:40,870 --> 00:44:45,280 So we've got things in there. 845 00:44:45,280 --> 00:44:47,300 How do I generate my code words? 846 00:44:47,300 --> 00:44:50,740 I just put in my message, carry out this multiplication, 847 00:44:50,740 --> 00:44:52,720 and see what I get for a code word. 848 00:44:52,720 --> 00:44:56,380 So for instance, if I my message is 1 849 00:44:56,380 --> 00:45:00,210 and all 0's, what's my code word? 850 00:45:00,210 --> 00:45:01,680 Well, I take this and I multiply it 851 00:45:01,680 --> 00:45:03,450 all the way through the matrix. 852 00:45:03,450 --> 00:45:06,990 Because of the special structure here, all of these are zeros, 853 00:45:06,990 --> 00:45:09,870 so the rows below the first one don't matter at all. 854 00:45:09,870 --> 00:45:14,910 What I get for a code word is 0101101. 855 00:45:14,910 --> 00:45:16,640 In other words, I get the first row of g. 856 00:45:19,810 --> 00:45:24,220 If I had a 1 and a 1 with 00, I'm going to get the sum 857 00:45:24,220 --> 00:45:26,980 of the first two rows of g. 858 00:45:26,980 --> 00:45:30,310 And all of these computations are done with the modulo 2 859 00:45:30,310 --> 00:45:32,260 arithmetic, so in GF2. 860 00:45:35,112 --> 00:45:37,320 So this is one way to think of what a linear code is. 861 00:45:37,320 --> 00:45:41,760 Another way to think of it is, every bit in your code word 862 00:45:41,760 --> 00:45:46,520 is a linear combination of the bits in the message. 863 00:45:46,520 --> 00:45:48,270 It's just that you have more bits here, 864 00:45:48,270 --> 00:45:51,290 so you're taking multiple linear combinations of the bits 865 00:45:51,290 --> 00:45:56,190 in the message to get the bits in the code word. 866 00:45:56,190 --> 00:45:59,430 So this is a highly structured kind of code. 867 00:45:59,430 --> 00:46:03,360 And the key fact about this is that the sum of any two code 868 00:46:03,360 --> 00:46:05,280 words is also a code word. 869 00:46:05,280 --> 00:46:07,540 And we'll leave you to look at that in recitation. 870 00:46:07,540 --> 00:46:11,000 So it's true that any code word generated 871 00:46:11,000 --> 00:46:12,940 this way plus any other code word 872 00:46:12,940 --> 00:46:17,260 generated this way will give you a code word generated this way. 873 00:46:17,260 --> 00:46:21,310 can, you deduce from that that the code word of all 0's has 874 00:46:21,310 --> 00:46:22,540 to be in any linear code? 875 00:46:25,120 --> 00:46:33,450 Why is it true that every linear code has 876 00:46:33,450 --> 00:46:38,310 to have the all zero code word? 877 00:46:38,310 --> 00:46:48,920 In this instance-- well, you can see 878 00:46:48,920 --> 00:46:51,860 it has to have the all zero code word. 879 00:46:51,860 --> 00:46:52,700 Why am I doing that? 880 00:46:57,660 --> 00:47:02,200 AUDIENCE: [INAUDIBLE] 881 00:47:02,200 --> 00:47:04,713 GEORGE VERGHESE: Yeah, so clearly from this picture, 882 00:47:04,713 --> 00:47:06,130 if your input is all zeros, you've 883 00:47:06,130 --> 00:47:07,600 got to have the zero code word. 884 00:47:07,600 --> 00:47:10,295 Can you tell me from this statement 885 00:47:10,295 --> 00:47:12,670 that the sum of any two code words has to be a code word? 886 00:47:12,670 --> 00:47:15,490 Can you deduce from that the all zero code 887 00:47:15,490 --> 00:47:17,338 word has to be in there? 888 00:47:17,338 --> 00:47:20,922 AUDIENCE: So that's [INAUDIBLE] subtraction. 889 00:47:20,922 --> 00:47:22,630 GEORGE VERGHESE: Subtraction or addition. 890 00:47:22,630 --> 00:47:26,360 So suppose I take a code word and add it to itself. 891 00:47:26,360 --> 00:47:29,450 What do I get? 892 00:47:29,450 --> 00:47:33,500 In GFW, if I take a code word and add it to itself, 893 00:47:33,500 --> 00:47:35,840 I get the all zero code word. 894 00:47:35,840 --> 00:47:38,460 So the all zeros has to always be in there. 895 00:47:38,460 --> 00:47:40,310 If you don't have the all zeros code word, 896 00:47:40,310 --> 00:47:41,893 you know you don't have a linear code. 897 00:47:46,550 --> 00:47:49,220 Now it turns out that for a linear code, 898 00:47:49,220 --> 00:47:52,310 it's easy to determine the minimum distance between-- 899 00:47:52,310 --> 00:47:55,400 the minimum Hamming distance between words, which we saw 900 00:47:55,400 --> 00:47:57,800 was crucial to establishing what the error correction 901 00:47:57,800 --> 00:47:59,480 or detection properties were. 902 00:47:59,480 --> 00:48:03,170 If you've got a linear code to determine the minimum distance 903 00:48:03,170 --> 00:48:05,660 between words, you only have to look for the distance-- 904 00:48:05,660 --> 00:48:09,440 the minimum distance between the zero code word and all 905 00:48:09,440 --> 00:48:11,810 the other code words. 906 00:48:11,810 --> 00:48:14,330 So it turns out that in a linear code, 907 00:48:14,330 --> 00:48:17,660 the minimum distance that you find between any two code words 908 00:48:17,660 --> 00:48:19,490 is the same as the minimum distance you'll 909 00:48:19,490 --> 00:48:23,180 find between the zero code word and any other code word. 910 00:48:23,180 --> 00:48:25,580 Now what's the distance between the zero code 911 00:48:25,580 --> 00:48:27,560 word and some other code word? 912 00:48:27,560 --> 00:48:30,200 It's just the number of ones in that other code word. 913 00:48:30,200 --> 00:48:31,880 So all you have to do for a linear code 914 00:48:31,880 --> 00:48:34,370 to determine the minimum Hamming distance is 915 00:48:34,370 --> 00:48:36,200 look at all the non-zero code words 916 00:48:36,200 --> 00:48:40,490 and see which one has the minimum number of ones. 917 00:48:40,490 --> 00:48:48,260 So let's see, it's not obvious that these are necessarily 918 00:48:48,260 --> 00:48:57,170 linear codes, but they turn out to be. 919 00:48:57,170 --> 00:49:01,880 In this particular case, here's a code with n equals 3. 920 00:49:01,880 --> 00:49:05,310 We've got only two messages being sent, 921 00:49:05,310 --> 00:49:08,078 so what's the value of k? 922 00:49:08,078 --> 00:49:11,000 Two messages means k equals 1, because 2 to the k 923 00:49:11,000 --> 00:49:12,590 is the number of messages. 924 00:49:12,590 --> 00:49:18,380 So n is 3, k is 1, and the minimum Hamming distance 925 00:49:18,380 --> 00:49:22,430 is 3, which is the weight of the-- 926 00:49:22,430 --> 00:49:24,920 smallest weight you find among the non-zero words. 927 00:49:24,920 --> 00:49:26,090 Here's another instance. 928 00:49:26,090 --> 00:49:28,107 This is again a linear code. 929 00:49:28,107 --> 00:49:29,190 Well, is it a linear code? 930 00:49:29,190 --> 00:49:31,340 Yeah. 931 00:49:31,340 --> 00:49:34,310 The minimum weight you see among the non-zero code words is 2, 932 00:49:34,310 --> 00:49:35,990 so the Hamming distance is 2. 933 00:49:35,990 --> 00:49:39,800 So the way we denote this code is the value of n 934 00:49:39,800 --> 00:49:44,090 is 4, because there are four different code words-- 935 00:49:44,090 --> 00:49:47,210 sorry, the four different bits here. 936 00:49:47,210 --> 00:49:48,590 4 bits, sorry, in the code words. 937 00:49:48,590 --> 00:49:51,140 It's not four different code words. 938 00:49:51,140 --> 00:49:54,900 2 is the value of k, because 2 to the 2-- 939 00:49:54,900 --> 00:49:57,950 4 is the number of messages that you have. 940 00:49:57,950 --> 00:50:00,315 And the minimum Hamming distance is 2. 941 00:50:00,315 --> 00:50:01,940 So with each of those, you can actually 942 00:50:01,940 --> 00:50:03,350 compute the associated rate. 943 00:50:09,590 --> 00:50:15,170 And just to wind up, these are not linear codes. 944 00:50:15,170 --> 00:50:17,940 How do we know they're not linear codes? 945 00:50:17,940 --> 00:50:20,550 Well, some two of them, and you'll 946 00:50:20,550 --> 00:50:23,000 discover that in some instances, you 947 00:50:23,000 --> 00:50:26,280 don't get the remaining one in the set. 948 00:50:26,280 --> 00:50:29,810 This is the code set that I put up earlier. 949 00:50:29,810 --> 00:50:31,950 It turns out to be a linear code. 950 00:50:31,950 --> 00:50:33,780 So if I claim that it's a linear code, 951 00:50:33,780 --> 00:50:36,255 can you tell me what the minimum Hamming distance 952 00:50:36,255 --> 00:50:37,560 is between code words here? 953 00:50:40,780 --> 00:50:42,280 3. 954 00:50:42,280 --> 00:50:45,282 You find a code word here of weight 3, 955 00:50:45,282 --> 00:50:47,740 and you don't find any code words-- here also another one-- 956 00:50:47,740 --> 00:50:50,500 you don't find any code words of weight less than 3. 957 00:50:53,860 --> 00:50:58,330 All right, so this is enough to get you going. 958 00:50:58,330 --> 00:51:01,220 We'll quit with this, you'll continue in recitation, 959 00:51:01,220 --> 00:51:03,880 and we'll pick it up again in lecture next time. 960 00:51:03,880 --> 00:51:05,730 Thank you.